How to measure whether multi-agent system development is working
How do you measure multi-agent system development?
Measure a multi-agent system on completed tasks, not conversations: the share of runs reaching a correct final outcome without human correction, plus cost and latency per completed task. Everything else, including per-agent accuracy, is diagnostic. If only one number reaches the business, make it task completion rate.
Measure a multi-agent system on completed tasks, not conversations: the share of runs that reach a correct final outcome without human correction, plus cost and latency per completed task. Everything else, including per-agent accuracy, is diagnostic. If only one number is reported to the business, make it task completion rate.
This article sets out the full measurement stack we build for multi-agent programmes: what to instrument, which metrics genuinely predict production quality, which ones flatter a system that is quietly failing, and how to turn all of it into an evaluation suite that gates releases instead of describing them afterwards.
Why per-agent accuracy is the wrong headline number
A multi-agent system is a set of specialised agents coordinated by an orchestrator, and its output is the product of several steps rather than the average of them. If a planner is right ninety-five per cent of the time, a retrieval worker is right ninety-five per cent of the time and a reviewer catches half of what slips through, the end-to-end completion rate is not ninety-five per cent. It is whatever the composition works out to, and it is always lower than the component that looks best in a demo.
That compounding is why multi-agent system development metrics have to start at the task boundary. A task is a unit of work with a definable correct outcome: this invoice was matched to the right purchase order, this ticket was resolved with the right refund, this claim was routed to the right queue with the right documents attached. If you cannot write the correct outcome down for a sample of a hundred real cases, you do not yet have a measurable system, and no dashboard will fix that.
The second reason is attribution. When a run fails, you need to know which agent, which handoff and which tool call caused it. That is a tracing problem, not a metrics problem. OpenTelemetry defines a trace as a tree of spans describing one request's path through a system, which is precisely the shape of a multi-agent run, and the tracing model it specifies is what makes per-step attribution possible at all.
The metrics worth reporting, and how each one lies
| Metric | What it answers | How it misleads |
|---|---|---|
| Task completion rate | What share of runs reached a correct final outcome unaided | Rises if you narrow the definition of a task; fix the definition first |
| Escalation rate | How often a human is pulled in | Low is not good. A system that never escalates is a system that never notices it is wrong |
| Correction rate | How often a human changes what the agent proposed | Falls when reviewers stop reading carefully, so track it alongside sampled audits |
| Cost per completed task | Whether the economics work at volume | Hides a rising failure rate, because failed runs still consume tokens |
| End-to-end latency | Whether the system fits the workflow it sits in | Averages conceal the tail; report p95 and the worst case |
| Handoff loop count | Whether agents are arguing with each other | Looks fine in aggregate while a small set of inputs loops until the cap |
| Groundedness of outputs | Whether claims trace to retrieved evidence | Passing on a golden set says nothing about unseen inputs |
| Tool call error rate | Whether integrations are healthy | Retries mask a failing dependency until it fails completely |
The four layers to instrument
Layer one: the task
Every run gets an identifier, an input snapshot, a final outcome and a verdict. The verdict is either automatic, when the outcome can be checked against a system of record, or human, when it cannot. This layer produces the numbers the business sees.
Layer two: the trace
Every agent invocation, tool call, retrieval and handoff becomes a span with timing, token counts and cost attached. When completion rate drops two points after a prompt change, the trace tells you which agent changed behaviour. Without it you will be reading logs and guessing. The practice is covered in LLM observability.
Layer three: the eval suite
A scenario set with expected outcomes, versioned in your repository, run in continuous integration on every prompt, model or tool change. This is the layer that turns a subjective judgement into a gate. An evaluation suite is not a test of the model; it is a test of your system's behaviour on cases you have decided matter.
Layer four: the business outcome
Hours returned, backlog cleared, cycle time reduced, error cost avoided. Measure a baseline before the agents exist, because you cannot reconstruct it afterwards. The AI agent ROI calculator gives you a structure for the arithmetic.
How do you build an evaluation suite for a multi-agent system?
Start from real history, not imagination. The steps below usually take a week of focused work and are the highest-return week in the whole programme.
- Sample a hundred to three hundred real cases across the last quarter, weighted to include the awkward ones rather than the clean median.
- Write the correct outcome for each, agreed with the person who does the work today, not with the project sponsor.
- Tag each case by intent, difficulty and whether it should end in an escalation. Cases that should escalate are the ones most suites forget.
- Set a pass threshold per agent and one for the whole task, and write down what a release-blocking regression looks like.
- Add adversarial cases: contradictory documents, missing fields, prompt injection attempts in user-supplied text, and tools that return errors.
- Run the suite in CI on every prompt change, treating prompts as versioned code rather than configuration.
- Re-sample quarterly, because the distribution of real inputs moves and a stale suite passes a drifting system.
Prompts belong under version control with the same rigour as application code, for the same reason: they change behaviour and need review. Prompt versioning and evaluation sets out the workflow.
Metrics that flatter a failing system
Three numbers appear in almost every agent dashboard and none of them should be reported upward on their own.
Deflection rate counts conversations that did not reach a human. It rises when the system is unhelpful enough that users give up, which is why a support agent can post excellent deflection and falling satisfaction at the same time. Count resolutions instead.
Average response time rewards speed over correctness, and a multi-agent system can always be made faster by skipping the reviewer. Latency belongs in the metric set, but only alongside completion rate, and always as p95 rather than a mean.
Model benchmark scores tell you about a model's general capability, not about your workflow. A model that leads a public leaderboard can still be the wrong planner for your task. Benchmark on your own eval suite, as described in model and vendor selection.
What measurement costs, and when to build it
Build it before the agents, not after. In our multi-agent systems programme, which runs from $24,500 to $84,000 or ₹16,00,000 to ₹56,00,000, the eval suite and tracing are part of the build rather than a later phase, because a system without them cannot be released safely and cannot be improved deliberately.
If you are not ready to commit to the full build, a three-week ProofRun at $6,250 or ₹4,00,000 produces a working eval harness and a measured result on your real data for the hardest path in the workflow. A ten-day Sprint Zero at $3,250 or ₹2,00,000, credited to the build, produces the metric definitions and the baseline. Both are listed on the pricing page.
After launch, measurement is ongoing work. Our Care Plan AI system add-on is $750 or ₹40,000 a month and covers evals, cost monitoring, prompt regression and re-indexing, which is roughly the cadence needed to keep a multi-agent system honest through model deprecations and data drift.
When heavy measurement is the wrong priority
If the system is a two-week internal prototype that three people will use and nobody will depend on, a full four-layer stack is over-engineering. Keep a small golden set, look at the traces by hand, and spend the time on the workflow instead.
If the workflow genuinely has no definable correct outcome, for example open-ended research assistance where usefulness is a judgement, forcing a completion rate onto it produces a number that measures nothing. Use sampled human ratings against a written rubric and be honest that the metric is qualitative.
And if you have not yet run the system in shadow mode alongside the people who do the work, elaborate dashboards are premature. Acceptance and correction data from shadow mode is the richest measurement you will ever get, and it costs nothing extra to collect.
What this looks like on a real system
On a field-service SaaS copilot, the meaningful unit was not a message but a job action: reassign this work order, find the nearest technician, close these jobs. Each action had a correct outcome that could be checked against the platform's own records, which made completion rate computable rather than debatable, and reassignments above a threshold went for approval so that correction data accumulated from day one. The in-app copilot case study describes the rollout.
A measurement checklist
- Define the task boundary and write the correct outcome for a hundred real cases
- Instrument traces before the first agent reaches staging
- Report completion rate, correction rate, cost per completed task and p95 latency together
- Set a pass threshold that blocks a release, and honour it
- Include cases that should escalate, and cases designed to break the system
- Record a pre-agent baseline for the business metric
- Re-sample the eval set every quarter and after any model change
- Name the person who reads the weekly escalation review
Related reading
How to measure an AI agent covers the single-agent version of this metric set, evals: the practice that separates AI demos from AI products explains why the suite comes first, and multi-agent systems explained describes the planner, worker and reviewer structure these metrics are measuring. If you want an eval harness built against your own history, tell us what the task is.
A multi-agent system you cannot measure end to end is not a product yet; it is a demonstration with an invoice attached.
Frequently asked questions
How do you measure multi-agent system development?
▾
Measure at the task boundary: the share of runs reaching a correct final outcome without human correction, plus cost and latency per completed task. Add tracing for per-agent attribution, an eval suite that gates releases, and a business baseline recorded before the agents went live.
What is a good task completion rate for a multi-agent system?
▾
There is no universal figure, because it depends entirely on how narrowly you define a task. The useful target is the rate the human process achieves today on the same cases, measured the same way. Beat that with a lower error cost and the system is working.
Why is a low escalation rate not a good sign?
▾
Escalation is how a multi-agent system admits uncertainty. A system that rarely escalates is either genuinely reliable or unable to detect its own failures, and only the correction rate and sampled audits distinguish the two. Track escalation alongside completion and correction, never on its own.