How to measure an AI agent: the six metrics that matter

AI agent metrics go wrong in one of two directions. Either the team reports model-level numbers, token counts, response quality scores, that nobody outside engineering can act on, or it reports a single headline such as "handles 70% of requests" that hides the intent doing all the damage. Six numbers, reported per intent and paired with an evaluation suite, cover what finance, operations and engineering each need to know. This article defines each one, explains how to collect it honestly, and shows how they fit together on one page.

Why AI agent KPIs need to be per intent

An agent that books appointments, reschedules them and cancels them is three different systems from a risk point of view. Booking is low-risk and high-volume; cancellation is where a wrong action costs a patient a treatment. An average across the three tells you nothing you can act on and invites the wrong decision, such as expanding autonomy on cancellations because the blended completion rate looks good. Every metric below is collected and reported per intent. The average can go in a footnote.

The six metrics in one table

1. Autonomous completion rate: the real agent success rate

A task counts as completed when the agent reached the defined end state, no person touched it, and no reversal or reopen followed within an agreed window. The window matters. A refund that is disputed a week later was not a successful refund; a booking that is cancelled the next day by a confused patient was not a successful booking. Reopens and reversals are subtracted automatically. Tasks the agent was never allowed to complete, because the intent is gated by design, are excluded from the denominator and reported separately, so that a deliberately human intent does not look like a failing one.

2. Escalation rate and reasons

Escalation is not failure; a well-designed agent escalates exactly the cases it should. The number alone is uninformative. The reasons are the point. Every escalation carries a reason code chosen by the agent, and every human override carries one chosen by the person, from a short list: missing information, outside policy, ambiguous intent, tool error, customer requested a person. Clustered weekly, the reasons become the backlog. "Missing information" is a tool to add. "Outside policy" that keeps recurring is a threshold to revisit. "Ambiguous intent" is a prompt or a routing fix. This review is the same one that drives promotion in Shadow mode.

3. Error rate on the eval set

Production metrics tell you what happened; the evaluation suite tells you why, and tells you before release. The suite is a set of scenarios per intent, each with an input, the expected tool calls, the expected outcome and the expected escalation decision, including adversarial cases. It runs on every prompt, model or tool change, and a regression blocks the release. Error rate here is the engineering number: it is what makes model-agnostic routing safe, because a cheaper model that passes the same suite can be adopted with evidence rather than hope. The suite grows every week from production escalations, so the two views converge over time.

4. Cost per completed task

Cost per message flatters agents, because a task that fails after nine tool calls cost as much as one that succeeded. Divide total running cost, model calls, tool calls, retrieval, orchestration, and the platform's share of infrastructure, by autonomously completed tasks. Track it per intent and per model, so that routing decisions are visible: moving a worker to a smaller model should show up here without moving the error rate. Step budgets and spend limits keep the tail bounded. Budgeting for this is covered in How much does an AI agent cost?.

5. Latency

Two numbers, at the median and at the ninety-fifth percentile: time to first useful output, and time to completion. The first decides whether a person waits for the agent's proposal or starts the work themselves. The second decides whether a customer stays on the line or in the chat. Report the tail, because the median hides the runs that loop or wait on a slow tool. For voice, latency is the product; the constraints are in Latency in voice AI.

6. Human hours returned

This is where the value shows up, and it is the number most often faked. Measure handling time per intent before launch from the helpdesk, dispatch or finance system, then after, and include the time people spend reviewing proposals and escalations. Hours returned is the difference. If review time eats the saving, the agent is not yet paying and the escalation reasons say why. Present this with the intents deliberately kept human listed alongside, so the figure is credible rather than inflated.

Metrics that do not belong on the page

• Deflection. It counts people who gave up. See AI ticket deflection is the wrong metric.
• Token counts. Useful for engineering, meaningless for anyone deciding whether to expand the agent.
• Model benchmark scores. A vendor's benchmark says nothing about your intents.
• Conversation length. Short can mean efficient or abandoned.
• A single blended success rate. It hides the failing intent and invites the wrong promotion.

A worked example

A hospital network's multilingual voice agent handled booking, rescheduling and cancellation. Reported as one number, completion looked healthy. Split by intent, booking was strong, rescheduling moderate, and cancellation was escalating most calls, which was correct: cancellations inside a short window were gated to a person by design. The escalation reasons on rescheduling clustered around one language where the speech recognition struggled with dates, which became a targeted fix and a new set of eval cases rather than a general "improve accuracy" ticket.

Cost per completed task was tracked per language and per model, and the latency tail showed a slow appointment-system lookup that the hospital's IT team fixed on their side. Hours returned were measured at the call centre with review time included. The multilingual voice agent case study describes the deployment; the service is on the voice agents page.

Team and timeline

The measurement framework is built in the first two weeks of any agent programme, before shadow mode, so a baseline exists. An AI engineer owns the evaluation suite and the per-intent dashboard; a backend engineer wires tracing so every run produces the fields the metrics need; the operational owner on your side supplies the baseline handling times and chairs the weekly escalation review. This is included in the multi-agent systems service from $24,500 or ₹16 lakh and in every customer service or voice agent build. After launch, a Care Plan reruns the eval suite on model changes and maintains the dashboard; Standard is $2,500 or ₹1,60,000 a month. Prices are on the pricing page.

Before you start: a checklist

• Define the end state and the reopen or reversal window for each intent
• Separate intents gated by design from intents expected to run alone
• Collect baseline handling time per intent from the existing systems
• Write the escalation and override reason list, kept to under ten codes
• Build fifty or more eval scenarios per intent, including adversarial cases
• Instrument tracing so cost, latency and tool calls are captured per run
• Agree who reads each metric and what decision it drives
• Fix the report format so month-on-month trends are visible

Questions clients ask

• What completion rate is good? It depends on the intent and the cost of an error. A status lookup should be very high; a gated financial action is measured on escalation quality, not completion.
• How often should we look at this? Escalation reasons weekly in the first quarter, the full page monthly, the eval suite on every change.
• Can we compare vendors on these six? Yes. Ask any vendor for completion per intent with reopens subtracted and cost per completed task. Vague answers are informative.
• Should the agent report on itself? No. Metrics come from tracing and from the systems of record, never from the model's own assessment of its work.

Related reading

How to build an AI agent that is safe to run unattended covers the tracing that feeds these metrics, Multi-agent systems explained covers evaluating each worker separately, and the OpenTelemetry documentation is the primary source for the tracing standard most of this instrumentation is built on.

Six numbers, per intent, with an eval suite behind them: that is enough to run an agent, expand it, and defend it to the board.