Shadow mode: the right way to launch AI agents in production

Launching AI agents in production is where most agent projects fail, and they fail in the same way: a demo that impressed a steering committee is switched on for real customers, something goes wrong in week one, and the agent is switched off for good. Shadow mode is the alternative. The agent runs on live traffic from day one but only proposes; a person approves, edits or rejects every action. When the record shows the agent is right often enough on a given intent, that intent, and only that intent, is released to run alone.

This article describes the stages, the evidence needed to move between them, and the mistakes that turn shadow mode into theatre.

Why an AI agent rollout needs shadow mode

An evaluation suite tells you how the agent behaves on the cases you thought of. Production tells you about the cases you did not. Shadow mode is the bridge: the agent sees everything real traffic contains, including the malformed, the unusual and the adversarial, while a human still owns every outcome. Nothing reaches a customer or a ledger that a person did not approve.

It also does something an eval cannot: it builds trust in the team that will live with the agent. Support leads, dispatchers and finance analysts who spent three weeks approving proposals know exactly what the agent does well and where it stumbles. Their sign-off means something. A release forced on them from above does not. The guardrail design that makes this safe is in How to build an AI agent that is safe to run unattended.

The four stages of staged autonomy

Stages are per intent, never for the whole agent. "Order status" may reach stage three in a fortnight while "cancel subscription" stays at stage one for a quarter, and some intents are designed to remain at stage one permanently.

Stage 1: what agent shadow mode looks like day to day

The agent receives the same inputs a person would: the ticket, the call, the document, the dispatch request. It produces a complete proposed action, the tool calls it would make, and a short explanation with the evidence it used. That proposal appears in the human's existing tool, alongside the case, with three buttons: approve, edit, reject. Rejections and edits require a reason from a short list, which is the most valuable data the project will generate.

Latency matters here. If the proposal arrives after the person has already started the work, they ignore it and the shadow record is worthless. The proposal has to be there before they open the case, which usually means triggering the agent on ticket creation rather than on assignment.

Reading the shadow record

Each week, the team reviews acceptance, edit and rejection rates per intent, and clusters the rejection reasons. Some clusters are prompt problems and are fixed and re-evaluated. Some are missing tools: the agent could not see the information it needed, and a new tool contract is added. Some are policy problems, where the humans themselves disagree, and those are the most useful finds of all, because they surface rules that were never written down. Each rejection becomes a scenario in the evaluation suite, so the suite grows from real failures rather than imagined ones. The metrics are set out in How to measure an AI agent.

Stage 2: gated autonomy and moving the thresholds

When an intent's acceptance rate is high and its rejections are understood, it moves to gated autonomy. The agent acts alone inside limits set by the business owner: below a value, for verified customers, outside a sensitive window. Everything above the limit is still proposed for approval. Over the following weeks the limits widen as sampled runs stay clean.

Limits live in configuration, not prompts, and are changed through a review with the owner. A widening that follows a bad week is reversed. The specific gate design for money-moving actions is in Policy-gated actions.

Stage 3: full autonomy, and why review never stops

At full autonomy on an intent, the agent acts and the human sees escalations plus a random sample. The sample is not optional. Models are updated by vendors, prompts drift as teams tweak them, upstream systems change their responses. The evaluation suite catches regressions before release; the sample catches the ones that reach production anyway. A rising error rate on the sample demotes the intent a stage until the cause is found.

Mistakes that make shadow mode theatre

• Approving without reading. If approvers rubber-stamp, acceptance rate is meaningless. Track time-to-approve and spot-check approved proposals against the evidence.
• No rejection reasons. A reject button without a reason produces a number, not a diagnosis. Reasons are mandatory.
• Whole-agent promotion. Promoting every intent at once because the average looks good hides the intent that is failing.
• Shadow mode with fake traffic. Replayed tickets are evals, not shadow mode. The point is live, unfiltered input.
• No end date. Shadow mode that runs indefinitely because nobody owns the promotion decision burns approver goodwill. Set review cadences and decision owners.
• Skipping stage 2. Jumping from propose-only to full autonomy misses the cheapest safety net available.

A worked example

A direct-to-consumer brand launched a WhatsApp agent for order queries, exchanges and returns. Stage zero built an evaluation suite from a few months of real conversations. In shadow mode the agent drafted every reply and action, and support staff approved or edited from their existing console. Order-status replies were accepted almost immediately and promoted within weeks. Exchange requests revealed a missing tool: the agent could not see warehouse stock, so it proposed exchanges that could not be fulfilled. The tool was added and the intent re-entered shadow.

Returns above a value threshold were designed to stay gated permanently, because the business wanted a person on any high-value decision. Personalised product recommendations, running on the same platform, were released only after a controlled comparison. The personalisation and WhatsApp agent case study describes the deployment; the customer service agents page describes the service.

Team and timeline

Shadow mode typically runs two to four weeks per intent group and is built into every agent programme we deliver rather than added after. On our side, an AI engineer runs the weekly review and the eval updates, and a backend engineer maintains the approval interface and limits. On yours, the operational owner (support lead, dispatch manager, finance controller) chairs the review and signs promotions. This is included in the multi-agent systems service from $24,500 or ₹16 lakh and in customer service agents from $12,500 or ₹8 lakh. After launch, a Standard Care Plan at $2,500 or ₹1,60,000 a month covers the ongoing sample review and eval reruns on model changes; see the pricing page for the full set. If you are not yet sure which intents to include, a ten-day Sprint Zero produces the intent list and stage plan.

Before you start: a checklist

• List intents and decide which may reach full autonomy and which stay gated forever
• Build the evaluation suite from real traffic before shadow mode begins
• Put the proposal in the approver's existing tool, before they open the case
• Make rejection reasons mandatory and short
• Name the operational owner who chairs the weekly review and signs promotions
• Set limits per intent in configuration with a documented change process
• Define the demotion rule: what sample error rate sends an intent back a stage
• Agree a review cadence and a decision date so shadow mode has an end

Questions clients ask

• How long does shadow mode take? Two to four weeks for simple intents with clean data; longer where rejections reveal missing tools or unwritten policy.
• Does shadow mode slow the team down? Slightly, in week one. By week two approving a good proposal is faster than doing the work, and the team is net ahead.
• What acceptance rate is enough? It depends on the cost of an error for that intent. The operational owner sets the bar, informed by rejection reasons rather than a single number.
• Can we run shadow mode on voice? Yes, for outcomes: the agent proposes the booking or follow-up, and the person confirms. Live conversation cannot be shadowed turn by turn, so voice relies more on evals and gated autonomy.

Related reading

AI agent vs chatbot explains why agents need a rollout that chatbots never did, AI agent orchestration covers where approval signals live in the architecture, and Google's Site Reliability Engineering material on canarying and gradual rollout is the primary source the staged approach borrows from.

Let the agent earn each permission on the record, intent by intent, and the launch that everyone fears becomes a series of small, boring decisions.