Five ways self-hosted AI agents projects fail, and how to avoid each

Self-hosted AI agent projects fail for five reasons: the model is chosen before the evaluation set exists, capacity is sized for the average rather than the peak, tools are given broad write access instead of narrow contracts, the agent goes live without a shadow period, and nobody owns the system after launch. Each is a decision, not an accident.

This piece takes each pattern in turn: how it presents, what causes it, and the specific engineering or governance decision that prevents it. All five are cheap to avoid at the start and expensive to fix once the system has write access to production.

The five patterns at a glance

Failure one: choosing the model before the evaluation set

This is the most common and the most expensive. A team picks an open-weight family because it fits the GPUs they have, tunes prompts for six weeks, and has no way to tell whether Friday's version is better than Monday's. Every discussion becomes an argument about anecdotes.

The cause is sequencing. The evaluation set is treated as testing work to be done near the end, when it is actually the specification. Without one hundred to three hundred real cases with agreed correct outcomes, there is no definition of working, so there is nothing to sign off and no basis for widening autonomy. The symptom to watch for is a team that reports progress in features shipped rather than in a score moving.

The fix is to build the scored set first, then benchmark two or three open-weight candidates against a frontier API as a control. If the best local candidate is materially worse on your data, that is a finding worth having in week three rather than month four. Evals: the practice that separates AI demos from AI products covers how to construct and maintain the set.

Failure two: sizing capacity for the average

Agents are not chat interfaces. A single task can involve several planning turns plus tool results read back into context, so tokens per task are often an order of magnitude above what a demo suggests. Teams that size from an average daily token count discover the gap on the first busy Monday, when requests queue, latency climbs and people go back to doing the job by hand.

The remedy is arithmetic, not hardware generosity. Estimate peak concurrent tasks, multiply by your longest realistic context, and size the serving layer against that, then add standby headroom you expect to waste. Run a batching inference server so the accelerator stays busy across concurrent requests rather than idling between them. GPU sizing for private AI works through the calculation, and the LLM inference cost calculator turns it into a monthly figure.

Failure three: broad credentials instead of narrow tools

An agent handed a database connection or a general-purpose service account is a single planning error away from a wide, hard to reverse change. It usually happens for speed: wiring one integration is faster than writing six contracts, and nobody expects the agent to do anything unusual until it does.

Excessive agency is a named risk in the OWASP Top 10 for Large Language Model Applications, which recommends limiting the functions, permissions and autonomy an agent is granted rather than relying on the prompt to constrain behaviour. That is the right instinct in production: each capability becomes a narrow tool with one verb, explicit limits, identity propagation so existing access control still applies, idempotency keys because agents retry, and an audit record written whether the call succeeds or fails.

Then add approval gates by action class. Anything irreversible, above a threshold, or touching a regulated record goes to a person first. How to build an AI agent that is safe to run unattended sets out the full pattern.

Failure four: going live without shadow running

A project with a launch date and no shadow period is betting that the first production week will be quiet. It rarely is. An agent that looked strong in demos meets the long tail of real cases, makes a visible mistake with write access, and gets switched off. The technical damage is small and the political damage is total: the second task family never gets funded.

Shadow running is not a delay, it is the evidence-gathering phase. The agent proposes, reviewers accept or correct, and acceptance rates by action class tell you exactly which capabilities are ready. Three to six weeks is typical, driven by task volume rather than calendar. When acceptance holds at the threshold agreed before launch, one class at a time moves to autonomous, starting with the reversible ones. Shadow mode describes the mechanics.

Failure five: nobody owns it after launch

A self-hosted agent is a running system with an operating cost in attention as well as money. Models get deprecated, documents move, prompts drift out of step with policy, and inference spend creeps upward without anyone noticing. Six months after a successful launch, an unowned agent is quietly worse than it was on day one and nobody can say when that started. The drift is rarely dramatic; it shows up as a slowly rising escalation rate that everyone attributes to harder cases until somebody checks the traces and finds a prompt that has not matched the current refund policy since March.

Ownership means three specific things: someone re-runs the evaluation suite on every model, prompt or tool change; someone watches cost per completed task and escalation rate weekly; someone reviews the traces behind escalations and feeds fixes back. LLM observability is what makes the third possible, and Eazyware Care Plans from $1,000 or ₹68,000 a month, with the $750 or ₹40,000 AI system add-on covering evaluations, cost monitoring, prompt regression and re-indexing, exist to carry it. Maintenance and support sets out what each tier includes.

Early warning signs

• Nobody can name the metric that would tell you the agent is working
• The evaluation set is still a plan in week six
• A demo is the only artefact anyone has seen
• The agent authenticates as itself rather than as the person it acts for
• Infrastructure sizing came from a vendor deck rather than your own peak
• The launch plan has a go-live date but no shadow period
• No named owner for evaluations after handover
• Cost is reported per API call instead of per completed task

The sixth failure: self-hosting when you should not

Some projects fail because the whole approach was wrong. At low or irregular volume, capacity you own sits idle and costs more than per-token billing while adding patching, capacity planning and model upgrades to your team's workload. If no contract, regulator or client questionnaire requires data to stay inside your perimeter, the honest answer is to build on a hosted API and move in-house when the bill or the obligation justifies it.

Self-hosting also fails when the underlying process is undefined. An agent applies a procedure; it does not invent one. If three people do the task three ways and none can say which is correct, spend the money on defining the work first. Saying no to a private deployment at that point is cheaper for everyone than proving it in production.

What a well-run engagement does differently

It front-loads the decisions. A ten-day Sprint Zero, sold as the AI discovery sprint at $3,250 or ₹2,00,000 and credited to the build, produces the task list, the integration inventory and the evaluation plan. A three-week ProofRun at $6,250 or ₹4,00,000 tests the hardest task on real data before anyone commits to the full build, which for self-hosted agentic AI runs from $31,500 or ₹20,80,000 to $105,000 or ₹72,00,000 plus infrastructure. All published figures are on the pricing page.

It also treats the first task family as the platform investment. In the KYC document intelligence work for an NBFC, most of the engineering went into extraction accuracy, permissioned retrieval and reviewer workflow rather than the model itself, which is the pattern in private deployments: the second capability is far cheaper than the first because the serving, permission and review layers already exist.

Related reading

Self-hosted AI agents: a practical implementation guide covers the build stage by stage, and how to measure whether self-hosted AI agents is working covers the instrumentation that catches four of these five patterns before they become incidents.

None of these failures is a model problem, which is the useful thing to notice: private agent projects succeed or fail on evidence, permissions and ownership, and all three are decisions you make in the first fortnight.