Care plans for AI systems: what maintenance involves

Maintenance for a live AI system means running the evaluation suite on a schedule, regression testing every prompt change, re-indexing knowledge as source content changes, migrating before models are deprecated, watching cost and latency drift, and answering incidents within a named response time. It is recurring measurement work, not occasional bug fixing.

Below is the actual calendar of that work, what sits in each Eazyware Care Plan tier with published prices, the AI-specific add-on and what it buys, and the case where a care plan is the wrong purchase.

Why an AI system needs different care from an ordinary application

A conventional web application degrades for reasons you can see. A dependency has a vulnerability, a certificate expires, a query gets slow as a table grows. Those things are true of AI systems too, and security patching cadence applies unchanged.

What is different is that an AI system degrades for reasons nothing in your monitoring reports. The knowledge base drifts away from what the retrieval index contains. A provider ships a new model version and the tone of answers changes. Users start asking a category of question the system has never seen because your business launched a product. A prompt edit made to fix one complaint quietly breaks four cases nobody is testing. None of these raise an alert, and all of them lower answer quality.

So the core of AI maintenance support is a measurement habit: a frozen evaluation set, run regularly, whose scores are reported whether or not anything changed. Everything else on the calendar exists to act on what those scores show.

The maintenance calendar

This is the shape of the work across a typical month on a production retrieval or agent system. Some items are automated, some are judgement, and the judgement items are why a care plan has engineer hours attached rather than being a monitoring subscription.

• Continuous: error and latency alerting, cost per request tracking with budget alerts, trace capture on every call so any answer can be reconstructed later.
• Weekly: evaluation suite run and score report, review of the worst-scoring answers, escalation and fallback rate review, a knowledge-gap report showing what users asked that the system could not answer.
• On every change: prompt regression against the frozen set before merge, with the change rejected if scores fall. Prompts are versioned and reviewed like code.
• Monthly: re-indexing as source documents change, permissions reconciliation so retrieval still respects access rights, dependency and security patching, cost review against budget.
• Quarterly: eval set refresh with new real cases, review of approval thresholds and whether any gated action has earned more autonomy, capacity and retention review.
• On provider events: model deprecation migration, re-benchmarking candidate models, and a scored comparison before anything is switched.
• On incident: triage, containment, root cause in the trace, fix, regression test, and a written note of what changed.

The weekly knowledge-gap report is the item clients underestimate. It converts maintenance from a cost into a product input, because the questions your system cannot answer are a list of content your business is missing. The practice is described in knowledge-gap reports.

What each Care Plan tier covers

Eazyware publishes three tiers plus an AI-specific add-on. The differences that matter are cover hours, response time and included engineer hours, not a longer feature list.

Response time is not resolution time, and any supplier who conflates them is selling you a number that cannot be met. Response is when a human acknowledges and starts work; resolution depends on what broke. The distinction is unpacked in SLAs that mean something and in the glossary entry on response versus resolution. Full pricing for every plan is on the pricing page.

How do you choose a tier?

Choose on the cost of the system being wrong for an hour, not on the size of your engineering budget. A knowledge assistant used internally can wait until morning; a voice agent taking patient appointments cannot.

Essential fits internal tools and systems with a human between the AI and the customer, where a slow answer is an inconvenience. Standard fits customer-facing systems with meaningful volume, where four hours is tolerable but overnight is not. Enterprise fits systems where the AI acts on customer accounts, money or clinical workflows, and where you want a named engineer who already knows your architecture at two in the morning.

Add the AI add-on whenever the system uses a language model in the live path. Without evals, prompt regression and re-indexing, you are paying for someone to fix outages while quality erodes unmeasured, which is the more expensive failure. Most clients stay on a plan for six to twelve months after launch; some stay indefinitely, and some take the work in-house with our documentation, which is a legitimate outcome.

The first month is not the steady state

The weeks immediately after launch are hypercare, not maintenance. Real users ask questions no eval set anticipated, edge cases surface at a rate they never will again, and thresholds set in testing turn out to be too tight or too loose. Expect to consume more included hours in month one than in month four, and expect the eval set to grow noticeably from real traffic. Budget for that rather than treating it as a sign something went wrong; a system that produces no surprises in its first month is usually a system nobody is using.

The model deprecation problem

Model providers retire model versions on published schedules, and a system pinned to a retired version stops working on a date somebody else chose. OpenAI, for instance, maintains a public deprecations page listing retirement dates and recommended replacements, and other providers do the same.

Handling this well is straightforward but it has to be somebody's job. You track the announcements for every model in your stack, benchmark the candidate replacements against your own eval set rather than the vendor's, budget for prompt adjustments because behaviour shifts between versions, and switch behind a flag so you can revert. Being model-agnostic by design makes this an afternoon rather than a project, which is why we route across providers from the start. The same flag that lets you switch forward is the one that lets you switch back when a new version scores worse on your own cases than the vendor's benchmarks suggested it would.

The clients who get hurt here are the ones whose original supplier finished the build, handed over a repository and left. Nobody was watching the deprecation notice, and the first sign of trouble was an error in production.

When a care plan is the wrong purchase

There are cases where we tell clients not to buy one.

You have an internal platform team already doing this. If you run your own observability, your engineers own the eval suite and someone is accountable for model changes, a care plan duplicates capability. Take the documentation and the handover instead, and consider a smaller quarterly review.

The system is a genuine experiment. A pilot used by a handful of internal people, with no customer exposure and no commitment to keep it, does not need a monthly contract. Wait until it is load bearing.

You want the plan to substitute for unfinished work. A care plan is not a way to finish a build that was never completed. If the system does not yet meet its acceptance criteria, that is remaining build scope and should be priced and closed as build scope. We will say so rather than take a monthly fee to keep something limping.

What this looks like in practice

The multilingual voice agent we built for a hospital network, described in the voice agent case study, is the kind of system where maintenance is not optional. Appointment booking is time sensitive, callers speak several languages, and speech models and telephony both sit in the live path. Monthly work includes re-running the evaluation set across every supported language, reviewing calls that ended in escalation, and watching per-minute costs against budget, alongside the ordinary patching.

None of that is dramatic. That is the point: maintenance that is visible is usually maintenance that arrived too late.

Before you sign any AI maintenance contract

• Confirm the evaluation suite is included and ask to see a sample score report
• Check that response time is defined separately from resolution time
• Ask what happens when a model you depend on is deprecated, and who tracks that
• Confirm re-indexing cadence and who owns the source content
• Check cost monitoring is included, with a named budget threshold and alert
• Agree what unused included hours do at month end
• Confirm you keep code, prompts, eval data and infrastructure regardless of the plan
• Ask what a plan exit looks like and how long handover takes

Related reading

What a care plan should cost, and what it should include takes the commercial view of the same subject, and application maintenance contracts covers the general AMC structure an AI plan sits inside. For the measurement habit at the centre of all of it, read Evals: the practice that separates AI demos from AI products.

An AI system does not break loudly; it drifts, and a care plan is simply the decision to keep measuring after the launch party.