Model drift

Model drift describes an AI system that performed well at launch and performs worse months later without any deliberate change. In classical machine learning it has two forms: data drift, where the distribution of inputs shifts, such as a new product line or a new customer segment, and concept drift, where the relationship between inputs and the correct output changes, such as fraud patterns evolving. The model is unchanged; the world is not.

LLM applications add a third source: the model itself moves. A hosted provider updates a version and the same prompt produces different outputs. Retrieval corpora go stale as policies change. Users learn to phrase requests in ways the prompt never anticipated. The effect is the same slow decline, and it is often invisible because no error is thrown.

Drift is not the same as a bug. A bug is a defect present from the start; drift is a system becoming wrong over time. Detecting it requires a baseline, usually an evaluation suite run periodically and production metrics such as escalation rate or human-override rate tracked against launch values.

• Data lead: the primary reason to monitor production accuracy rather than trusting a launch-time evaluation.
• CTO / Head of Engineering: dictates that evals run on a schedule and on provider updates, not only when the team changes code.
• Support manager: rising escalation or reopen rates are often the first visible sign that an AI agent has drifted.
• CFO: a drifting model erodes the return on the AI investment silently, so detection protects the business case.

The term exists because the most common failure of AI in production is not a dramatic outage but a quiet decline that nobody owns. A forecasting model trained before a market shift, a support agent grounded in last year's policies, a classifier facing a new document format: each keeps producing confident answers that are increasingly wrong. Naming drift makes it a monitored condition with a response, such as retraining, corpus refresh or prompt revision, rather than a surprise. The trade-off is ongoing effort: periodic evals, metric tracking and someone reading them. That is far cheaper than the cost of decisions made on a model that stopped being right.

• A retailer whose demand forecast degrades after a new store format launches, caught by tracking forecast error weekly against the launch baseline.
• A fintech fraud model whose false-negative rate creeps up as fraud tactics change, detected through monthly evaluation on labelled recent cases.
• A SaaS support agent whose resolution rate falls after the help centre is reorganised, traced to stale retrieval chunks.
• A hospital voice agent that misroutes more calls after a new department opens, flagged by a rise in human transfers.
• A document-extraction pipeline for a lender that starts missing fields when a bank changes its statement layout.

ConceptA phenomenon to monitor for rather than a skill to hire. Eazyware tracks drift through scheduled evaluations and production metrics as part of Maintenance & Support care plans, with retraining, corpus refresh or prompt revision as the response.

Eazyware service that covers it: Software Maintenance & Support. Starting prices are on the pricing page.