How to handle hallucinations in production systems
How do you reduce LLM hallucinations in production systems?
Reduce hallucinations with grounding, structured outputs, evals, confidence thresholds and human review where the stakes require it. No single technique removes them; a production system layers five defences so a fabricated answer is rare, detected when it happens, and never reaches a customer where it matters.
To reduce LLM hallucinations in production you need layered defences rather than a better prompt. A language model produces fluent text whether or not it has the facts, and it does not know the difference. Production systems therefore ground the model in retrieved or supplied data, constrain outputs to schemas, check that answers stay inside the sources, measure the fabrication rate on a golden set, route low-confidence cases to a person, and design the interface so an unsupported answer is easy to spot. Each layer catches what the previous one missed. This article sets out the five layers as we build them under the LLM applications service, and how to decide how many a given feature needs.
What a hallucination is, and why it matters more in some places
A hallucination is an output not supported by the model's inputs or by fact: an invented policy clause, a wrong renewal date, a contact name that does not exist. The model is not lying; it is completing a pattern. The consequence depends on where the output goes: noise in a brainstorming tool, a complaint in support, a liability in finance, medicine or law. The design question is not "how do we eliminate hallucinations?" but "how many defences does this feature need for the harm a wrong answer causes?"
The five defences and what each catches
| Defence | What it does | What it catches | What it misses |
|---|---|---|---|
| Grounding | Answer only from retrieved or supplied context; cite it | Answers invented from general knowledge | Wrong or stale context; faithful summaries of bad sources |
| Structured outputs | Constrain answers to schemas with enums, nullable fields and evidence | Invented field values; free-text drift | Valid schema with wrong content |
| Groundedness checks and evals | Verify every claim traces to a source; measure fabrication rate on a golden set | Claims outside the context; regressions across changes | Cases the golden set does not contain |
| Confidence thresholds | Score each answer; below a threshold, decline or route to review | Uncertain answers presented as certain | Confidently wrong answers with high scores |
| Human review and UX | People check what matters; interface shows sources and uncertainty | Everything above, at a cost in time | Reviewer fatigue on high volumes |
Defence 1: grounding the LLM in your data
Grounding means the model answers from context you supply, not from what it absorbed in training. For knowledge questions that is retrieval: the passages that answer the question are fetched and placed in the prompt, and the model is instructed to answer only from them and to say when they do not contain the answer. For transactional questions it is tool use: the order status comes from the order system, not from the model's guess. Grounding removes the largest class of hallucinations, but it moves the risk to retrieval. If the wrong passage is retrieved, the model will faithfully summarise the wrong passage. That is why retrieval quality is measured separately, as set out in why basic RAG fails in production.
Two details matter. Instruct the model to quote or closely paraphrase and to cite each claim; models given that instruction fabricate less than models asked to "use the context". And give it an explicit way out: "not found in the provided documents" is a success, and the eval should reward it on questions the corpus cannot answer.
Defence 2: structured outputs that leave no room to improvise
Free text invites elaboration. A schema with an enum for the answer type, a nullable field for values that may be absent, and an evidence field that must contain a passage from the context removes most of the space in which fabrication happens. A model that must fill "source_quote" with text from the context will, far more often, either find it or return null. The validation layer then rejects any record whose quote does not appear in the supplied context, which turns a hallucination into a countable validation failure. The mechanics are in structured outputs and function calling.
Defence 3: groundedness checks and the fabrication rate
A groundedness check asks, for each claim in an answer, whether it is supported by the context. It can be run by a second model with a narrow rubric ("is this sentence entailed by these passages?"), by string matching for extracted values, or by a person on a sample. In production the automated check runs on every response where the stakes justify the cost, and flags or suppresses unsupported claims before the user sees them.
The same check on a golden set gives the fabrication rate: the share of answers containing at least one unsupported claim. This is the number to track across prompt, model and retrieval changes, per category, because a change that improves helpfulness often increases fabrication and only the measurement shows it. Frameworks such as Ragas implement faithfulness and related metrics for retrieval-based systems; the eval discipline itself is described in how to measure RAG quality.
Defence 4: confidence thresholds and routing to review
Confidence is a score attached to each answer that predicts whether it is right. It can come from retrieval strength, from the groundedness check, from the model's own stated confidence (useful but not reliable alone), or from a small classifier trained on reviewed outcomes. Above a threshold set per feature the answer goes out; below it the system declines, asks a clarifying question, or routes to a person with the draft and sources attached.
Thresholds are set from data, not intuition. Run the golden set, plot confidence against correctness, and choose the threshold that gives the acceptable wrong-answer rate for that feature's harm level. Revisit it whenever the model or retrieval changes, and watch the share routed to review; a threshold that sends half of all answers to people has not solved the problem. The stance is the one we apply to agents: shadow mode first, then autonomy on the cases the data says are safe.
Defence 5: human review and an honest interface
Where a wrong answer causes real harm, a person reviews before it goes out. The system's job is to make review fast: show the draft, its sources, the claims the groundedness check could not support and the confidence score, so the reviewer reads the risky parts rather than the whole answer. Review outcomes go back into the golden set, and categories with a consistently clean record move to sampling rather than full review.
The interface matters even without review. Citations that open the source, visible uncertainty and a clear "I could not find that" all make an unsupported answer easier for a user to catch. An assistant that presents every answer with equal confidence trains users either to trust it blindly or to distrust it entirely.
A worked example
A B2B SaaS company had an in-app assistant that answered configuration questions from product documentation. Users liked it, and then a customer configured an integration according to a step the assistant had invented: the documentation described a similar feature and the model had filled in the gap plausibly. The support cost of that one answer exceeded a month of the assistant's savings.
The fix layered the defences. The prompt was changed to answer only with quoted, cited steps; the eval showed this cut fabrication substantially but not to zero. Outputs moved to a schema with a per-step source quote validated against the retrieved context. A groundedness check stripped unsupported steps and replaced them with a link to the relevant page. Confidence gated whether the answer was shown as a procedure or as "here are the pages that seem relevant". Questions that changed customer data were routed to support with the draft attached for two months, then sampled. The fabrication rate on the golden set fell to a level the product team accepted. The build is described in the in-app copilot case study, and the pattern is standard in the SaaS copilots we ship.
Team and timeline
On a new build, grounding, structured outputs and the groundedness eval are part of the first two weeks; confidence thresholds are set once the golden set exists, usually by week four; review workflows and interface work follow. Retrofitting to an existing feature is typically one AI engineer for two to four weeks, with a product designer for the interface changes. The work is included in every LLM applications build, from $21,000 / ₹13.6L, and a three-week ProofRun at $6,250–10,500 will measure the fabrication rate on your data before you commit. Current figures are on the pricing page; the golden set, checks and thresholds belong to the client.
Before you start: a checklist
- Write down what a wrong answer costs for each feature; that sets the number of defences
- Ground every factual answer in retrieved context or a tool result, with citations
- Give the model an explicit "not found" path and reward it in the eval
- Use schemas with evidence fields validated against the context
- Build a golden set including questions the corpus cannot answer
- Measure the fabrication rate per category on every change
- Set confidence thresholds from the data and track the share routed to review
- Design the interface to show sources and uncertainty, and feed review outcomes back
Questions clients ask
- Can hallucinations be eliminated? No. They can be made rare, detectable and harmless for the decisions that matter, which is the production standard.
- Does a bigger model fix it? Stronger models fabricate less but still do; grounding and checks move the rate far more than model choice.
- Will fine-tuning help? Rarely for factual accuracy; it teaches style and format, not your current data. Retrieval and grounding do that.
- What about answers that are grounded but the source is wrong? That is a data-quality problem, surfaced by citations; the knowledge-gap loop fixes the source.
Related reading
See what makes an LLM application production-ready for the wider set of practices, LLM observability for finding the answer that went wrong, and evals for the measurement discipline behind the fabrication rate.
Ground it, constrain it, check it, score it and let a person see what matters; the model will still occasionally invent, and your system will catch it.
Frequently asked questions
What is the most effective way to reduce LLM hallucinations?
▾
Grounding: answer only from retrieved context or tool results, with citations and an explicit not-found path. It removes the largest class of fabrications, and structured outputs plus a groundedness check catch most of what remains.
How do you measure hallucinations in production?
▾
Run a groundedness check on a golden set to get the fabrication rate per category, track it on every change, and run the same check on live responses where the stakes justify it. Review outcomes and user feedback feed new cases into the set.
When should a human review AI answers?
▾
When a wrong answer causes harm the business cannot absorb: financial, medical, legal or customer-data changes. Route by confidence threshold so reviewers see the risky cases, and move clean categories to sampling over time. Our LLM application builds design this in.