Why basic RAG fails in production and how to fix it

Retrieval-augmented generation is the most common architecture for enterprise AI and the most commonly disappointing. The demo works: a few PDFs, a vector store, a prompt, impressive answers. Then real users arrive with real documents and the system returns confident nonsense, misses the obvious passage, leaks a file to the wrong person or answers from last quarter's policy. The gap between 70% and 95% relevance is not a better model; it is six engineering decisions. This article walks through each failure, why it happens and what fixes it, and ends with how to measure the result so you know the fixes worked.

Failure 1: parsing destroys the document

Most pipelines extract text naively. Tables become word soup, headings lose their hierarchy, multi-column layouts interleave, footnotes attach to the wrong paragraph, and scanned pages are skipped. The model then retrieves fragments that no longer mean what the document meant. Fix: layout-aware parsing that preserves structure, tables as tables, headings as context, with OCR or a vision-language model for photographed and scanned pages. Open tools like Docling handle much of this; the point is that parsing is a stage to be engineered and tested per document type, not a library call.

Failure 2: one chunk size for everything

Fixed 500-token windows cut contract clauses in half, merge unrelated ticket comments and split a table from its header. Fix: chunk by structure. Contracts by clause, manuals by section, tickets by message, spreadsheets by row group. Attach metadata (document, section, date, owner) to every chunk so filters and citations work. Chunking strategy is decided per source, and revisited when retrieval evals show a category failing.

Failure 3: vectors alone miss exact matches

Embeddings capture meaning and miss specifics: a part number, an error code, a person's name, a legal term. Users searching for exactly those get semantically similar but wrong passages. Fix: hybrid search, keyword (BM25) plus vector retrieval, merged and passed through a re-ranker that scores the candidates against the actual question. Hybrid retrieval is the single largest relevance improvement most systems can make, and it is cheap.

Failure 4: the index goes stale

Policies change, prices change, tickets close. An index built once answers from the past. Fix: refresh on a schedule and on events, with versioning so you can see which document version an answer came from. For high-change sources like ticket systems, incremental indexing on change; for slow ones, nightly.

Failure 5: no permissions at retrieval

The most dangerous failure: a user asks a question and the system retrieves a document they were never allowed to see, because the index does not know who is asking. Fix: permission-aware retrieval. Every chunk carries the access control of its source; every query is filtered by the user's entitlements before ranking. This must be enforced in the retrieval layer, never in the prompt, and tested with negative cases ("user X must not receive document Y").

Failure 6: nobody can measure relevance

Without a measurement, every change is a guess and every complaint is an anecdote. Fix: a golden set of one to three hundred real questions with verified answers and source passages. Measure retrieval recall (did the right passage appear?), precision (how much noise?), and answer groundedness (did the answer stay inside the retrieved text?). Run it on every change. This is also how you decide between fixes 1–5: the eval tells you which category is failing. See RAG vs fine-tuning for how the same set informs the fine-tuning decision.

A production RAG architecture, in one table

Vector database choice matters less than you think

pgvector inside Postgres handles most enterprise workloads with the benefit of transactions and existing backups; dedicated stores like Qdrant, Weaviate or Pinecone earn their place at scale or when filtering and multi-tenancy demand it. Choose the store after the retrieval design, not before. The pgvector project documents its limits honestly.

GraphRAG: when relationships matter more than similarity

Questions like "which suppliers are affected if this component fails?" are about connections, not similar text. GraphRAG extracts entities and relationships into a graph and retrieves along it. It is more work to build and worth it only for relationship-heavy domains; we cover the decision on the Retrieval & Knowledge Engineering page.

What good looks like

Answers that cite the passage, with a click-through to the source. Recall and precision numbers on a golden set instead of anecdotes. An index that refreshes on schedule. Zero permission violations, verified by test. A system that scales from a pilot corpus to millions of documents without changing architecture. The KYC document-intelligence case study shows this shape in a regulated setting. If your first RAG attempt underperforms, a three-week ProofRun builds the golden set and tells you which of the six failures you have.

A diagnostic you can run this week

Take twenty real questions your users asked and, for each, find the passage in your documents that answers it. Run them through your current system and record three things: did the right passage appear in the retrieved set, was the answer inside that passage, and did the user have permission to see it. Twenty questions is enough to see the pattern. If the right passage is missing, the problem is parsing, chunking or search. If it appears but the answer wanders, the problem is grounding and prompting. If a passage appears that the user should not see, stop and fix permissions before anything else.

Refresh and versioning in practice

Give every document a version and every chunk the version it came from. Index on change for fast-moving sources, nightly for slow ones. Show the version in citations so a user can tell a current policy from last year's. Keep the previous version queryable for a period so audits can reproduce an answer. This is a few days of engineering and removes an entire class of "the AI said something out of date" complaints.

Who this is for

• Organisations with large, changing document sets: policies, contracts, manuals, tickets
• Products where grounded, citeable answers are a requirement, not a nice-to-have
• Teams whose first RAG attempt demoed well and disappointed in production
• Regulated industries that need permission-aware retrieval and an audit trail

The knowledge-engineering offering, including GraphRAG and document intelligence, is on the Retrieval & Knowledge Engineering page; enterprise search and internal assistants often pair it with a SaaS copilot. Starting prices are on the pricing page, and the LLM application practices for evals and observability apply to every RAG system we ship.

Team and timeline

Production RAG is typically a retrieval-focused AI engineer, a data engineer for ingestion and refresh, and an architect for the permission model, over four to ten weeks. The first two weeks build the golden set and the ingestion pipeline for each document type; the middle weeks tune retrieval against the set; the last weeks add permissions, refresh, observability and the answering layer with citations. The single biggest schedule risk is discovering a new document type late, which is why the corpus audit comes first.

Before you start: a checklist

• Inventory document sources, formats and how often each changes
• Write down who may see what; get the permission model signed off
• Collect 100–300 real questions with verified answers
• Decide chunking per document type, not globally
• Plan hybrid search and re-ranking from the start
• Schedule refresh per source and version every chunk
• Put recall, precision and groundedness on a dashboard

The answering layer: what happens after retrieval

Retrieval delivers passages; the answering layer decides what to say. It should quote or paraphrase only from the retrieved text, cite each passage, state a confidence, and, critically, say "I could not find that" when nothing relevant was retrieved rather than improvising. It should also handle multi-part questions by retrieving per part and composing, and refuse to answer from documents the user is not entitled to even if they were retrieved by mistake, as a second line of defence. This layer is where hallucinations are prevented or permitted, and it is evaluated with the same golden set: groundedness measures whether the answer stayed inside the sources. Systems that skip this discipline produce fluent answers with invented details, which users detect quickly and trust never returns.

Related reading

Related reading: RAG vs fine-tuning for when retrieval is not enough, Text-to-SQL accuracy for the structured-data cousin of this problem, and What is an AI agent? for how retrieval feeds agents.

A last word on expectations. Ninety-five percent relevance on your golden set is achievable for most enterprise corpora with these six fixes; one hundred is not, and a system that claims it has stopped measuring. Design the answering layer to say "not found" gracefully, and users will trust the ninety-five.