azyware
Business

RAG vs long context windows: which to choose and when

EZ
Eazyware
· 7 min read
Quick answer

What is the difference between RAG and long context windows?

Retrieval-augmented generation finds the relevant few thousand tokens and sends only those to the model. A long context window sends everything and lets the model find them. RAG wins on cost, latency and corpus size; long context wins on simplicity and on documents that must be read whole.

Retrieval-augmented generation finds the relevant few thousand tokens and sends only those to the model. A long context window sends everything and lets the model find them. RAG wins on cost, latency and corpus size; long context wins on simplicity and on documents that must be read whole. The dividing line is usually how big the corpus is.

This article works through the arithmetic that settles most of these arguments, compares the two approaches across the dimensions that matter in production, names the cases where each clearly wins, and describes the hybrid that nearly every serious system ends up using.

Start with the arithmetic, not the architecture

Every question you ask a model has an input bill and an output bill. With retrieval, a typical question sends a few thousand tokens: the user's message, a system prompt and perhaps six retrieved passages. With a long context window, the same question might send two hundred thousand tokens because you pasted the whole handbook in. The output is identical in length. The input is fifty times larger, and you pay for it on every single question, not once.

Latency follows the same curve. Time to first token grows with input size, so a long-context answer that takes eight seconds where a retrieval answer takes two is a product decision as much as a cost one. Prompt caching changes this materially when the same large prefix is reused across many requests, which is why caching is the first thing to check before dismissing long context. Run your own numbers through the LLM inference cost calculator before you commit to either shape.

The third variable is corpus size. A million-token context window sounds enormous until you measure a real knowledge base. Ten years of support tickets, a product catalogue, a policy library or an engineering wiki will exceed any window that ships this year, and no amount of waiting changes that, because your corpus grows too.

What each approach actually is

Retrieval-augmented generation

Retrieval-augmented generation is a pattern in which a system searches a corpus for the passages most relevant to a question, then puts those passages into the model's prompt so the answer is grounded in them. The engineering sits in the search: chunking, embeddings, hybrid keyword and vector matching, reranking, permission filters and citations. What is RAG covers the pattern in business terms, and chunking strategies covers the decision that most affects answer quality.

Long context windows

A context window is the maximum number of tokens a model can consider in one request, shared between everything you send and everything it generates. Frontier models now offer windows measured in the hundreds of thousands of tokens, and some reach roughly a million. Anthropic's documentation on context windows sets out how the limit is shared between input and output, which is the detail that catches teams out when a long input leaves no room for a long answer. The context window glossary entry gives the short definition.

RAG vs long context: a side-by-side

DimensionRAGLong context window
Corpus size it handlesUnbounded; millions of documentsWhatever fits the window, per request
Cost per questionLow and roughly flatScales with how much you paste in, unless cached
Time to first tokenFast; small promptsSlower; grows with input length
Build effortWeeks: pipeline, index, evaluationDays: load the document and ask
FreshnessAs fresh as the last index runAlways current, because you send the file
CitationsNatural; you know which passage was usedPossible, but the model must be asked to quote
Access controlEnforced at retrieval, per userYou must filter before assembling the prompt
Failure modeRight answer exists but is not retrievedDetail present but overlooked in a long input
Best fitLarge, permissioned, changing corporaOne long document read end to end

When is a long context window the better choice?

Long context wins when the unit of work is one document and the whole document matters. Reviewing a hundred-page contract for inconsistencies, summarising a full deposition, comparing two versions of a policy, answering questions about a single financial filing: in all of these, chunking actively destroys the thing you need, because the answer depends on how clause nine interacts with clause forty-one. Retrieval would hand the model six passages and hide the relationship.

It also wins on speed to first result. If you are still deciding whether a task is feasible at all, paste the document in and find out this afternoon. We do exactly that during discovery, because there is no point building an index for a task the model cannot perform.

When RAG is the better choice

Retrieval wins whenever the corpus is larger than a window, changes often, or carries per-user permissions. A support assistant over ten years of tickets, an internal knowledge assistant across departments, a product search, a policy assistant in a regulated firm: none of these can send everything, and none of them should, because different users are allowed to see different things. Retrieval is also the only approach that gives you citations for free, and citations are what make an answer auditable.

There is a quality argument too. A well-tuned retriever that puts six precisely relevant passages in front of a model usually beats a window stuffed with two hundred thousand tokens of mostly irrelevant text, because precision in the prompt is precision in the answer. Hybrid search explains why vector similarity alone is rarely enough to reach that precision.

The hybrid almost everyone ends up with

The mature answer is not one or the other. Retrieve at the document level, then use a large window generously on what you retrieved. Instead of six three-hundred-token chunks, fetch the four documents that matter and send them whole, which preserves internal structure while keeping the prompt a tenth of the corpus. Add prompt caching for any prefix you reuse, and semantic caching for repeated questions. This shape gives you the coherence of long context and the economics of retrieval, and it is what we build by default.

A four-question test

  • Does the whole corpus fit in a window today, with room to grow for two years? If not, you need retrieval regardless of anything else.
  • Do different users have different access rights to the content? If yes, retrieval with permission filtering is the only safe design.
  • Does the answer depend on relationships across the whole document? If yes, send documents whole rather than chunked.
  • How many questions per day, multiplied by prompt size? Multiply it out before the architecture meeting; the number usually decides the argument.
  • Can you cache the large prefix? If the same corpus is sent repeatedly, caching can make long context affordable where raw pricing would not.

Where neither is the right answer

If the model needs to learn a style, a format or a specialised vocabulary rather than facts, neither retrieval nor a bigger window is the tool; that is a fine-tuning question, and our RAG versus fine-tuning comparison sets out the split. If the underlying content is contradictory, out of date or scattered across four systems nobody maintains, both approaches will faithfully return the wrong answer. Fixing the knowledge base is unglamorous work that no architecture substitutes for.

And if the real task is calculation or lookup against structured data, put the model in front of a query engine instead of a document store. Asking a language model to add up figures buried in a long context is a reliable way to produce confident arithmetic errors. Text to SQL over a governed semantic layer answers those questions correctly and shows its working, and it costs a fraction of either approach discussed here.

What it costs to change your mind

Starting with long context and moving to retrieval is common and manageable. The prompts, the evaluation set and the interface survive; the new work is the ingestion pipeline, the index, the reranker and the permission model. For a single corpus, budget three to six weeks. Going the other way is easier still: you already have the documents and the evaluation set, so you delete the retriever and send more. The expensive mistake is neither of these; it is building a retrieval pipeline before anyone has checked whether the model can do the task at all, which is what a three-week ProofRun at $6,250 or ₹4,00,000 exists to prevent.

What a build looks like

Retrieval and knowledge engineering at Eazyware starts at $14,000 or ₹8,80,000 and runs to $49,000 or ₹32,00,000 depending on corpus size, permissions and the number of source systems, with all starting prices on the pricing page. Our KYC document intelligence case study shows the document-level approach in a regulated setting, where whole documents mattered and data could not leave the client's environment. After launch, the AI add-on to a care plan at $750 or ₹40,000 a month covers re-indexing, evaluation runs and cost monitoring, which is where long-context bills quietly grow if nobody watches them.

The comparison hub collects the neighbouring decisions, and why basic RAG fails in production is the post to read before you assume retrieval is the easy option. If you are choosing an index rather than an architecture, the comparison of pgvector, Pinecone and Qdrant covers that decision separately.

Size the corpus and multiply out the tokens before anyone draws an architecture diagram; that number, not the model's headline window, decides this.

Frequently asked questions

Do long context windows make RAG obsolete?

▾

No. Real corpora outgrow any window, they change daily, and different users are allowed to see different parts of them. Retrieval also keeps cost and latency roughly flat per question and produces citations. Long context complements retrieval by letting you send whole documents once they have been selected.

Is RAG cheaper than a long context window?

▾

Per question, almost always, because you send a few thousand tokens instead of hundreds of thousands. Prompt caching narrows the gap when the same large prefix is reused across many requests. Multiply your daily question volume by prompt size for both designs before deciding; the arithmetic usually settles it.

Can you use RAG and long context together?

▾

Yes, and this is the pattern we build most often. Retrieve at the document level rather than the chunk level, then send the selected documents whole into a large window. You keep the coherence of full documents and the cost profile of retrieval, with prompt caching applied to any reused prefix.