Chunking

Before documents can be indexed for RAG, they are cut into chunks, each of which gets its own embedding. The chunk is what the retriever returns and what the model reads, so its boundaries decide whether an answer arrives whole or truncated. Strategies include fixed token windows with overlap, splitting on headings and paragraphs, table-aware splitting, and parent-child schemes where a small chunk is matched but its larger parent section is passed to the model.

The right strategy depends on the document. A contract needs clause-level chunks that keep the clause number attached. A policy manual splits by heading. A spreadsheet or invoice needs rows or line items kept together with their header. Chat transcripts chunk by exchange. Applying one window size to all of them is the most common cause of a RAG system that retrieves the right document but the wrong sentence.

Chunking is not an OCR or parsing step, though it depends on both; garbage from a bad PDF parse is chunked into garbage. It is also not a one-time decision: chunk metadata (title, section, page, version) is what allows citations and filtered retrieval later.

• CTO / Head of Engineering: Chunking is the highest-leverage, lowest-glamour variable in retrieval quality and is cheap to change early and expensive to change late.
• Product manager: Whether a citation lands the user on the exact clause or on a whole 40-page PDF is decided at chunking time.
• Compliance officer: Chunks that carry version and section metadata make each answer traceable to a specific document revision.
• Data lead: Chunk strategy is testable; retrieval recall against a golden set moves measurably when it changes.

Models have a limited context window and retrieval works on similarity, so you cannot embed a 200-page manual as one item and expect a question about one clause to match it. Chunking exists to give the retriever units that are small enough to be specific and large enough to be understood alone. The trade-off runs both ways: too small and chunks lose their subject ("it must be filed within 30 days" of what?), too large and the model receives pages of irrelevant text around the answer, raising cost and lowering accuracy.

• Clause-level chunking of loan agreements and RBI circulars for an NBFC's policy assistant
• Heading-based chunking of a SaaS help centre so copilot citations link to the exact article section
• Table-aware chunking of hospital formularies and dosage tables so rows stay attached to their headers
• Module and lesson chunking of course material for an AI tutor, with prerequisite metadata
• Line-item chunking of supplier catalogues and rate cards for a logistics procurement assistant

Technique / practiceA retrieval engineering practice decided per document type and validated against a golden question set. It is one of the first things Eazyware's Retrieval & Knowledge Engineering team examines when a RAG system underperforms.

Eazyware service that covers it: Retrieval & Knowledge Engineering. Starting prices are on the pricing page.