Reranking

First-stage retrieval, whether vector, keyword or hybrid, is built for speed over millions of chunks and returns a rough top 50 or 100. A reranker then reads each candidate together with the query and produces a much more precise relevance score. Cross-encoder models do this by attending to query and passage jointly, which is far more accurate than comparing two independent embeddings but too slow to run across the whole corpus.

The reranked top 5 to 10 go into the prompt. Because the model only sees a handful of passages, the ordering matters: a good reranker lifts the passage that actually contains the answer above near-duplicates and topically similar but unhelpful ones. It also lets the pipeline retrieve generously at stage one without paying for the extra context at generation time.

Reranking is not a fix for retrieval that never found the answer; it can only reorder what stage one returned. It is also distinct from the LLM itself deciding relevance, which works but is slower and more expensive per query.

• CTO / Head of Engineering: A reranker is a bounded, swappable component that typically lifts answer accuracy more than any prompt change.
• CFO: Sending fewer, better passages to the model reduces tokens per answer, which offsets the reranker's own inference cost.
• Product manager: Better top-ranked passages mean fewer answers where the correct source is present but buried, which users experience as the assistant "missing the obvious".
• CISO: Rerankers can be small open-weight models run inside your VPC, so adding one need not send data to a new vendor.

Fast retrieval and accurate retrieval pull in opposite directions. An index that scans millions of vectors in milliseconds must use cheap similarity, which ranks loosely. Reranking exists to add a slow, precise judgement only where it is affordable: on the short list. The trade-off is added latency (tens to a few hundred milliseconds) and another model to host or pay for. When the first stage is already returning the right passage at rank one most of the time, reranking adds little; measure before adding it.

• Selecting the right help-centre section for a SaaS copilot when several articles cover overlapping features
• Ranking the correct policy circular above superseded versions in an NBFC's compliance assistant
• Prioritising the exact clinical protocol over related guidelines in a hospital's private assistant
• Choosing the most relevant product page among near-identical variants in a retail search engine
• Ordering delivery-exception precedents for a logistics support agent so the closest match leads

Technique / practiceA retrieval engineering practice using a cross-encoder or hosted reranking model. Eazyware adds and benchmarks rerankers within Retrieval & Knowledge Engineering builds, measuring the lift on the client's golden question set before keeping it.

Eazyware service that covers it: Retrieval & Knowledge Engineering. Starting prices are on the pricing page.