Reranking in RAG: Why the Right Document Beats More Documents

More context is not the same as better context

When a RAG assistant gives a vague answer, the instinct is to give it more. Retrieve 20 chunks instead of 5. Use a model with a bigger context window. Surely the answer is in there somewhere.

It often is in there, and that is the problem. Put the one relevant paragraph among nineteen that are merely on-topic and the model has to find it, weigh it against near-misses and resist blending them. Models are generally better at using information near the start or end of their context than buried in the middle, and every extra chunk costs tokens and time on every request.

Reranking takes the opposite approach: retrieve broadly, then choose carefully, and pass the model only the handful of passages that genuinely answer the question.

What a reranker actually does

First-stage search, whether keyword, vector or hybrid, has to be fast across the whole collection. Vector search compares a question's embedding with pre-computed document embeddings, and the question and document never meet directly. That is efficient and approximate.

A reranker, typically a cross-encoder model, reads the question and each candidate passage together and scores how well that passage answers that question. It is far too slow to run on a million chunks, but perfectly quick on fifty.

1. First-stage retrieval returns the top 30 to 100 candidates
2. The reranker scores each candidate against the question
3. Candidates are reordered by reranker score
4. The top 3 to 8, above a minimum score, go to the LLM
5. If nothing clears the minimum, the assistant says it could not find an answer

Why it improves answers so noticeably

The last row is underrated. A reranker's score is a far better signal of whether anything relevant exists than a vector similarity score, which makes it the natural place to decide when an assistant should decline to answer rather than improvise.

Options for reranking

• Hosted reranking APIs from several search and model providers. Simplest to add, a small cost per query, and data leaves your environment.
• Open-source cross-encoders run on your own servers. More setup, and a GPU helps at volume, but data stays in-house.
• LLM as reranker. Ask a small, fast model to score or order passages. Flexible and can follow business rules, but slower and costlier than a dedicated reranker.
• Rule-based boosts layered on top: prefer current document versions, the user's region, or official policies over meeting notes.

Business rules matter more than people expect. A reranker cannot know that the 2026 price list supersedes the 2025 one unless metadata tells it, so combine model scores with simple boosts and filters.

Keeping latency under control

Reranking adds a step, and users notice delays. It is usually manageable.

• Rerank 30 to 50 candidates rather than 200; gains flatten out quickly
• Truncate long passages to the most relevant portion before scoring
• Run retrieval and any query rewriting in parallel where possible
• Cache reranked results for frequent identical questions, scoped by user permissions
• Show a searching indicator and stream the answer, so the wait feels shorter

In most of the assistants we build, reranking adds a fraction of a second and saves more than that in generation time, because the prompt is shorter. Our post on latency in AI apps covers the wider picture.

When reranking will not help

A reranker can only reorder what first-stage retrieval found. If the correct passage is not in the top 50 at all, because it was chunked badly, never indexed, or phrased in jargon the search does not connect, reranking changes nothing. Measure recall at the candidate stage first: how often is the right passage anywhere in the candidates?

It is also unnecessary for tiny collections where a few dozen documents can be passed whole, and for structured questions that belong in a database query.

One more caution. General-purpose rerankers are trained on broad web data and can misjudge relevance in specialist domains, such as clinical protocols, engineering standards or dense legal drafting. Test on your own questions before assuming a gain. Where a general reranker underperforms, an LLM-based reranker with a short description of what counts as relevant for your business, or a cross-encoder fine-tuned on a few thousand labelled question and passage pairs, is usually the fix.

How SpiderHunts evaluates reranking

When SpiderHunts tunes a retrieval pipeline, we measure two numbers on a set of real questions: whether the right passage is among the candidates, and whether it ends up in the final handful sent to the model. The first tells us whether to fix search and chunking; the second tells us whether a reranker will pay off. It avoids adding components on faith.

Retrieval tuning is part of every assistant we build through our AI chatbot development work, and it is often where the biggest quality gains are hiding.

Frequently asked questions