Hybrid Search: Combining Keyword and Semantic Search in RAG

The question vector search keeps getting wrong

A service engineer asks the company assistant about fault code E-417 on a particular boiler model. The assistant replies with a confident, general answer about pressure faults. The manual page describing E-417 exists, is indexed, and was never retrieved.

This is the classic failure of pure semantic search. Embeddings capture meaning well, but they are fuzzy about exact strings. To an embedding model, E-417 and E-471 look very similar, and a part number carries almost no meaning at all. Business documents are full of exactly these tokens: SKUs, contract references, policy numbers, customer names, error messages.

Keyword search has the opposite profile. It nails exact terms and misses paraphrases. Hybrid search runs both and lets each cover the other's blind spot.

Keyword vs semantic vs hybrid search

If your users only ever ask conversational questions about narrative documents, pure semantic search can be enough. The moment identifiers appear in queries, which in most businesses is quickly, hybrid earns its keep. For a primer on the retrieval side of RAG, see what RAG is and how it works.

How hybrid retrieval works

The mechanics are simpler than the name suggests.

1. The user's query goes to a keyword index, which returns its top results with BM25 scores
2. The same query is embedded and sent to a vector index, which returns its top results by similarity
3. The two ranked lists are merged into one
4. The merged list, often after reranking, goes to the model as context

The interesting part is step three, because BM25 scores and cosine similarities are on completely different scales and cannot simply be added together.

Merging results: RRF or weighted scores

Reciprocal rank fusion ignores raw scores and uses positions. A document ranked first in one list and fifth in the other gets credit from both, and documents appearing in both lists rise to the top. It needs almost no tuning and is the sensible default.

Weighted score fusion normalises both sets of scores and combines them with a weight, say 0.3 keyword and 0.7 semantic. It can outperform RRF once tuned on your own evaluation set, but it is sensitive to the normalisation method and the weight drifts in value as content changes.

Our advice is to start with RRF, build an evaluation set of real questions with known correct documents, and only move to weighted fusion if the numbers show a clear gain. Most teams never need to.

Practical tuning for business documents

• Retrieve generously, then cut. Pull 30 to 50 candidates from each method before fusion, then pass the best handful onward.
• Look at your tokeniser. Keyword search can split 'E-417' into pieces. Check how identifiers are indexed and add a field for exact codes if needed.
• Use metadata filters for things that should never be fuzzy: product line, region, document status, access rights.
• Detect identifier-heavy queries and lean harder on keyword results when a query is mostly codes.
• Add a reranker after fusion. It is often the single biggest quality gain once hybrid is in place.

We cover that last step separately in reranking in RAG, because it deserves more than a bullet point.

Where to run it

You rarely need a separate search engine and vector database glued together any more. PostgreSQL with full-text search and pgvector can do both in one query for modest collections. Elasticsearch, OpenSearch, Weaviate, Qdrant and several managed search services support hybrid queries natively.

The choice mostly follows what you already run. A collection of tens of thousands of documents does not justify new infrastructure, and a single PostgreSQL database is easier to secure, back up and reason about. Millions of chunks with heavy traffic is a different conversation, which our vector database comparison goes into.

How much work is it to add hybrid search?

Less than most teams fear. If you already run a vector index, adding a keyword index over the same chunks is usually a few days of work, plus the fusion step and some testing. The larger cost is the evaluation set you need to prove it helped, and you should have that anyway.

Expect a modest increase in storage and a small amount of extra latency, because two searches run instead of one. Run them in parallel and the difference is rarely noticeable to users. The ongoing cost is keeping both indexes in sync when documents change, which matters more than it sounds: a keyword index that lags behind the vector index produces confusing, inconsistent results.

When hybrid search will not save you

Hybrid search improves retrieval. It cannot retrieve what is not there, and it cannot fix documents chunked so badly that the answer is split across three fragments. If an audit of failed questions shows the right content is missing, stale or mangled, fix the content first.

When SpiderHunts investigates a disappointing RAG system, we log retrieval separately from generation and check the failures by hand. It is common to find that most bad answers trace back to the search step, and that a week on hybrid retrieval and chunking beats a month on prompts. Our chatbot development projects include that retrieval evaluation as standard.

Frequently asked questions