Hybrid search for RAG: vectors plus keywords

If you are building retrieval-augmented generation (RAG), you quickly find out that vector search alone is not enough. Our first knowledge bases searched with embeddings only: every text becomes a series of numbers, and the query finds the passages that sit mathematically closest to it. That works well for questions in plain language, but it misses exactly what lawyers, builders and care professionals often look for: an exact article number, a case citation (ECLI) or a fixed term. In this post I show how our retrieval works now: semantic search and keyword search side by side.

The semantic illusion: why vectors alone fall short

Embeddings are designed to capture meaning, not literal precision. They place concepts in a mathematical space where texts about the same subject sit close together. For a question like "what are the rules for terminating a lease?" that is exactly what you want.

But if you search for Article 14(2)(b) of the GDPR, then Article 14(1)(a) and Article 15 sit in almost the same place in that space: they are all text about information duties in data processing. Vector search can then hand you the almost-right article. In legal work, the almost-right article is the wrong article. The same goes for a case citation: a lawyer who searches their own case documents in LEO, our legal AI colleague, for one ruling wants that ruling, not a ruling that resembles it.

Back to basics: keyword search for literal precision

That is why, since February 2026, we search two ways at once by default. A knowledge base can search by meaning only, by keywords only, or hybrid; hybrid has been the default for new knowledge bases since then.

Keyword search does not care about the deeper meaning of a sentence. It looks at which words literally appear in a passage. For an exact term such as "7:658 BW" or an ECLI number, the passage with that exact text scores high. Keyword search has the opposite problem too: if someone searches for "end my lease" and the document says "termination of the tenancy agreement", it misses the link, while the semantic route finds it. That is exactly why we combine the two.

How it works at Prudai

Every knowledge base is split into passages when it is ingested. Those passages go to two places:

Semantic route. The passages get an embedding from the gemini-embedding-001 model, through Google Vertex AI in the europe-west4 region, and are stored in Weaviate. A query is converted the same way and finds the passages closest to it in meaning.
Keyword route. The same passages are stored in a keyword index in Meilisearch, on our own servers. In our code this is the BM25 route. It searches for the literal words of the query.

In a hybrid knowledge base both routes run for every query. We merge the results without elaborate score fusion: keyword hits first, then semantic hits, and a passage that appears in both lists counts once. We do not use a separate re-ranking step with an extra model (a reranker).

One document must not take every slot

In practice we saw that one document that closely resembled the query could fill every slot in the results. A second document that was also relevant then no longer showed up at all.

Since August 2026 we therefore first fetch more candidates than we need, and select from them with a cap per document: one document gets at most half of the slots. Leftover slots are filled with the best skipped candidates. A question about one document loses nothing, and a question about several documents shows all of them.

A fixed budget for the language model

You cannot put an unlimited number of passages into a language model's prompt. Even models with a large context window use information in the middle of a long prompt less well than information at the start or the end; this is known as "lost in the middle". So the language model receives a fixed maximum number of passages. Around each passage we find, we can add the neighbouring passages from the same document, so a passage does not start mid-sentence, but always within that budget.

When one route falters

If the semantic route fails, for example because the embedding service is temporarily unavailable, a hybrid knowledge base still answers with the keyword hits instead of failing the whole query.

Retrieval supplies the knowledge; how it works together with MCP, which lets an agent take actions, is covered in Beyond the buzzwords: supercharging AI with MCP and RAG.

Stop blindly implementing standard RAG tutorials. Search by meaning and by the letter.

Sources

Want to compare notes on retrieval-augmented generation? Contact us.

Why We Abandoned Simple Vector Search in RAG for Hybrid Search