If you are building retrieval-augmented generation (RAG), you quickly realize that simple vector search is a trap. When we designed the first version of Zia, our knowledge agent, we relied entirely on dense embeddings and cosine similarity. It worked fine for generic questions like "how do I reset my password", but failed miserably the moment a user searched for a specific error code like ERR_CONNECTION_RESET or an exact transaction ID. In this post, I will break down why we ripped out our naive vector search and replaced it with a hybrid BM25 pipeline, backed by Cohere's Rerank API.
The Semantic Illusion: Why Vectors Fail Hard
The standard RAG tutorial tells you to pass all your documents through a model like OpenAI's text-embedding-3-large or an open-source alternative like bge-large-en. You store the resulting 3072-dimensional float arrays in a vector database, calculate the cosine similarity between the user's query and your chunks, and grab the top 5. This conceptual model is beautiful, right until you push it to production.
Dense embeddings are designed to capture semantic meaning, not lexical precision. They map concepts to a mathematical space. If a user in our agent platform LEO asks about "config.yaml line 42 syntax error", the concept of "syntax error" dominates the vector. The search query then pulls up all sorts of Python scripts and irrelevant documentation purely because they are semantically about error handling, completely ignoring the exact string config.yaml.
We saw an even more dangerous failure mode in Vera, our compliance agent. When an agent needed to retrieve the specific rules for "Article 14.2b of the GDPR", the vector search returned Articles 14.2a, 14.1, and 15. Why? Because in the vector space, they are virtually identical: they are all legal texts about data processing. For a Chief Executive Agent (CEA) making decisions based on compliance rules, retrieving the almost correct article is catastrophic. Semantic search creates an illusion of relevance.
Bringing Back the 90s: BM25 for Lexical Precision
To solve this, we had to fall back on technology that has been around for decades: BM25 (Best Matching 25). BM25 is a sparse bag-of-words retrieval function based on TF-IDF (Term Frequency-Inverse Document Frequency). It doesn't care about the deeper "meaning" of a sentence. It purely looks at how often a search term appears in a document, adjusted for the length of the document and how rare that word is across the entire dataset.
For exact matches like UUID-9081 or Article 14.2b, BM25 mercilessly scores the document containing that exact string the highest. The problem with using only BM25 is the vocabulary mismatch problem. If a user searches for "forgot password" and the document says "password reset", BM25 misses the link entirely, whereas a dense vector picks it up effortlessly.
The solution is hybrid search. In our current architecture, we run two queries in parallel on our vector database. We retrieve the top 50 results via dense vector search (for the conceptual matches) and the top 50 via BM25 (for the exact keyword matches). Then, we have to merge these two entirely different scoring systems. For this, we use Reciprocal Rank Fusion (RRF). The formula is simple: score = 1 / (k + rank). By combining the rankings of both systems, we get a robust pool of 100 documents that are both semantically and lexically relevant.
The Context Window Bottleneck and RRF Limitations
Now we have a combined list of 100 documents. However, you cannot stuff 100 chunks of 512 tokens (over 50,000 tokens) into the context window of your LLM for retrieval-augmented generation. Yes, models like gpt-4o or claude-3-5-sonnet have context windows of 128k or even 200k tokens, but massive context significantly degrades reasoning capability. This is known as the "lost in the middle" phenomenon: LLMs often ignore crucial information located in the middle of a gigantic prompt.
Then there are the costs. If Zia processes 10,000 queries a day and we send 50,000 tokens to a model that costs $5 per 1 million input tokens every time, we are burning cash unnecessarily. We need the absolute top 5 chunks, not the top 100.
The problem is that the RRF method we used to merge dense and sparse results is not calibrated for actual relevance to the user's intent. RRF is a blind mathematical trick that zips two lists together. It doesn't understand the nuance of the query. We needed a more intelligent way to prune those 100 documents down to the best 5.
Cohere Rerank: The Final Sorting Layer
This is where we introduced Cohere Rerank into our pipeline. Specifically, we use the rerank-multilingual-v3.0 model, which is crucial for us as a Dutch AI company processing both Dutch and English documentation seamlessly.
Unlike the bi-encoders we use for the initial vector search (where the query and document are vectorized separately and we only measure the angle between them), Cohere Rerank is a cross-encoder. A cross-encoder feeds the search query and the document text simultaneously through the transformer network. This allows for full self-attention between the tokens of the query and the tokens of the document.
Because of this, the model can establish much deeper relationships. If the query is: "Which port should I open for the database connection?" and the document contains the sentence "Ensure port 5432 is open for PostgreSQL", the cross-encoder understands exactly that "port 5432" is the answer to "which port".
Cross-encoders are computationally extremely heavy. You cannot possibly run your entire database of millions of chunks through a cross-encoder for every search query. But you can do it for the top 100 results that come out of your hybrid (Vector + BM25) search. This reranking step adds about 150 to 250 milliseconds of latency, but the jump in precision is massive. Our Mean Reciprocal Rank (MRR) at top-3 in Zia shot up from 0.61 to 0.88 after adding this layer.
Production Architecture at PrudAI
Our RAG architecture now looks like this:
- Query Ingestion: The user asks a question.
- Parallel Retrieval: We simultaneously execute a dense vector search (via
text-embedding-3-small) and a BM25 sparse search on our database. Both retrieve 50 chunks. - Fusion: We combine the results via Reciprocal Rank Fusion.
- Reranking: We send the combined list of up to 100 chunks along with the original query to Cohere Rerank.
- Generation: We grab the top 5 chunks with the highest relevance score from the reranker and inject them into the LLM prompt.
This pipeline ensures that our agents, from Zia to the CEA, operate with the highest possible contextual precision. If you are building agents that have to make business-critical decisions, you cannot rely on a simple dot product calculation between two vectors. You have to build for the edge cases, the exact keywords, and the complex semantics.
Stop blindly implementing standard RAG tutorials. Build a real search pipeline.
Sources
Want to compare notes on retrieval-augmented generation? Contact us.