Hybrid search, reranking and query rewriting

Arjun kept a list. Every time the handbook bot gave a store a wrong or useless answer, he pasted the question into a file called failures.txt. By Friday it had eleven entries, and Priya asked him to read them out.

“‘What is SOP-14?’ It gave the oven cleaning procedure, SOP-12. ‘cake came back from customer, smells off, what now’. It answered with the pest control schedule. ‘my cake came late, can I get money back?’ Pest control again.” He looked up. “Why does it love pest control so much?”

Meera, who had come in for the end of it, laughed and then stopped laughing. “The Viman Nagar manager told a customer we’d send the pest control team. For a late cake.”

Priya took the list and sorted it into three piles on the table. “Three different diseases,” she said. “Exact codes that meaning search can’t tell apart. Rankings that put a half-relevant chunk above the right one. And questions worded so differently from the handbook that nothing matches well. Each has a standard cure. None of them is a bigger model.”

Disease one: every SOP code looks the same to an embedding

“Start with SOP-14,” Priya said. “What does an embedding do with ‘SOP-14’?”

An embedding (a list of numbers that captures what a text means) is built to place similar meanings close together. To an embedding model, “SOP-12 Oven cleaning”, “SOP-14 Allergy incident procedure” and “SOP-15 Fire evacuation” all look like the same kind of thing: a standard operating procedure with a number. The numbers themselves carry almost no meaning for it.

Arjun ran “What is SOP-14?” against a twelve-chunk sample of the handbook and printed the cosine similarity scores (how closely two embeddings point the same way; 1 is identical, 0 unrelated):

Chunk	Cosine similarity
SOP-14 Allergy incident procedure	0.503
SOP-12 Oven cleaning	0.378
SOP-15 Fire evacuation	0.371
SOP-41 Allergen labelling	0.337

In the sample, the right chunk still won, but only just. The real handbook has 40 numbered SOPs, and with that many lookalikes, SOP-14 slid to seventh place and out of the top 4. Meaning search is the wrong tool for an exact identifier.

“So what is the right tool for finding an exact word?” Priya asked.

Arjun thought of the back of a cookbook. “The index. Look up ‘SOP-14’, get the page.”

“That’s keyword search. The standard algorithm is called BM25.”

BM25 (short for “Best Match 25”, a scoring formula from the 1990s that still powers most search boxes) is a smarter version of a book index. In plain words, it gives a chunk a high score when:

• the chunk contains the question’s words, and more mentions count for more, with diminishing returns;
• those words are rare across all chunks. “SOP-14” appears in one chunk out of 640, so finding it is strong evidence. “the” appears in every chunk, so it counts for almost nothing;
• the chunk is not padded with lots of other text, which would dilute the match.

On the same sample, BM25 gave the SOP-14 chunk a score of 1.74 and the next-best chunk 0.33. No contest. BM25 needs no model, runs in milliseconds and finds exact codes, product names (“Rational combi oven”), supplier names and employee IDs perfectly.

It also has the opposite blind spot. “Money back” will never match “refund”. BM25 sees letters, not meaning. Hence hybrid search: run both, and combine the results so each covers the other’s blind spot.

Arjun’s first merge, and why it collapsed

Arjun had BM25 working within the hour. To combine, he did the obvious thing: for each chunk, add its BM25 score to its cosine similarity, and sort.

For “cake came back from customer, smells off, what now”, the result put pest control first by a mile:

Chunk	BM25	Cosine	Sum
Pest control (“Keep the back store accessible”)	2.18	0.298	2.48
Customer returns of spoiled or off-smelling products	0.89	0.571	1.46

Reciprocal rank fusion: merge positions, not scores

“Different scales,” Priya said. “So stop using the scores. Use the positions.”

Reciprocal rank fusion (RRF) gives each list a vote. A chunk at position 1 in a list gets a big vote, position 2 a slightly smaller one, and so on:

RRF score of a chunk = the sum, over every list it appears in, of 1 ÷ (60 + its position in that list)

The 60 is a smoothing constant from the paper that introduced RRF; it stops first place from completely dominating second. A chunk missing from a list gets nothing from that list. Because only positions are used, the scale of the scores no longer matters.

Here is the cake question, traced through RRF with the real top 5 from each method:

Chunk	BM25 position	Vector position	RRF score
Pest control	1	2	1/61 + 1/62 = 0.0325
Spoiled products (the right one)	4	1	1/64 + 1/61 = 0.0320
Refunds above ₹2,000	2	(not in top 5)	1/62 = 0.0161
Greeting customers	3	(not in top 5)	1/63 = 0.0159
Fresh-cream 4-hour rule	(not in top 5)	3	1/63 = 0.0159

Arjun groaned. “Pest control still wins.”

“By a hair, this time,” Priya said. “And look at the SOP-14 question through the same formula: the SOP-14 chunk is first in both lists, so it wins clearly with 0.0328. For codes and names, RRF is exactly what you want: chunks both methods like rise to the top.” She circled the pest control row. “But RRF has a weakness you just found. It only knows positions. It can’t see that the vector search was much more confident about spoiled products. For that, we need someone to actually read the candidates.”

This is an important, honest result. Hybrid search is not a free upgrade. On questions full of everyday words, BM25 adds noise, and fusion can let that noise through. What hybrid search reliably does is put the right chunk somewhere in the top handful for a much wider range of questions. Picking the best of that handful is the next step’s job.

Disease two: the ranking needs a careful reader

Why is vector search fast? Because the question and each chunk are embedded separately. The 640 chunk vectors were computed last night; at question time you embed only the question (about 19 milliseconds on Arjun’s laptop) and compare numbers. A model used this way is called a bi-encoder: two texts, encoded independently, compared afterwards.

The price of that speed is that the model never reads the question and the chunk together. It cannot notice that “came back” in the question is about a returned product and “back store” in the chunk is about a room.

A cross-encoder takes the question and one chunk as a single input and reads them together, word against word, using the attention mechanism you will study in Module 30. It outputs one relevance score. It is far more accurate, and far slower, because it has to run once per (question, chunk) pair at question time. Nothing can be computed in advance.

Priya put it in hiring terms. “You have 640 applicants. You can’t interview all of them. So first a quick CV screen picks 20, then proper interviews rank those 20. The CV screen is your hybrid search. The interview is the cross-encoder.” Using a slow, accurate model to reorder a short list from a fast one is called reranking, and the model is a reranker.

Arjun ran the five RRF candidates for the cake question through a small, free cross-encoder, cross-encoder/ms-marco-MiniLM-L-6-v2:

Chunk	Cross-encoder score
Spoiled products	−1.68
Greeting customers	−11.19
Pest control	−11.39
Refunds above ₹2,000	−11.46
Fresh-cream 4-hour rule	−11.47

The right chunk now leads by almost ten points. Pest control dropped to third, where it belongs.

Do not worry that the scores are negative. This model outputs raw scores (sometimes called logits) rather than probabilities: higher means more relevant, and the absolute numbers mean little on their own. Compare scores only with other scores from the same model. Even so, keep them: a top score far below the usual range is a strong hint that nothing relevant was found, which you will use in the next lesson to decide when the bot should say “I don’t know”.

Disease three: the question is worded nothing like the answer

The last pile of failures was questions like “my cake came late, can I get money back?”. The handbook’s answer is written in policy language: “Delayed orders: if an order arrives more than 60 minutes after the promised slot, refund the delivery fee and offer a 20% voucher.” Vector search actually found it (top, at 0.424), but BM25 latched onto “back” again and RRF promoted pest control. And the best cross-encoder score was only −8.95: the question and the answer do not look alike.

Query rewriting uses an LLM to turn the user’s words into better search queries before retrieval. There are three common forms.

1. Standalone rewriting, for conversations. In a chat, the follow-up “and what about Sundays?” means nothing on its own; it has no searchable content. Rewrite it using the chat history (“What are the Kothrud store’s opening hours on Sundays?”) and search with that. This is the one rewriting step almost every chat-based RAG system needs.

2. Multi-query. Ask the LLM for three differently worded versions of the question, search with each one (and the original), and fuse every result list with RRF. One phrasing will hit the handbook’s vocabulary even if the others miss. For the cake question, the LLM’s rewrites and their effect:

Query	Vector score for “spoiled products”	BM25 top result
cake came back from customer, smells off, what now	0.571	Pest control
procedure for customer returns of spoiled products	0.669	Spoiled products
customer returned a cake with a bad smell	0.631	Delivery handover
handling an off-smelling product complaint	0.616	Spoiled products

Fusing all eight lists (four queries, two methods each) with RRF puts the spoiled-products chunk first with 0.130, and the next chunk at 0.095. Pest control’s lucky “back” match is outvoted.

3. HyDE (Hypothetical Document Embeddings). Questions and handbook passages look different: short and casual versus long and formal. HyDE asks the LLM to write a fake handbook paragraph that would answer the question, and searches with that paragraph instead of the question. For the cake question, a paragraph like “If a customer returns a cake that smells spoiled, apologise and offer a replacement or a refund. Keep the product sealed and report it to the store manager…” matched the real chunk at 0.777, against 0.571 for the original question, and the runner-up was far behind at 0.338.

“But the fake paragraph might be wrong,” Arjun said. “It invented ‘report it to the store manager’.”

“It doesn’t matter,” Priya said. “Nobody ever sees it. It’s a search probe, not an answer. It only has to sound like the right page, so it lands near the right page on the map. The answer still comes from the real chunk.”

The catch with all three: every rewrite is an extra LLM call before the user sees anything, typically half a second to a couple of seconds, plus the tokens. Multi-query also multiplies the searches.

The whole pipeline, and what each box costs

flowchart LR
  Q["Question"] --> R["Rewrite: standalone, multi-query or HyDE"]
  R --> K["BM25 top 20"]
  R --> V["Vector top 20"]
  K --> F["RRF fusion"]
  V --> F
  F --> X["Cross-encoder rerank"]
  X --> T["Top 4 chunks"]
  T --> L["LLM answer with citations"]

A rough latency budget for one question on Arjun’s laptop with the handbook’s 640 chunks:

Step	Typical time	Money
Rewrite (one LLM call)	0.5 to 2 s	A fraction of a rupee
BM25 over 640 chunks	Under 10 ms	Free
Embed question + vector search	About 20 ms	Free with a local model
Rerank 20 candidates	About 70 ms	Free with a local model
Final LLM answer	1 to 3 s	A fraction of a rupee

Hybrid search and reranking are nearly free. Rewriting roughly doubles the time before the answer starts. That shapes the order in which you add things: hybrid search first (cheap, fixes codes and names), reranking second (cheap, fixes sloppy ranking), rewriting last and only where your test questions show it helps. Standalone rewriting for follow-ups is the exception; if your bot is a chat, you need it from day one.

Building it, one piece at a time

You already have chunks in a Chroma collection from the previous lessons. Here each chunk is a dictionary with an id and text, stored in collection with the same IDs.

Step 1: keyword search with BM25.

# uv add rank-bm25 chromadb sentence-transformers anthropic pydantic python-dotenv
import re
from rank_bm25 import BM25Okapi

def tokenize(text: str) -> list[str]:
    # Lowercase, and keep codes like "sop-14" as one token instead of "sop" + "14"
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks])   # built once, at start-up

def keyword_search(question: str, n: int = 20) -> list[str]:
    scores = bm25.get_scores(tokenize(question))           # one score per chunk
    best = sorted(range(len(chunks)), key=lambda i: scores[i], reverse=True)
    return [chunks[i]["id"] for i in best[:n] if scores[i] > 0]

keyword_search("What is SOP-14?") should return the SOP-14 chunk first. The tokenizer matters more than it looks: a plain split() would keep “SOP-14?” with its question mark, and many tokenizers split it into “sop” and “14”, both much weaker clues.

Step 2: vector search, and fusion.

from collections import defaultdict

def vector_search(question: str, n: int = 20) -> list[str]:
    return collection.query(query_texts=[question], n_results=n)["ids"][0]

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for position, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] += 1 / (k + position)          # the RRF vote
    return sorted(scores, key=scores.get, reverse=True)

def hybrid_search(question: str, n: int = 20) -> list[str]:
    return rrf([keyword_search(question, n), vector_search(question, n)])[:n]

Step 3: rerank.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")   # downloads once
text_by_id = {c["id"]: c["text"] for c in chunks}

def rerank(question: str, candidate_ids: list[str], top_n: int = 4):
    pairs = [(question, text_by_id[cid]) for cid in candidate_ids]
    scores = reranker.predict(pairs)                    # one score per pair
    ranked = sorted(zip(candidate_ids, scores), key=lambda p: p[1], reverse=True)
    return ranked[:top_n]                               # [(chunk_id, score), ...]

q = "cake came back from customer, smells off, what now"
print(rerank(q, hybrid_search(q)))

You should see the spoiled-products chunk first with a score far above the rest. Hosted rerankers exist too (several providers sell reranking as an API) if you would rather not run a model yourself.

Step 4: multi-query and HyDE.

import anthropic
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()
client = anthropic.Anthropic()

class Rewrites(BaseModel):
    queries: list[str]

def multi_query_search(question: str) -> list[str]:
    response = client.messages.parse(
        model="claude-opus-5-5",
        max_tokens=500,
        messages=[{"role": "user", "content":
            "Rewrite this bakery staff question as 3 search queries using the formal "
            f"wording a store operations handbook would use.\nQuestion: {question}"}],
        output_format=Rewrites,                         # typed output, no parsing by hand
    )
    queries = [question] + response.parsed_output.queries
    lists = []
    for q in queries:
        lists += [keyword_search(q), vector_search(q)]
    return rrf(lists)[:20]

def hyde_search(question: str, n: int = 20) -> list[str]:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=300,
        messages=[{"role": "user", "content":
            "Write a short passage in the style of a bakery operations handbook "
            f"that answers: {question}"}],
    )
    fake_passage = next(b.text for b in response.content if b.type == "text")
    return vector_search(fake_passage, n)               # search with the fake, never show it

Either function’s output goes straight into rerank, because they all return the same thing: a list of chunk IDs. That is the retriever idea from the previous lesson paying off.

Where it breaks

A week in production taught Arjun four more things.

• Hinglish. A Hadapsar staff member typed “cake kharab ho gaya customer ne wapas kiya”. The English-only embedding model and the English handbook had nothing to match. Options: a multilingual embedding model, or a rewriting step that translates to English first. Know who your users are.
• Everyday words in BM25. “back”, “came”, “now” caused most of the pest control mess. Some teams remove very common words before BM25 (a stop-word list); others accept the noise and rely on the reranker. Measure both.
• Thresholds that do not travel. The cake question’s correct chunk scored −1.68, the late-delivery one only −8.95. A fixed cut-off like “abstain below 0” would have refused a perfectly answerable question. Thresholds must be chosen from your own test questions, which is the next lesson’s job.
• Stacking everything by default. Arjun briefly ran multi-query, HyDE and reranking on every question. Answers got slightly better and noticeably slower, and the bill for rewriting calls doubled. He kept standalone rewriting for follow-ups, turned the rest off, and switched them back on only for the question types where they measurably helped.

Try it yourself

Take five questions: one with a code, one with a rare name, one casual, one using different words from the document (“money back” versus “refund”), and one follow-up (“and on Sundays?”). For each, record whether the correct chunk is in the top 4 for (a) vector only, (b) hybrid with RRF, © hybrid plus rerank, (d) hybrid plus rerank with multi-query.

Show the typical pattern

Vector-only usually misses the code question once you have many lookalike codes, and sometimes the rare name. Hybrid fixes those, because BM25 matches the exact token. The casual question often has the right chunk in the top 20 but not the top 4; hybrid can even make it worse when BM25 latches onto everyday words, and reranking pulls it back up. The different-words question benefits most from multi-query. The follow-up fails in every column until you add standalone rewriting, because “and on Sundays?” contains nothing to search for. The lesson: each technique fixes a specific failure, so diagnose first, then add the matching cure.

Common mistakes

• Adding raw BM25 and cosine scores. They are on different scales; the bigger one wins. Use RRF.
• Reranking too few candidates. If the right chunk is not in the candidate list, the reranker cannot find it. Retrieve 20 to 50, rerank down to 3 to 5.
• Tokenizing codes badly. “SOP-14” split into “sop” and “14” throws away BM25’s best trick.
• Reading cross-encoder scores as probabilities. They are raw scores; compare them only with each other.
• Showing the HyDE passage to users. It is invented text, used only to find real text.
• Adding every technique at once. You will not know which one helped, and you will pay for all of them on every question.

Recap

Arjun’s failure list went from eleven entries to two. Meera asked him what had changed, “in words the store managers would understand”. He tried:

• We now search two ways at once: by exact words, like the index at the back of a book, and by meaning, like a librarian who understands the question. Each catches what the other misses.
• We combine the two lists by position, not by score, because their scores are on different scales.
• Then a slower, more careful model reads the top 20 results next to the question and picks the best 4. That is what finally beat pest control.
• For badly worded questions, the AI first rewrites the question in handbook language, or drafts what the answer might look like, and we search with that.
• Each step costs time, so we only add the ones our tests show we need.

Meera listened, then asked the question he had been dreading. “Eleven down to two on your list. How do you know it isn’t worse on questions that aren’t on your list?”

Arjun did not have an answer. Priya smiled. “That,” she said, “is the most important question in this whole module.”

Go deeper

• LangChain documentation (retrievers): https://python.langchain.com
• Hugging Face documentation (Sentence Transformers and cross-encoders): https://huggingface.co/docs
• The original RAG paper: https://arxiv.org/abs/2005.11401