Reranking and Retrieval Optimization

Learn why a second ranking stage improves precision, how cross-encoders differ from bi-encoders, how a reranking pipeline works, and how contextual compression trims passages.

20 minIntermediateCode Examples
The Core Idea: First-stage retrieval is fast but rough: it must compare a query with millions of chunks, so it uses cheap scoring. Reranking adds a second, more careful look at a small shortlist. A slower and smarter model reads the query together with each candidate and puts the truly best passages on top, so the model sees fewer, better chunks.

1. Beginner: Why Rerank?

Think of hiring. A quick screen of thousands of resumes uses simple rules and lets in more people than you will hire. Then interviewers spend real time on the shortlist. You cannot interview thousands, and you should not hire from the rough screen alone. Retrieval and reranking work the same way.
Stage 1, retrieve
Cheap and wide. Fetch the top 20 to 100 candidates. Goal: do not miss the answer, which means high recall.
Stage 2, rerank
Costly and sharp. Re-score only the shortlist and keep the top few. Goal: put the best passages first, which means high precision.
Why it helps: the best chunk is often retrieved but ranked fourth or tenth. Without reranking it may be cut off by a small top-k, or buried in the prompt where the model uses it less reliably.

2. Intermediate: Bi-Encoders vs Cross-Encoders

The difference lies in how the two texts are processed.
AspectBi-encoder (embedding search)Cross-encoder (reranker)
ReadsQuery and chunk separately, each becomes a vectorQuery and chunk together in one pass
Chunk vectors precomputedYes, stored in the indexNo, computed per query
SpeedVery fast over millions of chunksSlow, so only for a shortlist
AccuracyGoodUsually higher, it sees word interactions
Typical roleCandidate retrievalReranking
Because a cross-encoder sees both texts at once, it can notice things a single vector cannot: that a negation applies to the question topic, that an answer is about a different product, or that a number matches. Other rerankers include hosted reranking services and prompting a language model to score or order the passages.

3. Intermediate: The Reranking Pipeline

1. Retrieve N candidates, for example N equals 50, from dense, sparse or hybrid search.
2. Score each candidate together with the query using the reranker.
3. Sort by the new score and keep the top M, for example M equals 5.
4. Optionally drop passages below a minimum score so weak matches never reach the prompt.
The cost is simple to estimate: reranking time is roughly the number of candidates times the time per pair. If a reranker needs about 10 milliseconds per pair, then 50 candidates take about 500 milliseconds when scored one at a time. Batching the pairs on a GPU shortens this a lot, but the principle stands: more candidates improve recall at the price of latency, so choose N from measurements.
Tuning N and M: raise N until the right chunk reliably appears among the candidates, then lower it again until latency is acceptable. Keep M small, because only a few high quality passages are needed in the prompt.

4. Intermediate: Contextual Compression and Retrieval Filtering

Even a well-ranked chunk usually contains sentences the question does not need. Contextual compression shortens each retrieved passage to the parts that matter for this question, either by extracting relevant sentences or by asking a model to summarize them. It saves prompt tokens and reduces distraction.
Retrieval filtering removes whole passages that score below a relevance threshold, or that duplicate each other. Together these steps turn a long list of loosely related chunks into a short, dense context.
Risk: compression can delete the sentence that qualifies the answer, such as an exception or a date. Compare answers with and without compression on your test questions before turning it on.

5. Advanced: Measuring Gains and Respecting Limits

Reranking should earn its place. Measure it on your labeled questions, as in the retrieval fundamentals lesson: compare hit rate in the final top few passages and the average rank of the first relevant chunk before and after. A metric that rewards putting relevant results near the top, such as nDCG, is also common.
QuestionHow to check
Does the reranker improve final quality?Compare MRR and hit rate at the prompt size, with and without it
Is latency acceptable?Measure end-to-end time at realistic candidate counts
Is the extra cost worth it?Compare the cost per query against the quality gain
Does order matter to the model?Test best-first against best-at-the-edges ordering in the prompt
Reranking cannot fix a failure of stage 1. If the right chunk is not among the candidates, no reranker will find it. Always diagnose recall first, then precision.

6. Visual Explanation: From a Wide Net to a Sharp Shortlist

Panel A shows the pipeline. Panel B shows how the code below reorders two candidates. Panel C is an explorer of the main reranking choices.
Panel A · The PipelineEach stage narrows the list
Query→Retrieve 50→Rerank→Keep top 5→Compress and filter
Panel B · Before and AfterFrom the code in section 7
DocumentStage 1 word countReranker scoreFinal order
A, subscription pricing3, first0.56second
B, how to cancel a plan1, second1.12first
Repeating a word many times won the first stage, but the reranker saw that only B answers the question
Panel C · ChoicesClick a choice to open it
Click to see why stage 1 still matters▼
A reranker can only reorder what it is given. If stage 1 fails to retrieve the right passage, the reranker never sees it, so both stages need to be good.
How to read it: Panel A is the flow, Panel B shows a real reordering, and Panel C compares the choices you will face.

7. Python Implementation: A Two-Stage Ranker with Compression

The program stands in for both stages with simple scorers. Stage 1 counts how many words of the query appear in each text, which favors repetition. Stage 2 reads the query and each candidate together: it maps synonyms such as plan to subscription, rewards candidates that cover all query terms, and gives a small bonus when the matched terms are close together. Finally it compresses the winning passage to the sentences that matter.
rerank_demo.pyPython 3.10+ · Standard library only
synonyms = dict([("plan", "subscription"), ("membership", "subscription"), ("stop", "cancel")])stop_words = set(["how", "to", "a", "the", "your", "and", "can", "i", "do", "press", "open"]) documents = [    ("A", "Subscription pricing: compare each subscription tier and see how to upgrade a subscription."),    ("B", "To cancel your plan, open Settings and press Stop. Cancellations take effect at the end of the billing period. Support can also help by chat."),    ("C", "Invoices for every plan are emailed on the first day of the month."),] def terms(text):    cleaned = "".join(ch if ch.isalnum() else " " for ch in text.lower()).split()    return [synonyms.get(w, w) for w in cleaned if w not in stop_words] def first_stage(query, text):    q = set(terms(query))    return sum(1 for w in text.lower().replace(",", " ").replace(".", " ").split() if w in q) def rerank_score(query, text):    q = set(terms(query))    t = terms(text)    coverage = len(q & set(t)) / len(q)    first = [i for i, w in enumerate(t) if w in q]    closeness = 1 / (1 + (max(first) - min(first))) if len(first) > 1 else 0    return round(coverage + 0.5 * closeness, 2) query = "how to cancel a subscription"print("Query:", query)print("")print("Stage 1, simple word counts:")stage1 = sorted(documents, key=lambda d: -first_stage(query, d[1]))for doc_id, text in stage1:    print("  ", doc_id, first_stage(query, text))print("")print("Stage 2, rerank the top 2 by reading query and text together:")top2 = stage1[:2]reranked = sorted(top2, key=lambda d: -rerank_score(query, d[1]))for doc_id, text in reranked:    print("  ", doc_id, rerank_score(query, text)) best_text = reranked[0][1]sentences = [s.strip() for s in best_text.split(".") if s.strip()]kept = [s for s in sentences if set(terms(query)) & set(terms(s))]print("")print("Compression keeps", len(" ".join(kept)), "of", len(best_text), "characters")
Expected output
Query: how to cancel a subscription Stage 1, simple word counts:   A 3   B 1   C 0 Stage 2, rerank the top 2 by reading query and text together:   B 1.12   A 0.56 Compression keeps 49 of 141 characters
Stage 1 puts the pricing document first only because it repeats the word subscription three times. The second stage notices that it never mentions cancelling, while the cancellation document covers both ideas, so it moves to the top. Compression then keeps a single relevant sentence and drops most of the characters. Real rerankers are neural models, but the structure of the pipeline is the same.

Knowledge Check

Q1. Why are cross-encoders used only on a shortlist?
Q2. What does stage 1 retrieval mainly aim for?
Q3. What is a risk of contextual compression?

Key Points

✓
Reranking adds a slower, more accurate second ranking over a short candidate list.
✓
Stage 1 aims for recall; reranking restores precision.
✓
Bi-encoders embed texts separately and can be precomputed; cross-encoders read pairs together and are more accurate but slower.
✓
A typical pipeline retrieves tens of candidates, reranks them and keeps a handful.
✓
Latency is roughly the number of candidates times the time per pair, so choose the candidate count by measurement.
✓
Contextual compression and score filtering shorten the context but can drop qualifiers.
✓
A reranker cannot recover a chunk that stage 1 failed to retrieve.

The Big Picture

The Retrieval phase is now complete: you can search by meaning and words, apply rules, and sharpen the shortlist. The retrieval side of RAG is a funnel, and each stage has one clear job. The next phase improves what goes into that funnel, the query itself, and what comes out of it, the context.
Retrieve broadly → Filter → Rerank → Compress → Context
Up Next: Query Transformation. You will learn how to rewrite, expand, normalize and decompose user questions, and how multi-query retrieval improves recall.