The Core Idea: Retrieval is the step that decides which pieces of your knowledge the model gets to see. If the right passage is not retrieved, no later step can recover it. This lesson explains how a retriever ranks chunks, what top-k means, how a system moves from many candidates to a few final passages, and how to measure retrieval quality with recall and precision.
1. Beginner: What Retrieval Is
Retrieval takes a query and returns the chunks from the knowledge base that are most likely to help answer it. It is the librarian of the RAG system: it does not write answers, it fetches the right pages.
Input
A user question, often transformed or rewritten first.
Process
Score every candidate chunk, or the promising ones, against the query.
Output
A ranked list of chunks with scores and metadata.
Why it is the bottleneck: the generator can only use what it is given. A perfect model with the wrong passages still answers wrongly or admits it does not know.
2. Beginner: Similarity Search and Top-k
In the simplest retriever the query is embedded, compared with every stored chunk vector, and the k best-scoring chunks are returned. This number k is called top-k.
| Choice | Too small | Too large |
|---|---|---|
| Value of k | The needed passage may be missed | Noise fills the prompt and costs more |
| Typical starting range | A few results | Tens of results for later reranking |
A score threshold adds a second control: results below a minimum similarity are dropped even if they are in the top k. This lets the system say it found nothing relevant instead of returning weak matches.
3. Intermediate: From Candidates to Context
Real systems rarely go straight from search to prompt. They work as a funnel that starts wide and narrows. Early stages are cheap and favor not missing anything, later stages are precise but more expensive.
Stage 1, candidate retrieval. Fetch a generous set, for example the top 30 to 100 chunks, using fast similarity search.
Stage 2, filtering. Remove chunks that break rules such as wrong date, status or permission.
Stage 3, reranking. Re-score the survivors with a more accurate model and keep the best few.
Stage 4, context selection. Choose, order and trim the final passages to fit the prompt budget.
The next lessons in this phase cover each stage in turn.
4. Intermediate: Recall vs Precision
Two simple questions describe how good a ranked list of results is. How much of what is good did we find? And how much of what we returned is good?
They pull in opposite directions. Returning more results raises recall but usually lowers precision. In RAG the first stage should favor recall, because a missed passage is lost forever, while later stages such as reranking restore precision.
5. Advanced: Measuring Retrieval Properly
Retrieval should be evaluated on its own, before the generator is involved. Build a small test set of real questions, each labeled with the chunks that answer it. Then compute three numbers.
| Metric | Meaning | Question it answers |
|---|---|---|
| Hit rate@k | Share of questions with at least one relevant chunk in the top k | How often does retrieval give the model a chance? |
| Recall@k | Share of all relevant chunks that appear in the top k | Did we find everything needed? |
| MRR | Average of 1 divided by the rank of the first relevant result | How near the top does the first good chunk appear? |
A question whose first relevant chunk is ranked 1, 1, 1 and 2 gives . When a metric is poor, look at the failures one by one: was the chunk badly cut, badly embedded, filtered out, or just outranked?
6. Visual Explanation: The Retrieval Funnel and the Recall Trade-off
Panel A shows the funnel from many candidates to a few passages. Panel B uses the results from the code below to show how metrics change as k grows. Panel C is a failure explorer.
Panel A · The Retrieval FunnelWide first, precise later
Knowledge base→Top 50 candidates→Filter→Rerank→Top 4 passages
Panel B · Metrics as k GrowsFrom the code in section 7
| k | Precision@k | Recall@k | Hit rate@k |
|---|---|---|---|
| 1 | 0.75 | 0.62 | 0.75 |
| 2 | 0.62 | 1.00 | 1.00 |
| 3 | 0.42 | 1.00 | 1.00 |
| 4 | 0.31 | 1.00 | 1.00 |
Recall climbs and then flattens, while precision keeps falling
Panel C · When Retrieval FailsClick a symptom to open it
Click to see why precision matters even when recall is perfect▼
Every extra weak passage uses prompt space, adds cost and gives the model a chance to be distracted. Good systems retrieve broadly, then narrow before generating.
How to read it: Panel A is the process, Panel B shows the metric trade-off in numbers, and Panel C turns symptoms into next steps.
7. Python Implementation: Measuring Precision, Recall, Hit Rate and MRR
The program builds a tiny retriever that scores chunks by the number of shared words, using a stop word list and a crude plural rule. It then evaluates the retriever on four labeled questions for k from 1 to 4. Finally it reports the rank of the first relevant chunk for every question and the resulting MRR.
retrieval_metrics.pyPython 3.10+ · Standard library only
stop_words = set(["how", "do", "i", "is", "my", "the", "to", "a", "when", "can", "are", "your", "from", "long", "does", "fast"]) documents = [ ("d1", "refund policy allows returns within 30 days"), ("d2", "refund requests are processed within 5 business days"), ("d3", "standard shipping takes 5 to 7 business days"), ("d4", "express shipping takes 2 days"), ("d5", "premium members get free shipping"), ("d6", "support is open monday to friday"), ("d7", "reset your password from the account page"), ("d8", "passwords must contain 12 characters"),] tests = [ ("how long do refunds take", set(["d2"])), ("how fast is shipping", set(["d3", "d4"])), ("how do i reset my password", set(["d7"])), ("when can i call support", set(["d6"])),] def stem(word): if word.endswith("s") and len(word) > 3: return word[:-1] return word def terms(text): return set(stem(w) for w in text.lower().split() if w not in stop_words) def retrieve(question, k): q = terms(question) scored = [(len(q & terms(text)), doc_id) for doc_id, text in documents] scored.sort(key=lambda s: (-s[0], s[1])) return [doc_id for score, doc_id in scored[:k]] print("k | precision@k | recall@k | hit rate@k")for k in (1, 2, 3, 4): p = r = h = 0.0 for question, relevant in tests: got = retrieve(question, k) found = relevant & set(got) p += len(found) / k r += len(found) / len(relevant) h += 1 if found else 0 n = len(tests) print(k, "|", round(p / n, 2), "|", round(r / n, 2), "|", round(h / n, 2)) rr = 0.0for question, relevant in tests: got = retrieve(question, len(documents)) rank = min(got.index(d) for d in relevant) + 1 rr += 1 / rank print(" ", question.ljust(28), "first relevant rank:", rank)print("MRR:", round(rr / len(tests), 2))Expected output
k | precision@k | recall@k | hit rate@k1 | 0.75 | 0.62 | 0.752 | 0.62 | 1.0 | 1.03 | 0.42 | 1.0 | 1.04 | 0.31 | 1.0 | 1.0 how long do refunds take first relevant rank: 2 how fast is shipping first relevant rank: 1 how do i reset my password first relevant rank: 1 when can i call support first relevant rank: 1MRR: 0.88At k equal to 1 the retriever misses a relevant chunk for some questions, so recall and hit rate are below 1. From k equal to 2 every question has at least one hit, and recall reaches 1, while precision keeps shrinking because extra results are mostly irrelevant. The refund question shows a subtle point: two chunks tied on score, and the tie was broken by chunk id, which put the useful one second. Ties, and how you break them, can change your metrics.
Knowledge Check
Q1. What happens to recall and precision when you raise k?
Q2. Which stage of the retrieval funnel should mostly favor recall?
Q3. What is a score threshold for?
Key Points
✓
Retrieval selects the chunks the model will read, so it caps the quality of the whole system.
✓
Top-k returns the k best-scoring chunks, and a score threshold drops weak ones.
✓
Real systems work as a funnel: wide candidate retrieval, filtering, reranking, then context selection.
✓
Precision measures how much of the result is relevant; recall measures how much of the relevant material was found.
✓
The first stage should favor recall; later stages restore precision.
✓
Evaluate retrieval separately with hit rate, recall@k and MRR on a labeled test set.
✓
Inspect failures one by one to find whether chunking, embeddings, filters or ranking is at fault.
The Big Picture
Retrieval is a search problem with a quality ruler. Once you can measure hit rate and recall, every change to chunking, embeddings or ranking becomes an experiment with a result instead of an opinion.
Query → Candidates → Filter → Rerank → Select context → Measure
Up Next: Dense, Sparse and Hybrid Retrieval. You will compare embedding-based dense search with keyword-based sparse search such as BM25, and learn how hybrid search with reciprocal rank fusion combines their strengths.