The Core Idea: The pipelines in this course follow a fixed route. Agentic RAG gives a language model control of the route: it decides what to search for, which tool to use, whether the evidence is enough and when to stop. At the same time, research keeps improving the retrieval models themselves. This final lesson maps both directions, with a clear view of what is proven, what is promising, and what is only fashionable.
1. Intermediate: From Pipelines to Agents
In a pipeline, the developer fixes the steps. In an agent, the model chooses the next step at run time by repeating a simple loop: plan what is needed, act by calling a tool, observe the result, and decide whether to continue or answer.
| Aspect | Fixed pipeline | Agentic RAG |
|---|---|---|
| Who chooses the steps | The developer | The model, step by step |
| Handles unexpected question types | Poorly | Better |
| Predictability, cost, latency | High | Lower, varies by question |
| Debugging | Simple | Needs full traces of every step |
Rule of thumb: use a fixed pipeline whenever it solves the problem, and use an agent only for questions whose steps cannot be known in advance.
2. Intermediate: Agent Patterns for Retrieval
Planning plus retrieval
The agent first breaks the task into steps, then retrieves for each, and revises the plan when results surprise it.
Tool-based retrieval
Search, SQL, calculators, web search and APIs are all tools. The agent picks the right one for each sub-question.
Iterative retrieval
The agent reads, notices what is missing, and searches again with a better query until the evidence is sufficient.
Multi-step reasoning workflows
Several steps with intermediate results, such as finding a price and then computing a yearly total.
Agents bring risks that fixed pipelines avoid. Errors compound across steps, loops can run without end, costs are unpredictable, and every tool and every retrieved document is a new path for prompt injection to cause real actions.
Cap the number of steps, tool calls and total cost per request.
Restrict tools to the minimum, with read-only access by default.
Approve sensitive actions with a human.
Trace every thought, call and result.
Evaluate whole trajectories, not just the final answer.
3. Intermediate: Fine-Tuning Retrieval and Embeddings
Off-the-shelf embedding models are general. If your domain has special vocabulary, such as legal, medical or internal product language, fine-tuning the retriever or the embedding model on your own examples can raise retrieval quality noticeably.
Collect pairs of questions and the passages that answer them, from logs, experts or carefully reviewed synthetic data.
Add hard negatives passages that look relevant but are not, so the model learns to tell them apart.
Train with a contrastive objective that pulls matching pairs together and pushes others apart.
Compare against the original model on a held-out golden set.
Re-embed the whole index with the new model, since vectors from different models cannot be mixed.
Try the cheaper things first: better chunking, hybrid search and reranking often give comparable gains without training. Fine-tune when evaluation shows that the embedding model itself is the bottleneck, and you have enough good data.
4. Intermediate: ColBERT, Late Interaction and Learned Sparse Retrieval
Standard dense retrieval compresses a whole passage into one vector. ColBERT, a late-interaction model, keeps one vector per token and compares query and document tokens at search time. The score sums, for each query token, its best match among the document tokens, an operation called MaxSim:
This keeps fine-grained detail that a single vector blurs, at the price of storing many more vectors per document. It is often used to rerank or to retrieve with specialized indexes.
Learned sparse retrieval goes the other direction. A model learns which words matter and which related words to add, and produces sparse weighted vectors. They keep the inverted-index machinery and exact-match strengths of keyword search, with some semantic reach added.
| Approach | Representation | Strength | Cost |
|---|---|---|---|
| Single-vector dense | One vector per passage | Simple, fast, small | Blurs fine detail |
| Late interaction (ColBERT) | One vector per token | Fine-grained matching | Much more storage |
| Learned sparse | Weighted words with expansions | Exact terms plus some meaning, uses inverted indexes | Needs a trained model |
5. Advanced: Research Directions
The field moves fast. These themes are active, and each deserves a measured, evidence-first attitude.
Retrieval-augmented pretraining
Training language models with a retriever in the loop, so knowledge lives partly in an external store from the start. Research such as REALM and RETRO explored this.
RAG plus fine-tuning
Fine-tune the generator to use retrieved context faithfully and to ignore irrelevant passages, while retrieval supplies the facts.
Long context versus RAG
Larger windows do not make retrieval obsolete: cost, latency, control and reliability still favor selecting the right text. Expect the two to be combined.
Self-improving RAG
Systems that learn from feedback, such as clicks and corrections, by updating queries, indexes or models. Promising, but needs safeguards against learning the wrong lessons.
How to stay sane: for any new technique, ask what failure of your current system it addresses, test it on your golden dataset, and measure quality, cost and latency together. Many ideas look strong on a paper benchmark and add little to a well-tuned simple system.
6. Visual Explanation: The Agent Loop and Token-Level Matching
Panel A shows the agent loop. Panel B shows the trace and the MaxSim scores from the code below. Panel C is a technique explorer.
Panel A · The Agent LoopRepeat until the evidence is enough or the budget ends
Plan→Choose a tool→Act→Observe→Enough evidence? Answer
If not enough, return to Plan with what was learned, within a step limit
Panel B · From the CodeA tool-using agent and late interaction
| Item | Result |
|---|---|
| Step 1, search tool | Found: Pro plan costs 25 dollars per month |
| Step 2, multiply tool | 25 times 12 equals 300 |
| Step 3 | Finish: 300 dollars per year |
| MaxSim: budget airfare france | 2.95, the better match |
| MaxSim: cheap hotel paris | 2.80, shares words but wrong topic |
Panel C · Technique ExplorerClick a technique to open it
Click to see why the simple baseline still matters▼
A well-chunked, hybrid, reranked pipeline with good evaluation solves a surprising share of real problems. Advanced techniques should be justified by a measured gap, not by novelty.
How to read it: Panel A is the control loop, Panel B shows both ideas on tiny examples, and Panel C reminds you of the cost of each.
7. Python Implementation: A Tiny Tool-Using Agent and MaxSim Scoring
The first half is a minimal agent with two tools, search and multiply, and a step limit. The decision rule is written by hand so the program is deterministic, while a real agent would let a language model choose each action. It answers what the Pro plan costs for a year by searching for the monthly price, then multiplying it by 12. The second half scores two candidate passages for a query with MaxSim, using a tiny hand-made table of token vectors in place of a trained model.
agent_and_maxsim.pyPython 3.10+ · Standard library only
import math docs = [ "Pro plan costs 25 dollars per month.", "Basic plan costs 10 dollars per month.",] def search(query): q = set(w for w in query.lower().replace("?", "").split() if len(w) > 2) return max(docs, key=lambda d: len(q & set(d.lower().replace(".", "").split()))) def find_number(text): for word in text.split(): if word.isdigit(): return int(word) return None def multiply(a, b): return a * b def agent(question, max_steps=4): price = None trace = [] for step in range(1, max_steps + 1): if price is None: result = search("pro plan costs month") price = find_number(result) trace.append("step " + str(step) + ": search -> " + result) else: total = multiply(price, 12) trace.append("step " + str(step) + ": multiply " + str(price) + " x 12 = " + str(total)) trace.append("step " + str(step + 1) + ": finish") return total, trace return None, trace total, trace = agent("What does the Pro plan cost for a year?")for line in trace: print(line)print("Answer:", total, "dollars per year") # Late interaction (ColBERT style): one vector per token, scored with MaxSimtoken_vectors = dict([ ("cheap", [1.0, 0.0]), ("budget", [0.95, 0.3]), ("hotel", [0.3, -0.95]), ("flights", [0.0, 1.0]), ("airfare", [0.05, 0.998]), ("paris", [0.6, 0.8]), ("france", [0.58, 0.81]),]) def cosine(a, b): dot = sum(x * y for x, y in zip(a, b)) return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))) def maxsim(query_tokens, doc_tokens): return sum(max(cosine(token_vectors[q], token_vectors[d]) for d in doc_tokens) for q in query_tokens) query = ["cheap", "flights", "paris"]candidates = [("budget airfare france", ["budget", "airfare", "france"]), ("cheap hotel paris", ["cheap", "hotel", "paris"])]print("")for name, tokens in candidates: print(name.ljust(24), "MaxSim", round(maxsim(query, tokens), 2))Expected output
step 1: search -> Pro plan costs 25 dollars per month.step 2: multiply 25 x 12 = 300step 3: finishAnswer: 300 dollars per year budget airfare france MaxSim 2.95cheap hotel paris MaxSim 2.8The trace shows the plan, act and observe cycle: the agent searches, reads the price, calls the multiplication tool and finishes in three steps, well within its limit of four. In the MaxSim part, the passage budget airfare france scores higher than cheap hotel paris, although it shares no word with the query. Each query token finds its best partner among the passage tokens: cheap with budget, flights with airfare and paris with france. The second passage shares two words but has no match for flights, which is the kind of detail a single averaged vector can blur.
Knowledge Check
Q1. What distinguishes agentic RAG from a fixed RAG pipeline?
Q2. Why must an agent have step and cost limits?
Q3. What does MaxSim compute in late interaction?
Key Points
✓
Agentic RAG lets the model choose steps and tools in a plan, act, observe, decide loop.
✓
Use a fixed pipeline where it is enough, and an agent for questions whose steps are not known in advance.
✓
Control agents with step, tool and cost limits, minimal permissions, approvals for sensitive actions and full traces.
✓
Fine-tuning retrieval or embeddings helps with domain vocabulary, needs good question-passage data with hard negatives, and requires a full re-embed.
✓
ColBERT keeps one vector per token and scores with MaxSim, trading storage for fine-grained matching.
✓
Learned sparse retrieval gives weighted, expanded term vectors that work with inverted indexes.
✓
Judge every new technique on your own golden dataset for quality, cost and latency before adopting it.
The Big Picture
You have walked the whole path: from why RAG exists, through data, embeddings, retrieval, queries, context and evaluation, to production and research. The pattern that links everything is the same simple loop: make a change, measure it on real questions, and keep what helps. That habit will outlast any particular technique.
Foundations → Data → Embeddings → Retrieval → Query and Context → Advanced RAG → Evaluation → Production → Research
Up Next: Your next project. Build a small RAG system on documents you care about, create a golden dataset of twenty real questions, and let the measurements decide which of the ideas in this course to add.