The Core Idea: Advanced agents extend the same loop with richer inputs and bigger toolboxes: they retrieve knowledge and tools on demand, see and operate screens, work with images, audio and documents, and write and run code. The loop is the same. The observations and actions get richer.
- Beginner: Agentic RAG, Revisited
You met agentic RAG earlier: the agent decides when to retrieve, rewrites queries, checks scores and retries. Advanced systems add more control:
| Technique | Idea |
|---|---|
| Multi-hop retrieval | Use the first answer to form the next search, for questions that need several facts |
| Source routing | Choose between several knowledge bases or tools, such as documents, a database or the web |
| Relevance and support checks | Verify that retrieved text really supports the answer before writing it |
| Citations | Attach the source to every claim |
| Fallback to the user | Ask a clarifying question when retrieval keeps failing |
Remember: Each extra control improves hard questions but adds latency and cost. Add them only where simple retrieval fails.
- Beginner: Tool-Rich Agents
A real company may have hundreds of tools. Putting every tool description in the prompt wastes tokens and confuses the model. A tool-rich agent first retrieves the few tools relevant to the request, then shows only those.
In the code: 30 tools would need 330 tokens in every prompt. For the request show me the list of orders, the agent retrieves just three order tools and uses 33 tokens, a tenth of the size.
| Way to find tools | How |
|---|---|
| Keyword or embedding search | Match the request to tool descriptions |
| Categories (routing) | Pick a group such as billing, then show its tools |
| Learned from usage | Prefer tools that worked in similar tasks |
- Medium: Computer-Use and Browser Agents
A browser agent works with web pages, and a computer-use agent works with a whole desktop. Instead of calling an API, it looks at the screen and performs actions: click, type, scroll, press a key.
In the code: The page is a list of elements with ids. The agent types the email, fills the password from a secret store and clicks Sign in. The page then changes to a dashboard.
Risks: These agents can click the wrong thing and can be misled by hostile content on a page. Run them in a sandboxed browser, use limited accounts, never expose real passwords to the model and ask for approval before purchases or messages.
- Medium: Multimodal Agents
A multimodal agent understands more than text: images, screenshots, charts, PDFs, audio and sometimes video. This opens new tasks:
| Input | Example task |
|---|---|
| Screenshot | Operate an app or find a button |
| Chart or photo | Describe a trend or check a damaged product |
| PDF or scanned form | Extract fields and fill a system |
| Audio | Transcribe a call and create follow-up tasks |
- Advanced: Code Agents and Choosing the Design
A code agent writes code, runs it and reads the results. This ReAct-and-reflection pattern is why coding assistants work well: the test runner provides exact, objective feedback.
| Code agent ingredient | Why |
|---|---|
| Repository search and file tools | Understand a large codebase |
| A sandbox to run code and tests | Safe execution |
| Test results as feedback | Self-correction with real errors |
| Version control | Review, undo and compare changes |
| Human review of the result | Safety for important changes |
Choosing: Pick the narrowest agent type that solves the problem. A plain API tool is faster, cheaper and safer than a browser agent, which is itself safer than controlling a whole computer.
- Visual Explanation: Four Advanced Agents Share One Loop
The figure shows what each advanced agent observes and does, then orders the approaches from the narrowest and safest to the broadest.
Click to see why screens are a last resortCompare ▼
Part 1 compares the observations and actions of four agent types. Part 2 is a ladder from API tools up to computer use. Open the bar to see why screens are a last resort.
- Python Implementation: Tool Retrieval and a Browser-Style Action Loop
This script shows two ideas. First, a tool-rich agent retrieves the three most relevant tools from a catalog of 30 and compares the prompt sizes. Second, a browser-style agent observes a page as a list of elements, acts by typing and clicking, and observes the changed page.
# 1. A tool-rich agent: retrieve only the few relevant tools instead of showing all of themverbs = ["get", "create", "delete", "update", "list"]nouns = ["invoice", "ticket", "customer", "order", "report", "calendar"]TOOLS = [(v + "_" + n, v + " a " + n + " record in the system") for v in verbs for n in nouns] def words(text): return {w.rstrip("s") for w in text.lower().replace("_", " ").split()} def select_tools(query, k=3): q = words(query) scored = sorted(TOOLS, key=lambda tool: len(q & words(tool[0] + " " + tool[1])), reverse=True) return [name for name, _ in scored[:k]] def prompt_tokens(tools): return sum(round(len((n + " " + d).split()) / 0.75) for n, d in tools) query = "show me the list of orders"print("Tools available:", len(TOOLS), "| tokens if all are shown:", prompt_tokens(TOOLS))chosen = select_tools(query)print("Query:", query, "-> shown to the model:", chosen, "| tokens:", prompt_tokens([t for t in TOOLS if t[0] in chosen])) # 2. A browser agent: the page is turned into a list of elements the agent can act onpage = {"title": "Login", "elements": [ {"id": 1, "role": "input", "label": "Email", "value": ""}, {"id": 2, "role": "input", "label": "Password", "value": ""}, {"id": 3, "role": "button", "label": "Sign in"}]} def observe(page): return page["title"] + ": " + ", ".join("[" + str(e["id"]) + "] " + e["role"] + " " + e["label"] for e in page["elements"]) def act(page, action): if action["type"] == "type": next(e for e in page["elements"] if e["id"] == action["id"])["value"] = action["text"] elif action["type"] == "click" and all(e.get("value", "x") for e in page["elements"]): return {"title": "Dashboard", "elements": [{"id": 4, "role": "text", "label": "Welcome, Ana"}]} return page plan = [{"type": "type", "id": 1, "text": "ana@example.com"}, {"type": "type", "id": 2, "text": "(from the secret store)"}, {"type": "click", "id": 3}]print("Observe:", observe(page))for action in plan: page = act(page, action) print("Act:", action, "->", observe(page))The retrieval step cuts the tool prompt to one tenth. In the browser loop the observation is a compact text list of elements rather than a picture, which is cheaper and easier for the model, and the final observation shows the dashboard. Real browser agents use a browser automation library for the act step.
Knowledge Check
Key Points
The Big Picture
Same loop, richer senses and hands, always the narrowest tool that works
Each new capability brings new risks, so add power only where it earns its place.
Up Next: The last lesson looks at the research frontier. In Advanced and Research-Level Agents you will see self-improving agents, trajectory optimization, agent fine-tuning, simulation and swarms.