The Core Idea: An agent is judged on what it achieved (task success), which tools it used and how it got there (trajectory). Logs and traces record every step so you can debug failures instead of guessing.
- Beginner: Agent Evaluation
Evaluating an agent is harder than evaluating a single answer, because an agent takes many steps and can reach the goal in different ways. You need to look at three levels:
| Level | Question | Example metric |
|---|---|---|
| Outcome | Did it achieve the goal? | Task success rate |
| Tool use | Did it use the right tools correctly? | Tool precision |
| Trajectory | Was the path sensible and efficient? | Extra steps vs. a reference path |
Remember: An agent can succeed by luck or waste ten steps getting there. Outcome metrics alone hide both problems.
- Beginner: Task Success
Task success is the share of tasks the agent completes correctly. To measure it you need a test set of realistic tasks with a clear way to check each result.
| Way to check | Example |
|---|---|
| Exact check | The right order id, the right number |
| Programmatic test | The generated code passes its unit tests |
| State check | The database now contains the new record |
| Judge model or human | Is the summary accurate and complete? |
In the code: Three of four tasks succeeded, so the success rate is 75%. Always report it on many tasks, and run each task several times because agents are not fully repeatable.
- Medium: Tool-Use and Trajectory Evaluation
Tool-use evaluation checks the calls: right tool, right arguments, no needless calls. Trajectory evaluation compares the whole sequence of steps with a reference or with rules.
| Metric | Meaning | Warning sign |
|---|---|---|
| Tool precision | Share of calls that used an appropriate tool | t3 scored 0.0: it called only the calculator |
| Exact match | Same steps as the reference | Too strict, since several paths can be valid |
| Extra steps | How many more steps than needed | t2 repeated a search |
| Repeated calls | Same call again and again | A loop in the making |
- Medium: Logging and Tracing
Logging records events. Tracing links the events of one run into a timeline, so you can follow a task from the user request to the final answer, including every model call and tool call.
| Record | Why |
|---|---|
| Run id on every line | Links all steps of one task |
| Step, tool and arguments | Shows what the agent did |
| Latency in ms | Finds slow steps |
| Status and error | Finds failures |
| Tokens and cost | Finds expensive steps |
| Model input and output (carefully) | Explains decisions |
Privacy: Logs can contain personal data and secrets. Redact them and set retention limits.
- Advanced: Debugging Agents
When an agent fails, use a repeatable method instead of guessing:
In the code: The first failing step of the failed run is step 3, and it is also the slowest at 2100 ms. The trace points straight at the problem.
- Visual Explanation: Measuring Agents and Debugging Them
The figure shows the three evaluation levels with the numbers from the code, then a simple debugging loop.
Click to see why one run is not enoughCompare ▼
Part 1 gives the result for outcome, tool use and trajectory. Part 2 shows the debugging loop. Open the bar to see why repeated runs matter.
- Python Implementation: Success, Tool Precision, Trajectories and Traces
This script evaluates four recorded agent runs. It computes the success rate, the tool precision, exact trajectory matches and extra steps, prints a structured trace as JSON lines, and finds the first failing and the slowest step of a failed run.
import json runs = [ {"task": "t1", "success": True, "tools": ["search", "read", "answer"], "reference": ["search", "read", "answer"]}, {"task": "t2", "success": True, "tools": ["search", "search", "read", "answer"], "reference": ["search", "read", "answer"]}, {"task": "t3", "success": False, "tools": ["calculator", "calculator", "calculator"], "reference": ["search", "answer"]}, {"task": "t4", "success": True, "tools": ["search", "answer"], "reference": ["search", "answer"]},] # 1. Task success: did the agent reach the goal?success_rate = sum(r["success"] for r in runs) / len(runs)print("Task success rate:", round(success_rate * 100), "%") # 2. Tool-use evaluation: were the tools it called the right ones?def tool_precision(run): right = sum(1 for tool in run["tools"] if tool in run["reference"]) return right / len(run["tools"]) print("Tool precision per task:", {r["task"]: round(tool_precision(r), 2) for r in runs}) # 3. Trajectory evaluation: how close was the path to the ideal one?for r in runs: exact = r["tools"] == r["reference"] extra = len(r["tools"]) - len(r["reference"]) print(r["task"], "| exact match:", exact, "| extra steps:", extra) # 4. Logging and tracing: one structured line per step, tied together by a run idtrace = [ {"run": "t3", "step": 1, "tool": "calculator", "ms": 40, "status": "ok"}, {"run": "t3", "step": 2, "tool": "calculator", "ms": 35, "status": "ok"}, {"run": "t3", "step": 3, "tool": "calculator", "ms": 2100, "status": "timeout"},]for line in trace: print(json.dumps(line)) # 5. Debugging: find the first problem and the slowest step in a failed runfirst_bad = next(s for s in trace if s["status"] != "ok")slowest = max(trace, key=lambda s: s["ms"])print("First failing step:", first_bad["step"], first_bad["tool"], "| slowest step:", slowest["step"], str(slowest["ms"]) + " ms")The headline number (75%) hides the story: t3 failed because it used the wrong tool three times, and t2 succeeded but wasted a step. The structured trace of the failed run shows that its last step timed out and was also the slowest.
Knowledge Check
Key Points
The Big Picture
Test set + success rate + tool and trajectory metrics + traces = agents you can improve with evidence
What you cannot measure you cannot improve, and what you cannot trace you cannot debug.
Up Next: One agent can only do so much. In Introduction to Multi-Agent Systems you will learn why and when several specialized agents work better than one.