The Core Idea: Agent costs grow quickly because every step re-sends the whole history. The main levers are fewer tokens, cheaper models for easy steps, caching, parallel tool calls and hard budgets. Measure first, then optimize the biggest cost.
- Beginner: Where Time and Money Go
An agent run is many model calls plus tool calls. You pay for tokens (input and output) and wait for latency (model time plus tool time). Both grow with the number of steps.
| Cost driver | Why it grows |
|---|---|
| Input tokens | Every step re-sends instructions, tool descriptions and the growing history |
| Output tokens | Long reasoning and long answers (output tokens usually cost more than input) |
| Number of steps | More loops mean more calls |
| Model size | Larger models cost more per token and are slower |
| Tool time | Slow APIs add waiting time |
Rule: Measure before you optimize. Log tokens, cost, latency and step count for every run, and fix the biggest item first.
- Beginner: Latency and Token Usage
To cut latency, reduce the number of sequential model calls and the time spent waiting on tools. To cut tokens, send less to the model:
| Technique | Effect |
|---|---|
| Shorter, sharper instructions | Fewer tokens on every call |
| Fewer, better tools | Smaller tool descriptions |
| Compact tool results | Return only needed fields |
| Trim or summarize history | Stops quadratic growth |
| Concise output formats | Fewer output tokens |
Why history matters: In a 12-step run, step 12 sends the system prompt plus 11 steps of history. The total input grows roughly with the square of the number of steps, which is why long runs get expensive fast.
- Medium: Model Selection
Not every step needs the strongest model. A common design uses different models for different jobs:
| Step type | Model choice |
|---|---|
| Planning and hard reasoning | Larger, stronger model |
| Routine tool calls and formatting | Smaller, cheaper, faster model |
| Classification and routing | Very small model |
| Final answer quality | Stronger model if quality matters |
Be careful: Test a cheaper model with your evaluation set before switching. A saving that lowers success rate is not a saving.
In the code: Using a model that costs a tenth for half of the steps cuts the run cost to about 67% of the baseline.
- Medium: Caching and Parallel Execution
Caching avoids paying twice for the same work. Prompt caching discounts a repeated prefix, such as instructions and earlier history. Result caching stores tool results for identical calls.
Parallel execution runs independent tool calls at the same time. In the code, four 1.5-second calls take 6.0 seconds one after another but only 1.5 seconds in parallel.
| Technique | Saving | Condition |
|---|---|---|
| Prompt caching | Large (34% of baseline cost in the code, with an example 90% discount) | Stable prefix at the start of the prompt |
| Result caching | Time and tool cost | Same arguments, data not stale |
| Parallel tool calls | Latency | Independent calls |
- Advanced: Tool Optimization and Cost Control
Tools are a hidden cost. A search tool that returns a whole web page can flood the context, and each extra call adds a model round trip.
| Tool optimization | Why |
|---|---|
| Return compact, structured results | Saves input tokens on every later step |
| Merge tiny tools into one | Fewer round trips |
| Batch calls (look up ten IDs at once) | Fewer steps |
| Truncate and say so | Controls size without confusing the model |
| Cache slow, stable lookups | Saves time and money |
Finally, add cost controls so a bad run cannot become a bad bill:
| Control | Example |
|---|---|
| Per-run budget | Stop after a maximum number of tokens or dollars |
| Step limit | Hard cap on loops |
| Per-user quotas | Limit usage per account |
| Alerts | Notify on unusual spend |
| Cost in your logs | Know the cost per task type |
Do not forget quality: Always measure success rate next to cost. Cheaper but wrong is not an optimization.
- Visual Explanation: What Each Optimization Saves
The figure compares the cost of one run under each optimization using the example prices in the code, then lists the biggest levers.
Click to see the cost for 10,000 runsCompare ▼
Part 1 shows each variant as a share of the baseline cost. Part 2 groups the savings by lever. Open the bar for the cost of 10,000 runs.
- Python Implementation: A Cost Model for a 12-Step Agent Run
This script models the input and output cost of a 12-step run where every step re-sends the history. It compares the baseline with prompt caching, history summarization, a cheaper model for half of the steps, and all three together, and shows the latency gain of parallel tool calls. The prices are made-up examples.
# Cost of one agent run: every step re-sends the growing historyIN_PRICE, OUT_PRICE = 3.0, 15.0 # example prices in USD per million tokens (not real quotes) def run_cost(steps=12, system=1000, per_step=400, output=150, cached=False, summarize_cap=None, cheap_share=0.0): total = 0.0 for i in range(1, steps + 1): history = (i - 1) * per_step if summarize_cap: history = min(history, summarize_cap) # old steps are compressed into a short summary prefix = system + history if cached: input_cost = prefix * IN_PRICE * 0.1 + per_step * IN_PRICE # repeated prefix at an example 90% discount else: input_cost = (prefix + per_step) * IN_PRICE step_cost = (input_cost + output * OUT_PRICE) / 1_000_000 if i / steps <= cheap_share: step_cost *= 0.1 # easy steps use a model that costs a tenth total += step_cost return total baseline = run_cost()variants = [ ("baseline", baseline), ("+ prompt caching", run_cost(cached=True)), ("+ summarize history (cap 800)", run_cost(summarize_cap=800)), ("+ cheap model for 50% of steps", run_cost(cheap_share=0.5)), ("all three together", run_cost(cached=True, summarize_cap=800, cheap_share=0.5)),]for name, cost in variants: print(name.ljust(32), "USD", round(cost, 4), "per run |", round(cost / baseline * 100), "% of baseline | USD", round(cost * 10000, 2), "per 10,000 runs") # Latency: independent tool calls can run in paralleltool_seconds = [1.5, 1.5, 1.5, 1.5]print("4 tool calls one after another:", sum(tool_seconds), "s | in parallel:", max(tool_seconds), "s")Caching gives the largest single saving in this model, and combining all three cuts the cost to about one sixth. The model is simplified, because real providers have their own caching rules, so use it to build intuition and then measure your own runs.
Knowledge Check
Key Points
The Big Picture
Measure → send fewer tokens → route to cheaper models → cache → parallelize → cap the budget
Cheaper agents are not just a nicety. They decide whether a product is affordable to run.
Up Next: Now push agents further. In Autonomous and Long-Horizon Agents you will see how agents run for hours, remember across steps and recover from failure.