The Core Idea: Tools will fail: networks drop, services time out, models make bad calls. A reliable agent expects failure: it retries temporary errors, uses fallbacks, enforces timeouts, recovers with a plan, and stops loops before they burn money.
- Beginner: Tool Failures
Every external tool can fail. Knowing what kind of failure it is decides the right response:
| Failure type | Example | Temporary? | Typical response |
|---|---|---|---|
| Network or service error | 503 service unavailable | Yes | Retry with backoff |
| Rate limit | 429 too many requests | Yes | Wait, then retry |
| Timeout | No answer in 30 seconds | Maybe | Retry once, then fall back |
| Bad request | Missing or wrong argument | No | Fix the call, do not repeat it |
| Permission denied | No access to the file | No | Stop or ask for access |
| Wrong result | Tool returns empty or nonsense | Maybe | Validate and try another tool |
Key idea: Do not treat every error the same. Retry what is temporary. Fix or report what is permanent.
- Beginner: Retries
A retry simply tries again. Done badly, it makes outages worse. Done well, it hides small glitches from the user. Use exponential backoff: wait 1 second, then 2, then 4, and so on, with a small random delay (jitter) so many clients do not retry at the same instant.
In the code: The flaky tool fails twice and succeeds on the third try. The agent waited 1 s and 2 s between attempts, 3 seconds of simulated time in total.
Never retry blindly: Do not retry errors that cannot change, such as a bad argument. And never retry an action with side effects, such as a payment, unless it is idempotent (safe to repeat).
- Medium: Fallbacks and Timeouts
A fallback is a plan B when the main option keeps failing: another tool, cached data, a simpler method, or an honest message. A timeout limits how long to wait, so one slow tool cannot freeze the whole agent.
| Fallback | Example |
|---|---|
| Cached data | Show yesterday's prices, marked as possibly old |
| Alternative tool | Search engine B if search engine A is down |
| Simpler method | Summarize titles only if the full page cannot be fetched |
| Ask the user | Request the missing information |
| Honest failure | Tell the user what failed and what was tried |
In the code: The slow tool took 12 seconds against a 5 second limit, so the call is cut off with a clear timeout error that the agent can react to.
- Medium: Recovery Strategies
When something fails, the agent chooses a recovery strategy. Roughly from cheapest to most expensive:
- Advanced: Preventing Infinite Loops
The most expensive failure is an agent that never stops. Defend in layers:
| Guard | How it works |
|---|---|
| Maximum steps | A hard limit on the number of loop iterations |
| Repeat detection | If the last three actions are identical, stop or change approach |
| Budgets | Limits on tokens, time and money per run |
| No-progress check | If the state has not changed for several steps, escalate |
| Retry caps | A fixed number of retries per tool |
In the code: The confused agent repeats the same search. The loop detector fires at step 3 and stops it. Without a guard, this would repeat until the budget is gone.
- Visual Explanation: Four Defenses and a Recovery Ladder
The figure shows the four defenses from the code, then orders the recovery strategies from cheapest to most expensive.
Click to see why retries need jitterCompare ▼
Part 1 maps each failure to its defense. Part 2 is a ladder you climb only as far as needed. Open the bar to learn why jitter matters.
- Python Implementation: Retries, Fallbacks, Timeouts and a Loop Guard
This script uses a fake clock so it runs instantly. It retries a flaky tool with exponential backoff, falls back to cached data when the primary tool stays down, cuts off a slow tool with a timeout, and detects an agent repeating the same action.
# Error handling: retries with backoff, fallbacks, timeouts and loop detectionclass FakeClock: # so the demo runs instantly instead of really waiting def __init__(self): self.now = 0 def sleep(self, seconds): self.now += seconds clock = FakeClock()attempts = {"flaky": 0} def flaky_tool(): # fails twice, then works attempts["flaky"] += 1 if attempts["flaky"] < 3: raise ConnectionError("503 service unavailable") return "fresh data" def with_retry(fn, max_attempts=4, base_delay=1): for attempt in range(1, max_attempts + 1): try: return fn() except ConnectionError as error: if attempt == max_attempts: raise delay = base_delay * 2 ** (attempt - 1) # 1, 2, 4 seconds... print(" attempt", attempt, "failed (" + str(error) + "), waiting", delay, "s") clock.sleep(delay) print("1. Retry with backoff ->", with_retry(flaky_tool), "| simulated waiting:", clock.now, "s") def primary_tool(): raise ConnectionError("primary down") def fallback_tool(): return "cached data (may be old)" def with_fallback(): try: return with_retry(primary_tool, max_attempts=2) except ConnectionError: return fallback_tool() print("2. Fallback ->", with_fallback()) def slow_tool(): return "report", 12 # result and the seconds it took def with_timeout(tool, limit): result, seconds = tool() if seconds > limit: raise TimeoutError("took " + str(seconds) + "s, limit is " + str(limit) + "s") return result try: with_timeout(slow_tool, limit=5)except TimeoutError as error: print("3. Timeout ->", error) def detect_loop(actions, window=3): last = actions[-window:] return len(last) == window and len(set(last)) == 1 actions = []for step in range(1, 10): actions.append("search:weather today") # a confused agent repeats itself if detect_loop(actions): print("4. Loop detected at step", step, "-> stop and change approach") breakEach of the four defenses fires on its own kind of failure: the retry recovers a temporary error, the fallback covers a longer outage, the timeout bounds slow work, and the loop detector stops the repeating agent. Together they turn crashes into controlled outcomes.
Knowledge Check
Key Points
The Big Picture
Expect failure → classify it → retry or fall back → replan or escalate → never loop forever
Resilient agents turn most failures into small delays instead of broken tasks.
Up Next: Some actions should not happen without a person's say-so. In Human-in-the-Loop and Guardrails you will learn approvals, permissions and validation.