The Core Idea: Controlled autonomy means the agent works alone on safe tasks and asks a human for risky ones. Rules in code, such as permissions, limits and validation, enforce this, because instructions in a prompt can be ignored.
- Beginner: Human Approval
Human-in-the-loop (HITL) means a person reviews or approves certain steps. It is the simplest and strongest safety tool, especially for actions that cannot be undone.
| Action type | Example | Approval needed? |
|---|---|---|
| Read only | Look up an order | Usually no |
| Reversible change | Add a note to a ticket | Sometimes |
| Hard to undo | Send an email to a customer | Yes |
| Costly or sensitive | Issue a refund, delete data | Yes, plus limits |
How it works: The agent proposes the action with the reason. A human sees the exact details, and approves, edits or denies. Only then does the system run it.
- Beginner: Permission Systems
A permission system decides what each tool and each user is allowed to do. It lives in code, not in the prompt.
| Level | Meaning | Example from the code |
|---|---|---|
| auto | Runs without asking | read_orders |
| approval | A human must approve | send_email, issue_refund |
| forbidden | Never allowed | delete_account |
| unknown tool | Treated as forbidden | format_disk |
Default to no: Anything that is not explicitly allowed should be blocked. A list of what is permitted is safer than a list of what is forbidden.
- Medium: Safety Checks and Action Validation
Before running any action, validate it in code:
| Check | Example |
|---|---|
| Is the tool allowed? | delete_account is forbidden |
| Are the arguments valid? | The amount is a positive number |
| Are the values within limits? | Refunds up to 100 |
| Does it match the user's rights? | The user owns the order |
| Is it consistent with the goal? | A refund tool is not needed to look up shipping |
In the code: A 500 refund is blocked by the limit even if a human would approve it. Limits are a second safety net behind approvals, because people also click approve by mistake.
- Medium: Output Validation
Output validation checks what the agent is about to say or send. It can catch mistakes and leaks:
| Check | Catches |
|---|---|
| Pattern scan | API keys, card numbers, emails, passwords |
| Format check | Missing fields or broken JSON |
| Policy check | Forbidden topics or promises |
| Fact check | Claims not supported by tool results |
In the code: The reply contained a card number, an email address and an API key. All three were found and redacted before the text left the system.
- Advanced: Controlled Autonomy
Trust is earned. A good rollout increases autonomy step by step:
Avoid approval fatigue: If people approve hundreds of trivial actions, they stop reading. Ask only when it matters, and show clear summaries.
- Visual Explanation: Permission Outcomes and Layers of Safety
The figure shows how six requests are handled by the permission system in the code, then the layers of safety around every action.
Click to see why prompts are not enoughCompare ▼
Part 1 groups the outcomes into allowed, needs a human, and blocked. Part 2 shows the four layers. Open the bar to learn why prompts alone are not enough.
- Python Implementation: Permissions, Limits, Approval and Output Redaction
This script defines a permission table (auto, approval, forbidden), a refund limit and a human approver function, then runs six requests through it. It also redacts API keys, card numbers and emails from a reply before it is sent.
import re PERMISSIONS = {"read_orders": "auto", "send_email": "approval", "issue_refund": "approval", "delete_account": "forbidden"}REFUND_LIMIT = 100 def check_action(name, args, approver): level = PERMISSIONS.get(name, "forbidden") # unknown tools are forbidden by default if level == "forbidden": return "BLOCKED: this action is never allowed" if name == "issue_refund" and args["amount"] > REFUND_LIMIT: return "BLOCKED: refund above the limit of " + str(REFUND_LIMIT) if level == "approval" and not approver(name, args): return "DENIED by the human reviewer" return "ALLOWED" approve_all = lambda name, args: Truedeny_all = lambda name, args: False tests = [ ("read_orders", {}, deny_all), ("issue_refund", {"amount": 40}, approve_all), ("issue_refund", {"amount": 40}, deny_all), ("issue_refund", {"amount": 500}, approve_all), ("delete_account", {}, approve_all), ("format_disk", {}, approve_all),]for name, args, approver in tests: print(name.ljust(15), str(args).ljust(18), "->", check_action(name, args, approver)) # Output validation: remove secrets and personal data before text leaves the systemPATTERNS = { "api key": r"sk-[A-Za-z0-9]{8,}", "card number": r"\b(?:\d[ -]?){13,16}\b", "email": r"[\w.+-]+@[\w-]+\.[\w.]+",} def validate_output(text): found = [] for label, pattern in PATTERNS.items(): if re.search(pattern, text): found.append(label) text = re.sub(pattern, "[REDACTED " + label + "]", text) return text, found reply = "Refund sent. Card 4111 1111 1111 1111, contact ana@example.com, key sk-abcdef123456."clean, found = validate_output(reply)print("Redacted:", found)print(clean)Reading orders runs automatically, a refund needs and gets approval, a denied refund stops, a 500 refund hits the limit, and forbidden or unknown tools are blocked. The output validator catches all three kinds of sensitive data in the reply.
Knowledge Check
Key Points
The Big Picture
Allow explicitly → validate in code → ask a human for risky actions → check the output
Safe agents are not the ones that never fail, but the ones whose failures are contained.
Up Next: You can now make agents safer. But how do you know they work? In Agent Evaluation and Observability you will measure success and trace what agents do.