Human-in-the-Loop and Guardrails

Learn how to keep agents safe: human approval for risky actions, permission systems, safety checks, output validation, action validation, and how to grant autonomy gradually.

28 minIntermediateCode Examples

The Core Idea: Controlled autonomy means the agent works alone on safe tasks and asks a human for risky ones. Rules in code, such as permissions, limits and validation, enforce this, because instructions in a prompt can be ignored.

  1. Beginner: Human Approval

Human-in-the-loop (HITL) means a person reviews or approves certain steps. It is the simplest and strongest safety tool, especially for actions that cannot be undone.

Action typeExampleApproval needed?
Read onlyLook up an orderUsually no
Reversible changeAdd a note to a ticketSometimes
Hard to undoSend an email to a customerYes
Costly or sensitiveIssue a refund, delete dataYes, plus limits

How it works: The agent proposes the action with the reason. A human sees the exact details, and approves, edits or denies. Only then does the system run it.

  1. Beginner: Permission Systems

A permission system decides what each tool and each user is allowed to do. It lives in code, not in the prompt.

LevelMeaningExample from the code
autoRuns without askingread_orders
approvalA human must approvesend_email, issue_refund
forbiddenNever alloweddelete_account
unknown toolTreated as forbiddenformat_disk

Default to no: Anything that is not explicitly allowed should be blocked. A list of what is permitted is safer than a list of what is forbidden.

  1. Medium: Safety Checks and Action Validation

Before running any action, validate it in code:

CheckExample
Is the tool allowed?delete_account is forbidden
Are the arguments valid?The amount is a positive number
Are the values within limits?Refunds up to 100
Does it match the user's rights?The user owns the order
Is it consistent with the goal?A refund tool is not needed to look up shipping

In the code: A 500 refund is blocked by the limit even if a human would approve it. Limits are a second safety net behind approvals, because people also click approve by mistake.

  1. Medium: Output Validation

Output validation checks what the agent is about to say or send. It can catch mistakes and leaks:

CheckCatches
Pattern scanAPI keys, card numbers, emails, passwords
Format checkMissing fields or broken JSON
Policy checkForbidden topics or promises
Fact checkClaims not supported by tool results

In the code: The reply contained a card number, an email address and an API key. All three were found and redacted before the text left the system.

Another model as a checker
A second model call can review output against a policy. It is useful but fallible, so pair it with simple rules for the things that must never leak.

  1. Advanced: Controlled Autonomy

Trust is earned. A good rollout increases autonomy step by step:

1
Shadow mode: The agent proposes, a human does the action. Measure how often they agree.
2
Approve everything: The agent acts only after approval.
3
Approve the risky: Safe actions run alone. Risky ones need approval.
4
Monitor: Review logs and a sample of automatic actions.
5
Stay ready to pull back: A kill switch and the ability to lower autonomy quickly.

Avoid approval fatigue: If people approve hundreds of trivial actions, they stop reading. Ask only when it matters, and show clear summaries.

  1. Visual Explanation: Permission Outcomes and Layers of Safety

The figure shows how six requests are handled by the permission system in the code, then the layers of safety around every action.

Part 1 · Permission Decisions in the CodeSix requests, six outcomes
AllowedRuns
read_orders (auto)
↓
issue_refund of 40, approved
↓
ALLOWED
Needs a humanWaits
issue_refund of 40, human says no
↓
DENIED by the reviewer
BlockedNever runs
refund of 500 over the limit
↓
delete_account (forbidden)
↓
format_disk (unknown)
↓
BLOCKED
Part 2 · Three Layers of SafetyDefense in depth
Permissions
Before
Is this tool allowed at all?
Limits
During
Are values within bounds?
Approval
Gate
Does a human agree?
Output check
After
Does the text leak anything?
Click to see why prompts are not enoughCompare ▼
A prompt that says never delete accounts is only a request, and a model can ignore it or be tricked. A permission check in code cannot be talked around, so important rules must live in code.

Part 1 groups the outcomes into allowed, needs a human, and blocked. Part 2 shows the four layers. Open the bar to learn why prompts alone are not enough.

  1. Python Implementation: Permissions, Limits, Approval and Output Redaction

This script defines a permission table (auto, approval, forbidden), a refund limit and a human approver function, then runs six requests through it. It also redacts API keys, card numbers and emails from a reply before it is sent.

guardrails.pyPython 3.10+ · standard library only
import re PERMISSIONS = {"read_orders": "auto", "send_email": "approval", "issue_refund": "approval", "delete_account": "forbidden"}REFUND_LIMIT = 100 def check_action(name, args, approver):    level = PERMISSIONS.get(name, "forbidden")             # unknown tools are forbidden by default    if level == "forbidden":        return "BLOCKED: this action is never allowed"    if name == "issue_refund" and args["amount"] > REFUND_LIMIT:        return "BLOCKED: refund above the limit of " + str(REFUND_LIMIT)    if level == "approval" and not approver(name, args):        return "DENIED by the human reviewer"    return "ALLOWED" approve_all = lambda name, args: Truedeny_all = lambda name, args: False tests = [    ("read_orders", {}, deny_all),    ("issue_refund", {"amount": 40}, approve_all),    ("issue_refund", {"amount": 40}, deny_all),    ("issue_refund", {"amount": 500}, approve_all),    ("delete_account", {}, approve_all),    ("format_disk", {}, approve_all),]for name, args, approver in tests:    print(name.ljust(15), str(args).ljust(18), "->", check_action(name, args, approver)) # Output validation: remove secrets and personal data before text leaves the systemPATTERNS = {    "api key": r"sk-[A-Za-z0-9]{8,}",    "card number": r"\b(?:\d[ -]?){13,16}\b",    "email": r"[\w.+-]+@[\w-]+\.[\w.]+",} def validate_output(text):    found = []    for label, pattern in PATTERNS.items():        if re.search(pattern, text):            found.append(label)            text = re.sub(pattern, "[REDACTED " + label + "]", text)    return text, found reply = "Refund sent. Card 4111 1111 1111 1111, contact ana@example.com, key sk-abcdef123456."clean, found = validate_output(reply)print("Redacted:", found)print(clean)
EXPECTED OUTPUT (your numbers may differ slightly)
read_orders {} -> ALLOWED
issue_refund {'amount': 40} -> ALLOWED
issue_refund {'amount': 40} -> DENIED by the human reviewer
issue_refund {'amount': 500} -> BLOCKED: refund above the limit of 100
delete_account {} -> BLOCKED: this action is never allowed
format_disk {} -> BLOCKED: this action is never allowed
Redacted: ['api key', 'card number', 'email']
Refund sent. Card [REDACTED card number], contact [REDACTED email], key [REDACTED api key].

Reading orders runs automatically, a refund needs and gets approval, a denied refund stops, a 500 refund hits the limit, and forbidden or unknown tools are blocked. The output validator catches all three kinds of sensitive data in the reply.

Knowledge Check

Question 1: Why should safety rules live in code and not only in the prompt?
Question 2: What should the default be for a tool not on the permission list?

Key Points

✓Human approval is the strongest safeguard for irreversible or costly actions.
✓Permission systems live in code and default to forbidden.
✓Validate every action: tool allowed, arguments valid, values within limits, user has the right.
✓Validate outputs: scan for secrets and personal data, check format and policy.
✓Limits are a second net behind approvals, because people make mistakes too.
✓Grow autonomy gradually from shadow mode, and avoid approval fatigue.

The Big Picture

Guardrails in One Line

Allow explicitly → validate in code → ask a human for risky actions → check the output

Safe agents are not the ones that never fail, but the ones whose failures are contained.

Up Next: You can now make agents safer. But how do you know they work? In Agent Evaluation and Observability you will measure success and trace what agents do.