Advanced and Research-Level Agents

Explore the research frontier of agents: self-improving agents and agent learning, trajectory optimization, agent fine-tuning from successful trajectories, simulation environments, agent swarms and emerging autonomous-agent architectures.

30 minAdvancedCode Examples

The Core Idea: At the research frontier, agents learn from their own experience: they write down lessons, sample many trajectories and keep the best, fine-tune on their successes, practice in simulations, and sometimes work in swarms. These methods are promising but young, so measure carefully.

  1. Beginner: Self-Improving Agents and Agent Learning

A self-improving agent gets better at its task over time without a human rewriting its prompts. The simplest form is reflection memory: after a failure the agent writes down what went wrong, stores the lesson and reads it on the next attempt.

In the code: The first booking attempt fails with an invalid date format. The agent stores Dates must be written as YYYY-MM-DD. The second attempt, with that lesson in context, succeeds.

Way an agent learnsWhat changesNeeds training?
Reflection memoryNotes added to the promptNo
Skill librarySaved, tested code or procedures it can reuseNo
Prompt optimizationThe instructions are rewritten and testedNo
Fine-tuningThe model weights changeYes

Caution: Learning from experience can also learn bad habits or store false lessons. Review what the agent saves.

  1. Beginner: Trajectory Optimization

A trajectory is the full record of a run: every thought, action and observation. Because agents are random, the same task gives different trajectories. Trajectory optimization uses this: sample many trajectories, score them, and keep or learn from the best.

TechniqueIdea
Best-of-NRun N times, pick the highest-scoring result
Search over stepsExplore several possible next actions and keep promising branches
Scoring by outcomeReward a successful result, minus a small cost per step
Scoring by processJudge each step as good or bad, not just the final outcome

In the code: Out of 20 sampled trajectories, 11 succeeded. A reward of 10 for success minus 0.3 per step picked trajectory 1 as the best.

  1. Medium: Agent Fine-Tuning

Instead of only changing prompts, you can train the model on agent behavior. A common recipe is rejection sampling: run the agent many times, keep only the successful and efficient trajectories, and fine-tune on them, so the model learns to act more like its own best runs.

In the code: 8 of the 20 trajectories were kept: they succeeded and used no more than the median number of steps.

IngredientDetail
Checkable successTests, rules or a verifier that can say success or failure
Many attemptsNeeded to find enough good trajectories
FilteringRemove lucky, wasteful or unsafe runs
Evaluation on new tasksCheck it generalizes and did not memorize
Related ideas
Reinforcement learning with verifiable rewards trains agents directly on checkable outcomes, such as passing tests. It is an active research area.

  1. Medium: Agent Simulation

Real-world practice is slow and risky. Simulated environments let agents practice safely and cheaply: a fake website, a mock company with fake customers, a sandboxed code repository, or even simulated users played by another model.

Use of simulationBenefit
TrainingMany safe attempts, with automatic scoring
TestingReplay the same scenarios after every change
BenchmarksCompare agents fairly
Red-teamingRehearse attacks such as prompt injection

The gap: A simulation is never the real world. Agents that excel in a simulator can still fail on real systems, so validate on real tasks before trusting them.

  1. Advanced: Agent Swarms and Emerging Architectures

In a swarm, many simple agents work in parallel and their outputs are combined by voting, ranking or a judge. This is the multi-agent idea scaled up, and it is closely related to majority voting.

In the code: One agent is right about half the time. A swarm of 15 agents taking a majority vote is right about 93% of the time on the same kind of question, because wrong answers are scattered while the right answer repeats.

Be honest about noise: In the self-improvement test, 20 trials per prompt version picked the wrong winner (v2). With 400 trials it still tied v3 and v4 at 0.64 and picked v4, although v3 is truly best. Noisy tests can mislead any learning loop, so use enough trials and keep a held-out test set.

Emerging directionIdea
Hybrid plannersA strong model plans and cheaper models execute
Self-evolving skill librariesAgents save and refine reusable tools
Agent-to-agent protocolsStandards for agents from different vendors to cooperate
Long-horizon trainingTraining on very long task sequences
Better evaluationHarder, cleaner benchmarks
Staying current
This area moves fast. Treat claims skeptically, ask how a result was measured, and test ideas on your own tasks.

  1. Visual Explanation: The Self-Improvement Cycle and Two Honest Numbers

The figure follows the cycle of sampling, scoring, filtering and learning with the numbers from the code, then shows what swarms and noisy tests did in the experiments.

Part 1 · The Self-Improvement CycleTry, score, learn
Sample20 trajectories
Same task, different runs
↓
11 succeeded
ScoreReward
10 for success minus 0.3 per step
↓
Best: trajectory 1 (8.8)
FilterKeep the best
Successful and efficient only
↓
8 of 20 kept for training
LearnImprove
Reflection notes or fine-tuning
↓
Better next time
Part 2 · Numbers From the CodeSwarms and noisy tests
One agent
Accuracy
About 50%.
Swarm of 15
Majority vote
About 93%.
20 trials
Test per version
Picked the wrong version (v2).
400 trials
Test per version
Picked v4, tied with v3.
Click to see why swarms work, and when they do notCompare ▼
Voting helps when each agent is right more often than any single wrong answer, because mistakes spread out while the correct answer repeats. If agents share the same blind spot, they all make the same mistake and voting cannot fix it.

Part 1 shows the cycle step by step. Part 2 compares a single agent with a swarm and shows how test size changes a decision. Open the bar for why swarms work and when they fail.

  1. Python Implementation: Reflection, Trajectory Sampling, Swarms and Noisy Tests

This script demonstrates five research ideas in miniature: reflection memory, scoring and selecting trajectories, building a fine-tuning set by rejection sampling, a swarm of voting agents, and a self-improvement loop that compares prompt versions on noisy trials. Everything uses a fixed random seed.

frontier_agents.pyPython 3.10+ · standard library only
import randomrng = random.Random(7) # 1. Learning from failure: reflect, store a lesson, use it next timelessons = []def attempt(lessons):    if any("YYYY-MM-DD" in lesson for lesson in lessons):        return True, "booked"    return False, "error: invalid date format" ok, info = attempt(lessons)print("Attempt 1:", info)lessons.append("Dates must be written as YYYY-MM-DD")          # the reflection step writes a lessonok, info = attempt(lessons)print("Attempt 2 with the lesson:", info) # 2. Trajectory optimization: sample many tries, score them, keep the besttrajectories = [{"id": i, "success": rng.random() < 0.4, "steps": rng.randint(4, 14)} for i in range(20)]reward = lambda t: (10 if t["success"] else 0) - 0.3 * t["steps"]best = max(trajectories, key=reward)successes = [t for t in trajectories if t["success"]]print("Successful trajectories:", len(successes), "of", len(trajectories), "| best trajectory:", best["id"], "with reward", round(reward(best), 1)) # 3. Training data for agent fine-tuning: keep only successful, efficient trajectories (rejection sampling)median_steps = sorted(t["steps"] for t in successes)[len(successes) // 2]training_set = [t["id"] for t in successes if t["steps"] <= median_steps]print("Fine-tuning examples kept:", len(training_set), "of", len(trajectories), "| ids:", training_set) # 4. Agent swarm: many noisy agents vote (4 answer options, one correct)def one_agent(p_correct=0.5):    return 0 if rng.random() < p_correct else rng.randint(1, 3) def swarm_vote(n_agents):    votes = [one_agent() for _ in range(n_agents)]    return max(set(votes), key=votes.count) == 0 questions = 2000single = sum(one_agent() == 0 for _ in range(questions)) / questionsswarm = sum(swarm_vote(15) for _ in range(questions)) / questionsprint("One agent accuracy:", round(single, 3), "| swarm of 15 voting:", round(swarm, 3)) # 5. Self-improvement loop: test prompt versions on noisy trials, keep the best (noise can mislead)true_skill = {"v1": 0.50, "v2": 0.58, "v3": 0.66, "v4": 0.61}for trials in (20, 400):    scores = {v: sum(rng.random() < p for _ in range(trials)) / trials for v, p in true_skill.items()}    print("With", trials, "trials per version, estimated:", {v: round(s, 2) for v, s in scores.items()}, "-> picks", max(scores, key=scores.get))
EXPECTED OUTPUT (your numbers may differ slightly)
Attempt 1: error: invalid date format
Attempt 2 with the lesson: booked
Successful trajectories: 11 of 20 | best trajectory: 1 with reward 8.8
Fine-tuning examples kept: 8 of 20 | ids: [0, 1, 2, 4, 5, 7, 9, 15]
One agent accuracy: 0.504 | swarm of 15 voting: 0.927
With 20 trials per version, estimated: {'v1': 0.6, 'v2': 0.8, 'v3': 0.45, 'v4': 0.65} -> picks v2
With 400 trials per version, estimated: {'v1': 0.49, 'v2': 0.58, 'v3': 0.64, 'v4': 0.64} -> picks v4

Each experiment is tiny, but the pattern is the same as in real research. Note the final two lines: even with 400 trials the noisy test could not separate two close versions, which is why careful measurement is the foundation of every agent that learns.

Knowledge Check

Question 1: What is rejection sampling for agent fine-tuning?
Question 2: Why can a swarm of agents beat a single agent?

Key Points

✓Self-improving agents learn through reflection memory, skill libraries, prompt optimization or fine-tuning.
✓Trajectory optimization samples many runs, scores them and keeps or learns from the best.
✓Rejection sampling builds fine-tuning data from successful, efficient trajectories.
✓Simulations give safe, cheap practice and testing, but never match the real world exactly.
✓Swarms combine many agents by voting. They work when errors are independent.
✓Noisy tests can pick the wrong winner, so use enough trials and a held-out test set.

The Big Picture

The Frontier in One Line

Sample → score → keep the best → learn from it → test honestly

Research ideas change quickly, but the habit of measuring carefully stays valuable.

Up Next: You have finished the AI Agents series. A good next step is to build one real agent project end to end: pick a narrow task, add two or three tools, add guardrails and evaluation, and improve it from its traces.