The Core Idea: At the research frontier, agents learn from their own experience: they write down lessons, sample many trajectories and keep the best, fine-tune on their successes, practice in simulations, and sometimes work in swarms. These methods are promising but young, so measure carefully.
- Beginner: Self-Improving Agents and Agent Learning
A self-improving agent gets better at its task over time without a human rewriting its prompts. The simplest form is reflection memory: after a failure the agent writes down what went wrong, stores the lesson and reads it on the next attempt.
In the code: The first booking attempt fails with an invalid date format. The agent stores Dates must be written as YYYY-MM-DD. The second attempt, with that lesson in context, succeeds.
| Way an agent learns | What changes | Needs training? |
|---|---|---|
| Reflection memory | Notes added to the prompt | No |
| Skill library | Saved, tested code or procedures it can reuse | No |
| Prompt optimization | The instructions are rewritten and tested | No |
| Fine-tuning | The model weights change | Yes |
Caution: Learning from experience can also learn bad habits or store false lessons. Review what the agent saves.
- Beginner: Trajectory Optimization
A trajectory is the full record of a run: every thought, action and observation. Because agents are random, the same task gives different trajectories. Trajectory optimization uses this: sample many trajectories, score them, and keep or learn from the best.
| Technique | Idea |
|---|---|
| Best-of-N | Run N times, pick the highest-scoring result |
| Search over steps | Explore several possible next actions and keep promising branches |
| Scoring by outcome | Reward a successful result, minus a small cost per step |
| Scoring by process | Judge each step as good or bad, not just the final outcome |
In the code: Out of 20 sampled trajectories, 11 succeeded. A reward of 10 for success minus 0.3 per step picked trajectory 1 as the best.
- Medium: Agent Fine-Tuning
Instead of only changing prompts, you can train the model on agent behavior. A common recipe is rejection sampling: run the agent many times, keep only the successful and efficient trajectories, and fine-tune on them, so the model learns to act more like its own best runs.
In the code: 8 of the 20 trajectories were kept: they succeeded and used no more than the median number of steps.
| Ingredient | Detail |
|---|---|
| Checkable success | Tests, rules or a verifier that can say success or failure |
| Many attempts | Needed to find enough good trajectories |
| Filtering | Remove lucky, wasteful or unsafe runs |
| Evaluation on new tasks | Check it generalizes and did not memorize |
- Medium: Agent Simulation
Real-world practice is slow and risky. Simulated environments let agents practice safely and cheaply: a fake website, a mock company with fake customers, a sandboxed code repository, or even simulated users played by another model.
| Use of simulation | Benefit |
|---|---|
| Training | Many safe attempts, with automatic scoring |
| Testing | Replay the same scenarios after every change |
| Benchmarks | Compare agents fairly |
| Red-teaming | Rehearse attacks such as prompt injection |
The gap: A simulation is never the real world. Agents that excel in a simulator can still fail on real systems, so validate on real tasks before trusting them.
- Advanced: Agent Swarms and Emerging Architectures
In a swarm, many simple agents work in parallel and their outputs are combined by voting, ranking or a judge. This is the multi-agent idea scaled up, and it is closely related to majority voting.
In the code: One agent is right about half the time. A swarm of 15 agents taking a majority vote is right about 93% of the time on the same kind of question, because wrong answers are scattered while the right answer repeats.
Be honest about noise: In the self-improvement test, 20 trials per prompt version picked the wrong winner (v2). With 400 trials it still tied v3 and v4 at 0.64 and picked v4, although v3 is truly best. Noisy tests can mislead any learning loop, so use enough trials and keep a held-out test set.
| Emerging direction | Idea |
|---|---|
| Hybrid planners | A strong model plans and cheaper models execute |
| Self-evolving skill libraries | Agents save and refine reusable tools |
| Agent-to-agent protocols | Standards for agents from different vendors to cooperate |
| Long-horizon training | Training on very long task sequences |
| Better evaluation | Harder, cleaner benchmarks |
- Visual Explanation: The Self-Improvement Cycle and Two Honest Numbers
The figure follows the cycle of sampling, scoring, filtering and learning with the numbers from the code, then shows what swarms and noisy tests did in the experiments.
Click to see why swarms work, and when they do notCompare ▼
Part 1 shows the cycle step by step. Part 2 compares a single agent with a swarm and shows how test size changes a decision. Open the bar for why swarms work and when they fail.
- Python Implementation: Reflection, Trajectory Sampling, Swarms and Noisy Tests
This script demonstrates five research ideas in miniature: reflection memory, scoring and selecting trajectories, building a fine-tuning set by rejection sampling, a swarm of voting agents, and a self-improvement loop that compares prompt versions on noisy trials. Everything uses a fixed random seed.
import randomrng = random.Random(7) # 1. Learning from failure: reflect, store a lesson, use it next timelessons = []def attempt(lessons): if any("YYYY-MM-DD" in lesson for lesson in lessons): return True, "booked" return False, "error: invalid date format" ok, info = attempt(lessons)print("Attempt 1:", info)lessons.append("Dates must be written as YYYY-MM-DD") # the reflection step writes a lessonok, info = attempt(lessons)print("Attempt 2 with the lesson:", info) # 2. Trajectory optimization: sample many tries, score them, keep the besttrajectories = [{"id": i, "success": rng.random() < 0.4, "steps": rng.randint(4, 14)} for i in range(20)]reward = lambda t: (10 if t["success"] else 0) - 0.3 * t["steps"]best = max(trajectories, key=reward)successes = [t for t in trajectories if t["success"]]print("Successful trajectories:", len(successes), "of", len(trajectories), "| best trajectory:", best["id"], "with reward", round(reward(best), 1)) # 3. Training data for agent fine-tuning: keep only successful, efficient trajectories (rejection sampling)median_steps = sorted(t["steps"] for t in successes)[len(successes) // 2]training_set = [t["id"] for t in successes if t["steps"] <= median_steps]print("Fine-tuning examples kept:", len(training_set), "of", len(trajectories), "| ids:", training_set) # 4. Agent swarm: many noisy agents vote (4 answer options, one correct)def one_agent(p_correct=0.5): return 0 if rng.random() < p_correct else rng.randint(1, 3) def swarm_vote(n_agents): votes = [one_agent() for _ in range(n_agents)] return max(set(votes), key=votes.count) == 0 questions = 2000single = sum(one_agent() == 0 for _ in range(questions)) / questionsswarm = sum(swarm_vote(15) for _ in range(questions)) / questionsprint("One agent accuracy:", round(single, 3), "| swarm of 15 voting:", round(swarm, 3)) # 5. Self-improvement loop: test prompt versions on noisy trials, keep the best (noise can mislead)true_skill = {"v1": 0.50, "v2": 0.58, "v3": 0.66, "v4": 0.61}for trials in (20, 400): scores = {v: sum(rng.random() < p for _ in range(trials)) / trials for v, p in true_skill.items()} print("With", trials, "trials per version, estimated:", {v: round(s, 2) for v, s in scores.items()}, "-> picks", max(scores, key=scores.get))Each experiment is tiny, but the pattern is the same as in real research. Note the final two lines: even with 400 trials the noisy test could not separate two close versions, which is why careful measurement is the foundation of every agent that learns.
Knowledge Check
Key Points
The Big Picture
Sample → score → keep the best → learn from it → test honestly
Research ideas change quickly, but the habit of measuring carefully stays valuable.
Up Next: You have finished the AI Agents series. A good next step is to build one real agent project end to end: pick a narrow task, add two or three tools, add guardrails and evaluation, and improve it from its traces.