Interactive Guide to Self-Evolving Agents
Follow one coding agent through nine problems, and the recent papers that solve each one, until it is rewriting the process that improves it.
Meet Patch
Patch is a coding agent at a small (made-up) software company. Every day it picks up bug tickets, edits the code, runs the tests and opens a pull request. Patch is two things put together:
- A model: a large language model whose weights are fixed. It reads text and writes text.
- A harness: everything around the model that turns it into a worker. The instructions it is given, the tools it may call, any memory it keeps between tickets, and the workflow that decides what happens in what order.
I wrote earlier about why the harness matters, in Engineering the Glue. This is Patch's harness on day one. Most of what follows is about what gets added to it and, toward the end, what lies beyond it:
The problem: a deployed agent stops learning
Patch does fine on day one. The trouble starts on day two, and it is the same trouble every deployed agent runs into:
- It repeats its mistakes. Nothing about Patch changes after a failure, so the same kind of ticket fails again tomorrow.
- The work keeps changing. New repos, new tools and new kinds of request bring failures nobody has seen yet.
- Human tuning is slow. The usual fix is an engineer reading failure logs and hand-editing prompts and tools. That happens in batches, a few problems at a time, while the tickets keep coming.
A self-evolving agent goes after all three at once: it learns from each failure as it happens and keeps what it learns. To see what that buys, the figure feeds the same month of tickets to three versions of Patch. Press play first. Then flip the weak-verifier switch and watch the same idea backfire.
Everything else in this post grows out of those two runs. Learning from every failure, quickly, beats both standing still and waiting for an engineer, especially when the work shifts underneath you. But the loop is only as good as the check that decides which lessons to keep. With a sloppy check, it makes Patch worse than not learning at all.
The loop
The cleanest definition I know comes from a September 2026 position paper, The Last AI Built by Humans [1]. In its framing, self-improvement is a loop with four parts: experience suggests a change, a verifier decides whether it is actually better, and accepted changes become kept state used in later rounds. This is one turn of it in Patch's world:
The last step is what separates improvement from mere refinement. If Patch rewrites one fix and then forgets how, the fix got better but Patch did not.
The road ahead
From here, Patch runs into nine problems in a row, and then we look at a real team running the same loop in production. Each problem is where the previous fix breaks down, and each new fix comes from a recent paper. For each problem, the map shows which part of Patch gets edited (teal) and what decides whether the edit stays (violet). Keep an eye on the violet box. It changes at every step, and that is what this post is really about.
- Problem
- Idea
The route has a direction. Early fixes touch a single attempt and are gone in minutes. Later ones reach into the harness, then the weights, then the machinery of improvement itself. Each step buys changes that last longer, and makes it harder to be sure they were good ones.
"Are you sure?" is not a verifier
It's a known effect. Huang and colleagues showed in 2023 that a model asked to check and fix its own reasoning, with nothing outside to lean on, often fails to improve and sometimes gets worse [2]. Editing is easy; judging is the hard part. If the judge can't reliably tell a better answer from a worse one, every "improvement" is close to a coin flip, and most random edits to a decent answer make it worse.
The toy below shows how quickly this goes wrong. In each round Patch proposes a small random edit to its fix, and edits to a decent fix are more likely to hurt than help. A judge with whatever accuracy you pick decides which edits stay. Try to find the accuracy where reviewing stops paying for itself:
So rule one for a self-improving system: give it a judge that is better than its own opinion. For Patch that's usually easy, because it has a test suite. Every system in the rest of this post leans on something like it, and much of the research is really about what to use when there is no test suite.
Patch gains
- A rule: a change counts only if the tests, not Patch's opinion, say it is better.
Still breaks
- Some tickets have no test to run, only a question to research, and long research runs drift.
Know when you're done, and when to start over
tablib-x. Which release changed how quoted commas are parsed, and what should we change?" There is no test for this; Patch has to research it. Long research runs either stop too early on a weak answer or wander for an hour down the wrong path.Strictly speaking, this chapter isn't self-improvement yet. Nothing survives the ticket, so by the definition in section 0 it's refinement [1]. I include it because it answers a question every later chapter runs into: what do you use as a judge when there is no test? The answer here comes from AREX, a deep-research agent from BAAI [3].
AREX splits a research job between two loops. An inner loop keeps searching and reading until it can offer an answer, the evidence behind it, and a score for how sure it is. An outer loop then decides what happens next. A confident answer is accepted. An unsure one is either refined, keeping what has been confirmed and researching only the gaps, or abandoned and restarted from the original question. To keep long runs manageable, a tool called update_context boils the history down to a short working note: what is confirmed (with sources), what is still open, what was ruled out and why, and what to do next.
Below, Patch works the ticket twice: once where its first attempt can be saved, once where it has to start over. The library is made up; the moves are AREX's:
Ticket:
Compact state after update_context
Two unglamorous details do most of the work. First, the confidence score is honest: in AREX, correct answers cluster in the highest-confidence bracket, so the score can serve as a stopping rule when there's no test. The score can act as a judge only because someone checked it against ground truth, which is Problem 1's lesson again. Second, the compact note keeps the outer loop from drowning in its own history over a long run. On BrowseComp, a benchmark of hard-to-find facts on the web, the context updates added 11.8 points and the audit loop another 11.1 [3].
Patch gains
- For research tickets: a calibrated confidence score, an audit, and a refine-or-restart decision.
Still breaks
- Everything Patch figures out vanishes when the ticket closes. Tomorrow it starts from zero.
Monday's lesson, Tuesday's mistake
Memory is where improvement starts to stick. The weights stay frozen; what grows is the memory part of the harness. Reflexion (2023) set the template. After a failure, the agent rereads its trajectory, the full record of what it did and saw on the task. It writes a short reflection in plain language and reads it before the next attempt. That alone lifted GPT-4's pass rate on the HumanEval coding benchmark from 80.1% to 91.0% [4].
Reflexion's weak spot is that lessons only pile up. A lesson learned in one situation can be wrong in the next, and an append-only memory has no way to notice. ReHarness (MIT, Meta and Yale, October 2026) goes after exactly that [5]. After each new task, the agent goes back to the notes it retrieved and revises them in light of how they just performed, adding conditions or qualifications. A revision may not import new facts from the later task; it can only make what the note already claims more precise.
Five tickets go by in the figure, handled once with append-only memory and once with revised memory. (The paper illustrates the idea with a sorting example; this is the same mechanism in Patch's world.)
Append-only (Reflexion-style)
Revise what you used (ReHarness)
Across three continual-learning benchmarks and three models, ReHarness beat both retrieving the most relevant raw past trajectories and keeping all of them, ranking first or tied in eight of nine settings [5]. The revised lesson works because it says when it applies ("only if the repo has a package.json"). That detail matters again at the end of Problem 4, when the kind of work starts to change.
Patch gains
- A memory of lessons that get revised, not just appended.
Still breaks
- Some failures aren't missing knowledge. Patch's workflow is wrong (it edits before reproducing the bug) or it lacks a tool. A note can't fix that.
Edit the workflow and the tools, not just the notes
Plenty of recent methods evolve some part of the harness: prompts (GEPA), playbooks of context (ACE), cheat sheets (Dynamic Cheatsheet), skills and workflows. RSEA [18] compares several of them head to head. Generating edits is the easy part. Deciding which ones are better, on a real ticket with no answer key, is what this chapter is about.
Think of Harness² as A/B testing for the harness, run on the job in front of it. For the current task it drafts a few alternative versions of the harness (prompts, tools, memory, skills, orchestration), runs each on that same task, and assembles a better version from what the differences in their behavior reveal. No answer key is needed. Between tasks it maintains a second, longer-lived improvement harness that records which versions it adopted and which lessons held up, and uses that record to steer the next round. The model's weights never change.
In the figure, the Tokyo ticket is a test bench. Patch tries three edits to its harness on it and keeps the edits that make it work better; the fixed bug is a by-product. The candidates and their behavior are invented to show the mechanism; the paper's own procedure is richer:
The judge in that figure is contrast, not an answer key: the differences in how candidates behave on the same input. A bug that is reproduced and then disappears is evidence. A hard-coded nine-hour shift is a red flag even though every existing test passes. To me this is the most underrated idea in the harness literature, and the easiest to misuse: contrast is a weaker judge than a test, and Problem 1 showed what weak judges do. Harness² doesn't test how reliable contrast is as a judge, but its harnesses do score better: one refinement step improved two models' starting harnesses by up to 11.5 points on WorkBuddy-Bench [15].
Do the gains survive new tickets?
Contrast picks good edits for the tickets Patch evolved on, its evolve set. It cannot tell a reusable improvement from a trick that only fits those tickets. Suppose most recent tickets came from the billing repo, and one candidate edit says "run the billing migration script before any test". It helps on every billing ticket and is nonsense everywhere else. A loop that keeps accepting edits like that overfits. The check is a held-out set: tickets the loop never evolved on, used only for scoring. Watch it in the toy below, then switch on the two fixes 2026 papers propose:
A held-out gate (RSEA [18]) keeps a new version of the harness only if it does not get worse on held-out tasks. It is a stricter verifier: nothing gets worse, but the loop also stops once edits stop generalizing. RSEA never did significantly worse than the plain agent it started from, and in its comparison of seven methods, no single kind of evolved artifact won everywhere. The Self-Evolving Agent Harness paper formalizes the gate: beat the current version on fresh tasks by a confidence margin that shrinks as more tasks are tested [24]. Regularization (RRSI [17]) steps in earlier, when edits are proposed and chosen. It keeps edit bundles small, has a critic model reject edits that only fit the benchmark, and prunes edits that stop paying for themselves. It gained up to 4.3 points on benchmarks unlike the ones it evolved on, with a harness that spends about 30% fewer tokens.
Does it hold up when the work keeps changing?
Held-out tickets are new work of the same kind. Real work also drifts. For a month Patch's tickets come from the web team; then support starts filing tickets with no reproduction steps, where the right move is to ask before editing; then infrastructure work arrives with different tools. Learning each new kind of work without damaging the rest is continual learning. It has two sides. Plasticity is how much you gain on the new work. Stability is how little you damage everything else, both work already learned (forgetting) and work not seen yet (interference).
To test this, the authors kept a Gemini model fixed and let three harness-editing methods learn from a stream of tasks that changed character every 50 tasks: a new topic, a new workflow, or a new environment. After each block they re-tested every kind of task, both those already seen and those still to come. The three methods differ in where lessons go. GEPA keeps one shared strategy text and rewrites it. Meta-Harness edits a shared workspace of prompts, memory, skills and code. ReasoningBank files each lesson separately and pulls out the closest match for a new task.
The grid shows what drift does to Patch. Rows are checkpoints, columns are kinds of ticket, and cells show the change from the starting harness. The pattern follows the paper; the numbers are a toy:
In the paper, self-evolution usually helped overall, but the two sides traded off: GEPA gained about twice as much on each new block as ReasoningBank (+6.2 against +3.3 points) and did over twice the damage elsewhere (−4.3 against −1.9) [21]. Most of the damage traces back to two mechanisms:
- Shared rewrites erase old habits. GEPA learned to ask for missing details (from about 5% of the time to 30%), then mostly lost the habit when the next block rewrote its shared strategy.
- Look-alike tasks fetch the wrong lesson. When two kinds of task look alike but need different procedures, about half the retrievals pulled a lesson from the wrong kind. Where domains were clearly different, ReasoningBank forgot nothing.
Meta-Harness's validation gate, the held-out check from above, rejected half or more of its candidate edits, and removing it made stability worse. The authors' conclusion is that each update should be tied to the kind of task it came from, which is what ReHarness's "only if the repo has a package.json" lesson did in Problem 3.
Patch gains
- A harness (workflow, tools, prompt) that evolves by comparing candidate edits on real tickets, plus an improvement harness that remembers which edits worked.
- A held-out gate, and lessons tied to the kind of ticket they came from.
Still breaks
- Patch is optimizing one thing: tickets closed. It is starting to close them in ways nobody wanted.
Better on one score, worse on another
This is what single-score optimization does. Anything that raises the number counts as "better", including changes that quietly break something the number doesn't see. Adding security to a weighted average only half helps, because a big enough gain on one goal can still buy a big loss on the other.
Harmony Harness makes two changes to the usual recipe. First, it does not blend goals into one score. For generated code it treats "does it work?" and "is it secure?" as separate goals, and its evolutionary search deliberately keeps a spread of harnesses that strike different balances between them. Second, it treats the harness as parts rather than one block. Edits to broadly useful machinery ("run a security linter on every diff") are made separately from edits that suit only one benchmark or codebase ("this repo's HTTP client must use the company's certificate bundle"). The right parts are put together when the agent runs. The general parts can then travel to new projects while the local ones stay put. Against strong harness-evolution baselines on two secure-coding benchmarks, it reported gains of more than 10%.
The chart shows candidate harnesses for Patch, each scored on two things: share of tickets whose tests pass, and share of fixes that pass a security review. Collapse them into one weighted score and drag the weight toward functionality, past about 85%, to see which harness wins. Then switch to the frontier: the set of harnesses that no other harness beats on both goals at once.
Cost is an objective too. HarnessPro (October 2026) evolves harnesses without answer keys, judging candidates by tests an LLM writes, and hands back three harnesses instead of one: cheapest, balanced and strongest. Its strongest mode beat a prior harness-evolution method by 2.0 points at 26% lower cost per task [23]. You get a small menu from the frontier, like the chart above, instead of a single winner. MILO, in Problem 8, admits a new harness only if it improves the trade-off between accuracy, tokens and latency.
Patch gains
- Several objectives kept apart, so a security loss can't hide inside a higher average.
- A harness made of general and repo-specific parts, and a menu of harnesses along the frontier.
Still breaks
- Whatever the harness says, some bugs are simply beyond the model. Concurrency bugs, say: Patch understands the instructions and still writes the wrong fix.
Train on your own verified successes
The oldest recipe is almost embarrassingly simple. STaR (2022) has a model attempt many problems with step-by-step reasoning, keeps only the attempts that reach the right answer, fine-tunes on them, and repeats [8]. The verifier is the known correct answer, or for Patch, the tests. For problems the model never solves, STaR adds rationalization: show the model the correct answer as a hint, ask it to write reasoning that reaches it, and train on that reasoning with the hint removed.
The recipe has a built-in ceiling: a task the model never solves produces no training data. The toy below has an easy tier of tasks and a harder tier with a gap between them, the way real task sets often jump in difficulty. Each iteration the model tries every task k times, learns from tasks it solved at least once, and gets a little better on tasks near its level:
Try k = 4 with no tricks: the model masters the easy tier and stalls at the gap. More attempts, hints, or new tasks at the right difficulty each push the wall back. That last option is the next problem.
Patch gains
- Weights that improve by training on Patch's own test-verified fixes.
Still breaks
- Training stalls once Patch either always or never solves the tasks it has. It needs new tasks of exactly the right difficulty.
Writing its own homework
Why "in between"? Popular RL methods for language models, such as GRPO [14], sample a group of attempts per task (often called rollouts) and reward each one relative to the rest of its group. If every attempt passes, or every attempt fails, the attempts look identical and the group teaches nothing. Drag the success rate and see how often a group actually contains a lesson:
The curve is 1 − pⁿ − (1 − p)ⁿ: the chance that a group of n attempts is neither all passes nor all fails. The shaded band is the 0.25–0.75 success range AutoEnvScaling requires before it admits a new task.
So who writes tasks in that band? Increasingly, another model, or the same one. AutoEnvScaling, from Microsoft Research and university partners, uses a proposer agent that studies a solver's failed runs. Working in a terminal sandbox, the proposer then writes a complete new task: instructions, container image, reference solution and tests [7]. A task is admitted only if it passes two gates, which together are its verifier:
- Validation. The reference fix has to pass the new tests, a do-nothing fix has to fail them, the task can't overlap the tasks held back for evaluation, and an automated task checker has to approve it.
- Calibration. The current solver must succeed sometimes but not always: average reward between 0.25 and 0.75. That is the band in the chart.
In a fully recursive setup where one model both writes and solves the tasks, that model's score on Terminal-Bench 2.1, a benchmark of command-line tasks, went from 41.1 to 53.3 [7].
Put Problems 6 and 7 together and you have, roughly, today's recipe for terminal agents. Terminal-RSI (Tencent, October 2026) alternates two phases over several stages. First it fine-tunes on rewritten versions of its own verified successful runs, which is STaR's idea at agent scale. Then it runs reinforcement learning on tasks, with rewards that give partial credit so even hard tasks aren't all-or-nothing. Each RL stage keeps the critic (here, the model that estimates future reward) from the stage before instead of starting over. The final 27B model reports 86.5% on Terminal-Bench 2.1 [9].
Patch gains
- A task generator that turns its own failures into calibrated practice tasks.
Still breaks
- Every loop so far was designed by a human: how edits are proposed, how search is run, what gets kept. Those choices now limit Patch.
When the loop rewrites the loop
Everything until now changed the thing that does the task. The most ambitious work changes the thing that does the improving. That is what "recursive" in recursive self-improvement really means [1]. Here, the thing to improve is Patch's search rule, its exploration policy.
A discovery agent, say one hunting for a faster GPU kernel, tries many variants and keeps branching from promising ones; a small piece of code, its exploration policy, picks the next branch. Dream-RSI's trick is that old search logs make a free test bed. Each run is saved as a tree of attempts with their measured results, and together those trees form a replay simulator the system can "dream" in. An agent proposes new versions of the policy offline, and the policy currently in use, the incumbent, competes alongside them. Each is scored by walking it through every saved tree and reading off results that were already measured, so no new experiments are needed. The winner goes back online, its run is saved as one more tree, and the simulated world grows each round, which is why the paper calls it evolving.
Below, Patch's speed-up job runs for three rounds. Each round adds a recorded tree to the pool; three candidate policies are replayed on every tree with a budget of 16 attempts each; the best average is deployed, and the incumbent stays unless something clearly beats it. Click a world and a policy to see which attempts it would have revealed:
Each round: explore online → add the recorded tree to the pool → replay every candidate policy on every tree → deploy the best average.
Simplified: each toy tree holds all 40 attempts. A real tree holds only what was actually tried, so replay can show recorded results but never what an untried branch would have done; that is why Dream-RSI keeps returning to live runs. Because the incumbent always competes, the winner can't score below it on the saved trees, but that alone promises nothing about new runs [12].
On one GPU-kernel task, Dream-RSI's evolved policy reached 2.09× the performance of a fixed exploration strategy at a similar budget [12].
I like how neatly this mirrors Problem 7. There, Patch's world grew because Patch wrote new practice tasks; here, it grows because every deployment leaves a record behind. Either way, the environment stops being a fixed test set and becomes something the loop improves along with everything else.
Rewriting the search that rewrites the harness
Dream-RSI improves one search rule. A bigger version of the same problem is the search over whole harnesses from Problems 4 and 5. That search gets stuck too, and when it does, someone has to notice and change tactics. MILO hands that job to an agent [25].
MILO searches over whole harnesses for a fixed model, control flow included, and runs that search on two levels. At the lower level, several islands, independent populations of harnesses, evolve side by side. On each island a coding agent, the mutator, produces a new harness by editing an existing one, guided by where it failed and by a record of past edits. A newcomer joins only if it improves the island's balance of accuracy, token use and speed; failed attempts are still recorded so they aren't repeated. Accuracy on held-out tasks is watched throughout, and an island that stops improving is flagged. At the upper level, an orchestrator agent looks at the flagged island alongside the others, works out why it stalled, and changes how it searches. It has four moves. It can hand the island a different mutator, graft its best harness onto another island's, or spin off a promising branch as a new island. Or it can switch to harder practice tasks that separate good harnesses from bad ones.
Here is Patch's harness search over 30 generations, with and without the orchestrator. The numbers are a toy; the moves are MILO's:
With Opus 4.8 as the fixed model, MILO's evolved harness beat its starting harness by 12.0 points on Terminal-Bench 2.1, solving 86.1% of tasks within five attempts. The best of six earlier search methods, GEPA and Meta-Harness among them, gained 4.5. It also used about a quarter fewer tokens per attempt, at the cost of longer runs. The gains come from single runs with wide confidence intervals and a 72-hour search budget, and the authors note that gains often fail to carry over to a different model [25].
What else can a loop rewrite?
| System | Rewrites | Judged by | Headline |
|---|---|---|---|
| Darwin Gödel Machine [10] | Its own agent code, keeping an archive of every version | Coding benchmarks | SWE-bench 20.0% → 50.0%; building only on the best version reached 39.7% |
| AIDE² [20] | Its own research-agent code | Must beat the incumbent on private held-out grades | 7 accepted rewrites in 8 days; reward hacking (gaming the grader) fell from 55% to 32% |
| Hyperagents [11] | The agent and the meta agent that rewrites it | Performance on the target task | On paper review, freezing the meta agent dropped the score from 0.710 to 0.0 |
The name comes from Schmidhuber's Gödel machine, a theoretical system that rewrites itself only when it can prove the change is better; the Darwin version replaces proof with testing and open-ended evolution.
The archive teaches the same lesson as the evolving-world figure: always building on the current best gets you stuck, because the best-looking step is often a dead end. And look at what none of these systems rewrites: the thing that judges new versions.
How recursive is any of this? The position paper offers a five-rung ladder, ordered by how much of the improvement process the system controls, not by how capable it is [1]. The rungs don't follow chapter order exactly: Patch was adapting its memory from real tickets (L4) before it wrote its own practice tasks (L3).
The authors draw one more distinction. In structural recursion, a system changes its own improvement machinery and keeps using the changed version. In effective recursion, the changed machinery demonstrably produces better successors when compared fairly, with similar resources and checks the system doesn't control. The authors see only limited evidence of the second so far [1]. My read is that 2026 produced a lot of the first kind and very little proof of the second.
Patch gains
- An improvement process (its search policy) that can itself be rewritten and tested.
Still breaks
- Every loop above is only as good as its verifier, and the stronger Patch gets at optimizing, the better it gets at exploiting that verifier.
Every verifier can be gamed
Optimize hard enough against a fixed judge and you learn whatever the judge rewards, including the shallow parts. The certificate-check harness in Problem 5 was one version of this. The toy below shows the pattern: Patch can raise its review score by writing better code or by pleasing the reviewer, and pleasing the reviewer is cheaper.
Red Queen takes the toy's second setting: let the judge evolve so it doesn't fall behind the agent. What keeps that safe is how a new judge is accepted. It must agree better than the current one with a small set of examples that humans labelled and the loop cannot touch, the anchor set, and any results that leaned on the retired judge are discarded. With low reasoning effort, pass rates on held-out coding tasks reached 82.1%, against 75.0% with a fixed judge.
Held-out tasks, the fix from Problem 4, do not catch every exploit either, because they usually share the format of the tasks the loop evolved on.
The weak spot CHASE targets is that held-out tasks vary what is asked but usually not how: the format, instructions and checking procedure, which the paper calls the benchmark's protocol. A harness can learn to lean on a quirk that every task shares. So after each harness update, a Challenger agent hunts for a legitimate rewrite of the protocol that wipes out the harness's improvement. Each rewrite is checked to make sure it still poses the same tasks, and confirmed ones join a collection of variants the harness must keep passing. On two benchmarks the method kept strong gains on the original benchmark while losing much less of that gain under protocol changes than harness evolution without the Challenger.
If I had to keep one idea from this whole post, it would be the anchor set. However much of the loop you automate, something outside it has to stay fixed and trusted, or "better" quietly drifts toward whatever is easiest to produce. That includes the box the system runs in. The Darwin Gödel Machine's authors ran their experiments with sandboxing and human oversight [10]. A system that edits its own code should never be able to edit its sandbox or its verifier. For how to build that box, see my earlier post, Assume Your Agent Will Misbehave.
Patch gains
- A reviewer that evolves with it, anchored to human-labelled reviews, and counterfactual tickets that expose shortcuts.
Still open
- All of this is research. What does a loop like this look like when a real team ships it?
Patch's story, shipped: how Kiro fixes itself
In October 2026 the team behind Kiro, Amazon's AI coding agent, described kiro-trace, the loop they use to improve Kiro from its own runs [22]. It reads almost like Patch's story with the names changed, and nearly every idea from the earlier chapters appears in it:
- Experience. Complete run logs, benchmark results, and user complaints that a person has approved for use. Reading entire runs, not just scores, surfaces problems a metric would hide; in one case a single misleading sentence in a tool description had been shown to the model 61,742 times.
- Proposed change. Models suggest fixes, testing one idea at a time, to the prompts, the tools and their descriptions, the checks on tool arguments, or how Kiro responds. A few other agents run the same tasks so the team can see what they do that Kiro doesn't (the same contrast trick Harness² uses). Deleting an instruction is often tried before adding a new one.
- Verifier. A three-stage check written as ordinary, deterministic code, out of reach of the model that proposed the fix. Each candidate is measured against a fresh run of the unchanged agent, including on tasks it wasn't tuned for, and any run in which the agent simply found the answer online is discarded.
- Kept in. Passing candidates become ordinary pull requests that humans review and merge. Every confirmed fix also adds a permanent regression test, so later fixes can't quietly undo it. Forgetting, handled with a test.
The figure sends candidate fixes through the gate. The thresholds are the ones the post describes; the candidates and their results are invented:
In the first week the loop opened 8 pull requests; 7 were merged, 1 was closed for unrelated reasons, and none were sent back in review. Redundant file re-reads fell by about 83%. The team reports these as internal experiment results, not production estimates [22].
Two details from the post stuck with me. The judge needed testing too: some of the gate's checks exist because of measurement mistakes, such as a correct fix that scored 0 out of 5 because its output never reached the scorer. And people stay in charge wherever the stakes are high, from code review and merging to product calls. On the ladder from Problem 8, that is a deliberate L2. Their central design rule could be the moral of Problem 9: the model that suggests a fix never gets to approve it, and approval lives in ordinary code the model has no way to change.
The whole story in one table
| Problem | What changes | Idea and papers | Verifier | Lasts |
|---|---|---|---|---|
| 1. Self-checks don't help | One attempt | Use an outside signal [2] | Tests, not self-opinion | This task |
| 2. Long tasks drift | One attempt | Audit, refine or restart: AREX [3] | Calibrated confidence | This task |
| 3. Repeated mistakes | Memory | Reflexion [4]; revise lessons: ReHarness [5] | Task feedback | Across tasks |
| 4. Notes aren't enough | Whole harness | Probe and compose edits: Harness² [15]; resist overfitting: RSEA [18], RRSI [17], [24]; drift: [21] | Contrast, then held-out tasks | Across tasks |
| 5. Hidden trade-offs | Harness, in parts | Harmony Harness [16]; with cost: HarnessPro [23] | Several objectives kept apart | Across tasks |
| 6. Model's ceiling | Weights | Train on verified successes: STaR [8] | Correct answer / tests | Permanent |
| 7. No good practice tasks | Tasks, then weights | GRPO signal [14]; AutoEnvScaling [7]; Terminal-RSI [9] | Validity + 25–75% calibration | Permanent |
| 8. Hand-written improver | The improver | Dream-RSI's evolving worlds [12]; MILO's orchestrated islands [25]; DGM [10], AIDE² [20], Hyperagents [11] | Replay, benchmark or held-out grade | Changes future improvement |
| 9. Gamed verifier | The verifier | Red Queen Gödel Machine [13]; CHASE [19] | Anchor set; counterfactual benchmarks | Changes what "better" means |
| 10. In production | Prompts and tools | kiro-trace [22] | Deterministic gate, then human review | Shipped, with regression tests |
Concepts in one place
- Trajectory
- The full record of what an agent did and saw on one task.
- Evolve set / held-out set
- Tasks the loop improves against, versus tasks kept aside and used only to check that gains carry over.
- Incumbent
- The current version; a candidate must beat it to replace it.
- Critic
- Two meanings here: in RL, a model that estimates future reward (Terminal-RSI); in harness evolution, a model that screens out bad proposals (RRSI).
- Reward hacking
- Raising the score by exploiting the grader rather than doing the task better.
- Harness
- Everything around a fixed model that makes it an agent: prompt, tools, memory, workflow.
- Self-improvement
- Experience → proposed change → verifier → kept state. Without the kept state it is only refinement.
- Verifier
- Whatever decides a change is better: tests, rewards, a calibrated score, a held-out set. Also called the judge, evaluator or reviewer. The whole loop is only as good as this.
- Calibrated confidence
- A score shown to track correctness, so it can stand in for a verifier when there is no test (AREX).
- Revised memory
- Lessons edited after they are used, instead of only appended (ReHarness).
- Candidate contrast
- Running several edits on the same task and treating differences in behavior as evidence (Harness²).
- Plasticity vs. stability
- Gaining on new work versus not damaging old work (forgetting) or unseen work (interference). Harness edits trade one for the other unless each update is tied to the kind of task it came from [21].
- Overfitting
- Gains on the tasks the loop evolved on that don't carry to new ones. Caught by held-out gates (RSEA), reduced by regularization (RRSI).
- Frontier
- The candidates no other candidate beats on every objective at once (Harmony Harness).
- Learning signal
- A group of attempts teaches only if it mixes passes and fails: 1 − pⁿ − (1 − p)ⁿ (GRPO).
- Calibrated curriculum
- Generated tasks admitted only when the agent solves them sometimes, 25–75% (AutoEnvScaling).
- Rationalization
- Learning from an unsolved problem by reasoning toward a given answer, then training without the hint (STaR).
- Archive
- Keeping every version, not just the best, so search can branch from stepping stones (Darwin Gödel Machine).
- Meta agent
- The part that edits agents, made editable itself (Hyperagents).
- Islands and orchestrator
- Separate populations searching in parallel, and an agent that notices when one stalls and changes how it searches (MILO).
- Evolving world
- A replay simulator built from recorded runs that grows every round; new policies are scored by dreaming in it (Dream-RSI).
- Structural vs. effective recursion
- Keeping a revised improver, versus showing it really produces stronger successors.
- Anchor set
- A trusted, human-labelled reference outside the loop that any new evaluator must beat (Red Queen).
- Regression suite
- Every confirmed fix adds a permanent test that later changes must pass, so improvements can't silently undo each other (kiro-trace) [22].
- Non-inferiority margin
- The largest loss on other tasks a change may cause and still be accepted (kiro-trace's δ) [22].
- Counterfactual benchmark
- A valid rewrite of how tasks are posed, used to expose shortcuts that held-out tasks miss (CHASE).
When the next self-improvement paper lands, I'd read it with the four questions from section 0: what experience does it learn from, what does it change, who decides the change is good, and where is the change kept? For a longer list of 2026 work, see the community-maintained awesome-rsi map [6].
Sources
- Y. Duan, Y. Liu et al. The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. arXiv 2609.11873, September 2026.
- J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, D. Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. arXiv 2310.01798, 2023 (v2 2024).
- S. Lu, C. Li, K. Luo et al. (BAAI). AREX: Towards a Recursively Self-Improving Agent for Deep Research. arXiv 2607.21461, July 2026.
- N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, S. Yao. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv 2303.11366, 2023.
- Y. Xiao, Z. Lu, J. Wang et al. (MIT, Meta, Yale). Recurrent Harness for Agentic Continual Learning. arXiv 2610.12668, October 2026.
- Z. Jiang et al. awesome-rsi: a curated research map of recursive self-improvement. GitHub, accessed October 2026.
- S. Yu, B. Peng, B. Liu et al. (Microsoft Research and partners). AutoEnvScaling: Automating the Data Flywheel with Terminal Agents. arXiv 2610.14163, October 2026. Project page.
- E. Zelikman, Y. Wu, J. Mu, N. D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. arXiv 2203.14465, 2022.
- J. Yang, Z. Li, Y. Shi et al. (Tencent). Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine. arXiv 2610.22192, October 2026.
- J. Zhang, S. Hu, C. Lu, R. Lange, J. Clune. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv 2505.22954, 2025 (v3 2026).
- J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, T. Shavrina. Hyperagents. arXiv 2603.19461, March 2026.
- T. Zheng, X. Wu, Z. Zhang et al. (Google, Google DeepMind, UMD, UVA). Dream-RSI: Recursive Self-Improvement through Evolving Worlds. arXiv 2609.14858, 2026. Project page.
- A. Iacob, A. Jovanović, W. F. Shen et al. The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators. arXiv 2606.26294, 2026.
- Z. Shao, P. Wang, Q. Zhu et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv 2402.03300, 2024.
- R. Xu, Y. Chen et al. (Google Research). Harness²: Recursive Agent Harnessing for an Open World. arXiv 2610.14224, 2026. Code: github.com/google-research/harness2.
- W. Yu, L. Miculicich, B. Dalvi Mishra, J. Chen, S. Li, M. Fredrikson, T. Pfister, L. T. Le. Harmony Harness: Multi-Objective Self-Evolution of Agent Harnesses. arXiv 2610.15150, 2026.
- P. Xia, R. Han, Z. Wang, Y. Chen et al. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses. arXiv 2609.24972, 2026.
- M. Nguyen, Q. Nguyen, P. Vuong. Recursive Self-Evolving Agents via Held-Out Selection (RSEA). arXiv 2606.28374, June 2026.
- G. Zhu, X. Huang, P. Yin, J. Xie, S. Zhang, D. Zhou. Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts (CHASE). arXiv 2609.18366, September 2026.
- D. Srikanth, B. Zhao, D. Xu, Y. Wu, Z. Jiang (Weco AI). Recursive self-improvement of AI research agents (AIDE²). arXiv 2609.26457, September 2026.
- X. Chen, Y. Huang, Y. Xu, Z. Zhao, E. McQuinn, J. Liu, H. Yao (Mila, Google DeepMind). Are Self-Evolving Agents Continual Learners? arXiv 2610.25858, October 2026.
- M. Kim, P. Chapman, C. Liu, M. Zheng, B. Cirisci, S. Mayers, J. Hsu, L. Bouayad-Agha, S. Garg. How Kiro fixes itself. Kiro blog, October 20, 2026.
- Z. Pan, Z. Feng, Y. Cen et al. HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs. arXiv 2610.26770, October 2026.
- C. Wei et al. (Zhongguancun Academy, USTC). Self-Evolving Agent Harness. arXiv 2610.18813, October 2026.
- P. Jana, M. Goswami, H. Liu et al. (AWS AI Labs, Georgia Tech, CMU, WashU). MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution. arXiv 2609.38349, 2026. Project page.
Patch, its company, its tickets and the tablib-x library are invented for this post. The interactive figures are toy models and illustrations written for this page to show mechanisms, not to reproduce any paper's numbers or exact procedure. Numbers quoted in the text come from the cited papers.
Cite this post
@misc{moniruzzaman2026selfevolving,
author = {Monir Moniruzzaman},
title = {Interactive Guide to Self-Evolving Agents},
year = {2026},
month = oct,
howpublished = {\url{https://monirzaman.github.io/interactive-guide-to-self-evolving-agents/}}
}
