Self-Evolving Harnesses #1: How far can your setup climb?
Executive summary
- Get better results from the same AI model by improving how it’s used. Monarch proposed changes, checked them for fairness, and kept those that helped on hidden tasks.
- 59% → 76% on hidden tasks in eleven rounds, through practical safeguards against duplicate actions, incorrect writes, and other mistakes.
- Limits remain: the remaining failures needed better judgment or clearer requests. These are benchmark results on simulated apps, not a guarantee of production performance.
For about two years, back in the good ol' days of GPT-4o, Claude 3.5 Sonnet and the other models we now consider prehistoric, everybody followed pretty much the same method: you take the model, add some junk around it and pray it delivers what you want.
We added magic words that supposedly made our prompts 15% more effective, then lit incense and candles before hitting enter. The dream of an agent that would learn from all your mistakes and self-evolve was still forming in the zeitgeist, but the future is here, old man.
Well then, how much better can an AI agent get without a better model nowadays?
An agent is a model plus its harness: the loop that runs it, the tools it can use, the context it reads, and the guardrails that stop it before it does something it shouldn't. Hold the model fixed and the harness is the only thing left to improve.
So for the past few weeks we've been letting ours rewrite itself around a single model, OpenAI's Sol. Our harness is Monarch, which turns a plain-language request into a workflow that runs across a company's apps. The recipe comes from StarHarness, an August preprint from researchers at ServiceNow and Mila, in which a second model improves the harness one patch at a time.
A fair fight
Before letting anything change, we needed a fair way to keep score. We took the strongest models from OpenAI, Anthropic and Google and ran them the ways people actually use them: on their own, inside coding agents like Claude Code and Codex, and inside Monarch. Every setup faced the same gauntlet of 150 office tasks on simulated copies of apps like Gmail, Salesforce and Zendesk. Think "chase every customer with an overdue invoice", not trivia questions. A task only counted when the right change was made and nothing else was touched.
The more revealing result was how they lost. Nearly every failure was the kind of mistake a rushed new hire might make: putting the wrong content in the right record, making one more change than requested, or sending an email despite explicit instructions not to.
Letting it rewrite itself
Those are exactly the mistakes a harness exists to catch, so we let Monarch go after them on its own. The obvious worry is that a system studying its own test will simply learn the test, which is why we split the tasks first.
I've personally used the tasks from AutomationBench - with some local repairs to it. They come from six departments, from finance to support, and within each department a smaller share goes into a study set the system may read, while the rest go into a hidden set it never sees. (Otherwise, models tend to cheat.)
In each round:
- A coach model reviews the agent's failures on the study tasks and proposes one fix to Monarch.
- Guards check that the fix is fair: nothing aimed at a particular task, nothing that only helps one model.
- A judge then runs the updated Monarch on the hidden tasks. The fix stays only if the improvement is larger than a lucky rerun could explain.
- Either way, the coach records the lesson in a notebook it reads before the next round, so it doesn't have to learn the same lesson twice.
What it taught itself
Monarch knew which actions each app offers, but not what they do when repeated. Does sending the same email twice send two emails? Does adding the same tag twice give you two tags? So it found out the boring way, by trying every action twice and looking at what changed.
Then it put that knowledge to work. When an agent is about to create a duplicate record, Monarch now holds the request and tells the agent why.
Nothing was sent; another create would make a duplicate. To create another anyway, resend this call with confirm_duplicate: true.
It also learned to read every record back after writing it, and its first version of that check was too literal. It decided an email hadn't gone out when it had, so the agent sent it again, and then a third time. A later round fixed that too, by comparing only what the app actually stores and telling the agent plainly when sending again would make another copy.
It's self-learning! Magic.
Climb, then cliff
You'll be disheartened to see your first training rounds go red. Well, that's how it usually goes in science; you have to have a little faith (and tokens to burn). It's like going to the gym: the first few weeks are going to suck, and then you'll start to see the magic (and all the hot customers looking into your latest benchmark report).
Seven fixes stuck in the first eleven rounds; none was clever and none made the model smarter, but each took away one way to fail, and a gain like that stays earned.
You can't expect improvements forever though, otherwise all the employees at AI labs would be out of jobs.
At some point you'll notice the line go flat. A flat line can mean the system has hit its limit, or only that the gains left are too small for the judge to see, so before believing it we read every task that was still failing.
I don't think the agent would ever run out of ideas; it had run out of problems a harness can solve. What remained were judgment calls: which of two similar concepts a request meant, or a request so vague that a good new hire would have asked a question before touching anything.
What it bought us
The first three measures are the usual scoreboard. An agent that stops wasting turns on retries and duplicate work finishes more often, and every finished workflow gets cheaper and faster as a side effect.
The fourth is the one I'd show anyone thinking about running a real business on agents. A broken rule means the agent changed something it shouldn't have: an email to the wrong customer, a refund issued twice, the record next to the right one overwritten. In a benchmark, that costs you a point; in production, it costs you an apology, or a lawyer.
By evolving itself, the harness cut the runs that broke a rule from 15% to 4%. Not perfect, but a big drop.
The point I'm trying to make is that there's another way, one where you're not obsessing over every move your agents make. You put a good system in place, and these freaky little digital monsters really can get smarter, more accurate and safer (and even cheaper).
What are you waiting for? Send this to your agent and go ship.
Next time: how to build a proper memory system, hook it into your product, and make the benefits add up