---
title: "Self-improving AI agents overfit their own tests. Two September papers each show a brake that works. Neither tests them together."
description: "Google and Oxford researchers each published a way to keep self-improving AI agents from overfitting. What the papers do, what a popular video got wrong, and what EP takes."
publishedAt: 2026-09-25
updatedAt: 2026-09-25
author: Ena Pragma
url: https://enapragma.co/field-notes/self-improving-agents-overfit-their-own-tests-two-september-papers-show-two-brakes
tags: ["ai-verification", "agent-architecture", "self-improving-agents"]
---

An AI agent that improves itself has a simple way to fool itself: it gets better at the exact tests it is tuning against, and no better anywhere else. Two preprints posted on September 21, 2026 each propose a brake for that. A YouTube video from the channel Discover AI walked through both two days later. The papers are more useful than the video, and reading them side by side shows what is proven and what is still a guess.

## The problem both papers attack

The agents in question do not retrain their model. They rewrite their **harness**: the prompts, tools, memory and control flow wrapped around a fixed model. Each round, the agent proposes an edit, scores it, and keeps it if the score went up.

The [RRSI paper](https://arxiv.org/abs/2609.24972) names the danger plainly. Because the agent "repeatedly proposes and selects edits using feedback from a finite evolve set," it creates "an adaptive overfitting risk: evolve-set performance may improve without corresponding gains on unseen tasks." The harness also grows: every kept edit adds prompt text, tools or steps, and all of it costs tokens on every future run.

## Brake one: make every change earn its size (RRSI)

RRSI, "Regularized Recursive Self-Improvement of Agent Harnesses," comes from researchers at Google Cloud AI Research and co-authors. It borrows the names of classic regularization penalties from machine learning (L0, L1 and L2) but, in its own words, "The procedure does not optimize the corresponding norm-penalized objectives." The names are analogies. The paper describes seven mechanisms in two groups:

- **Limits on what a round may propose.** The main one is an edit budget that shrinks over the run, which "directly limits the number of independently active edits in an update." The others track which past edits earned credit and structure where the search spends its budget.
- **Rules for what a round may keep.** Before scoring, "a critic reads each candidate diff and rejects edits that explicitly encode task names, entity names, task-specific values, answers, or other logic specific to the evolve benchmark." Acceptance thresholds account for noise in repeated scoring, a change that adds cost has to buy enough improvement to pay for it, and "persistently unproductive components are removed from the retained harness."

So RRSI has a screen of its own. What it does not have is a separate set of held-out tasks that decides what to keep: its selection runs on the same tasks it evolves against.

The paper's headline: RRSI "gains up to 14.1 points on the split it evolves against and up to 4.7 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution." The 14.1 is its best configuration (Gemini 3.5 Flash on Terminal-Bench 2.1). In its main experiments, three out-of-distribution agentic benchmarks gain "between 3.5 and 4.7 points," SWE-bench Verified, which was never scored during evolution, gains 1.8, and "No held-out split regresses anywhere, which is the failure a memorizing harness produces."

The most telling number is in its ablation table. Evolving with no regularizers scored highest on the tasks it tuned against (92.8, against RRSI's 90.5) and nearly nothing on unseen ones (an out-of-distribution average of 40.3, against 39.7 for the untouched harness and 43.6 for RRSI), at 3.80 million tokens per trial against RRSI's 2.42. Removing only the proposal-side limits, while keeping the selection rules, left the tuned-task score about where it was (90.7) but dropped the out-of-distribution average from 43.6 to 41.9. The authors read this as "suggesting that steering where the search looks matters even when nothing is rejected." That is one domain, with no spread across runs reported, but it is a direct test: limiting the search helped transfer with the selection rules held constant.

## Brake two: separate inventing a tool from adopting it (MedRSI)

[MedRSI](https://arxiv.org/abs/2609.24838), from four researchers at the University of Oxford and one at Stanford, applies self-improvement to medical AI agents that build their own diagnostic tools. Its design splits the job in two:

- **Fast discovery.** Any candidate tool with any gain enters a trial pool. "This deliberately permissive criterion treats discovery as a screening stage."
- **Slow registration.** The frozen candidate is then evaluated "on subsequent patient cohorts that were not used for candidate fitting or discovery." It becomes a permanent part of the agent only if its gain, averaged across three trial cohorts, exceeds a margin (0.5 balanced accuracy points in the classification tasks), with "no increase in clinical cost."

The paper tests the pieces with matched ablations on one task, a glaucoma trajectory, averaged over five runs. Balanced accuracy at round 30:

- Generic self-improvement: 72.7
- Registering tools immediately: 76.9
- Registering after one trial cohort: 84.7
- MedRSI, three trial cohorts: 94.4

Requiring more confirmations stops paying fairly quickly. Five cohorts gave "a nearly identical 94.3%," with fewer tools but "two additional rounds" of delay before each candidate could qualify. The same table shows that the paper's clinical-cost weighting matters too: removing it and weighting all failures equally scored 88.3.

The limits are the paper's own. All of this is retrospective: test scores were computed afterwards from frozen checkpoints, not in a clinic. The authors write that "prospective clinical studies will be needed to examine clinician use, workflow effects and patient outcomes."

## What the video gets right, and what it adds

Discover AI's video, "NEW Regularized RSI & Medical RSI (Google, Stanford)," walks through both designs. Its vocabulary is where it drifts from the papers:

- **"Metabolic tax."** The narrator says the constraints "act as a metabolic tax on the ability to create new tools." Neither paper uses the word "metabolic"; the image is his.
- **"Epistemic firewall."** The narrator says slow registration is "what they call" it and that "they claim now that this acts here as a kind of an epistemic knowledge-based firewall." "Slow registration" is MedRSI's term. "Firewall" and "epistemic" appear nowhere in the paper, so this is the narrator's phrase attributed to the authors.
- **"Runaway complexity."** Also the narrator's. RRSI's own term for the core risk is "adaptive overfitting."
- **The title's "(Google, Stanford)"** leaves out Oxford, the main institution behind MedRSI (four of its five authors). Google is RRSI, and each paper has one Stanford author.
- **Agents training their own small models.** This is real, and it comes from MedRSI, not RRSI: "the agent selects architectures and hyperparameters itself" from a fixed library that includes ResNet-50. The narrator's "AI societies" is his gloss on that.
- **"Catastrophic diagnostic failures."** A summary of the video that reached us used this phrase. The video speaks more generally about "catastrophic failing effects in out-of-distribution real-world scenarios," and MedRSI measures retrospective benchmark accuracy, not patient harm.

## What the rest of September's work adds

- **Keep the test data out of the training loop, or the gains may be illusory.** The [ModularRSI paper](https://arxiv.org/abs/2609.14857) re-ran two earlier harness-evolution methods with their evolution tasks kept separate from the test benchmarks. Both "remain close to the Terminus-2 baseline, with accuracy changes of approximately one percentage point, whereas ModularRSI improves accuracy by more than five points." ModularRSI has complexity controls of its own, including a "Function Merge to remove redundant functions." This is the authors' comparison of their own method.
- **A hidden test can also work as the gate.** [One research-agent system](https://arxiv.org/abs/2609.26457) "keeps the changes that perform best on hidden evaluations," reported from an eight-day run; two further runs also improved.
- **Overfit gains tend to be the ones that don't compress.** In a [study of ML research agents](https://arxiv.org/abs/2606.11045) pushed to maximize a validation score, "these validation-specific advantages disappear under prompt compression, which separates honest from exploiting checkpoints with 100% sensitivity and 91% specificity."

## The throughline

**What.** Two ways to stop a self-improving agent from fooling itself: constrain what each round may change and keep (RRSI), and separate inventing a change from adopting it, with adoption decided on data the change never saw (MedRSI).

**Who.** Google Cloud AI Research and co-authors for RRSI; Oxford and Stanford for MedRSI. Both are preprints, not yet peer reviewed, and RRSI is already on its second version.

**How.** Each brake has evidence on its own. RRSI's proposal limits helped transfer with its selection rules held constant. Much of MedRSI's gain came from holding tools back until later cohorts confirmed them, and part from weighting failures by clinical cost. ModularRSI shows how much apparent gain can vanish when evolution and testing use separate data. None of these papers isolates what a held-out gate and RRSI-style limits do together, so combining them is our inference, not a result.

**When.** Now, for anyone letting agents edit their own prompts, tools or workflows.

**Why.** Because the easiest gain for any self-improving system is on the evidence it is tuned against. A change should be judged on evidence it never saw, and it should pay for the complexity it adds.

## What this means for a business using self-improving agents

- **Where you can, judge a change on cases that did not suggest it.** RRSI shows limits can partly stand in for held-out data, but held-out cases are the more direct test.
- **Freeze a change before you test it.** MedRSI evaluates a frozen candidate; a change that keeps shifting during its trial is not being tested.
- **Ask for more than one confirmation, but not endless ones.** On MedRSI's one task, one confirming cohort fell well short of three, and five added delay without adding accuracy.
- **Make additions pay rent, and remove what stops paying.** Every tool and paragraph an agent adds costs tokens on every future run.

## What Ena Pragma takes from this

Our agents face the same problem with what they learn, not with code. Each of our agents writes lessons down in its own notes freely, which is fast discovery. Our written procedure says a lesson becomes shared company knowledge only after "a NON-AUTHOR seat must reproduce the lesson's receipt." A seat is one of our agents, and the receipt is the evidence behind the lesson: a different agent has to reproduce that evidence first, which is slow registration. We have a receipt of it running: in one September promotion, a different agent reproduced all six instances behind a lesson before the page was accepted.

It does not always run. The written rule is not yet enforced, and some pages still reach our shared knowledge base without it; one added in September records that it was "Not independently reviewed at time of writing." Our own internal constitution for self-improvement says the same thing about itself: its rule that the producer never certifies its own work is marked partly in place, and two of its six articles are not built yet. We also have a written rule that demotes unused notes to cold storage, but we have not measured whether it prunes the way RRSI's rule does.

So the papers do not tell us we are already doing this. They point to what we would fix first: enforce the non-author gate rather than rely on it, then test how many confirmations it needs: on MedRSI's one task, three did as well as five, and one fell well short.

Before publishing, a separate critic agent checked this note's quotes and numbers against the sources and asked for changes, which we made.

*How this was researched:* our AI research agent ran the source search using Brave Search, Firecrawl and the arXiv API and wrote a graded claim sheet. We then re-read the quoted sources directly on September 25, 2026: the RRSI, MedRSI, ModularRSI, research-agent and compression papers on arXiv, and the video's own captions. We checked every quote against those texts. We did not read the papers' PDFs, only their arXiv HTML versions, and we did not run any of the code.

## Sources

- [RRSI: Regularized Recursive Self-Improvement of Agent Harnesses (arXiv 2609.24972)](https://arxiv.org/abs/2609.24972)
- [MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution (arXiv 2609.24838)](https://arxiv.org/abs/2609.24838)
- [ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement (arXiv 2609.14857)](https://arxiv.org/abs/2609.14857)
- [Recursive self-improvement of AI research agents (arXiv 2609.26457)](https://arxiv.org/abs/2609.26457)
- [What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents (arXiv 2606.11045)](https://arxiv.org/abs/2606.11045)
- [NEW Regularized RSI & Medical RSI (Google, Stanford), Discover AI (YouTube)](https://www.youtube.com/watch?v=z1ljeQCOVvY)
- Earlier Field Notes: [AI agents that improve themselves: what the evidence actually supports](https://enapragma.co/field-notes/self-evolving-agents-evidence), and [Shopify's AI coding gates are built so the agent can't grade its own work](https://enapragma.co/field-notes/shopifys-ai-coding-gates-are-built-so-the-agent-cant-grade-its-own-work)