ena pragma
·9 min read·agent-memory · multi-agent · ai-verification

Long-running AI agents turn rumors into facts. The team behind a Supercell lab project proposes a lab-bench fix, and is careful not to say it worked.

A talk on Project Paradox shows how AI agents lose track of where facts came from, and proposes a controlled test loop. What it shows, what it doesn't, and what to do now.

Contents

AI agents that run for days, not minutes, start to go wrong in a quiet way: they keep the gist of what they were told and lose where it came from. A hedged rumor comes back later as a fact. A talk from the 2026 AI Engineer World's Fair, posted to YouTube on September 26, describes this happening inside a simulated village of AI characters, and proposes a way to engineer against it. The proposal is sensible. The speaker is also unusually clear about what it has not yet shown, and that part is worth reading closely.

#What the team built

The talk, "Autoresearch in a Multi-Agent AI Village," is by Erina Karati and Arunachalam Manikandan. The conference schedule lists Karati as Supercell and formerly Microsoft, and she introduces herself in the talk as a former engineer at both; Manikandan is listed at the University of Minnesota. Their system, Project Paradox, was, in the words of its public code repository, "Developed at Supercell's AI Innovation Lab." It is a framework for AI characters in games and simulations, shown in the talk as a village of characters that move around, react with simple emotions, keep their own memories, hold trust scores for each other, and talk.

In short-term play, in her words, it "worked pretty well." Over longer runs, Karati describes the problems that appeared. An agent "may remember the rough topic but lose the source of the topic." "A rumor may become certain instead of just a rumor." And an agent can know something and still fail to use it when it makes a plan. She illustrates this with one example, a rumor about a sale on mangoes. It is a description of what they saw, not a measured failure rate.

#The proposed fix: treat the village as a lab bench

Karati borrows the idea of "autoresearch" from a project by Andrej Karpathy, in which an AI agent repeatedly edits one training file, trains for five minutes, checks whether a single number improved, and keeps or discards the change. Its README sums up the design as "One GPU, one file, one metric."

Her version keeps the shape and changes the measuring. In her words, "project paradox is more like a lab bench and auto research becomes the experimental loop around it." Her own recipe has five steps; grouped our way, it comes to three parts:

  • Controlled scenarios instead of random interaction. Specific tests with a known right answer: does a public fact spread, does a rumor stay a rumor, does an agent replan when its route is blocked.
  • A scorecard, not one number. "Instead you want a balanced scorecard." She lists reach, source retention, uncertainty preservation, false-certainty rate, action consistency, time to replan, and privacy containment. Her reason: "optimizing only one metric can create bad behavior." Reward spreading facts alone, and "the agents may learn to overshare everything."
  • A small, fixed surface to change. The loop may edit only certain rules, such as how memories are written and retrieved, how agents talk, how trust and source attribution work, and when to replan. Everything else stays put: "freeze the harness, the scenarios, and the metrics."

Her headline lessons: "memory is not enough," and "roll back also is not optional." A change that improves one behavior can damage another, so the loop "should basically be like a ratchet": keep a change only if the scorecard improves and the guardrails hold.

#What the talk does not claim

This is the part a summary tends to drop. Karati says the team ran at least one of these loops, and reports one outcome: after a loop, an agent asked about the mango sale was "able to respond within context this time," where it had not before. Then she stops short: "I kind of want to be careful about our claims here," and, without repeated loop results, "I wouldn't say the system just generally improved." What she argues instead is that this is "the right kind of surface to expose" to an automated improvement loop.

That is an honest place to land, and it means this is a proposal with one anecdote behind it, not a demonstrated fix.

There is also a gap the talk, as spoken, does not address. The same frozen set of scenarios both suggests changes and judges them. We wrote recently about why that setup can reward changes that only help on the tests being tuned against, and about two brakes researchers use: limiting how much each change can add, and adopting a change only after it holds up on cases that did not suggest it. A third example: in a 2026 overview of self-improving agent harnesses, Lilian Weng describes a method in which "Candidates are accepted only if they have no regression on both held-in and held-out data." Karati does not mention a held-out set in the talk. We did not review its slides, and that does not prove the team lacked one.

#What recent research adds

A preprint posted in June, "Manufactured Confidence", tests the hardening problem directly, in a different setting: agents deciding whether to grant access requests. It is one author's work in constructed settings, and we found no independent replication, so treat it as a lead, not a settled result. Its findings are pointed:

  • When memory tools rewrite conversation into stored "facts," "a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact."
  • What the agents responded to was not where a claim came from: "It is the confidence of the phrasing." Even "reportedly" was "obeyed like a flat assertion on most models."
  • Simple fixes failed. "A passive 'unverified' tag is ignored," in the abstract's words (the detailed results show it unreliable, varying by model), and an active warning not to trust a memory "escalates even correct memory": the agent refuses to decide even when the memory is right.
  • What helped: "keep the tentative phrasing rather than upgrade it," which the author calls "hygiene, not a defense against an attacker who can simply write a confident lie." And "a single load-bearing memory is the hazard, and one redundant source restores correct decisions."

If that holds up, it matters for Karati's list of edits. Keeping the hedge in the words, the might in agent C might leave the village, may do more than adding a label such as secondhand next to them.

Two other September preprints are worth knowing, each in its own narrow domain:

  • A judge that reads the log can be fooled by it. Karati's loop has an AI layer that reads the full record of a run and scores it. In a study of video-generation agents, a log reporting success made three open-weight AI judges accept 78% to 90% of failed clips, up from 7% to 19%, while "Frontier closed judges are essentially unmoved." Karati does not say in the talk how its scoring works, so this is a risk to test, not a flaw found.
  • Revision loops can add hedging. In a paired experiment on AI agent teams writing business reports, a manager that could send work back for revision made the reports "hedge 53% more." A loop rewarded for preserving uncertainty could learn to hedge everything.

#The throughline

What. Agents that keep state over long periods lose track of where information came from, and uncertain information hardens as it is stored and retold. Karati's proposal is to stop tuning by feel and run controlled experiments against a scorecard, on a small set of editable rules, with rollback.

Who. A two-person talk about a project developed at Supercell's AI lab, presented at a practitioner conference. The research alongside it is mostly single preprints from 2026.

How. The engineering shape is sound and matches wider practice: a frozen test, a narrow thing to change, keep or revert. What is not shown yet is that it works in general, or that it holds up on scenarios the loop never tuned against.

When. Now, for anyone whose agents hand information to each other or summarize it into notes. As Karati puts it, "They maintain state over time," and that state shapes what they do next.

Why. Because the failure is quiet. Nothing crashes when a rumor becomes a fact. The only way to see it is to test for it on purpose.

#What this means for a business running AI agents

This is our advice, built from the sources above, not a tested method.

  • Test the handoff, not just the recall. Give one agent a hedged fact. After two handoffs or one summary, is it still hedged, and can you still tell where it came from?
  • Keep the hedge in the words. Any step that summarizes or stores information should keep tentative wording tentative. It is cheap to check on your own setup.
  • Don't rely on a label alone. In the study we cite, a passive "unverified" tag was ignored or unreliable, depending on the model.
  • Don't let one note carry a decision. If an action depends on a single stored claim, get a second, independent source first.
  • Keep some tests the improvement loop never sees, and only accept a change that does not make those worse.
  • If an AI judge reads an agent's own log, test it with a log that falsely claims success. Where you can, score from what actually happened, not from what the agent said happened.

#What Ena Pragma takes from this

Our research agent grades the claims it collects as KNOWN, ASSUMED or UNVERIFIED. That is a label, and this research suggests a label alone may carry less weight than we would like. So the label is not the last word in our process. Before a claim goes into one of these notes, the person or agent writing it re-reads each quote, figure and attribution in its original source, and a separate critic agent checks the draft against those sources before it is published. We have written before about where handoffs between AI agents lose information; this talk is the same problem, stretched over weeks instead of one exchange.

How this was researched: our AI research agent searched for sources on the talk and the surrounding research and wrote a graded evidence sheet. We then re-read the sources quoted here directly on September 26, 2026: the talk's captions, the conference schedule, the Project Paradox and autoresearch repositories, Lilian Weng's post, the arXiv abstracts of the three preprints, and the relevant sections of the full text of "Manufactured Confidence." We did not read the preprints end to end or review the talk's slides, and we did not run any of the code.

#Sources

Keep reading