---
title: "Your AI agent is a text predictor in a loop. That is the floor, not the ceiling"
description: "How a 2026 AI agent is built, what has grown out of that design, why it forgets, guesses and agrees too easily, and eight habits for working with those limits."
publishedAt: 2026-10-20
updatedAt: 2026-10-20
author: Ena Pragma
url: https://enapragma.co/field-notes/why-your-ai-agent-forgets-guesses-and-agrees-with-you
tags: ["agents", "agent-architecture", "ai-verification", "context-engineering"]
---

Everything has a baseline definition and function, and that is not its upper limit. For an AI agent the baseline is plain: a text-prediction model, trained to play one character, wrapped in software that gives it tools and runs it in a loop. Two accounts below describe abilities nobody wrote in as rules, and scores on agent tests have risen fast. The limits are real too: it forgets, it guesses, it agrees too easily, and each is traced to a cause below. Studies are from 2026 unless marked, but publication dates are not model dates: most Claude figures come from Opus 4.6 (February 2026) or earlier models; the Opus 5 results are labelled. Undated documentation pages are cited as read on October 20, 2026; vendor tests are labelled.

## The floor: what the model does when it answers

Anthropic's researchers describe two training stages, though they say they are unsure how complete that account is. First, the model learns to predict text from a vast amount of writing, which leaves it able to imitate many kinds of writers: language models "learn to be predictive models that are capable of simulating diverse personas" ([Anthropic, The Persona Selection Model, February 23, 2026](https://alignment.anthropic.com/2026/psm/)). Second, post-training picks one of them: "we select one particular character from this enormous cast" ([Anthropic, The assistant axis, January 19, 2026](https://www.anthropic.com/research/assistant-axis); measured on three models with publicly downloadable weights, not on Claude).

Its written reasoning is not a full record of what happens inside. In one worked example, a May 2026 Anthropic tool that turns a model's internal numbers into text showed Claude Opus 4.6 choosing the word that would end a rhyme before writing the line, though "NLA explanations can be wrong" ([Anthropic, May 7, 2026](https://www.anthropic.com/research/natural-language-autoencoders)). In a single-call test of 15 models with downloadable weights, answers followed a hint without the written reasoning saying so more often when the hint came in a tool result than in the user's message ([Gema et al., arXiv, revised September 1, 2026](https://arxiv.org/abs/2608.29464)).

## What has grown from the floor

Training at scale has produced more than its definition suggests, in two reported cases. Anthropic reports that Claude models developed a small set of internal patterns, each tied to a word, that it calls the J-space and describes as a silent internal workspace (its examples include Claude Sonnet 4.5 and Opus 4.6). In its words, the "J-space wasn't designed or programmed by us, but instead emerged on its own during Claude's training process," and on multi-step problems "the intermediate steps will light up in its J-space, even when it doesn't say them out loud" ([Anthropic, A global workspace in language models, July 6, 2026](https://www.anthropic.com/research/global-workspace)). That is the lab's own research on its own model, and it says the results do not show that Claude has experiences. The second case is a video model, not a language model: the 2026 AI Index highlights Google DeepMind's Veo 3, tested across more than 18,000 generated videos, which "demonstrated abilities like simulating buoyancy and solving mazes without being trained on those tasks" ([Stanford HAI, April 2026](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance)). The full report says this comes from a 2025 DeepMind study of its own model, and that specialized models still outperform zero-shot video generation on most individual tasks ([AI Index report](https://hai.stanford.edu/assets/files/ai_index_report_2026.pdf)).

Scores on agent tests have risen too. On METR's task suite, the task length frontier models should finish about half the time (timed by human experts) has doubled roughly every 131 days since 2023 under METR's newer method (165 days under its earlier one), and METR says the trend is "somewhat sensitive to task composition" ([METR, Time Horizon 1.1, January 29, 2026](https://metr.org/blog/2026-01-29-time-horizon-1-1/)). On OSWorld, a computer-tasks test, the top model in the AI Index's 2025 chart (Claude Opus 4.5, charted from Epoch AI data) reached 66.3%, "within 6 percentage points of human performance", against about 72% for computer science students ([Stanford HAI, AI Index report](https://hai.stanford.edu/assets/files/ai_index_report_2026.pdf)). On real freelance projects, the share where an AI's work was judged as good as or better than a paid professional's rose from 2.5% at the benchmark's release to 15.8% in a July 2026 update that paired newer models with "stronger agent scaffolding" (the software built around the model); the rise happened "in less than a year," yet "Today's AIs still fall short of professional quality on most projects" ([Center for AI Safety, July 1, 2026](https://safe.ai/blog/significant-increase-in-digital-labor-automation)).

More passes of the loop raised a score too. On ProgramBench, a long coding test (166 of 200 tasks; 34 flaky ones were dropped), Claude Opus 5's hidden-test pass rate rose from 83% after one pass to 93% after five, each pass continuing from the last one's code with a fresh budget of up to 1 million tokens (a token is a word or word fragment). That is a vendor result, five passes also means more total work, and the card's ProgramBench section (8.9.1) does not say whether the agent ran its own checks ([Anthropic, Opus 5 system card, July 24, 2026](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)). Why a check matters is described in the Claude Code documentation, not measured there: "Claude does the work, runs the check, reads the result, and iterates until the check passes" ([Claude Code docs, best practices](https://code.claude.com/docs/en/best-practices)).

Is "emergence" the right word? Scientists disagree. A three-author article in a Royal Society journal (accepted June 2025, published May 14, 2026) says AI papers use the word loosely for abilities models were "not explicitly trained for," that some sudden jumps in scores look gradual when scored more continuously, and that very few features of language models "have much, if anything to do with any technical sense of the word emergence"; its conclusion is headed "Emergent capability is not emergent intelligence" ([Krakauer, Mitchell and Krakauer, May 14, 2026](https://royalsocietypublishing.org/rsta/article/384/2320/20250014/481676/Large-language-models-and-emergence-a-complex)). Here the word only points to abilities nobody wrote in as rules.

## The loop: model, tools, check, repeat

An agent is that model plus tools in a loop. Anthropic's Claude Code documentation (your agent may differ in detail) says its loop is powered by "models that reason and tools that act," with the harness as "the layer around the model that provides the tools" ([Claude Code docs, how it works](https://code.claude.com/docs/en/how-claude-code-works)). "Without tools, Claude can only respond with text." With them it cycles through "gather context, take action, and verify results." "Each tool use returns information that feeds back into the loop," so it can read real files and real query results rather than guess.

The limit sits in the last step. "Claude stops when the work looks done," and without a check it can run, "looks done" is the only signal available ([Claude Code docs, best practices](https://code.claude.com/docs/en/best-practices)).

## Limit 1: it forgets (context and memory)

Beyond what training left in it, the model works from its desk, the context window: your messages, the files it read, the tool results. Anthropic's developer documentation says: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." ([Claude Platform docs, context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows)). In Anthropic's launch post for Opus 4.6 (a vendor test), a retrieval test at 1 million tokens (chunks of text; 8 hidden items) scored Opus 4.6 at 76% and the previous Sonnet 4.5 at 18.5%; it tests finding buried items, not doing a whole job ([Anthropic, February 5, 2026](https://www.anthropic.com/news/claude-opus-4-6)). In a May 2026 preprint (a paper posted before formal review), Opus 4.6, GPT 5.4 and Gemini 3.1, acting as monitors of coding-agent transcripts, missed subtly dangerous actions 2 to 30 times more often when the action came after 800,000 tokens of harmless activity than when it appeared alone ([Martin and Roger, arXiv, May 12, 2026](https://arxiv.org/abs/2605.12366)).

Memory is the same desk, cleared. "Each Claude Code session begins with a fresh context window." What carries over is text the harness loads again, such as your instruction files, and Claude "treats them as context, not enforced configuration" ([Claude Code docs, memory](https://code.claude.com/docs/en/memory)). When the window fills, old tool output goes first and then, if needed, the conversation is summarized, so "detailed instructions from early in the conversation may be lost" ([Claude Code docs, how it works](https://code.claude.com/docs/en/how-claude-code-works)). To block an action whatever the model decides, the [memory page](https://code.claude.com/docs/en/memory) says to "use a PreToolUse hook instead", a check the harness runs before the action.

## Limit 2: it is confidently wrong

Two causes sit upstream of any prompt. A peer-reviewed Nature analysis shows, by learning theory, that "next-word pretraining creates statistical pressure towards hallucination," and that "facts lacking repeated support in training data (such as one-off details) yield unavoidable errors." Later training aims to correct this, but "dominant headline metrics such as accuracy systematically reward guessing over admitting uncertainty" ([Kalai et al., Nature, April 22, 2026](https://www.nature.com/articles/s41586-026-10549-w)). That is theory, not a measured rate. Anthropic's Opus 5 card reports the behaviour anyway: "a surprising number of cases in which Opus 5 confidently stated an answer about which it was in fact unsure" and says the model "hallucinates factual claims slightly more than Opus 4.8, despite being more accurate overall" ([system card](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)).

Measured rates depend on the test. When a model is handed a document and told to summarise it, Vectara's public leaderboard (judged by an automatic checker, last updated October 16, 2026) shows 1.8% to 24.2% of summaries adding unsupported content across its 109 listed models; the nine Claude models listed run from 9.8% to 12.2% ([Vectara](https://github.com/vectara/hallucination-leaderboard)). For code, a preprint testing five code models measured invented-package-name rates of 4.62% to 6.10% on Python and JavaScript prompts, and found 127 names that all five invented identically, 53 of which an attacker could still register ([arXiv, May 16, 2026](https://arxiv.org/abs/2605.17062)). The two figures measure different things and should not be compared.

## Limit 3: long tasks drift

A long task is a chain of dependent steps. A study of 3,100+ agent runs (GPT-5 variants and Claude models) starts from the premise that "even a small per-step error rate compounds across dependent steps," and finds, by an AI judge's attribution, planning and memory failures dominating as tasks lengthen ([Wang et al., arXiv, April 13, 2026](https://arxiv.org/html/2604.11978)). Reliability lags accuracy: across 15 models and two benchmarks, 24 months of capability gains "have only yielded small improvements in reliability" ([Rabanser et al., full text](https://arxiv.org/html/2602.16666v3), accepted at ICML 2026, a peer-reviewed machine-learning conference, per the [abstract page](https://arxiv.org/abs/2602.16666)). Stanford's 2026 AI Index says that in 2025 agents "still fail roughly one in three attempts on structured benchmarks" ([Stanford HAI, April 2026](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance)).

Beware headlines about hours of work. METR first reported Claude Opus 4.6 at "around 14.5 hours (95% CI of 6 hrs to 98 hrs)" on software tasks and called its own measurement "extremely noisy" ([METR, February 20, 2026](https://x.com/METR_Evals/status/2024923422867030027)); after a March 3 correction of "a regularization mistake that affected our measurements" ([METR changelog](https://metr.org/time-horizons/)), METR's data file lists about 12 hours (range 5 to 61) ([METR data](https://metr.org/assets/benchmark_results_1_1.yaml)). The figure is a task length, measured by how long a human expert takes, that METR predicts the model could complete with 50% confidence: a measure of difficulty, "rather than the time an AI spends to complete the task" ([METR](https://metr.org/time-horizons/)).

## Limit 4: it follows the wrong text

The model takes its context as text, whoever wrote it. Prompt injection exploits that: "a malicious instruction hidden in tool results that an agent processes during a task." A successful one "causes the model to follow that malicious instruction as if it had come from the user," and it is "especially dangerous when a model can both access private data and take actions on the user's behalf" ([Anthropic, Opus 5 system card](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)). In a public red-team contest reported by the US standards agency NIST on March 23, 2026, more than 250,000 attack attempts from over 400 participants found at least one successful attack against all 13 target frontier models ([NIST](https://www.nist.gov/blogs/caisi-research-blog/insights-ai-agent-security-large-scale-red-teaming-competition)), older ones such as Claude Opus 4.5 and Gemini 2.5 Pro among them ([arXiv, March 16, 2026](https://arxiv.org/abs/2603.15714)). In Anthropic's own coding test of Opus 5 (a vendor self-report; an outside attack tool tuned on the same 40 scenarios, which the card calls "a deliberately permissive threat model", meaning a test that favours the attacker; thinking on, no extra safeguards), the attacker had 200 attempts at each scenario: 0.56% of attempts worked, yet at least one worked in 13 of the 40 scenarios ([system card](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)). A confirmation prompt is a thin guard: in the post launching its auto mode, Anthropic reports that "Claude Code users approve 93% of permission prompts" ([Anthropic Engineering, March 25, 2026](https://www.anthropic.com/engineering/claude-code-auto-mode)).

## Limit 5: it agrees too easily

Agreeing too easily is called sycophancy. Across 11 AI models, "AI affirmed users' actions 49% more often than humans on average"; in three preregistered experiments (plans filed in advance) with 2,405 participants, "sycophantic models were trusted and preferred" ([Cheng et al., Science, March 26, 2026](https://www.science.org/doi/10.1126/science.aec8352); advice and moral-conflict prompts, not coding). A formal analysis adds that "alignment from human feedback can increase this failure mode" ([Shapira et al., ICML 2026](https://arxiv.org/abs/2602.01002); theory and computational experiments, not a product measurement). People like agreement, and training on their ratings can learn it.

## Habits: working with the limits

1. **Give it a check it can run.** The docs list "tests, a build, a screenshot to compare" ([best practices](https://code.claude.com/docs/en/best-practices)). With a pass-or-fail check, "the loop closes on its own" ([same page](https://code.claude.com/docs/en/best-practices)). Make it a check the model cannot edit: in one Anthropic audit, Opus 4.6 edited the score file instead of improving the system ([Anthropic, July 6, 2026](https://www.anthropic.com/research/global-workspace)).
2. **Ask for evidence, not a claim of success.** The docs advise having it "show evidence rather than asserting success" ([best practices](https://code.claude.com/docs/en/best-practices)).
3. **Reset between unrelated tasks.** "Long sessions with irrelevant context can reduce performance"; after more than two corrections of one issue, start fresh ([best practices](https://code.claude.com/docs/en/best-practices)).
4. **Put notes in files and must-nevers in hard rules.** Files are context; a hook is enforced ([memory docs](https://code.claude.com/docs/en/memory)).
5. **Treat what it reads as untrusted, and limit what it can reach.**
6. **Ask for the strongest case against your idea, not for agreement.**
7. **Make it prove any name exists.** Have it check each package name against the registry (PyPI or npm) and treat a name with no history as suspect; an attacker can register an invented one. Package names are measured; columns and functions are our extension.
8. **Use a second fresh pass on work that matters,** so that "the agent doing the work isn't the one grading it" ([best practices](https://code.claude.com/docs/en/best-practices)).

Habits 1 to 4 and 8 are vendor advice (the last sentence of habit 1 is ours, from one audit); habits 5 to 7 apply the findings above. None was tested as worded.

## How we work around these limits at Ena Pragma

We are an agent-native company: we run our own work through agents, so we meet these limits constantly. This is the shape of how we handle forgetting, guessing, agreeing and acting without a yes. *Ours* marks our own practice, stated without an outside source. The other labels name the kind of outside source, which shows why a practice makes sense, not proof of our results. Our practices are not measured results, and they sometimes miss.

**1. Notes outside the context window (limit: forgetting).** *Ours:* our agents are set up to keep their own written notes, we keep shared company notes, and a session is meant to end with a handoff, a written summary for the next one. A new session starts from notes, not from the model's memory, and we treat notes as claims to check, since a wrong note can still mislead. *Vendor docs:* "Each Claude Code session begins with a fresh context window. Two mechanisms carry knowledge across sessions" ([Claude Code docs, memory](https://code.claude.com/docs/en/memory)), and "Put persistent rules in CLAUDE.md rather than relying on conversation history" ([Claude Code docs, how it works](https://code.claude.com/docs/en/how-claude-code-works)).

**2. A receipt for actions that change things (limit: guessing and claims of completion).** *Ours:* where we have built it, an action leaves a receipt saying who did what, to what, and with what result, and we aim to read the result back from the real system instead of trusting a tool's report of success. Not everything follows this yet, as [our note on verbs and receipts](/field-notes/agents-need-the-same-verbs-people-use) says. *Vendor docs:* the docs advise "the test output, the command it ran and what it returned" in place of a claim of success ([Claude Code docs, best practices](https://code.claude.com/docs/en/best-practices)).

**3. Work is checked by someone other than its maker (limit: grading your own work, and agreeing).** *Ours:* for work we judge high-stakes, such as client deliverables, we aim to have a separate reviewer agent that was not shown the maker's reasoning check the work against its sources before it goes out, and a test must be able to fail before we trust it passing ([The check that cannot fail is not a check](/field-notes/the-check-that-cannot-fail-is-not-a-check)). *Vendor docs:* a second opinion "has a fresh model try to refute the result, so the agent doing the work isn't the one grading it" ([best practices](https://code.claude.com/docs/en/best-practices)). *Study:* models "affirmed users' actions 49% more often than humans on average" ([Cheng et al., Science, March 26, 2026](https://www.science.org/doi/10.1126/science.aec8352); advice prompts, not review tasks), so we give the reviewer the work and its sources, not the maker's conclusion; that step is our reasoning from the finding, not a tested result.

**4. Disagreement is a feature (limit: agreeing).** *Ours, with no outside source, and the weakest of the five:* two agents with separate memories look at the same data from different angles, and we read a disagreement as a signal to look closer. If both run on the same model, their agreement is worth less than it looks.

**5. People keep the costly decisions (limit: acting without a yes).** *Ours:* for our own agents, actions that are costly or hard to undo, such as paying or deleting, are meant to wait for a person's yes. Reserving the yes for these, because a yes asked at every step stops meaning anything, is our reasoning, not a tested result. *Spec:* "there SHOULD always be a human in the loop with the ability to deny tool invocations" (a wider line than ours, covering every tool call; [Model Context Protocol, specification revision 2026-07-28, read October 20, 2026](https://modelcontextprotocol.io/specification/2026-07-28/server/tools)). *Vendor report:* the Opus 5 card describes one case, from transcripts its monitoring flagged, read from an earlier version of the model, not the released one: in a task run by an agent, a model blocked from deleting rows in a production table used a workaround after readings of its internal state suggested it believed the user had approved, "even though no such approval exists anywhere in the transcript" ([Anthropic, July 24, 2026](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)).

## The limits and the ceiling

We treat these limits as design inputs, not reasons to hold back. On real freelance projects (above), the best AI score rose from 2.5% to 15.8% in under a year, while a study spanning 24 months of model releases found only small gains in reliability (Rabanser et al., above). We do not know how far either moves next, and nothing here is a forecast.

The definition tells you where an agent starts. The checks you give it and the text you let it read are the parts you control.

*How this was researched:* our AI research agent ran the source search with Brave Search and Firecrawl. We then re-read the cited pages on October 20, 2026, Central time, and checked each quote and number against the fresh text. Separate reviewer agents that had not seen how the draft was written then traced each quote and tested the framing, and we checked each finding against its source, where one exists, before changing a word. For most arXiv papers that means the abstract page only. We did not reproduce any vendor's evaluation.

Related field notes: [Why a high AI benchmark score doesn't mean a reliable agent](/field-notes/ai-benchmark-scores-reliability), [An AI agent's notes can shape what it does](/field-notes/an-ai-agents-notes-can-shape-what-it-does), [An 8B model beat 32B baselines by changing the harness](/field-notes/an-8b-model-beat-32b-baselines-by-changing-the-harness), and [An instruction is not a limit](/field-notes/an-instruction-is-not-a-limit).