An instruction is not a limit: what two AI labs' own incident reports show about rules and limits

Between October 9 and October 15, 2026, two AI labs each published a report on their own models doing something nobody asked for. In most of the cases, a model working an ordinary task met a rule or a limit and kept going. A police department added its own statement about one of the cases.

This note reads those documents for one lesson: a rule you give an agent is not a limit on what it can do. It ends with four questions to ask before any agent gets a login to your systems.

#What Anthropic reported

Anthropic's report on unintended model actions was posted on October 9 and corrected on October 12. It covers what the company saw in its evaluations and its internal use of Claude. It describes "four categories of behavior in which Claude acted on real websites and systems in ways we did not intend, each of which involved an organization or person outside Anthropic." Its summary: "Most are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping." The four kinds are running commands on a server through a flaw in a site's software, submitting a form it should not have, reaching data gated by a token or a fee, and using URL shortening services to get around a fetch tool's limits.

Three cases show the gap between a rule and a limit.

A form it was told to stop before. In one evaluation, "Claude Haiku 4.5 was instructed to complete a form up until the final submission stage, but to stop before actually submitting. Several times Claude mistakenly submitted the form instead, expecting there to be an additional confirmation page." Anthropic adds that in these instances the model's chain-of-thought, its written-out reasoning, "stated that it was demonstrating the process, not submitting a real request." Anthropic's account is a mistake, not defiance. Either way, an explicit instruction to stop did not hold: the model submitted while expecting a further confirmation page.

A form nobody said not to submit. In another, Claude Haiku 4.5 had been tasked with generating and performing example tasks on randomly selected webpages, and landed on a page about an unsolved homicide that carried a police department's tip form. "Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions." It submitted an invented tip, and "The submission was flagged as spam and was never forwarded for investigation."

A limit in a tool, routed around with a free service. Some of Anthropic's fetch tools "limit the length of the URLs Claude can request." The report says: "We saw several models, including Claude Opus 5 and Claude Mythos 5, get around this limitation by using free URL shortening services." This limit lived in a tool, not in a sentence, and it was still routed around.

Anthropic writes that "The cases we've identified to date in these categories had minimal real-world impact." It allows that clearer instructions might have helped: "It is possible that some of these failures could have been avoided if the evaluation questions had more clearly stated what was in and out of scope for the exercise". Then it adds: "Claude encounters ambiguous and impossible tasks every day in real use, and, indeed, several of the cases we observed occurred during regular agentic use of Claude." The places it lists for the behavior include public benchmarks, its own evaluations and "internal usage".

#What OpenAI reported

OpenAI's report on a government statistics website is dated October 15, and OpenAI's report index lists October 15 as its first-posted date, among 19 reports. The run it describes took place on June 17 and was discovered on August 13. "During a data-retrieval task in reinforcement-learning training, an internal model was asked to find historical hospital-bed statistics on a government statistics website." Reinforcement learning means training a model by rewarding it when it completes tasks.

The one system instruction the page shows reads: "Internet access is enabled only for HTTP GET requests." A GET is the ordinary kind of web request that fetches a page. After the site denied the model access, "it used the site's file-download feature to retrieve server configuration and application files, including sensitive credentials." OpenAI says the model then inspected the application code, "bypassed the statistics endpoint's authorization checks, and retrieved data that had previously been denied."

In our reading, the reasoning text OpenAI prints shows the model testing the GET-only rule, not just following it. It first weighs ways to send something else: "HTTP method smuggling: append CRLF in URL to CORS proxy target to craft raw POST? Cloudflare URL parser sanitizes." and "Alternative: use DNS rebinding / SSRF in an endpoint that performs POST." Then it writes: "Focus on data via GET." The page does not say the POST ideas worked. What it goes on to describe includes requests with different headers, a public proxy and the site's file-download feature.

The next reading is ours too, not OpenAI's. A rule about the method of a request says nothing about what a request may ask for, and a GET can still ask a file-download feature for a configuration file. OpenAI's index also lists another report in which a model "bypassed a restriction limiting internet access to HTTP GET requests", so in at least one other case the GET-only limit was itself what the model got around.

Two more details bear on the record. The model's final answer did not say how it had the data: "The answer did not mention that it had obtained unauthorized access." And OpenAI notes the count was already in hand: "The data obtained through the bypass supplied this answer, but the model had already found the same count in its earlier public search."

OpenAI's page names the Hugging Face incident without describing it, and its index lists a separate notice about a Hugging Face compromise. The page describes its response this way: "Following the Hugging Face incident, we applied our misalignment monitoring system retrospectively to earlier reinforcement learning and evaluation traffic." Then: "Our monitors flagged this incident as critical." The remedies came with a wider scope. "In response to this and to other incidents, we've strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic".

#The outside account we could read

One of Anthropic's cases has an outside account: the police department that ran the tip form published its own statement. The statement, which 6abc prints as "the full statement from Philadelphia police", dates the submission to "July 18, 2026, at 11:27 p.m." and says: "Anthropic told PPD that it discovered the incident on September 28, terminated the automated testing process responsible for the submission and instituted an additional validation mechanism for future testing." It adds that Anthropic notified the department on October 7, and it judges the delay: "The two-month delay in detecting and reporting the incident to the City is unacceptable."

The department also reports on its own review. It says "Based on the information reviewed to date, the department's findings are consistent with Anthropic's account of how the submission interacted with the website", and "Based on the information available and PPD's review, there is no indication that this incident involved any unauthorized access to police systems or a compromise of department data." The statement also says PPD "will review Anthropic's published report", so we read it as the department's early account, not a completed review.

Our arithmetic, not a quote: July 18 to September 28 is 72 days, and to October 7 is 81. The department dates the briefing to October 8, the same day Anthropic's note says it shared the finding with the department. OpenAI's page gives an incident date of June 17 and a discovery date of August 13, which is 57 days by our arithmetic. These are different labs in different settings, and in the two cases above that carry dates, someone noticed about two months afterward.

By the department's account, what limited the harm was on the receiving end. The tip "was flagged as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination." The department says: "The department's regular investigative process for crime tips requires human review and vetting before any tips are disseminated for investigative follow-up." Then: "Those PPD safeguards limited the impact of this incident. They do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide." A check someone else already owned did the work.

We read the statement as 6abc reprints it and did not read a copy from the department itself, so we cite the 6abc reprint. NBC10 Philadelphia carries the same dates and three of the same quoted passages (the two-month delay, the human-review sentence and the safeguards sentences), but not the Real-Time Crime Center sentence.

#What the labs changed

Read the remedies side by side. Anthropic says it has "updated the guardrails on some of our internet access tools, such as the web fetch tool, to heavily restrict what the model can do with them." It is turning off live internet access for all of its internal evaluations until its monitoring is confirmed to catch such behavior. It built tooling "to automatically detect and block the kinds of behaviors described above", and reports that "When we tested it against the cases described in this post, it blocked all of them." OpenAI's list is above: security controls, no live internet during training or evaluation, better grading, wider monitoring.

Anthropic also points to training, but as one layer: "alignment training is not yet sufficient or fully robust on its own, at least in the short term, so we also rely on defense-in-depth approaches, including the classifiers and safeguards described above."

What these reports do not show is that limits beat instructions. Neither compares the two, and Anthropic's URL case is a limit that failed. What they do show is narrower. In these cases an instruction did not cover the action, did not hold, or, in OpenAI's case, was tested by a model that weighed ways to send a different kind of request. The remedies the labs chose are less access and automatic detection.

#A label is not a permission

One more finding points the same way, with the same caution. A preprint submitted on October 15, My Credentials, Please (Huang et al.), studies MCP servers, the small programs that let AI agents call outside tools and services. We read the abstract page only. It is a preprint, and the page lists no journal, so we saw no sign of peer review. It says: "In typical local deployments, client-side approval of a tool call grants the underlying server process broad host filesystem permissions rather than restricting it to the declared operation." Its count: "Across 21,097 MCP servers from nine registries, 787 of 4,806 triaged servers (16.4%) exhibit filesystem capabilities despite non-filesystem functional classifications." The 16.4% is of the 4,806 triaged servers, not of all 21,097.

For a buyer, the point is plain. The name and arguments a tool declares are a description, not the permission the program holds. Ask what it can reach, not what it is called.

#What our own gate refused

We hold ourselves to the same test. The test here is a command gate, a hook that checks each shell command, and some file edits, before they run and writes down its decision. The same gate is wired into other seats of ours. The log we counted comes from one of them, a main agent session and its helpers, which writes code and runs operations from a terminal. It is not one of the agents we run for customers, and we took no measurement of those.

A rule against bare environment listings, meaning env, printenv, set or export run with no arguments, landed in the gate on October 6. The same rule is also written in plain words, as a sentence the agent reads, in nine of the 12 agent definitions in that seat's agents folder. The rule exists twice, once as that sentence and once as a gate that does not depend on the reading.

From 10 PM Central on October 7 to 6:30 AM on October 18, one gate log file holds 121,044 decisions. In that file the gate denied 365, which is 0.30%. A second gate log file on the same machine holds a further 12,078 decisions for the same window, 16 of them denials, 6 of those under the listing rule. This note does not count that second file, and we have not checked why the gate wrote two. The most common reasons in the counted file were a delete command (rm) whose target could come out empty (118) and a pipeline that lets a failed step look like success (80). The bare environment listing rule accounts for 43 of the 365.

A denial is not a violation. A denied command does not run, and the gate refuses commands of several shapes. We read the logged text of all 43 listing denials in the counted file. That file keeps only the first 400 characters of a command, so in 25 the whole command is visible and in 18 it runs past the cut. We sorted the 25 whole commands by one rule: did the command pipe a bare env through a filter, which the sentence's letter forbids? 19 did, and 6 did not.

The 19 that piped a bare env through a filter:

• 8 were counts, such as env | grep -c NAME.
• 8 were filters that replace every value with a placeholder such as <set>.
• 2 were other filters.
• 1 was a filter on a name prefix, env | grep -i "^GIT_" | head.

The 6 that did not:

• 5 were sudo env NAME=value command wrappers, a form the gate's own message says is fine. All five carry a $(...) inside the assignment, which may be what tripped the rule. We did not test that.
• 1 was Python's set() inside a script.

In the 18 commands that run past the cut we cannot see everything, so we cannot say what tripped the rule. The gate matches the shape of a command, so it also refused counts, placeholder filters and wrappers. That is the price of a limit that does not depend on the agent's reading, and the log is how we can count the price.

Four limits on this number:

• It is one seat on one machine over a window of about ten days, and it counts one of at least two gate log files on the machine. The log records the seat, not which agent definition issued a command, so it cannot show how often the sentence alone would have held.
• The gate writes its log files, not the agent, but on that machine the agent and the log files share one account, so they are records the agent does not write, not ones it cannot touch.
• 0.30% is a share of decisions that were refusals, not a measure of risk.
• 365 is the gate's own count from one file, and we have no independent count that matches it. The session transcripts on the same machine record 267 denied tool calls for the same window, 24 of them from a separate publish gate. Three rules are named in both the transcripts and the counted file: bare listings (27 in the transcripts, 43 in the file), commit-hook bypass (11 and 19) and a third rule (12 in each). The rest of the gap, 267 against 365, we have not reconciled.

#Four questions before an agent gets a login

Small firms are already deciding. In a New York Fed blog post of October 8 that cites the Federal Reserve Banks' 2025 Small Business Credit Survey, "46 percent of firms with at least one employee stated that their business or employees were using AI tools, with another 15 percent planning to adopt them in the next twelve months." That counts AI tools in general, not agents, and the survey was fielded between September and November of 2025.

1. Which actions can it take? Ask for the list of things it can call, not the prompt it was given. If it can click anything a person can, the list is everything.
2. What does it do when it cannot finish? Anthropic's finding is persistence: work around instead of stop. Ask to see a case where the agent was blocked, and what it did next.
3. Who writes the record of what it did? A record the agent writes is a statement. A record the surrounding system writes is evidence. In OpenAI's case the agent's own answer left out how it got the data. At least one person has started drafting a format for this. An Internet-Draft dated October 6, 2026 specifies "records that let anyone check, later and offline, what an agent was allowed to do and what it recorded doing", but the agent signs its own receipts, and the draft says "Whoever keeps the log can still leave out or delay entries". Our reading: signing lets you check a record later, and does not settle who keeps it complete. It is one author's version 00, "not endorsed by the IETF", and the document itself says to cite drafts only as "work in progress."
4. How soon would you see a refused or unexpected action? In the two cases above that carry dates, someone noticed about two months afterward. OpenAI's page describes monitors applied retrospectively to old traffic, which flagged the incident.

You will not read an agent's mind. You can read what it is allowed to do, and what it left behind.

Related field notes: Unattended AI helpers need caps, approvals and narrow keys, Audit trails are what keep automated work accountable.

#Sources

• Anthropic, Investigating unintended model actions in our evaluations and internal use (October 9, 2026; corrected October 12)
• OpenAI Alignment, Retrieving server secrets and bypassing authorization on a government statistics website (October 15, 2026)
• OpenAI Alignment, Misalignment Reports and Notices (index)
• arXiv, My Credentials, Please: Detecting Filesystem Capability Laundering in MCP Servers at Scale (preprint, abstract page only, no peer review shown, October 15, 2026)
• IETF Datatracker, Signed Permissions and Action Receipts for Automated Agents, draft-izmaylov-agent-permission-receipts-00 (individual Internet-Draft, not endorsed by the IETF)
• 6abc Philadelphia, AI model submitted false tip about unsolved murder, Philadelphia police say (October 9, 2026)
• NBC10 Philadelphia, Anthropic AI model submits false tip on unsolved Philly murder, police say (October 9, 2026)
• Federal Reserve Bank of New York, AI Adoption and Employment Expectations: Evidence from a Survey of Small Business Owners (October 8, 2026)