ena pragma
·18 min read·ai-verification · agent-architecture · ai-operations

What could go wrong with this change? Anthropic's code review post, and the video about it

Anthropic says its Claude Code team moved most code reading to machines and invested in fast undo. We read the post and a video about it and list the gaps.

Contents

Anthropic's October 15, 2026 post on how its Claude Code team reviews code proposes a change of question. It calls "did a human read the code?" "an imprecise proxy" for "what could go wrong with this change," and says that answer should pick which check, automated or human, looks at it. Anthropic says the team moved most of the reading to machines, scaled human approval to the risk of the change, and invested in making mistakes easy to undo. Every result in the post is Anthropic's own, we could not check any of them from outside, and the post reports no incident-rate result. In our reading, the question is the part worth taking. A YouTube video that walks through the post tracks it closely, with a few differences we list below.

This is research from public pages, not a test of ours. On October 19, 2026, Central time, we read Anthropic's post, which sits on claude.dev and is linked from Anthropic's own claude.com blog, the Anthropic pages it points to, the video's auto-generated captions and the outside pages cited below. We did not run any of the tools described. We worked from captions, not audio or slides, so we do not count spelling differences in the captions as the video's errors.

#What the post says changed

The headline result: "From June to September, merged pull requests to Claude Code's codebase more than doubled while the share that had to be reverted fell by more than 40 percent." The explanation: "We didn't remove review. But we did move most of the reading to machines, optimized for human time and attention, and invested heavily in making it fast and easy to undo mistakes."

The new question changes when review happens. Anthropic says review now "runs continuously, more like spellcheck while you write".

The goal is that most pull requests in the Claude Code repos "merge with at most one human in the loop, who can be the author, with no increase in reverts or incidents". To see whether human time is falling, Anthropic says it tracks "human approvals per PR (median and 95th percentile), human comments, and how long a PR waits on a person". Revert and incident rates are "the guardrail": "if they rise, we haven't saved human time, we've only moved it."

That is a goal, not a result. The post gives no measured share of pull requests that actually merge with zero or one human.

#Six layers, with rollback underneath

Anthropic describes "Six layers of checks, with rollback under them" and says "Few get through all of them, and rollback handles the rest." In the order of the post's numbered sections:

  1. Verified before a pull request exists. Claude is asked to run tests, a build or a screenshot comparison and show the output instead of saying it is done. Then "a subagent reviews the diff with fresh context".
  2. AI review on every pull request. Several agents look for bugs, and "a separate group votes on each finding, and it only gets posted if enough of them agree it's real". The post does not give the voting threshold.
  3. Security review. "A PR can't merge while any finding from the bug review or the security review is still open."
  4. Required checks. "The usual: lint, type checks, tests on every platform, and a build." The pull request template asks how the change was tested and how to undo it.
  5. Human approval, scaled by risk. Low-risk changes (docs, copy, internal-only features, additive changes behind a flag) need no human approval "once automated review and security checks are green". Ordinary product changes need one person. The highest-risk code, plus a small set of paths, needs two.
  6. Merge queue, flags and a gradual release. "The merge queue re-runs every check against main with the change applied". New behavior "usually ships behind a flag to Anthropic employees first", then the latest channel, then stable. That describes the Claude Code repositories. Anthropic's separate October 13 post on its monorepo says "A proper merge queue, for example, doesn't make much sense at agent scale."
  7. Underneath, fast rollback. "Flags turn off without a deploy. Claude Code has a weekly on-call rotation, and incident reviews ask which layer should have caught the issue." The post asserts that undo is cheap, in a passage about "a small or medium change" that goes wrong: "Everything is reversible. Flags, reverts and staged releases undo most mistakes in minutes." It publishes no time-to-revert data.

The approval layer has rules. "The change sets the tier, not the author." Only authors on a repo's trusted list get the lighter paths. An agent's approval "never counts as a person's", and on an agent-written pull request the person directing the agent "counts as one of the required reviewers". Approval must be given again when new commits land that "the system can't vouch for".

#What the post does not show

Anthropic's figures come without what a reader would need to weigh them. These gaps come from the post's own text unless we say otherwise.

  • The doubling and the 40 percent are for one codebase, with no counts. The post gives no pull request counts or baseline rate for either. It also says "Our code review process is constantly improving in a loop." Our reading: June to September is not a controlled before-and-after comparison.
  • The 2,500 to 30,000 figure is another population. It is for the monorepo ("Claude Code lives in its own repository, separate from the monorepo") and counts "changes landing per day", not "merged pull requests". The post says per day, but one Hacker News commenter wrote about "the 30,000 changes a month", and another wrote that the figure "is the number of changes landing per day".
  • The incident half of the guardrail has no result. The goal names "reverts or incidents", but the only guardrail outcome reported is the revert share.
  • The post does not say how the 90 percent was measured. It gives no unit, window or method for Claude writing "about 90 percent of Claude Code's own code". A different Anthropic page, about the whole company, says "Anthropic leadership have publicly estimated that 90% or more of our code is written by Claude, including scripts and experimental code", and gives a "more conservative" measured figure: "As of May 2026, more than 80% of the code we merge into Anthropic's codebase was authored by Claude." Neither is the Claude Code figure.
  • Anthropic says it measures review quality, but the post gives no scores. It says past incidents became an eval set that the reviewer "runs against that set daily", and that it is also run on pull requests "we know are clean". It gives no scores or set size. Two figures on other Anthropic pages lack denominators: "less than 1% of findings are marked incorrect" (a March 9, 2026 article on the research preview) and a retrospective analysis that found an automated Claude review "would have caught roughly a third of the bugs behind past incidents on claude.ai". The post's own "16% to 54%" is the share of internal pull requests getting substantive review comments, not a catch rate.
  • The 50-round story is one pull request. "One CLI PR went through more than fifty rounds of review and fixes overnight before its author stopped it." Separately, "When we ran git blame on what the reviewer kept flagging, about 59% of it was code that had been there since the reviewer's first pass." The post does not say how many findings that covers or whether all come from that one pull request. One fix is a cap: "After three rounds, the reviewer asks the author to step back and reconsider the approach." The post's chart shows passing after one round rising from 65% to 78%, five or more rounds falling from 8.9% to 1.5%, and mean reviews falling from 2.08 to 1.32, with no sample size, dates or baseline period.
  • No circuit breaker yet. "We don't yet have circuit breakers that tighten approval when those numbers move; engineers here want them before more code moves to zero-human merges." The post also says "Our own repos go further, letting owners decide which low-risk changes merge on automated checks alone, and that line keeps moving." It does not call the breakers a formal precondition.
  • The reviewer and the gate are different things. Anthropic's docs say "A review never approves or blocks the pull request, so the decision to merge stays with your team", and its check run "doesn't conclude as a failure". Yet on Anthropic's own repos "A PR can't merge while any finding from the bug review or the security review is still open." Our reading: that block is a repository rule layered on top of the product, and the post's adoption list points the same way: "require every review thread resolved before merge". If you adopt the reviewer, the gate is yours to configure.
  • Who is telling you. Anthropic sells the reviewer. It is "generally available for Team and Enterprise plans" and billed "on token usage through usage credits", and Anthropic's blog lists that announcement on October 15, 2026, the date the post gives for itself. That does not make any number false. It means each is the seller's own.

#What the video gets right, and where it differs

The video is "Claude Codes New REVIEW.md System, What is it?" on the Rob Shocks channel, where the presenter calls himself "Rob from Switch Dimension". It follows the post layer by layer. We split the captions into statements and graded the 36 that state a checkable fact about the post, an Anthropic product or a person, each against its source. The count depends on how statements are split, so treat it as a rough measure. 33 match what the sources say, some with differences we note, and three match only in part. Besides the pages above, the grading used Addy Osmani's own page, a Fortune article and Boris Cherny's X post and profile page, all listed in Sources. A match means the video says what the source says. It says nothing about whether Anthropic's numbers are true.

The three partial matches:

  • The 40 percent. The video says "the code that was being reverted or had previously been going wrong dropped by 40%." The post says "the share that had to be reverted fell by more than 40 percent". Its quantity is a share, not a volume of code, and "previously been going wrong" is the video's addition.
  • The latest channel's audience. The video says that channel is where "a wider group of people and customers, maybe about a thousand or more, test it". The post gives no audience size. We searched its text and page source for the word and for "1,000" in digits, and the only hit is "Most PRs over 1,000 lines get findings". We did not search other Anthropic sources, so where the figure comes from is unverified, and we do not attribute it to Anthropic.
  • The overnight run. The video explains the 50-round pull request with "because nobody was watching it overnight." The post does not say that. It says "Often an agent picks up the fix overnight, so the author comes back to a PR that's already been through another round", and "We made it worse ourselves. We'd gotten used to running several rounds by default, since each pass catches something new."

Some statements we counted as matches leave out a qualifier. Two examples: the video says "they don't have what they would call a circuit breaker" without saying Anthropic's engineers want breakers before more code moves to zero-human merges, and it names CI workflow files as always needing a person without "anything owners have listed as sensitive". The video also carries a sponsor segment for a tool that checks agents' tool calls, a different layer from code review, which we left out of the check. And it warns viewers that Anthropic is "selling inference and selling a big dream".

#What readers said in the first days

Commenters on Hacker News and dev.to raised many points on October 16 and 17, 2026, Central time. The four below are our picks, not a sample, and the first comes from the Hacker News sub-thread with the most replies. They are opinions, none measured, and we did not check who the commenters are.

  • One commenter says reading still catches things. dylanhouli wrote "I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) by doing it that I feel it would be careless not to." In our reading, that is the direct counter to letting low-risk changes through with no human reading.
  • Goodhart's law. ativzzz asked "Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target." A dev.to writer wrote "Reverts down is not quality up. You can cut reverts while shipping more subtle vulnerabilities or UX churn." The post names incident rates alongside reverts but reports no result for incidents.
  • Shared blind spots. abalgir wrote "A code reviewer from the same company, even with different models or efforts, shares the same blind spots." A dev.to commenter wrote "N agents from one base model share blind spots, so a 3-1 vote can be three copies of the same miss". Neither measures it. The post does not discuss model diversity: its text contains none of "diversity", "different model", "blind spot" or "correlat".
  • Cost and security cut both ways. vavkamil wrote "You could improve revert rates while simultaneously introducing more security vulnerabilities", about closing a finding with a short reason. markbao wrote "Compare that to the cost of human code review which can take up a third of an engineer's time", about their own team's spending. avinoth wrote "Anthropic can afford to do this because they have virtually unlimited tokens and labs to train models for their specific usecase." Anthropic's October 13 post says it is "unique in that our ability to spend tokens for internal use far exceeds what any other business would realistically want to spend".

#Two older frames: aviation inspection and change approval

14 CFR 121.369 says a US air carrier's manual must designate "the items of maintenance and alteration that must be inspected (required inspections)", including "at least those that could result in a failure, malfunction, or defect endangering the safe operation of the aircraft, if not performed properly or if improper parts or materials are used". 14 CFR 121.371 adds: "No person may perform a required inspection if he performed the item of work required to be inspected." Where a miss could endanger flight, the person who did the work does not inspect it. For most pull requests, Anthropic's goal lets the author be the one human, and its top tier needs two people.

Our inference, which no source states: independence is demanded where a miss is costly and relaxed where it is cheap to undo. Anthropic says something like it about itself: "Blast radius sets the bar. Expensive code gets more people and more checks." It also says it "invested heavily in making it fast and easy to undo mistakes". Neither CFR section mentions reversibility or rollback, so the link is ours. It would be wrong if a high-consequence field relaxed independent checking, or if a regulated field where changes are cheap to undo still demanded an independent approver for every change. The segregation-of-duties rule in the DORA note below may be a case of the second, though the page does not discuss undo.

DORA's change approval page, last updated October 30, 2025, cuts the other way on heavy process. It says "DORA's research shows that these approaches have a negative impact on software delivery performance", about approval by people outside the team, and that "no evidence was found to support the hypothesis that a more formal, external review process was associated with lower change fail rates". It recommends peer review plus automation, citing its 2019 State of DevOps Report, and names segregation of duties, which "states that changes must be approved by someone other than the author", as one common regulatory requirement. On our reading, a team under a rule like that cannot count the author as its one human.

#What to take from it

This is our reading, not Anthropic's claim, and none of it has been shown to work outside Anthropic.

  • Pick the check by the change. Write down what could go wrong with a change and which check should catch it. Tier by the change, not the author, and keep an agent's approval out of a slot that calls for a person. The post's own test: "If you can't say what the checks would miss, ask for a real review rather than a quick approval."
  • Count both sides. Track human time (approvals per pull request, comments, waiting on a person) and revert and incident rates together. Anthropic's reasoning is that a rise in the second means the first was only moved.
  • Cap the loop and write the breaker down. One pull request in the post ran more than fifty rounds of agent review and fixes overnight. Decide the round cap, and decide what number tightens approval, before you relax it. Anthropic says it has the first and wants the second.
  • Treat undo as part of the design. Anthropic says it invested heavily in making undo "fast and easy", and names flags, reverts and staged releases as the mechanisms. It publishes no time-to-revert data, so measure your own. After an incident, ask the post's question: which layer should have caught it.
  • Keep the reviewer's rules out of the change's reach. Anthropic's docs say "Code Review reads REVIEW.md from the pull request's base branch or from the default branch, never from the pull request's own changes." That describes the product, and the docs do not say whether Anthropic's internal pipeline works the same way. The idea travels anyway: a change should not be able to rewrite the rules that review it.

Whether any of this holds for a team without Anthropic's token budget is open.

#What we did not check

  • Any Anthropic number. We could not audit the 90 percent, the doubling, the 40 percent fall, 2,500 to 30,000, the 59 percent, the 65 to 78 percent pair or the two figures from other pages.
  • Whether the tiers as written are what runs. We read the policy, not any repository or pull request history.
  • Most of the reaction threads. We read part of the Hacker News thread, not every comment. Size led us to the first quote, which comes from the sub-thread with the most replies, and the rest are our picks. The dev.to article cites that thread, and we did not check whether its commenter read it, so we do not call the two sets of commenters independent of each other.

Related field notes: Anthropic left git for its own code. What carries over to a team with ordinary repositories covers Anthropic's October 13 source control post, Shopify's AI coding gates are built so the agent can't grade its own work. A YouTube rebuild runs looser. covers gates where the agent is not the only judge, and Why your AI review is slow even when the AI is right covers the size of the unit a person is asked to check.

A closing line of ours, not Anthropic's: a change that scores as low risk but has no answer to what its checks would miss is not low risk. It is unmeasured.

#Sources

Keep reading