ena pragma
·9 min read·ai-verification · ai-operations · agent-architecture

Anthropic's SDLC playbook has one rule worth copying everywhere: confirm the check fails first.

The AI-native SDLC playbook asks for a test that fails before a fix. We think the field data makes the case for putting that rule on every gate.

Contents

A YouTube video circulating this week walks through Anthropic's AI-native SDLC playbook. Its description credits the playbook to "Boris Cherny and the Anthropic team." The playbook itself, published on Anthropic's Claude blog on August 21, 2026, carries a different byline: Louis Claxton. (We worked from the video's published description, not its spoken content, since no link to the video itself was available to us.)

It is a small error, and a useful one to start with. A summary made a confident claim, the claim did not match its source, and what caught it was opening the source. That is what this piece is about: not writing code, but knowing whether the checks on it are real.

#What the playbook gets right

We walked through the playbook's six stages when it came out: plan, design, build, test, deploy, maintain, each ending in a version-controlled file the next stage reads. It is worth restating what makes it good.

It turns review policy into a file, REVIEW.md, so every pull request gets the same passes. It keeps the agent's working knowledge in the repository. It puts hooks at the gates that can allow, ask, or block an action. It says "each production incident gets an eval, written by the team that owned the incident, and stays in the suite as a regression test." It keeps a person at merge time: review findings "do not approve or block a PR on their own, and branch protection still requires approval from a code owner." And it draws one hard line: "the agent may act up to the production gate and cannot pass it."

Its summary of where people fit is the sentence to remember: "Human attention moves up a level, to whether the change does what the plan intended and whether the risk is acceptable."

#The rule worth reading twice

In the test stage is what we think is the playbook's best idea. For bug fixes, it says: "Ask Claude to reproduce the bug as a test, run it, and confirm it fails for the reason you expect. Commit that test. Only then ask Claude to make it pass without editing the test, with the test-file hook from the final step enforcing the restriction." That hook is spelled out later: "an agent fixing code must not be able to weaken the check on that code. A hook that blocks edits to test files during a fix task does this."

That is a strong answer to a hard problem. A test that has never failed has not shown it can catch anything. By making the test fail first, and then locking it, the playbook turns "the tests pass" from a hope into evidence, for that bug.

The playbook applies this rule to bug fixes. The rest of this piece is about why we think, given the field data, it belongs on every gate. One caution before the numbers: none of the studies below measures teams that follow this playbook. They measure agent-written code in general, which is the situation the playbook is trying to improve.

#Review is now the slow part, and its record is thin

LinearB's 2026 Software Engineering Benchmarks Report, built from more than 8.1 million pull requests across more than 4,800 organizations, puts the shift plainly: "AI PRs wait 4.6x longer before review – but are reviewed 2x faster once picked up." Acceptance rates for AI-generated pull requests were 32.7%, against 84.4% for manual ones. PRLens, reporting the same benchmark, gives the absolute numbers: over 16 hours for an AI-generated pull request to be picked up, against roughly 200 minutes for everything else, with AI-assisted changes running 408 lines at the 75th percentile against 157 for unassisted ones.

Google Cloud's DORA team reached the same place from the other direction. InfoQ's summary of DORA's ROI report says the report expects a temporary productivity dip before gains, with three causes, one of which is "the verification tax imposed by reviewing AI-generated code."

The most direct measurement of review itself that we found is a May 2026 study, These Aren't the Reviews You're Looking For, by researchers at Nicolaus Copernicus University in Toruń. They looked at agent-authored pull requests in GitHub repositories with at least 100 stars. Of 33,596 such pull requests, 61.38% had no recorded review activity, and 84.0% had either no recorded review or a review only by other agents.

The authors put that number in their abstract, but they do not stop there. In repositories that had both kinds of pull request, human-written ones showed no observable human involvement at almost the same rate as agent-written ones: 69.2% against 69.9%. The authors also note that "the absence of review comments does not imply that the code was not reviewed (e.g., it may have received a silent approval)." Their conclusion is that AI-generated contributions are "associated not with the disappearance of human involvement, but with measurable changes in how review effort is expressed and documented."

So this is not evidence that agents are escaping review. It is evidence that the review record, the thing a dashboard can count, is a weak signal of whether anyone checked anything, for code written by people and by agents alike. If you want to know that a change was checked, "a review exists" will not tell you.

#When tests run, they can grade themselves

The playbook's test stage also says: "Always give Claude a way to verify its own work, whether tests, a build, or a screenshot diff." As a first loop, that is sound. A session that runs its own tests catches its own typos before anyone else has to.

The risk is in what "its own work" contains. A model asked to test code it just wrote has the code in front of it, not the requirement behind it, so it can end up testing what the code does rather than what it should do. That is our reading. A developer who built a small auditor for AI-written tests, Alexey Spinov, names three shapes this takes. The recompute: the test retypes the implementation's own formula, so it "can't disagree with the code." The golden literal: an expected value copied from one run of the code, which pins the test "to whatever the code did on day one, bug included." And the smoke test: assert result is not None, which passes if the function returns anything. His tool scored a deliberately mirror-shaped suite at 50% and an honest one at 0%, which is what makes it useful: it can tell the two apart.

Thin tests are not only an AI problem. A 2025 paper, Rethinking Verification for LLM Code Generation, found that the test suites behind popular code benchmarks such as HumanEval and LiveCodeBench "often comprise only a limited number of homogeneous test cases, resulting in subtle faults going undetected." And some people skip the step entirely. A grey-literature review of 101 practitioner sources on "vibe coding", drawing on 518 firsthand accounts, found that "QA practices are frequently overlooked, with many skipping testing, relying on the models' or tools' outputs without modification, or delegating checks back to the AI code generation tools."

Most of those failures share a fix, and it is the playbook's own: make the check fail on purpose before you trust it to pass.

#Why this is our angle

What follows is our interpretation, not a claim any of these sources makes.

The playbook tells you where each gate goes, and for bug fixes it tells you how to prove a gate works. Outside bug fixes, the playbook does not ask for a gate to be shown failing: review passes, eval suites, pre-publish checks, the scripts that mark a task done. A gate that cannot go red looks exactly like a gate that is working, and we have written before about why that is the dangerous case.

We saw it this week in our own research for this article and a companion piece. A teammate's source sheet marked a benchmark comparison as verified from a vendor's own page. One of the two numbers had actually come from a search result and belonged to a different model than the row said; none of the teammate's reads of the cited page had returned that number for that model. The sheet's verification column said "known," and nothing in the column could show that the underlying read had come back incomplete. It was caught within the hour by fetching the source again, not by reading the sheet more carefully. The teammate added a rule that we think generalizes: a comparison counts as verified only if a single read returned both sides of it.

This matters more as vendors start to sell completed work. This week's launch of Anthropic's Claude Opus 5.5 was pitched as much on cost per task as on benchmark scores. A completed task is only as real as the check that marked it complete.

#What to do with this

  • Give every gate a failing test before you trust it. The playbook's bug-fix rule, generalized: make a deliberately bad change, confirm the gate stops it, and keep that case around.
  • Lock the check while the work is being done. The playbook's hook that blocks test edits during a fix is the model. An agent should not be able to weaken the check that grades it.
  • Write tests from the requirement, not the implementation. The playbook already produces the inputs for this: intent.md and spec.md.
  • Do not read "reviewed" off a dashboard. A review record shows that someone left a trace, not that someone checked.

How this was researched: every figure above was read directly from the linked page on September 22, 2026, using direct web fetches and web search, including the full text of the review study. We did not have access to the video's content beyond its published description, so nothing here is attributed to what the video says aloud.

#Sources

Keep reading