"No problems found" and "I never looked" are the same green

Anthropic's AI-native SDLC playbook is a map of where checks belong. Requirements become intent.md, design becomes spec.md, work becomes plan.md, then tests, agent evals, code review, hooks, deployment checks, production monitoring, and the issues feed back into the next plan. We wrote about that map three weeks ago, and the argument there was that faster models do not remove bottlenecks, they move them into the stages that still run at human speed.

We also argued, a month ago, that a check that cannot fail is not a check.

This note is the next claim, and we only have it because a week of running that playbook produced nine cases that neither earlier piece covers. The first claim is about the guard: a check must be able to fail. This one is about the answer:

A check must be able to say "I cannot answer this." Most cannot, and when they cannot, "nothing found" and "never looked" arrive as the same green.

#The case that made us write it

A writer on our team was assigned a blog post on the AI-native SDLC playbook. They proposed a thesis, pitched it as the angle they had found, and started work.

Both already existed on this site. The subject was their own article from three weeks earlier. The thesis was the title of another one from the month before.

Nothing in the publishing pipeline objected. They found it because a different subsystem refused: a memory tool would not let them file a ticket without running a search first, and the search returned their own article. They then did the step most people skip, which is that they opened the live page and read it, rather than trusting the index row that said it existed.

So we measured why the publishing gate had been silent. Our pre-publish script runs twenty checks. It validates frontmatter keys, dates, em-dashes, link presence, a sources section, description length, filename convention. Zero of the twenty can tell an author that a draft duplicates a published post, and the skill document never tells anyone to look.

Every one of the twenty checks is about the shape of a draft. None is about whether the draft is needed. A duplicate passes all twenty perfectly, because it is well formed.

#Nine shapes of a green that cannot fail

These are ours, from one week, each with the measurement that caught it.

1. A gate whose success criterion is not the thing it protects. An installer of ours installs a transcript library, confirms import youtube_transcript_api succeeds, and reports "capture dependency importable." That sentence is true. It is also true on machines where capture is impossible, because the platform refuses our datacenter IP address. Two separate true facts. What the check cannot say is "I verified an import, not a capability."

2. A receipt that declares instead of measuring. The same tool writes a capture receipt with four fields read from the run and one hardcoded string: the method field. The one value that would let anyone reproduce or bisect a capture is a literal, so a good run and a broken one are indistinguishable in the record.

3. A test suite green against a fake. A reviewer found tests passing because the fixture implemented four fields the real API does not return. The harness agreed with the code instead of with production. The rule we adopted: a fixture must reject what the real tool rejects.

4. A red arm that went red for the wrong reason. An engineer broke a guard on purpose to prove the guard was load-bearing. The test went red, so every downstream signal said the guard worked. It had gone red because the mutation crashed on a null value, not because the assertion caught anything. Red for a reason that proved nothing. Compiling the mutant catches one that cannot run; nothing but reading catches one that runs the wrong experiment.

5. A runner that printed success over a syntax error. The same engineer, minutes later, found their own mutation runner reporting "mutant compiles" while the compile had thrown a SyntaxError, because the compile's exit status was never checked.

6. A verified write of already-wrong bytes. A shell backtick executed as a command substitution and removed a key identifier from a message before it was sent. The API then confirmed the write: bytes sent equals bytes stored. Honest, and useless. It verified transport while the content was already wrong.

7. A read-back that proves landing, not survival. Two writers set the same field within thirty seconds. Both read their own write back. Both read-backs were true when taken. The second silently replaced the first. The fix is a baseline: assert was X, now Y, not now equals what I expected.

8. A document that omits its own reader. A per-machine config file is generated for each machine and every copy leaves out the machine it was generated for. On one host, the entry for that host's own service named two other hosts, one of them switched off two days earlier. Complete, correctly formatted, confident, and missing exactly the row the reader is standing on.

9. A confident number from a truncated source. An engineer measured a repository's history and got three commits and one contributor, then built a conclusion on it and escalated. The clone was shallow. The real numbers were 395 and ten. The command was correct. The object it answered about was not the one anyone cared about, and it could not report that it had been truncated.

#The pattern, and the part that is already settled

Our own canon already covers the diagnosis: a probe blind to something returns the same value whether that thing is present or absent, so you run it against a known positive before trusting any empty result. That is the control, and it works.

The new claim is about who carries the burden. A control is external. It requires that you already suspect the check. The nine cases above were all found by someone becoming suspicious, usually late, sometimes after telling a colleague something false.

A check that can return a third value does not need you to suspect it.

#Some of our tools already do this

This is not an argument that checks are worthless. Half of ours refuse correctly, and the half that do were built by people who had already been burned.

• Our fabric search prints "no WAL at this path. This is NOT 'no matches'", unprompted. Whoever wrote it anticipated the exact misreading someone was about to make, and it ships that way on every machine.
• Our publish gate exits with a named reason when a dependency is missing, rather than skipping quietly. It also now reports its skips individually, so an incomplete run is visibly incomplete.
• Our recall tool returns measured: false and an index age alongside its verdict, so you can see whether the answer rests on anything.
• And the best one: a worker was handed a specification citing a rate limit that did not exist. It refused to invent compliance with a rule it could not find.

That last case is the shape to copy. The check did not fail and it did not pass. It said the question could not be answered as asked.

#What to do on Monday

Four things, none of which require adopting anyone's playbook.

Give every check a third return value. Pass, fail, and could not determine. Then make callers handle the third one. Most codebases have exactly two, so every unknown gets rounded into whichever of the two is quieter, and the quieter one is almost always pass.

Make empty results prove they looked. Any check that can return "nothing found" should report what it searched: how many records, which index, how old. An empty result with no denominator is not a finding.

Break your red arms on purpose, and read the output. Not "does the test fail when I break the code" but "does it fail for the reason I think." Case 4 above passed the first test and failed the second.

Ask which machine the check ran on. Case 1 is invisible until someone asks that. A check that passes on the build box and skips in the place work actually happens is a check you do not have.

#The honest ending

Case 5 in that list is a check on a check that printed success over an error. Which means the fix for a check that cannot refuse is another check, and that one needs verifying too.

It is checks all the way down until somebody reads the output with their own eyes. The playbook cannot remove that person. It can only make sure they are looking at something worth reading.

#Sources

• Anthropic, Claude Code documentation
• Anthropic engineering, writing effective tools for agents
• Our earlier note on the AI-native SDLC and where bottlenecks move
• Our earlier note on checks that cannot fail
• Why accuracy is the wrong bar