ena pragma
·5 min read·reasoning · research · evaluation

A video explained a new reasoning paper accurately, then put a number in the title that the paper never reported

We checked a popular explainer against the paper it covers. Four claims hold, two do not, and both failures are in the title. A short method for reading AI claims.

Contents

A widely shared video explains State of Thought, the reasoning paper we looked at in a separate note. We checked its claims against the paper's full text, one row at a time. Of six, three hold, two do not, and one is neither a hit nor a miss because it is not a claim at all. Both of the failures are in the title, and everything below the title is accurate.

That split is the interesting part, and it is more common than outright error. The video is a reliable guide to how the method works and an unreliable guide to what it means. Those are different skills, they fail independently, and a reader who trusts the first tends to extend that trust to the second.

#What it got right, including one thing it did better than the paper

The mechanism is correct throughout.

The paper's controller reads four signals out of the model's internal state. The video names them as geometry, velocity, consistency and uncertainty. The paper names exactly four: concentrated versus diffuse structure, stepwise progress by magnitude, directional consistency, and local predictive uncertainty. Same four. The video's framing is arguably clearer than the paper's own abstract, which compresses all four into the phrase "a compact dynamics-geometric state".

The video describes a 577-parameter evidence head plus a 5-parameter stopping operator. The paper reports "only 582 parameters", giving the total without the split. 577 plus 5 is 582. The video decomposed a figure the abstract states only in aggregate, and got it right.

It also says token usage drops by "up to 60%". The paper reports 62.6%. Rounding down is the conservative direction, so that is fair.

#The row that is neither

One framing, proprioception for language models, an internal autopilot, has no counterpart in the paper, whose term is endogenous reasoning. It is not wrong, because it was never a claim. It is analogy, and analogy is how good explanations work.

It earns its own heading because it is the row a reader most needs to be able to spot. An explainer is built from claims you could check and images you could not, and the images are doing real work: the autopilot picture is why the mechanism lands at all. The skill is not discarding them. It is knowing which sentences you could take to an argument and which you could not.

#Where it breaks, and both are in the title

"Chain of Thought is dead." The paper does not say this. It positions its method as an alternative, arguing that gains "need not come primarily from prescribing a stronger external reasoning program or from scaling search". Chain of Thought appears as one of thirteen named baselines it measures against. A method you benchmark against is a comparison, not a casualty.

"Made AI 60% smarter." This is the load-bearing error, and it is a category error rather than a wrong number. The 62.6% figure is a token reduction. It measures how much less the model wrote, not how much better it answered. The accuracy results are reported separately.

The tell is unusually clean here: the video's own description states it correctly, one paragraph below the title, as "reducing token usage by up to 60%". The body of the work knew. The headline did not inherit it.

#The part worth keeping

The paper's actual accuracy gains are much larger than 60%. Against mean baselines it reports 89.8%, 58.6% and 46.0% on three models.

So relabelling the efficiency number as an intelligence number manages to fail in both directions at once. It overstates what was measured, because 62.6% was never a capability figure. And it understates what was reported, because the real capability figures are bigger than the one being misused.

That is the shape we keep meeting. A number gets moved from the column it was measured in into a column that sounds more impressive, and the move costs accuracy in both directions.

#What to do with this

Three checks, none of which require reading the paper.

Ask what the unit is. Percentages are not interchangeable. A percentage of tokens, a percentage of accuracy and a percentage over a baseline are three different quantities, and a headline that does not say which one is not yet a claim.

Read the source's own description against its title. Here they disagreed, and the description was right. When a summary contradicts its own headline, the headline is usually where the pressure was applied.

Separate mechanism from significance. Explanations of how something works and claims about what it means have different failure modes. An explainer can be excellent at one and unreliable at the other, and it usually will not tell you which mode it is in.

None of this requires distrusting the source. The video is a good explanation of the method. It is just not a source for what the method proves, and those are separate questions about the same forty minutes of material.

#Provenance

This compares the video's published title and description against the paper's full text. No transcript was captured, so claims here rest on what the video published in writing rather than on its spoken content. Anything resting on precise wording would want rechecking against a transcript. The substance of the four technical rows does not depend on it.

#Sources

Keep reading