---
title: "An 8B model beat 32B baselines by changing the harness"
description: "A separate paper found the benchmarks, not the models, were failing. Two September 2026 papers asked why models scored badly. Only one made a model better."
publishedAt: 2026-09-16
author: Ena Pragma
url: https://enapragma.co/field-notes/an-8b-model-beat-32b-baselines-by-changing-the-harness
tags: ["ai-verification", "ai-operations", "ai-benchmarks"]
---

A video this week covered two papers back to back, and the pairing turns out to be sharper than either one alone. The first had physics experts re-check model answers that automated grading had marked wrong, and found the questions and the graders were at fault far more often than the models. The second built a reasoning harness that let an 8-billion-parameter model outscore 32-billion-parameter models running the previous best methods. Both papers are real, both were submitted 11 September 2026, and we read both against their own text rather than the retelling.

They rhyme in a way worth naming precisely, and they are still not the same finding. Getting that distinction right is most of the value here.

## The benchmarks were failing, more than the graders were

The first paper is "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks" (arXiv:2609.13009), from Yale, Jump Trading Group, Cambridge, and USC. Researchers evaluated frontier models across six widely used physics benchmarks, then had physics faculty and graduate researchers audit failures. The four audit runs covered 502 questions across four of those six benchmarks, all using GPT-5.6-Sol at High reasoning effort. Expert review was then restricted to the rejections, the cases the evaluator had scored incorrect (on some benchmarks that meant a single attempt, on others all of up to five).

Of those 502, the evaluator had accepted 252 answers as correct; the other 250 were rejections sent to expert review. That review is where the number everyone quotes comes from: **238 of the 250 audited rejections, or 95.20%, were benchmark or grader errors rather than model errors.** The paper's own split matters more than the headline: 143 (57.20%) were benchmark errors, 95 (38.00%) were grader errors, and 12 (4.80%) were genuine model errors.

Read that split carefully, because it is easy to repeat wrong. The single biggest category is not a grading bug. It is **defective questions and wrong reference solutions**, where the grader was arguably doing its job against a broken answer key. And the twelve genuine model errors are twelve out of the 250 rejections experts actually reviewed, by one model, GPT-5.6-Sol High. The 252 the evaluator accepted were never re-checked, so this audit says nothing about whether any of those were wrongly marked correct.

The authors do generalize, and within physics they generalize hard: "Frontier models are not failing these physics problems. The benchmarks are failing to pose them." What the paper does not support is the leap the video reaches for, that almost all AI benchmarks on science are wrong. The 95.2% is a rate within an already-flagged rejection set on four physics benchmarks. The narrower claim is the more useful one anyway, and it is damning enough.

The correction moves scores a long way. On UGPhysics at pass@4, Fable 5 (High) went from a pre-audit 82.00% to a corrected 92.68%; GPT-5.6-Sol (High) from 85.00% to 93.90%; Gemini 3.1 Pro (High, no tools) from 88.00% to 93.90%. On the 54 retained CritPt challenges, corrected pass@4 lands at 90.74% for Fable 5 High, 94.44% for GPT-5.6-Sol (Max), and 68.52% for Gemini 3.1 Pro High (CritPt is one of the two benchmarks reviewed by a single expert per problem, as noted below). For scale, pre-audit CritPt figures were far lower, 32.29% for GPT-5.6-Sol Max and 17.71% for Gemini 3.1 Pro High, though those are mean@5 rather than pass@4, so part of that gap is the metric and not only the correction. The high numbers are the after-correction scores, not the original public ones.

It replicates outside its own methodology, which matters more than any single table. Anthropic's system card for Claude Fable 5.1 and Mythos 5.1 reports a separate expert correction of CritPt, revising 31 of 71 problems and scoring Fable 5.1 at an average pass@1 of 88.4% over 16 tool-enabled attempts. The re-grading authors call that broadly in line with their own corrected mean@4 of **87.5% for GPT-5.6-Sol Max and 78.2% for Fable 5 High**, while noting the two evaluations use different question sets, models, attempt budgets and judges, and are not directly comparable. Two independent groups corrected the same benchmark family and both found the corrected numbers much higher.

## The harness was the upgrade, not the model

The second paper is "Cognition on Graph: Navigating Massive Knowledge Space via Cognitive Cycles and Bidirectional Graph-Text Synergy" (arXiv:2609.12791), from Beijing University of Posts and Telecommunications, Zhongguancun Academy, and the Chinese Academy of Sciences institutes.

The idea is easy to state and genuinely good. Most systems treat a knowledge graph and a text corpus as two separate places to look things up. CoG makes each patch the other's blind spots at query time: a missing graph edge gets bridged by a fact found in text, and text search gets organized by graph structure. The paper calls this "deep bidirectional synergy between structured graph and unstructured text, where entities extracted from text dynamically guide graph exploration to bridge knowledge gaps." It is training-free, running as a continuous plan-explore-reflect cycle.

The control logic is the part an operator should notice. At each step the system decides one of three things: the evidence is sufficient, so stop; insufficient but useful, so continue; insufficient and useless, so change strategy. The paper contrasts this with ToG-2, which it treats as the current state of the art, not as an older approach: lacking a reflection mechanism, ToG-2 "blindly continues its beam search," pivoting to high-ranking but irrelevant entities and reaching "an irreversible dead end." (ToG-2 is a different, later paper than the original Think-on-Graph; worth keeping straight, since the two names get used interchangeably and only one of them is the comparison here.) Knowing when to abandon an approach is a different capability from searching harder.

The scale is real: a Wikidata graph of 89.3 million entities and 1.38 billion edges, against roughly 7.1 million Wikipedia articles totaling over 5 billion words.

The results, with their measurement attached. The metric is Exact Match. The four competing baselines all run on a Qwen3-32B backbone, while CoG and the plain direct-prompting control are swept across model sizes, and that comparison covers six datasets, excluding WebQSP because modern models already saturate its simple one-to-two-hop queries. On that basis, Qwen3-8B with direct prompting scores 23.91% and the same model running CoG scores 50.97%. Qwen3-32B goes from 33.36% to 57.15%, and DeepSeek-V3.2 with CoG reaches 62.64%. One caution about how these travel: that 57.15% is CoG's own 32B score, not a rival method's, and pairing it against the 8B number implies a contest that is not happening.

The striking comparison is a different one, and the paper states it outright: **"CoG-8B (51.0%) outperforms all strong 32B baselines (e.g., IRCoT 44.6%, ToG-2 45.9%)."** A better harness on a small model beat every incumbent harness running on a bigger one.

Two limits, honestly. The presenter's aside, that CoG is prompt structure and deterministic control rather than a trained cognitive model, is an editorial read rather than an authors' claim, though a defensible one: the framework is training-free, even though it does lean on existing learned components (bge-m3 for chunk retrieval, Qwen3-Embedding-4B for entity linking). And the paper was posted 11 September 2026; we found no independent reproduction or critical response, which is almost certainly its age rather than a verdict, and our search instruments for reception were weak enough that we would not call it confirmed absence.

## What the two papers actually share, and where they part

The tempting move is to call these one story. The more interesting truth is that they rhyme at one level and diverge at another.

Here is the rhyme. **Both teams asked why a model was scored wrong, and both concluded the instrument was mostly at fault.** The physics paper: 238 of 250 flagged rejections were benchmark or grader defects. The CoG authors run a smaller version of the same question over their own system's failures, attributing **71.1% to external factors**, benchmark and evaluation limits rather than model failure: 37.8% where the model completed the multi-hop reasoning correctly but failed strict string matching, 26.7% dataset flaws like typos, ambiguous prompts and noisy annotations.

Worth saying plainly, because the symmetry is tempting and only goes so far: these are not two equivalent audits. The physics review put subject-matter reviewers against six third-party benchmarks and states its denominator. The CoG figure is the authors' own manual analysis of a sampled subset of their own system's errors, and the paper does not give the sample size. One is much stronger evidence than the other, and it is still not a replication.

The physics protocol is worth reading closely too, because it is honest about its own limits in a way that proves the point of this article.

For the pooled 250, the reviewers were physics PhD students, endorsed by faculty advisors rather than being them; elsewhere the roster overlaps in a way worth noticing. Five of the twelve physics advisors appear among the thirty-nine data auditors, and so do two of the four core team members who carried out the study. The paper describes the data auditors as having conducted the expert audits and, where necessary, established or corrected reference solutions, in particular for CritPt and CMT-Benchmark; it gives no per-person assignments, so which of them touched which benchmark is not stated.

On CritPt, one of the two benchmarks outside that pooled 250, the official reference solutions were not public at all, so the reviewers solved the problems themselves to create the ground truth they then graded against. That is worth holding next to the CritPt figures quoted earlier. On both it and CMT-Benchmark, each problem went to a single reviewer, with no second independent expert label, though submissions did go to the project team for review.

Within the 250, 196 questions were reviewed twice and 54 got one review only, and where two humans did look at the same question they disagreed 28.57% of the time on a three-way label, with every disagreement settled by a third review. The reviewers were also shown an AI-generated preliminary review before labelling.

None of that undoes the finding, and the honest accounting is also the stronger one. By the paper's own Table 4, 34 of the 56 disagreements were benchmark versus grader, which land inside the 238 either way on any normal reading of that adjudication, though they do bear on the published 143 to 95 split, which for 34 of its items was decided by adjudication rather than by two reviewers agreeing. The other 22 touch the model-error boundary, and those are the ones that could matter. Even under the deliberately unfair assumption that every one of the 22 had been settled the other way, benchmark or grader would still account for at least 216 of the 250, about 86%. (That floor is our arithmetic from the paper's Tables 2 and 4, not a figure the paper states, and it is deliberately conservative. The paper never says how the 22 resolved, and at least two of them went the other way, since PRISM-Physics reports two model-touching conflicts and zero final model errors.)

So the instrument that audited the instrument has a partly measured error bar, and the paper published it. Partly, because the 28.57% is a first-pass rate, measured before a third review resolved all 56, and it covers the 196 double-reviewed questions while being structurally unmeasurable on the other 54. And the paper raises one exposure it does not measure. It reports no ablation and no agreement rate against that preliminary label, so nothing in the paper tells you whether it moved the reviewers at all. None of which is a reason to discount the finding. It is a reason to trust it for what it measured, which is the distinction this whole piece is about, and the paper is the party that made the distinction checkable.

Here is where they part, and it is the part a slogan would erase. The physics paper is about **measurement**: it corrected the scoreboard, and no model got better. The CoG paper is about **capability**: it built a harness that raised what an 8-billion-parameter model could actually do, and then separately noticed that even its improved scores are undercounted by strict string matching. One fixed the ruler. The other built a better tool and found the ruler was short too.

Collapsing those into "AI is better than we thought" would lose the useful half of both.

## What an operator does with this

If you evaluate models on benchmarks, the physics paper is a direct warning about your own evaluation stack. An automated grader nobody with domain knowledge has ever audited is an untested instrument, and this one, together with the answer keys it graded against, was wrong about 95% of what it flagged. The check is cheap and almost nobody runs it: have someone who knows the subject re-read a sample of the failures before you conclude anything about the model. Note which half of that number is yours to fix. A grader bug is yours; a defective answer key belongs to whoever built the benchmark, and you may be stuck citing a score you cannot repair.

If you are deciding where to spend effort, CoG is a real data point that the harness is undervalued relative to parameter count. One paper its authors say was accepted to EMNLP 2026, which we could not independently confirm, and no reproduction we could find, is not a law. It does mean the question is worth asking before you reach for a bigger model.

And the habit worth taking from both: when a number moves, check what changed. Sometimes the model got better. Sometimes the ruler did. Both papers this week changed an answer without touching a single weight.

## Sources

- [arXiv:2609.13009, How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks](https://arxiv.org/abs/2609.13009)
- [arXiv:2609.12791, Cognition on Graph: Navigating Massive Knowledge Space via Cognitive Cycles and Bidirectional Graph-Text Synergy](https://arxiv.org/abs/2609.12791)
- [CoG code and data, github.com/zhougengxian/CoG](https://github.com/zhougengxian/CoG)
- [arXiv:2407.10805, Think-on-Graph 2.0 (ICLR 2025), the ToG-2 baseline CoG is compared against above](https://arxiv.org/abs/2407.10805)
- [arXiv:2307.07697, Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024), the earlier paper ToG-2 succeeds](https://arxiv.org/abs/2307.07697)
- [arXiv:2509.26574, Probing the Critical Point (CritPt) of AI Reasoning, the physics benchmark re-graded in the study above](https://arxiv.org/abs/2509.26574)
- [Discover AI, "Beyond GraphRAG: Runtime Graph Repair (w/ Human Cognition)"](https://youtu.be/K2F_ViLU2Vs)