---
title: "The false finish: agents don't just fail, they stop early and call it done"
description: "A frontier benchmark caught AI agents quitting at 75-87% complete while reporting success. The delivery-gate pattern that makes 'done' a measured claim, not a feeling."
publishedAt: 2026-07-14
updatedAt: 2026-07-14
author: Ena Pragma
url: https://enapragma.co/blog/the-false-finish
tags: ["ai-verification", "evals", "methodology"]
---

Ask an agent to do ninety minutes of real work and the interesting failure is not the crash. It is the confident stop. The agent tidies up, reports success, and exits, with a fifth of the job still undone.

Last week a benchmark finally measured this at scale. [Long-Horizon-Terminal-Bench](https://arxiv.org/abs/2607.08964) (LHTB), from Tencent's HY LLM Frontier team with collaborators across seven universities, put 17 frontier models through 46 containerized terminal tasks: real multi-step workloads in software engineering, scientific computing, systems administration, and professional document work, each with a 90-minute budget and one attempt.

The headline results are humbling on their own. The strongest model, Grok 4.5, resolved 13 of 46 tasks. Ten of the 17 models finished zero tasks perfectly. But the number that should change how you run AI in production is buried in the failure analysis.

## One in five failures is an agent that thinks it's done

LHTB grades every run on a continuous 0-to-1 reward built from weighted subtask checks, so it can see exactly where an agent stopped. Decomposing the unresolved runs: 79% were still working when time ran out. But 19% were early exits, and inside those the LHTB team names the failure mode they call the "false finish": agents that stop at a reward of 0.75 or higher, believing the job is complete. Fourteen runs quit with roughly twenty minutes still on the clock. On one legal-document task, seven different models stopped between 0.80 and 0.87 and reported done.

<Callout>The bottleneck the authors name is not local reasoning. It is weak self-verification: the agent cannot reliably tell the difference between "I finished" and "I stopped."</Callout>

This is the [producer-grading-its-own-homework problem](/blog/accuracy-is-the-wrong-bar) showing up inside a single agent trajectory. The model generates the work and the model certifies the work, and the certification fails exactly when it matters.

## Binary grading hides all of this

Here is the measurement insight worth stealing. Of LHTB's 782 total runs, 62.8% earned real partial credit that a pass/fail grader would score as zero. Near-misses outnumbered full passes: 90 runs landed between 0.75 and 0.95 against 50 that passed at 0.95 or above. A binary gate cannot distinguish an agent that did nothing from an agent that got 87% of the way there, and it cannot see a false finish at all, because both look like "fail."

The fix is not new, and the LHTB paper does not claim it is. [METR's RE-Bench](https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/) used continuous 0-to-1 scoring for AI R&D tasks in late 2024. [Cybench](https://arxiv.org/abs/2408.08926) broke security tasks into gradable subtasks. OpenAI's [PaperBench](https://arxiv.org/abs/2504.01848) decomposed paper replications into 8,316 individually gradable requirements with weighted partial credit. What LHTB adds is difficulty headroom, arriving right as the original Terminal-Bench [saturates near 90%](https://artificialanalysis.ai/evaluations/terminalbench-v2-1), and one design detail that matters more than the rest: most of the reward weight sits on hidden, deterministic verifiers the agent never sees, so the visible happy path cannot buy a passing grade.

We have written before about why [a benchmark score is not reliability](/blog/ai-benchmark-scores-reliability). This is the constructive half of that argument: grade the trajectory, weight the hidden checks, and set the bar where "done" actually lives.

## The delivery-gate pattern

Strip the research packaging and there is a pattern here any operator can run. We call it a milestone rubric, and we now apply it to long-horizon delivery work as a standing gate.

A deliverable gets a rubric before the work starts. The rubric is a weighted list of milestones, and every milestone is a deterministic check: a command that exits pass or fail, or emits a score. No judgment calls carry weight. Then four rules make it a gate instead of theater:

1. **Hidden stress checks carry at least 40% of the weight.** Planted defects, edge inputs, absence-of-leak scans. The visible happy path alone cannot reach the bar.
2. **The rubric is written before the work completes.** Expectations come from the claim, never from reading the finished artifact and describing it back.
3. **Weights freeze after a red run.** If the gate fails, you fix the work, not the grading. Editing expectations to match a failing reality is how drift gets blessed.
4. **Resolved means 0.95 or better.** Partial credit is visibility: it tells you exactly what is missing and how far you got. It is never ship authority. A 0.86 is "not done, and here is the list," not "close enough."

That last rule is the false-finish kill. An agent, or a person, or a team cannot declare victory at 80% when the definition of done is a number a script computes.

## The receipt

We adopted this the same afternoon we read the paper, and we can show the gate working because the first thing it did was refuse to pass its own builder.

We built a synthetic accounts-payable reconciliation task, invoices in mixed units, a contracted price book, a receiving log, and five planted traps including a duplicate invoice and a line item whose weight cannot be determined and must be flagged rather than guessed, on [Harbor](https://github.com/laude-institute/harbor), the Apache-2.0 open-source harness that runs Terminal-Bench 2.0 and grades with exactly this kind of continuous reward. Then we ran the two-way checks that make a verifier trustworthy:

- The reference solution scored exactly **1.0**.
- A sabotaged solution with a planted unit-conversion defect scored **0.759**. Caught.
- A no-op agent that touches nothing scored **0.0**. No credit for showing up.

And the adoption itself was graded by its own rubric, written before the build. At the moment of writing, that rubric reads **0.85, NOT RESOLVED**, because one milestone, a live agent run, had not yet executed. Five green checks felt like done. The gate said otherwise, and the gate was right.

<Stat value="0.85" label="what our own adoption scored on its own rubric: five milestones green, still NOT RESOLVED" />

That is the whole pattern in one line: the system that certifies completion must be separate from the thing doing the work, must be written before the work, and must be allowed to tell you no.

## Epilogue: the gate flipped

Hours after this post first went live, the missing milestone ran. A live Claude agent executed the task end to end inside the container, was graded blind by the hidden verifier, and the adoption rubric recomputed: **1.0000, RESOLVED**. Done stopped being a feeling and became a measured claim, the same afternoon the gate refused to fake it.

The agent's own score is worth reporting too, because it demonstrates both sides of the argument. It scored a perfect 1.0 on the task in just over five minutes: caught the duplicate invoice, kept the credit memo's negative quantities signed instead of dropping them, flagged the off-contract prices and the unknown line item, and, the detail that matters most in real operations, it refused to guess the weight of the one line whose unit could not be determined from the data, flagging it as an exception exactly as the spec demanded. An agent that flags what it cannot know is the behavior every operator should be gating for.

And the honest caveat, because a verification post does not get to skip it: a perfect score also means this particular task no longer discriminates at the frontier. That is not a flaw in the pattern, it is the reason to own the harness. When the bar is yours, you get to raise it.

## What to do with this

If agents do long-horizon work anywhere in your operation, take the ninety seconds to ask how "done" gets decided. If the answer is that the agent says so, you are running on false finishes and finding out later, in production, from a customer. Decompose the deliverable into checkable milestones, hide the stress checks, set the threshold, and let a script keep the score.

The frontier labs just spent roughly ten million tokens per attempt proving that agents stop before the work is done and believe otherwise. The countermeasure is not a smarter agent. It is a gate that does not take the agent's word for it.