ena pragma
·14 min read·ai-verification · methodology · agent-architecture

Spec-driven development has early support. The course's biggest promises are barely tested.

A DeepLearning.AI and JetBrains course teaches writing the spec before an AI agent writes code. Benchmarks back clearer specs. Its own workflow is barely tested.

Contents

Anyone who has asked an AI coding agent to build something has met the same problem: it builds quickly, and it builds something slightly different from what you meant. A short course, Spec-Driven Development with Coding Agents, offers a fix. It was "built in partnership with JetBrains, taught by Paul Everitt, Developer Advocate at JetBrains", and is published by DeepLearning.AI. Its pitch is that writing the specification first, before the agent writes any code, "produces better results than vibe coding", the habit of steering an agent with loose prompts and hoping.

The idea has real support: in benchmark studies, clearer specs help agents. But the course's own workflow has barely been tested against working without a spec, and the one number we found attached to its promise does not survive a check of its own sources. Here is what spec-driven development is, what the evidence shows, and where it still breaks.

#What spec-driven development is

The course's workflow has three parts. First, a project "constitution": the mission, the technology stack and the roadmap, written down once so every agent session starts from the same place. Second, a loop for each feature: plan it in a written spec, let the agent implement it, then validate the result with a person in the loop. Third, periodic replanning, where the constitution and roadmap are updated as the project moves.

The tools are moving fast. GitHub's open-source Spec Kit had 138,893 stars and shipped version 1.0.12 on September 25, 2026. Kiro, from AWS, builds its coding agent around specs, and Tessl takes its own approach. GitHub's launch post for Spec Kit states the ambition plainly: a move from "code is the source of truth" to "intent is the source of truth."

We have written about the pieces around this before: the Agent Client Protocol and the context an agent carries, and Anthropic's playbook for the AI-native software lifecycle, which also puts a spec and a plan ahead of the build. This note is about the promises.

#The course's promises, checked

The course page lists three: specs "preserve context across agent sessions, improve intent fidelity, and reduce cognitive debt." A caution that applies to everything below: most of the research studies cited here are 2026 preprints on arXiv, not yet peer-reviewed; where one has been accepted at a venue, we say so.

1. Preserve context across agent sessions. The files do persist, which is real value: a new session can read them. Whether the agent then uses them is a separate question. Birgitta Böckeler, testing three spec tools for martinfowler.com in October 2025, described existing code in a spec and found "the agent ignored the notes that these were descriptions of existing classes." François Zaninotto of marmelab calls the related failure "Context Blindness": spec agents "often miss existing functions that need updates." One study has measured a fix for that repository-level blindness: Spec Kit Agents added hooks that ground each stage of a Spec Kit pipeline in the actual code, and judged quality rose by "+0.15 on a 1-5 composite LLM-as-judge score." That is a small gain, and it compares one spec pipeline with another rather than specs with no specs. There is also a ceiling on how much a model can follow at once: the July 2025 IFScale benchmark found "even the best frontier models only achieve 68% accuracy at the max density of 500 instructions", with a "bias towards earlier instructions." IFScale tested models on simple keyword instructions, not specs, so it is an indication, but a long spec loaded into every session may move the forgetting rather than end it.

2. Improve intent fidelity. This is the promise that matters most, and here the evidence splits in two. Narrow benchmark studies do find that clearer specs help agents build the right thing:

  • A study of multi-agent code generation stripped specification detail "from full docstrings (L0) to bare signatures (L3)". A single agent's accuracy fell "89% to 56%". In a recovery experiment with two agents each building part of the same class, "restoring the full specification alone recovers the single-agent ceiling (89%)."
  • CodeSpec compiles requirements into executable specifications. On a feature-development benchmark, removing all of its specifications cut its pass rate from 70.7% to 62.6% in the paper's own ablation test.
  • PaperCompiler turns research papers into repository specifications and reports "a 13.8% relative improvement in reference-based fidelity (from 3.64 to 4.15)", a model-judged 1 to 5 score of how closely the generated code matches the authors' own implementation.
  • In a study evaluated on production bugs from Google, an agent first wrote down each function's pre-conditions, post-conditions and undefined behaviors, then generated tests. That delivered a 9.8 percentage point improvement in bug detection rate over a standard test-generation agent (p = 0.0352). That is a controlled comparison, but of test writing, not of building features; the paper is to appear at the SpecOps 2026 workshop.

None of these is the course's method: a person writing a constitution and feature specs, reviewing each step, on a real project. We found one direct test of that kind of workflow against vibe coding, and it is small. A single-author case study built one banking application twice: once following the author's constitution-first spec method, and once without it, as what the author calls the "vibe coding" baseline, with "the same AI assistant (Claude) and the same developer." It reports that "constitutional constraints reduce security defects by 73%", which is 3 security violations against 11. It is one application, measured only on security, and its own threats section notes "a single development team with prior security training" and a possible "Hawthorne effect." No study we found tests intent fidelity this way. Another case study of one engineer working with four agents reports a project delivered "in half the planned time, with 90% acceptance of AI-generated code on first review."

The one number we found attached to the promise comes from a January 2026 technical report on arXiv, which says "Empirical studies [6, 5], though nascent, suggest that human-refined specs significantly improve LLM-generated code quality, with controlled studies showing error reductions of up to 50%." Neither source it cites contains that figure or a controlled study. The InfoQ article has no 50% figure. The Red Hat Developer article says the approach reduces errors, but its only figure is a goal: "Think 95% or higher accuracy in implementing specs on the first go."

The researchers themselves are candid about the state of play. An August 2026 paper on spec-driven development presents its framework "Given the immaturity of the evidence base" as "a first step" rather than "a validated theory." CURRANTE, a registered report whose protocol was accepted at SANER 2026, should shed more light on how people refine specs with AI; its results are still to come ("The results will provide empirical insights").

Self-reported speed-ups deserve the same care. In METR's 2025 study, experienced developers "expected AI to speed them up by 24%", yet with AI tools they took "19% longer." In February 2026 METR said its newer data "gives us an unreliable signal", while adding that "it is likely that developers are more sped up from AI tools now" and that its "raw results show some evidence for speedup." Careful researchers are still unsure how big the effect is, and a course or a vendor case study cannot settle it. After building with Spec Kit, Colin Eberhardt of Scott Logic went the other way: "I am a lot more productive without SDD, around ten times faster." That is one person and one project, which is the point: anecdotes run in both directions.

3. Reduce cognitive debt. The idea is that a written spec keeps people understanding what the agent built. We found no study that tests this. One classroom report, accepted at CSEE&T 2026, on third-year undergraduates using spec-driven development found that "while AI agent utilization increased implementation throughput, it also tended to encourage students to proceed with development without fully understanding the code." In that course "AI agents generate documentation and code during each phase", so the report cannot separate the effect of the spec from the effect of the agent, and it had no concurrent control group, only comparisons with earlier years. Its own conclusion is about people, not documents: "regular verification of code comprehension by instructors and appropriate feedback are essential."

#Consistency, measured

Spec Kit's own guide says that when "specifications drive implementation, pivots become systematic regenerations rather than manual rewrites." Böckeler saw the other side: "I have seen the non-determinism in action though, when I generated code multiple times from the same spec." That was in Tessl's spec-as-source mode, and her fix was to "make it more and more specific to increase the repeatability of the code generation."

A single-author preprint from June 2026 measured consistency directly, as lexical similarity between runs across 840 implementations and two models. It compared Spec Kit, OpenSpec and a framework called traceSDD, which makes every line of code cite the requirement it serves. On consistency, the paper reports "A clear ordering emerges in both models: uncited >> cited >> OpenSpec >> Spec Kit." Spec Kit came last. The citations had a use: when the author deliberately injected hallucinated lines citing fake requirement IDs, an automated check caught 86.4% of them with Claude and 88.0% with GLM, where the other setups had nothing to check. The author's conclusion: "citation annotations trade determinism for verifiability." It is one author, two small Python task sets and a similarity measure that also counts formatting, so treat it as a first measurement, not a verdict.

#Where advocates and critics disagree

Spec-driven development is not one settled method. Four disagreements matter to a buyer.

  • Which is the truth, the spec or the code. Spec Kit's guide says specifications "don't serve code" and that "code serves specifications." Eberhardt: "Code is law because it is formal language you can reason about. You can test it."
  • Whether to keep the spec after the build. Böckeler separates "spec-first" from "spec-anchored", where "The spec is kept even after the task is complete." Watching how Spec Kit behaves in use, she concluded that "spec-kit is still what I would call spec-first only." Eberhardt, six weeks later and going by how Spec Kit describes itself, wrote that it "adheres to SDD in the purest form (spec-as-source)." The same tool lands at opposite ends depending on whether you judge its stated aim or its behaviour. Eberhardt also asked, "how often do you re-visit a user story? Very rarely."
  • How big a spec should be. On a small bug fix, Böckeler found "the workflow was like using a sledgehammer to crack a nut", and "I'd rather review code than all these markdown files." Zaninotto cites a Spec Kit example "where a developer wanted to display the current date on a time-tracking app, resulting in 8 files and 1,300 lines of text." Kiro has since added smaller Bugfix and Quick specs, so some of these critiques describe the tools as they were in late 2025.
  • Whether it is waterfall again. Zaninotto calls it "an echo of the Waterfall era." Marc Brooker, an engineer at AWS, which makes Kiro, answers that it "isn't about pulling designs up-front, it's about pulling designs up", toward "explicit, versioned, living artifacts."

Even up-front planning itself is contested. Böckeler: "I'm very skeptical that lots of up-front spec design is a good idea, especially when it's overly verbose." Eberhardt: "I don't consider it a viable process, at least not in its purest form."

#The throughline, as we read it

What follows is our interpretation, not a claim any single source makes.

What. Clearer specs help agents; the benchmarks agree on that much. The unsolved part is keeping the spec and the code in agreement after the work ships.

Who. Tool makers, course providers, researchers and critics are all shaping the practice now, and the promises are running ahead of the measurements.

How. Two older practices show ways to keep a written record honest. One is a mechanical check in both directions. In aerospace software, a Parasoft guide to DO-178C traceability says that if source code "can't be traced to a requirement, then it's a risk and shouldn't be there", and "Maintaining traceability records on any sort of scale requires automation." (Parasoft sells traceability tools, so read that as an interested source.) The single-author preprint above tried a version of that idea with coding agents: requirement citations made planted errors detectable, at a cost in consistency. The other is to keep the record short and replace it rather than patch it: Michael Nygard's architecture decision records run "one or two pages long", and when a decision is reversed, the old record is kept but marked "superseded". Today's spec tools mostly rely on prose specs and human review. Spec Kit's converge command, added in June 2026, is a step toward the first approach: it flags code "not called for by the spec, plan, or tasks." But a model makes that judgment, and the command's instructions say it "MUST NOT" modify spec.md or plan.md, so it checks the code against the spec and never the spec against the code.

When. Now, because agents write code fast enough that the spec is where a person still has control. The spec is also where drift starts.

Why it matters. A spec that no longer matches the code can be worse than no spec, because people and agents both trust it.

#What this means for a business adopting it

  • Adopt it for the planning, not for a promised error rate. Writing down what you want before an agent builds it is sound practice, and benchmark studies support clearer specs. The one direct test we found of the full workflow against vibe coding is a single small case study, and the 50% figure should not appear in any business case.
  • Turn spec lines into checks where you can. Several of the measured gains involve specs that become something executable: pre-conditions that drive tests, or specifications a program can check. A requirement that stays prose depends on someone rereading it.
  • Keep specs small. A one-page spec for a bug fix is scaffolding; eight files for a date display is overhead. Size the spec to the change.
  • Measure it on your own work. Run a few comparable tasks with and without a spec and compare rework and review time. That tells you more than any course or vendor figure. It is the same principle behind our guide to reading AI claims: a number you cannot trace back is a claim, not a result.

How this was researched: our research teammate ran the source search using Brave Search and Firecrawl and wrote a graded claim sheet. We then re-read every quoted source directly on September 25, 2026, including the course page, the arXiv report and both of its cited articles (the Red Hat article through its Internet Archive copy, because the live page refused our request), and the critiques and tool documentation. An independent reviewer then ran a handful of arXiv searches, not an exhaustive literature review, and found the 2026 studies that the web search had missed. We read their abstracts and checked the specific passages we quote in the full text of four of them; the independent reviewer read further. Star counts and version numbers are as of September 25 and change weekly. We did not watch the course videos.

#Sources

Keep reading