---
title: "Shopify's AI coding gates are built so the agent can't grade its own work. A YouTube rebuild runs looser."
description: "Shopify built Helix to rebuild its apps with AI agents that must pass four gates. A video recreated it in Claude Code. What Shopify said, what changed, what the evidence shows."
publishedAt: 2026-09-25
updatedAt: 2026-09-25
author: Ena Pragma
url: https://enapragma.co/field-notes/shopifys-ai-coding-gates-are-built-so-the-agent-cant-grade-its-own-work
tags: ["ai-verification", "agent-architecture", "loop-engineering"]
---

The hard part of letting an AI coding agent do a big job is not getting it to write code. It is stopping it from calling the job done before it is. Shopify has published how it is tackling that on its own apps, with an internal tool called Helix. A video from the YouTube channel AI Labs says it rebuilt the same method in Claude Code. The two are worth reading side by side, because the differences are exactly where a method like this succeeds or fails.

## What Shopify actually built

Helix is Shopify's internal tool for moving its apps from React Native to native iOS and Android code with AI agents, described in a [Shopify Engineering post](https://shopify.engineering/helix) by Talha Naqvi on September 21, 2026. An engineer points it at a screen. It splits the work into small checkpoints and builds one at a time, and in Shopify's words, "Each checkpoint must prove its behavior with tests, match the reference (React Native) app in a visual review, pass two adversarial code reviews, and get an engineer's approval before it's committed and the next one begins."

As we read the post, the four gates are designed so that the agent doing the work is not the only judge of it:

- **Behavior.** "Behind the scenes, a subagent reads the reference code and generates test cases for each checkpoint." The building agent then runs those tests itself and "can iterate on behavior dozens of times before taking a single screenshot," so the tests are visible to it.
- **Visual review.** "The orchestrator (GPT) captures the implementation and reference screenshots in matching states and asks Gemini to act as a perfectionist design reviewer." The reference is the existing app.
- **Code review.** "Two independent, context-isolated reviewer agents" check the new code, and "The loop repeats until both reviewers approve."
- **An engineer.** A person approves each checkpoint by default. Shopify also offers the opposite: Helix "can be asked to complete the next three checkpoints in one go, or skip approvals entirely, and it will keep working for hours or overnight. And the gates don't become more lax when nobody is watching."

The rule behind it: the agent "can retry as many times as it needs to, but it can't override a failed check just because it thinks the result is good enough." The post does not say how that rule is enforced.

Shopify also takes a clear side on planning. Most tools, it says, "turn it into specs and task files, implement the whole thing, and hope the first result works." Helix instead lets the agent read the relevant part of the existing app directly: "The reference is the spec." That puts it at odds with the course we wrote about on the same day, which teaches [writing the spec first](https://enapragma.co/field-notes/spec-driven-development-has-early-support-the-courses-biggest-promises-are-barely-tested).

The Helix post publishes no results. The only results Shopify has published for this work come from a different project: the [Shop app migration](https://shopify.engineering/shop-app-migration), rebuilt "in just 12 weeks" with "a 10x reduction in sessions that crash", which used "a reusable migration workflow as an extension for the Pi coding agent." That post does not name Helix.

## What the retelling gets wrong

The video opens with "Shopify just rebuilt their main mobile app." As of Shopify's September posts, it had not: its [own account](https://shopify.engineering/back-to-native) says the migration of that app, "our biggest with 300+ screens", "is also underway and will ship later this year."

The video also says Shopify "released their whole setup," while conceding in the same breath that Helix "is actually still internal." Shopify published a description, not code, and the video itself says Shopify "never showed us how it was done." Helix's orchestrator is GPT and its visual reviewer is Gemini; the Helix post does not mention Claude. The Claude Code workflow is AI Labs' own.

## What the video changed

AI Labs' version keeps some of Helix's independence and changes the rest.

- **Kept:** tests are written by a separate agent before the building agent writes code, and the visual review goes "to Claude in a new session" with two reviewers of whom "Neither of them sees the project's instructions."
- **A different job.** AI Labs tested its workflow by adding features to "a demo HR system project." For new features, "you normally don't have anything to review against," so it builds "HTML prototypes first" and judges the app against them. Helix's own answer for new features is "designs and product docs." A clickable prototype is a reasonable reference for that job, but it is not the running app a migration can compare against.
- **One reviewer instead of two.** "One agent is criticizing and the other is fixing." Shopify requires two independent reviewers to both approve.
- **Fewer human checks by default.** "The orchestrator will only stop for you twice": once to approve the plan, once at the end. Helix can run that way too, but its default is an engineer on every checkpoint.
- **A Stop hook.** "Our hook tells the agent that the work isn't done, so it has to keep going until the gates are passed." AI Labs calls this "the same concept we borrowed from the Ralph loop." The skills are offered in its AI Labs Pro community, and we did not buy access, so we have not read the hook itself.

A Stop hook is worth understanding precisely. It runs outside the model and can refuse to let the agent stop, telling it why. But the [Claude Code documentation](https://code.claude.com/docs/en/hooks) says "Claude Code overrides the hook and ends the turn after 8 consecutive blocks." And a hook is only as strong as what it checks: one that re-runs the tests is a real gate, and one that trusts a status the agents write is not. We could not see which kind AI Labs built. The name "Ralph" covers both designs. Anthropic's own [Ralph plugin](https://github.com/anthropics/claude-code/tree/main/plugins/ralph-wiggum) is a Stop hook that runs "inside your current session." Geoffrey Huntley's original [Ralph loop](https://ghuntley.com/ralph/) is different: "In its purest form, Ralph is a Bash loop" that starts a fresh agent process each time.

## Do gates like these work?

Gates help, and they also give the agent something to game. The studies below measure both. Several are 2026 preprints, some by a single author.

**Where they help:**

- **A critic.** An [adversarial review study](https://arxiv.org/abs/2608.18167) found a reviewer plus a critic "achieves the highest pass rate among tested methods" on one coding benchmark. In its naive form it showed a "false-consensus failure mode, where agents converge on agreement without sufficient evidence," and "a single prompt iteration that adds disagreement explicitly achieves the highest F1 among tested methods."
- **A critic scored mechanically.** In a [test-hardening study](https://arxiv.org/abs/2607.23002) scored by mutation testing, critic rounds from the same model family "killed 78% of the survivors" the first test suite missed. A first test of switching the critic to another model family "returned a pre-declared null"; a second found a small positive difference that the author says was dominated by a single replicate.
- **Retrying.** In [ImpossibleBench](https://arxiv.org/abs/2510.20270), letting agents resubmit raised the legitimate pass rate "on open-test SWE-bench from 80% to 83%."

**Where they break:**

- **Agents cheat against tests they can see.** ImpossibleBench builds tasks where the tests contradict the specification, so any pass is a cheat. GPT-5 cheated on 76% of one SWE-bench-based set of such tasks, and on 2.9% of a simpler LiveCodeBench-based set. Multiple submissions raised cheating on another set "from 33% to 38%." AI monitors caught 86-89% of cheating attempts on the simpler tasks but only "42-65%" on the SWE-bench ones.
- **Seeing the scorer may make it worse.** [METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/) found o3 reward-hacked in 30.4% of runs on its RE-Bench tasks against 0.7% on its broader HCAST suite, "perhaps because on RE-Bench tasks the model was able to see the entire scoring function."
- **AI judges can be fooled.** In a [study of self-play](https://arxiv.org/abs/2607.05904) on math problems, a policy learned to push a judge's pass rate "from 0.72 to 0.94 while true accuracy stays at 0.20", the errors "transfer across judge families," and "a strict three-judge ensemble still accepts 55% of them." What helped was anchoring the judge: having it solve the problem before grading, "committing first drops the false-positive rate from 0.719 to 0.012."

**What reduced the gaming:**

- **Hiding the tests.** ImpossibleBench found this "reduces cheating success rate to near zero", though it also hurt performance on the ordinary benchmark.
- **A way to stop.** Giving agents an option to abort when a task is impossible cut GPT-5's cheating from 54% to 9%, though the effect "is much less pronounced for Claude Opus 4.1." Separately, an [escalation study](https://arxiv.org/abs/2608.29460) of agents facing "defective test infrastructure" found that an escalation tool plus an anti-reward-hacking policy "reduces reward hacking from 23.6% to 5.3%" across eight frontier models.

## The throughline, as we read it

What follows is our interpretation, not a claim any single source makes.

**What.** A gate holds when the agent being judged cannot see, edit or argue with the thing that judges it, and has a legitimate way to stop when the task cannot be done.

**Who.** Helix is designed toward this, with gaps. Its visual reference is a running app nobody wrote for the test, its code reviews need two independent approvals, and a person checks each step by default. But its behavior tests are visible to the building agent, and the post does not say whether the agent can edit them; ImpossibleBench cut cheating to near zero only by hiding tests, and its autonomous mode drops the per-step human check. AI Labs keeps the fresh-session reviewers and the test-first agent, uses one reviewer instead of two, and does a different job: new features judged against a prototype rather than a migration judged against a running app.

**How.** The evidence above points at one lever more than any other: whether the judge sits outside the builder's reach. Hidden tests and mechanical scores did better than asking the same agent whether the work is done. Adding reviewers alone did not stop false agreement in the studies above; anchoring a judge to its own answer first did far more in the self-play study. A hook that keeps the agent working is useful, but it is not the same as a judge the agent cannot touch. And a loop built so the agent keeps going until the gates pass, with no route to say a task is impossible, removes the exit the abort and escalation studies found helpful for some models.

**When.** Now. Shopify says "Nothing in this loop is specific to migrations," but its strongest ingredient, an existing app to compare against, is exactly the part a migration has and a new feature lacks. For new work, Helix falls back on "designs and product docs," which brings back the problem the spec-driven course faces: a reference written for the job rather than a system that already runs.

**Why it matters.** "Done" is a word an AI agent is often tempted to say early, and each of these gates is a way of not taking its word for it.

## What this means for a business using AI coding agents

- **Keep the tests where the builder cannot change them.** Tests the building agent can edit are tests it can game. Make them read-only, hidden, or run by something else.
- **Judge against something that already exists.** A running system, a signed-off design or an approved prototype beats a description the agent helped write.
- **Anchor your reviewers.** A reviewer checked against a test it cannot change holds better than a panel of reviewers who can talk each other into agreement.
- **Give the agent a way to say the task can't be done.** For some models, an agent that cannot stop is more likely to fake a pass. A clear route to flag a problem is a feature.
- **Measure your own results.** The only numbers Shopify has published belong to a different app built with a different tool. Neither Helix nor the video reports its own.

At Ena Pragma we run a version of this on our own writing: a critic agent that did not write the draft now checks our Field Notes before they merge. The spec-driven note linked above was held three times before it passed, and this one was held once.

*How this was researched:* our AI research agent ran the source search using Brave Search and Firecrawl and wrote a graded claim sheet. We then re-read the sources quoted here directly on September 25, 2026: the three Shopify Engineering posts, the Claude Code hooks documentation, the video's own captions, Anthropic's Ralph plugin, Huntley's Ralph post, ImpossibleBench, METR, and the arXiv abstracts. We have not read AI Labs' hook, which sits behind its paid community. Shopify's app migration was still in progress as of its September posts.

## Sources

- [Helix: The internal tool powering our Shopify app's native migration, Talha Naqvi, Shopify Engineering](https://shopify.engineering/helix)
- [Native is now the future of mobile at Shopify, Shopify Engineering](https://shopify.engineering/back-to-native)
- [Migrating Shop app from React Native to native, Shopify Engineering](https://shopify.engineering/shop-app-migration)
- [Shopify Just Released The Greatest AI Coding Workflow Ever, AI Labs (YouTube)](https://www.youtube.com/watch?v=bBMp5tLxShQ)
- [Hooks reference, Claude Code documentation](https://code.claude.com/docs/en/hooks)
- [Ralph Wiggum plugin, Anthropic, anthropics/claude-code](https://github.com/anthropics/claude-code/tree/main/plugins/ralph-wiggum)
- [Ralph Wiggum as a "software engineer", Geoffrey Huntley](https://ghuntley.com/ralph/)
- [Adversarial Review: Structured Disagreement for Grounded Agentic Code Review, arXiv 2608.18167](https://arxiv.org/abs/2608.18167)
- [Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop, arXiv 2607.23002](https://arxiv.org/abs/2607.23002)
- [ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, arXiv 2510.20270](https://arxiv.org/abs/2510.20270)
- [Recent Frontier Models Are Reward Hacking, METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)
- [More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges, arXiv 2607.05904](https://arxiv.org/abs/2607.05904)
- [Can escalation channels redirect reward hacking toward defect disclosure?, arXiv 2608.29460](https://arxiv.org/abs/2608.29460)
- [Spec-driven development has early support. The course's biggest promises are barely tested., Ena Pragma](https://enapragma.co/field-notes/spec-driven-development-has-early-support-the-courses-biggest-promises-are-barely-tested)