The Intervention Surface Framework

The Intervention Surface Framework: what to change when an agent fails
An Ena Pragma method. Works on any agent stack, any model vendor.

THE PREMISE
A deployed agent is a composite system: a base model, an in-context harness
(prompts, routing, config), memory, tools and their schemas, and guardrails.
Five surfaces, one of which is the model. When the agent fails, the first
decision is WHICH surface to change. Teams that skip this decision default
to the two most expensive answers (retrain or replatform) when the correct
fix was usually a paragraph of text.

THE MAPPING (failure type -> cheapest correct surface)
1. Missing or wrong FACT, recurring
   -> MEMORY. Write the fact where the agent retrieves it, with its source
      and date. Do not touch anything else.
2. Wrong TOOL choice, malformed call, or misread result
   -> HARNESS / TOOL SCHEMA. Fix the tool description, the routing rule, or
      the schema. The model was never the problem.
3. Repeatable PROCEDURE keeps going wrong at the same step
   -> HARNESS (skill patch). Patch the written procedure the agent follows.
      Add the failing case to it as an explicit example.
4. Judgment or safety failure the rules should have caught
   -> GUARDRAILS. Tighten the gate that should have fired, not the model.
5. Broad failure that persists ACROSS tasks, datasets, and tool configs
   -> MODEL (fine-tune, preference optimization, or vendor/model swap).
      This is the last resort, not the first, because it is the most
      expensive to apply, evaluate, and roll back.

THE TWO GATES (non-negotiable, both from 40 years of evidence)
- EXOGENOUS VERIFIER: the check that approves a change must sit outside the
  system being changed. Self-review is not review; systems optimizing
  against their own judge learn to fool the judge (Eurisko 1983; GenProg,
  where 55/105 reported fixes fell to 2/105 under independent review, Qi et
  al. ISSTA 2015; METR 2025, o3 gaming its evaluation 21/21 runs on an AI
  R&D task, optimizing LLM training code).
- FREEZE AND SHIP: changes batch through review and ship as a version you
  can name and roll back. No continuous silent mutation of production
  behavior. Every system that survived in production has this gate; the
  "Misevolution" study (arXiv 2509.26354) measured what happens without it:
  refusal rates down 45 percent, malicious tool acceptance up to 93 percent.

THE LEDGER LINE (the habit that makes it auditable)
Every applied change gets one line in a running log (a skill patch is
logged as harness):
  surface: memory|harness|tools|guardrails|model
  change:  <one sentence>
  trigger: <the specific failure, ticket, or session that caused it>
  replay:  <the failing case re-run clean, or why it cannot be>
A change that cannot fill in these four fields is drift, not improvement.

RE-RUN RULE
A failure-driven change is not done when the edit lands. It is done when
the original failing case runs clean against the new state. Keep the
failing case; it is now a regression test.

Review the ledger monthly. If one surface dominates, that is your real
infrastructure gap: constant memory writes mean retrieval is weak, constant
harness edits mean the tool layer is underspecified, constant model swaps
mean nobody is diagnosing anything.