Grade an AI agent by its decisions, not its wording

When a company wants a cheaper AI agent, the usual move is to take a big, expensive model and teach a small one to imitate it. Most distillation methods teach that imitation token by token, close to word for word: the paper describes both standard methods as ones that "supervise tokens without explicitly matching action semantics." A paper published on October 14, 2026 makes a simple argument against that. What the small model should copy is the decision, not the sentence.

The paper is "ActionOPD: Distilling Decisions, Not Tokens," from a team at Tsinghua University (arXiv 2610.19820, October 14, 2026). The code is public (GitHub, thu-nics/ActionOPD, last updated October 15, 2026), and a walkthrough video appeared two days later (Discover AI, October 16, 2026). Here is what it does, what it found, where it stops, and what it means for anyone running agents.

#Same decision, different words

An agent at work does one of two things at each step: it calls a tool, or it says something to a person. The authors' point is that the same decision can come out in many wordings. In their words: "A tool call expresses a decision through its tool name and arguments; a natural-language response conveys communicative intent and information. The same decision can be realized through different wording and reasoning."

Their own example: at the same point in a task, "Basic-economy tickets cannot be changed" and "Changes are not permitted for basic-economy tickets" say the same thing. Word-by-word imitation treats them as different. A decision-level grader treats them as one.

#How it works, in plain terms

The small model (the student) is put into real tasks and makes its own moves. At each point it reaches, two things happen:

• The student proposes four possible next actions.
• The big model (the teacher) is asked for three of its own at the same point.

Each student action earns credit if it means the same thing as a teacher action. A tool call is compared by tool name and arguments, checked by code, with a frozen AI judge for argument differences the code cannot settle. A reply is compared by intent and facts. The student is then trained toward the actions that earned credit.

Two consequences follow, and they are the practical part. The teacher can be a black box: the method needs only its outputs, not its internal probabilities. And the teacher and student do not need to come from the same model family. In the paper, a DeepSeek teacher trains a Qwen student.

#What the numbers say

The authors trained two small open models, Qwen3-4B and Qwen3-8B, on 6,697 tool-use tasks for 100 training steps, then tested them on four agent benchmarks they never trained on.

Source: ActionOPD, Table 1, October 14, 2026. Gains are percentage points, as the paper rounds them.

Three things in this table are easy to miss.

First, the students got better and the stronger teacher produced the stronger students, at both sizes.

Second, the students are still far from the teachers. Scored directly, DeepSeek-V4-Flash averaged 71.2 on the same benchmarks. The best student reached 40.0. This is a real improvement to a small model. It is not a small model that matches a big one.

Third, it is not a clean win over the older methods. The paper also ran a separate head-to-head with a different setup: a Qwen3-4B student, a larger Qwen3-32B teacher, 50 training steps, on the customer-service tasks of the τ²-bench benchmark. There the new method had the best single-try success on the test tasks, but only just: 40.0%, against 39.6% for token-level on-policy distillation (a lead of 0.4 points) and 37.5% for classic offline distillation. The classic method was the most reliable when the same task had to succeed four times in a row (19.2%, against 14.2% for the new method). The authors say so themselves: "While these results do not establish uniform superiority, ActionOPD offers a competitive alternative with a more flexible teacher interface."

#Where it stops

The authors list three limits, and the first one matters most for anyone deploying agents: "agreement does not guarantee correctness: teacher errors or suboptimal decisions can become learning targets." If the teacher makes a bad call, the student is trained to make the same bad call.

The second is the judge. An AI deciding whether two replies mean the same thing can be wrong in both directions. The third is long tasks. When many different actions are reasonable, a handful of samples may contain few matches, and the paper does not test whether drawing more samples fixes that.

#The same month, models that only make decisions

ActionOPD changes how agents are trained. A second development this autumn changes what some models produce. On September 15, 2026, TypeSafe AI released Jev, the first of what it calls System One models, built "to make fast, structured decisions that software can use directly" (TypeSafe AI, September 15, 2026). Jev does not write text. As IBM puts it, you give it predefined choices "and it assigns probabilities to each." Others followed, including Cloudflare's Clef models on October 1 (IBM Think, October 15, 2026).

One test shows that typed output does not make a model safe. Security firm HiddenLayer put a browser agent built on Jev in a fake online store and added a sponsored ad with a "Confirm Price" button. The agent clicked it instead of finishing the purchase it was told to make. The same trick worked on a regular language model set up to act like a decision model, though Jev fell for it less often. HiddenLayer's explanation: "they are, under the hood, LLMs, simply using a different decoding strategy" (The Register, October 15, 2026).

So two lines of work arrived at the same unit from different sides. One trains agents on decisions. The other makes decisions the output. Neither removes the need to check that a decision was right and was not steered from outside.

#What an operator should do now

1. Judge a cheaper model by its decisions on your own work. Before you move an agent to a smaller or cheaper model, replay real moments from your own tasks and check whether it would take the same action: the same tool with the same arguments, or a reply with the same meaning. A benchmark headline cannot tell you that, for the reasons in Why a high AI benchmark score doesn't mean a reliable agent.
2. Keep a record of every action an agent takes. Record the tool, the arguments, what came back and whether it worked. That record is what makes the check in step 1 possible. It also lets you count agreement only where the outcome was right, which helps address the paper's own first limit. We are building our own agents this way, with each action going through a typed verb that leaves a receipt. The longer case for that record is in Audit trails are what keep automated work accountable.
3. Do not treat agreement with a big model as proof of being right. A big model is a strong teacher, not a correct one. Where a decision touches money, a customer or a commitment, check the outcome, not the resemblance.
4. Treat typed output as a format, not a defense. A model that returns a clean yes or no can still be pushed to the wrong yes. Keep the same guards against untrusted page text and documents that you keep for any agent.
5. Expect small agents to improve, not to replace large ones. In this paper the students trained by the DeepSeek teacher closed roughly 12% to 17% of the gap to it. That is useful for cheap, frequent, well-defined steps, and it is not a reason to hand a small model the hard calls.

The words were never the job. The decision was, and now the research grades it that way.