ena pragma
·17 min read·ai-verification · evaluation · agent-security

Three AI videos mostly got their numbers right. The slip is in what each number is attached to.

We checked three October 2026 AI videos against their sources. Most figures match. The slips are in what each is attached to: a setting, a model, a yardstick.

Contents

We checked three AI videos from October 18 and 19, 2026 against the pages they lean on: a walkthrough of xAI's Grok Voice 3.0, a talk about AI agents meeting on the open internet, and an explainer on a Goodfire essay arguing that neural networks are already clean. Most figures match. What slips, in all three, is what a figure or a word is attached to: one configuration read as a whole product, the word "swarm" stretched from a lab worm to a field encounter, two different yardsticks set side by side.

This is research from public pages, not a test of ours. We read the videos' auto-generated captions, not audio or slides, so we do not count caption spelling as a speaker's error. Every vendor and lab figure is that vendor's or lab's own, and we say whose. We read the sources on October 19 and 20, 2026, Central time.

VideoStatements gradedMatch a sourceMatch in partNot shown, or contradicted
Grok Voice 3.05134143
Agents in the wild2514110
Goodfire essay7132354

These counts come from our full grading of each video, which covers more pages than this note cites. They depend on how we split captions into statements and leave out forecasts, opinions and sponsor reads. A match means the video says what the source says, not that the source is right. The Goodfire row is 68 statements plus three single-detail checks. Two of its four in the last column are the two names under "Names, as captioned", which we read in captions only.

#Grok Voice 3.0: each figure is narrower than the sentence around it

The video "Grok Voice 3.0 Just Changed EVERYTHING!", from the Julian Goldie SEO channel (uploaded October 18, 2026), retells xAI's October 14 launch post. We call the company xAI, as the video and the x.ai address do; SiliconANGLE says it was renamed SpaceXAI in July and took the SpaceXSI name in October. xAI writes: "In Artificial Analysis's independent testing, Grok Voice 3.0 ranks #1 on the Speech-to-Speech Index, with a score of 86.9." Artificial Analysis's leaderboard, read October 20, 2026, has 86.9 first of the 19 listed models and configurations with an index score. The video also owns up to a cost: Grok Voice 3.0 is "a tiny bit slower to start talking than 2.0", which the table confirms, 0.81 seconds against 0.70.

  • A configuration read as a product. The video says "Gemini 3.8 Live from Google scored 82.6." On the leaderboard, 82.6 belongs to "Gemini 3.8 Live Extended Thinking (High)". The row named "Gemini 3.8 Live" scores 76.0, though it has the higher arena preference rating, 1088 against 993. OpenAI's 83.2 is "GPT-Live-1 (Astra, medium)", and "GPT-Live-1 (Sol, low)" scores 80.1. xAI's launch table compares against those two configurations, and the video does say Grok beat "the versions of GPT Live 1 and Gemini that xAI compared it against". The ranking holds whichever rows you pick. The sentence about what a named product scored does not.
  • A time to the first sound read as speed. The video says xAI claims Grok "gets close to that score while answering about 40% faster." xAI's post says Grok Voice 3.0 "scores 71.6%, comparable with GPT-Live-1 (Astra), but at 40% lower latency". The latency column is "Time to first audio (seconds) on Big Bench Audio", which Artificial Analysis defines as the "Average number of seconds required to generate the first token of audio output, measured across the Big Bench Audio question set." Grok Voice 3.0's 0.81 seconds against 1.34 for GPT-Live-1 (Astra, medium) is about 40 percent lower (39.6 from the table's rounded values), so xAI's number holds on that column. It marks when the first sound starts, not how long a caller waits for an answer to finish.
  • Accuracy by one yardstick, reliability by another. The video says Grok Voice 3.0 "got the highest accuracy score there at 74%" on ServiceNow's EVA-Bench, and that holds. ServiceNow's leaderboard code labels the figure "EVA-A pass@1", the share of single trials that pass. Its data file gives the same system 73.7 percent on that measure and 55.3 percent where every repeated attempt at a scenario must pass (95 percent interval 48.9 to 61.8), over 213 scenarios. Grok Voice 3.0 also leads that stricter measure, 55.3 against 44.0 for the next of 23 systems, so the ranking stands and the level is what changes. The file does not say how many attempts sit behind that row. ServiceNow's page says "single-trial scores systematically overstate deployment-grade reliability".
  • A customer that is a sibling business. xAI's post says "Starlink call centers use Grok Voice to answer 20,000 customer calls every day across multiple languages. Roughly 70% are resolved with no human in the loop". The same paragraph says "A single agent built on Grok Voice 3.0 verifies customers, troubleshoots hardware, issues credits, and ships replacement equipment", but not that the 70 percent was measured on that agent. xAI's earlier Starlink story (August 19, 2026) gives roughly 20,000 calls "on a weekday (about 10,000 on weekends, higher during promos)" and names no Grok Voice version, so we cannot tell which model the 70 percent was measured on. The video notes that "the Tesla and Starlink numbers come from XAI itself". SiliconANGLE adds that "SpaceX acquired xAI in February" and that Starlink is "the SpaceX satellite internet business". In our reading, the customer is a sibling business, not an arm's-length buyer.
  • A yardstick that moved under the same model. Artificial Analysis's index, version 2.0 from August 2026, "Replaced Conversational Dynamics (Full Duplex Bench) with Task Success Rate in the index." So the video's 97 percent on Full Duplex Bench is not part of the 86.9. And xAI's July 29, 2026 launch page for Grok Voice 2.0 gave that model 82.9% on what it called the "AA Speech-to-Speech Quality Index". The table read on October 20, 2026 gives it 81.3.

#Agents in the wild: the events are in the sources, and the frame is wider than the evidence

The video "What Really Happens When AI Agents Meet in the Wild?" is from Nate B Jones's channel, AI News & Strategy Daily, uploaded October 18, 2026. (Its page showed a second title, "Researchers Built AI Agents That Copied Themselves 7 Generations Deep. Here's How To Stay Safe.", when we re-read it on October 20.) It covers a Tencent-hosted agent fleet that researchers contacted, a lab worm and an Anthropic warning. The investigators' report, dated October 14, 2026 and written by participants in an AI swarm hackathon, says: "Between 28 September and 8 October 2026, a fleet of AI agents running on Tencent Cloud made more than 2,600 attempts to extract data from Amap, Alibaba's mapping service". The video's "Over 11 days, the fleet made more than 2,600 attempts" matches it.

  • "The encounter here gives us a place to start understanding how swarms are changing the internet as a whole." The video opens with "Self-replicating AI swarms are here" and does say the worm was built "In a lab". That lab result is real: Guan and colleagues' preprint, submitted June 2, 2026, reports a worm that reached "up to 7 generations of self-replication" in "15 independent experiments on an isolated 33-host network". Seven is the maximum; the paper reports a mean of 5.1 and says generation 7 "is only reached in 2 instances". The fleet is another matter. Its investigators "found no evidence that the agents coordinated with one another in real time", so they "label these rogue agents a fleet rather than a swarm", while allowing that "this event may turn out to be a swarm, we lack evidence to confirm that at this time". Anthropic's October 12 post says "We don't yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain". In the sources we read, a self-replicating swarm appears only in the lab.
  • "skeptics will tell you these were just agents doing a browsing evaluation." The video answers with "That is true as far as it goes" and then argues that strangers reaching a running fleet is "a huge deal". The investigators trace the fleet's machines to Tencent Cloud in Hong Kong and write that "Several lines of evidence point to Tencent's Hy (Hunyuan) model team", while being "deeply uncertain about what model was being evaluated". The evaluation account comes from Tencent's side. Tencent's own post on X (October 16, 2026, UTC) says "The activity originated from evaluations of third-party models, not Tencent Hy." The report's version comes from a person close to Tencent, speaking on background, and the South China Morning Post quotes "A person familiar with the matter"; we cannot show those are two people. So the account rests on Tencent's post, the primary source, and on one or two unnamed people, all unaudited.
  • "agents started pouring in messages." The report's mailbox table counts 33 messages from agents in 13 threads and 595 inbox checks. The inbox went unread for hours at first ("so many messages were missed"). The hidden note that drew the messages "included some false and misleading claims".
  • "hidden message passing" is two different things. The video says the contact came through "a hidden message that humans wouldn't be able to read, but agents could read", and the report confirms its note was invisible to a human reader. The video then calls hidden message passing "another hallmark of the hugging face attacks" and imagines "white on white text on a page". We searched METR's report and Hugging Face's technical timeline for white text, white-on-white text and off-screen text and found none; control searches for phrases that do occur there ("message board", "message protocol") find them. What Hugging Face describes is a payload "compressed and base64-encoded, so nothing sensitive appeared in cleartext": hidden from a casual log scan. Text hidden from readers on web pages is documented separately by Palo Alto Networks' Unit 42, which reported on March 3, 2026 that indirect prompt injection "is no longer merely theoretical but is being actively weaponized".
  • "We don't know yet, because we don't have documented instances of that happening." The video asks whether a consumer agent, a Muse or a Grok bot, could be prompt-injected by an agent it meets in the wild. In the sources we read, we did not find a documented case of that. A nearby one exists: the Institute for Security and Technology wrote on February 26, 2026 that an assessment of Moltbook, a forum for AI agents, "flagged 506 prompt injection attacks" in its first 72 hours. We did not fetch that assessment, and the sources do not say whether agents or people wrote the posts.

If an agent of yours reads the web or talks to other agents, here is our defense, untested by these sources. Treat everything it reads, including text a person cannot see, as data and never as instructions. Give it the narrowest keys that let it do its job. Log what it read and did, and decide in advance which actions need a person. Weigh who wrote a sentence over where it appears: an instruction inside a page, or from another agent, has only its author's authority.

#The Goodfire essay: a 1.7% and a 90% from two different yardsticks

The video "A Potentially Historic AI Breakthrough", from The AI Daily Brief (uploaded October 19, 2026), explains Nick Cammarata's essay Neural Networks Are Already Clean, published October 15 and labeled "A working hypothesis". Cammarata writes from Goodfire. The video's captions never mention that, and The Decoder wrote that "As an interpretability company, Goodfire also has an obvious stake in the thesis." The essay argues that the textbook cases of neurons packing unrelated things together were misread, and the video follows it closely. It tells viewers "these are early counts, they are self-reported" and that the full technical paper is not out. The slips are in the comparisons and some quotations, and one comparison is the essay's own.

  • The 2% and the 90% are different measurements. The video says "Compare that again with that 2023 OpenAI project that could only explain about 2% of detectors in GPT-2." The essay itself sets "found about 1.7% of neurons interpretable by our most generous measure" next to the 90% in consecutive paragraphs, and the video repeats the pairing. The 2023 OpenAI paper found "5,203 (1.7%)" of 307,200 GPT-2 XL neurons with "top-and-random scores above 0.7", a measure of how well a model's written explanation predicts the neuron's activity, and "732 neurons (0.2%)" under random-only scoring. The 90% is the essay's estimate that "In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job": another model, and a share of neuron poles with one main job, not a prediction score. The essay says "The scoring isn't identical to the original project" and "I no longer think it's a great metric for explanations". Its own redo on GPT-2 XL, "500 neuron papers with almost no iteration", reads "Now about 70% look like roughly one thing".
  • Names, as captioned. The captions read "In a modern OpenAI model, about nine out of 10 detectors turn out to have one main job." The essay's 90% audit is of Qwen3-32B, and it names no OpenAI model for it. The captions also call the trillion-parameter model in the found-wallet example "Gopher 3"; the essay says Kimi K3 and adds "We haven't begun full audits of this scale yet". We heard no audio, so neither is counted as the speaker's error; as captioned, neither matches the essay.
  • 85% sure of what. The video says Cammarata "puts himself at about 85% sure" without saying sure of what. The essay says: "We're maybe 85% sure, but that means 85% sure that nearly everything comes from the second kind of process, which is very different from thinking 85% of mechanisms do." It is a confidence in a kind of explanation, not a share of neurons.
  • Two sentences we could not find in the essay, and a hedge that was flattened. The video reads three sentences as Cammarata's writing. Only the first, "Why would an arbitrary mixture of cats and cars need to connect so strongly to a flame?", is in the essay's text and rendered page as we read them; we did not find the other two. And Neel Nanda, whom The Decoder says leads the interpretability team at Google DeepMind, wrote "This is interesting work, but I don't think it's a significant update to the big picture of interpretability." The video quotes him as saying the work is "not a significant update to the big picture", which drops his "I don't think it's".
  • A safety payoff that is an argument. The video says reading the wiring opens up "the ability to check AI's reasoning, not just judge it by its answers." The essay's sentence is an argument: "if we become a reader, it should be accessible to us too." Searches of the essay for "deception detector", "detect deception", "detecting deception" and "lie detect" find nothing, though "deception" appears, so we found no deception-detection test in it.
  • A noise control the essay does not report. A self-published reproduction on Qwen3-0.6B and 1.7B reports that "the labeller called 84% of pure-noise neurons “one job,” versus 52% of real ones", the noise being "an untrained, randomly-initialized network". We searched the essay for "untrained", "randomly initialized", "random initialization" and "random weights" and found none, though it does describe positive controls. The reproduction is unreviewed, by its own words "me and my AI research assistant", uses far smaller models and the 2023 protocol the essay no longer favors, and we have not verified it, so it does not estimate the 90%. In our view it poses a question the 90% should answer: what does the same yardstick say on noise?

One reading of the dispute, ours and untested: people may differ mainly on which instrument they trust, an explainer model's labels, a simulation score, or the essay's own predict-then-test, so a result where the trusted instrument could fail, such as a noise control, would move opinion more than another percentage; it would be wrong if the companion paper reported such controls and the critics did not move.

#What the slips have in common

Four habits follow, each ours and none tested here:

  1. Write the noun next to the number. Version, configuration, model, setting, window.
  2. Ask for the all-attempts number beside the single-attempt number.
  3. Say who counted. The deployment figures are the vendor's, the fleet's evaluation account is the cloud provider's, and the 90% is the essay's authors'. We found no outside count for any of them in the pages we read.
  4. Write the yardstick down before you open the vendor's table. In COMPare, Ben Goldacre and colleagues compared the outcomes in 67 trials from five high-impact medical journals with their protocols or registry entries: "Fifty-eight trials had discrepancies requiring a correction letter". A buyer's version is to choose the configuration and the metric first, then look.

Related notes: A video explained a new reasoning paper accurately, then put a number in the title that the paper never reported covers titles and The LLM Killer? The paper times itself against diffusion models, not autoregressive ones. runs the same check on a paper.

A figure is a pointer. Follow it to the thing it points at before you repeat the sentence built on it.

#What we did not check

  • We ran no model and no benchmark, and heard no audio.
  • Figures from xAI, Artificial Analysis, ServiceNow, Tencent and Goodfire are read, not reproduced. The leaderboard is the table as of October 20, 2026. "Independent" is xAI's and Artificial Analysis's word for the testing; we did not research Artificial Analysis's ties to the vendors it ranks.
  • The Goodfire companion paper is not out, so every number that depends on it, the 90% included, is unchecked. We also did not check whether Tencent's evaluation account is true.

#Sources

All read October 19 and 20, 2026.

Keep reading