ena pragma
·16 min read·ai-verification · ai-vendor-selection · evaluation

Robot training in a scan of your site is on offer. Ask for the real-world table first

Niantic Spatial and Flexion, on NVIDIA software, show a robot trained in a scan of an office. We found no real-world trial count in their pages, and list seven asks.

Contents

If a vendor offers to scan your warehouse, office or shop into a simulated copy and train a robot in it, one question should come first. Has anyone compared how the robot does in the copy with how it does in the real place? For the newest offer, Niantic Spatial's, shown with Flexion and NVIDIA, we found no such comparison in the companies' own pages that we read. We are not saying the work is poor. We are saying that on the pages we read, the campaign's headline is larger than the measurement under it, and that flight simulators are held to a check we could not find for this offer.

#What was released

Here is what the companies released, stated plainly. On July 6, 2026, Niantic Spatial said "Scaniverse can now turn a 360 video capture into a simulation-ready USDZ file for NVIDIA Isaac Sim". On July 20 it and Flexion published a blog post on the full pipeline. A person walks a site once with a 360-degree camera. The pipeline builds a 3D Gaussian splat (a scene drawn from many small colored blobs, viewable from any angle) and a collision mesh from the same capture, and packs both into one file in NVIDIA's NuRec format. Flexion then trains a humanoid robot's navigation policy inside that scene in Isaac Lab. On September 25 Niantic Spatial announced a Places Library, "starting with a curated catalog of 100 high-fidelity, real-world 3D environments."

NVIDIA supplies the simulation software the blog names, Isaac Sim and Isaac Lab. GitHub's release for Isaac Sim 6.2.0 reads "Isaac Sim 6.2.0 GA," published October 14, 2026, 03:05 UTC, and the Isaac Lab 3.0.0 release notes say "Isaac Lab 3.0 is now generally available," published October 20, 2026, 02:05 UTC. The blog post is labeled "Co-authored by Niantic Spatial and Flexion," with NVIDIA named in its title.

We read the campaign page, that blog post, Niantic Spatial's robotics, plans and Places Library pages, Flexion's June 29 post, the Scaniverse post, the two NVIDIA releases, NVIDIA's Isaac Sim 6.2.0 release notes, two trade articles, five research papers and the FAA's rules for flight simulators, all on October 25, 2026, Central time. We fetched each again for this note and matched every quote and number to the fetched text. Release dates are GitHub's UTC timestamps. We ran none of the software and did not watch the livestream or the two launch videos (81 seconds each, uploaded to the Niantic Spatial and Flexion YouTube channels on July 20, 2026). Five pages show no date of their own (Niantic Spatial's robotics, plans and livestream pages, and the two 14 CFR Part 60 appendices), so for those we give the day we read them.

#What the campaign says, and what is measured

The page for a livestream with speakers from the three companies, which YouTube's listing dates August 12, 2026, lists a learning outcome: "Close the sim-to-real gap for humanoid robotics". The blog's own title is "Closing the Sim2Real Gap for Humanoids." Titles state goals, and a title that says closing can mean work under way. Still, set beside a 97.8% score, they can read as a real-world result. Two terms first. A rollout is one attempt at the task, start to finish; the blog's are simulated. Zero-shot means the policy ran on the real robot with no further training there. Here is each headline next to what the pages show.

The claimWhat is measuredWhat it is attached to
The blog's title, "Closing the Sim2Real Gap for Humanoids," and the page's learning outcomeSuccess rates in simulation, and a video of a real robot. We found no real-world success rate, trial count or site list on the blog, the robotics page or the campaign pageOne task, local navigation. The policy takes the camera view, body sensors and a goal and outputs a speed command. "A separate, in-the-loop pre-trained locomotion policy" turns that into motion
"97.8% versus 93.8%" in the Flexion office, "75.0% versus 70.9%" in the Niantic Spatial officeSimulation only: "1,024 rollouts" per policy, "in simulation across two reconstructed scenes"A policy trained in a reconstruction of the site, against a depth policy trained in a generated mesh. The blog says Flexion trains "inside a reconstruction of the specific deployment site." It does not say whether the areas scored were held out from training. That is a question to ask, not a finding
The policy "transfers zero-shot to a real office"One video, captioned "navigating the real office toward three commanded target poses." During nominal navigation through the test area the policy "performs on par with a depth-based policy." No trial count, success rate or fall countOne office. The blog adds that it held up to "scene changes such as rearranged furniture," again with no count
The robotics page says the policy "fell 13 times less often than the same architecture trained on a hand-built synthetic office"True in the Flexion office, where falls went from 27.1% to 2.1%, 12.9 times fewer. In the Niantic Spatial office they went from 10.9% to 9.0%, 1.2 times fewerBoth in simulation, 1,024 rollouts each. Our rough 95% intervals for the second pair, 9.0% to 12.8% and 7.2% to 10.8%, overlap (unpaired; the blog reuses the same poses across conditions, so this is not a test of difference)
The robotics page says "On the real robot in the real site, it handled cases where a depth-input policy failed"Three illustrative cases: a blue mat, thin structures such as tripods and railings, and glass. No count of trials or of the depth policy's failuresThe blog notes that none of those exact setups, in the locations shown, was present when the scene was reconstructed, so the policy did not meet them in that form in training

Several limits come from the companies themselves, and they deserve credit. The blog says the twin "captures a moment in time, so lighting, reflections, and object placement are baked in as observed during capture," that "areas skipped during the walkthrough will reconstruct less reliably," and that "RGB is not magic." Flexion's June 29 post says "The current system still operates within a bounded task distribution." Niantic Spatial's robotics page lists the step that would send results from deployed robots back into the twin, "Deploy and feed the loop," as Roadmap, not Available. What is missing is not candor about limits. It is the one table a buyer needs.

#Earlier work already published a real-world table

VR-Robo, posted to arXiv on February 3, 2025, already reported that "our framework achieves RGB-only sim-to-real policy transfer" from a Gaussian-splat twin, on a real quadruped robot. Its real-world protocol: three sets of cone positions, each with an easy, a medium and a hard task, and "For each task, we repeat the experiment 5 times." That is 15 trials per method per difficulty (3 tasks times 5 repeats, our arithmetic). On the easy, medium and hard tasks, the splat-trained policy succeeded 100.00%, 93.33% and 100.00% of the time, against 20.00%, 6.67% and 0.00% for one trained in a textured mesh. It is one scene and one task, but the denominator is on the page.

GaussGym, posted October 17, 2025, runs a splat renderer inside a physics simulator at "exceeding 100,000 steps per second on consumer GPUs," and reports "partial transfer to real-world scenarios." Its limitations section reads "Visual sim-to-real transfer remains a difficult and largely unsolved problem, and GaussGym offers a promising platform for developing algorithms to narrow this gap." Our reading: what the 2026 offer adds, on the pages we read, is packaging and a humanoid demo. There is a standard file format, an app that exports it, a catalog and a plan. Packaging is a real service. It is not a trial result.

#The check aviation already requires

For flight simulators, the FAA's rules require a written comparison against objective data. Under 14 CFR Part 60, the Qualification Test Guide is "the primary reference document used for evaluating" a simulator. The guide must include "A means of comparing the FFS test results to the objective data" (FFS means full flight simulator), along with the "Parameters, tolerances, and flight conditions" for each test, and the flight test data used to validate the simulator "must have been gathered in accordance with a flight test program". In plain words: the simulator's results are compared with objective data against stated tolerances. Where that data is flight test data, it must be gathered under a flight test program that includes a flight test plan. The appendices also allow objective data from "another approved source." This is an analogy: the rule covers flight simulators used for pilot training credit, not robot twins.

Robotics has a version of this. PolaRiS (December 18, 2025) scores robot policies in reconstructed scenes and in the real world, with 20 real rollouts per policy in each environment, and reports an "average Pearson correlation of r=0.9" between the two for manipulation policies on the DROID platform. A November 2025 paper reports a Pearson correlation above 0.9 for every policy in its soft-body tests, against 0.649 and 0.237 for an Isaac Lab baseline on two tasks. Its correlations are per task (0.90 to 0.94 for the full method), computed from 16 to 27 evaluation episodes per checkpoint. The baseline is a general-purpose Isaac Lab setup whose "support for deformable objects remains limited," and it was scored on two of the three tasks. A correlation near 1 means that policies scoring higher in the simulator also score higher in the real world. We found no such figure published for the Niantic Spatial and Flexion pipeline. That is a search result, not proof that none exists.

#Seven things to ask for or do

  1. Ask for the real-world table. Ask any vendor three questions: how many real trials, on which sites, and which routes or areas of the scanned site were held out. Flexion's launch-video description says its policies "transfer zero-shot to the real robot and environments they were trained for," so the test that matters is on that site, with part of it kept out of training. The public record we found for this pipeline is 1,024 simulated rollouts per policy per scene, one real-office video of three target poses, and three illustrative hard cases with no trial counts (blog, July 20, 2026). VR-Robo published 15 real trials per method per difficulty (February 3, 2025).

  2. Open the second scene before you repeat a ratio. The page's "fell 13 times less often" holds in one office (27.1% to 2.1%, 12.9 times) and not in the other (10.9% to 9.0%, 1.2 times, with intervals that overlap, though unpaired: the blog reuses the same poses across conditions, so this is not a test of difference). The page has a section headed "Proven in simulation" and a benchmark table for both offices. But the "13 times" sentence sits in a card headed "Proven policy transfer improvement" that says "Across 1024 simulation runs," it holds in one office, and neither the card nor the section names the office. When a headline ratio comes without naming its test site, read the others first (Niantic Spatial robotics page, read October 25, 2026; blog, July 20, 2026).

  3. Scan where the robot will walk, then test where you did not scan. Niantic Spatial says areas skipped in the walkthrough "will reconstruct less reliably" (July 20, 2026). A May 21, 2026 paper from another domain, self-driving, says reconstruction quality "degrades significantly under large viewpoint shifts" away from the recorded path (arXiv 2605.22420). Its authors propose a method they say "generalizes reliably to challenging unseen viewpoints." Plan two or more passes and keep one route out of training.

  4. Borrow the aviation idea. It is an analogy: the FAA uses tolerances, robotics papers use correlation. Pick at least five versions of the robot or agent you are testing. Five is our rule of thumb, because a correlation from two or three points tells you little. Score each version in the twin and in the real site on the same task, with a stated number of real trials. Set a threshold for the Pearson correlation before you run it. Use a low correlation to reject a twin, not a high one to certify it; five points leave a wide range. PolaRiS reports an average of 0.9 with 20 real rollouts per policy in each environment (December 18, 2025), and its worst environment was 0.81 against that average. If five physical versions are out of reach, ask the vendor for its table for a site like yours. This checks scores, not safety. The same idea applies to an AI agent tested in a copy of your systems: run the same tasks in the copy and in production, where that is safe to do, and compare.

  5. Price a refresh, not one scan. The twin is a frozen moment, as the blog says above. In our reading, a changed site means a new capture. The entry plan, Evaluate, is a 3-month minimum with "15 capture minutes a month (~1 place)" (plans page, undated, read October 25, 2026). Niantic Spatial's robotics page says "A deployment environment starts at a few hundred dollars" (Niantic's figure; we did not check it), but the plans page lists allowances, not prices, and a trade article says "Niantic Spatial does not publish prices" (Radiance Fields, October 4, 2026). Ask for a written quote for one capture, one re-capture after the site changes, and the re-test.

  6. Pin your software versions. Isaac Sim 6.2.0 (October 14, 2026 UTC) and Isaac Lab 3.0.0 (October 20, 2026 UTC) came out within a week. The Isaac Lab release notes never mention NuRec, Gaussian or splat; we searched the notes, and they do mention Newton, so the search could find text. That does not show NuRec fails in 3.0: the blog loads NuRec in Isaac Lab, and we did not test it. NVIDIA's Isaac Sim 6.2.0 release notes (last updated October 14, 2026) list isaacsim.replicator.nurec_utils moving from 0.1.2 to 0.1.3 with the change "Enable Gaussian accumulated depth and albedo in the default Sensor Processing Graph (SPG) render configuration," which Radiance Fields reported the same day. Keep a pinned pair of versions, render one known scene before and after any upgrade, and compare the metric.

  7. If you build agents, borrow Flexion's reliability arithmetic. Flexion's June 29, 2026 post says a navigation policy that works "95% of the time" and a grasp that works "90% of the time" "compound into failure." Flexion states the compounding. The 16-step figure is our arithmetic, not Flexion's: 0.95 to the 16th power is about 44%, assuming the steps are independent. Flexion reports a "16-step mission evaluation" in which "SFT reaches 38% end-to-end completion, while SFT+RL reaches 90%," where SFT is supervised fine-tuning and RL is reinforcement learning added on top. That is the company's own figure. The post does not say how many rollouts were run or whether they ran in simulation or on a robot. It also says that off-the-shelf models "emit the next logically plausible tool call too quickly" instead of checking that the previous step finished. Measure end-to-end completion on a fixed set of multi-step tasks, add a check on each step's result before the next call, and report an interval, not a single number.

Related field notes: Same task, twice gives a twenty-task test an owner can run on a savings claim, Grade an AI agent by its decisions, not its wording makes the case for judging an agent by what it decides rather than how it phrases it, and Three AI videos mostly got their numbers right checks what a figure is attached to.

A twin that looks like your site is now on offer. A table showing that it behaves like your site is what to ask for.

#What we did not check

  • We ran no software: no Isaac Sim, no sample scene, no policy, no robot.
  • We did not watch or transcribe the livestream, which YouTube lists at 4,065 seconds. Its question-and-answer part may answer questions raised here.
  • We did not watch the two launch videos (81 seconds each, July 20, 2026). We read only their descriptions.
  • The 12.9 and 1.2 ratios, the 44% figure and the intervals are our arithmetic. The intervals use a normal approximation and treat the 1,024 rollouts as independent, while the blog says the same spawn and target poses were reused across conditions.
  • We read two appendices of 14 CFR Part 60, not the whole rule. They allow objective data from "another approved source," and we did not review what the FAA approves.
  • Whether the scored areas were held out from training is an open question, not a finding. The blog does not say, and these questions have not yet been put to the companies.
  • The robotics, plans, livestream and two CFR pages show no date. What we quote from them is as of October 25, 2026.
  • We found no real-world trial table for this pipeline in the pages named above, and we did not search every venue. That is not proof that none exists.
  • We did not repeat the vendors' statements about training time, deployment time and cost, because we could not check them.

#Sources

All read October 25, 2026, Central time.

Keep reading