GR00T N1.7 succeeded on 25 of 45 layouts in simulation and 11 of 45 on the real robot. Yet the order of all five systems stayed the same. In the SO-101 Bench preprint, simulation ranked candidates, but its success rates were not substitutes for hardware acceptance scores.

Illustrated tabletop SO-101 and its digital twin; GR00T N1.7 drops from 25 of 45 to 11 of 45.

The tabletop rig is an original illustration. The counts are from the authors.

Same order, different scores

System Simulation Hardware Gap (pp)
GPT-6.1 Sol + PyRUA + Galaxea 0.5 38/45 29/45 20.0
Galaxea 0.5 30/45 24/45 13.3
GR00T N1.7 25/45 11/45 31.1
π0.5 22/45 8/45 31.1
MolmoAct2-SO101 11/45 7/45 8.9

Five paired success counts: 38 to 29, 30 to 24, 25 to 11, 22 to 8, and 11 to 7.

SO-101 Bench preprint, Table 6. Both domains share an axis of 0–45 successes.

The authors’ preprint, Section 5 and Appendix B reports these results. BotsTest did not replicate the experiment. The gaps come from raw counts, not rounded rates. A single discount would conceal the difference between gaps of 8.9 and 31.1 percentage points.

What the authors held constant

Four learned policies and one hybrid system each ran 45 matched layouts in each domain. For each system, the weights, execution harness, success criteria and time limits stayed consistent. The tests used one physical environment with fixed lights and cameras. The hybrid ran later in that same environment.

Forty-five layouts feed simulation and hardware; weights, scoring and time limits are shared.

Five task families × three object splits × three layouts per combination.

Matched layouts do not make the physics identical. The authors note that a light physical bin could shift when pushed; this does not explain every gap. Individual outcomes agreed on 56–91% of layouts, depending on the system. Ranking candidates and predicting each individual trial are separate questions.

A paired retest you can run

  1. Write the pass rule first. Define success, the time limit and how to score an intervention. Shortlist candidates in simulation.
  2. Run the same layout IDs on hardware. Record starting poses, weights, camera and scoring. Keep resets consistent and mark cases you cannot match.
  3. Report successes over attempts for both. Look separately at cases that passed in simulation and failed on hardware. Do not use the simulation ranking as a waiver if the hardware misses the acceptance target.

Set the acceptance line, shortlist, retest matching layouts, and log successes, attempts and mismatches.

BotsTest proposed retest workflow, not new measurements.

See the BotsTest SO-101 resource and LeRobot hardware guide.

Turn the shortlist into a decision

Write down the task, operating conditions, success criteria and total ownership costs. Ask shortlisted suppliers to demonstrate your representative task, including failure recovery. Use the same acceptance test for each candidate.

Explore the full robotics directory →