Start with the task, not the loss curve

robomimic’s official README describes a framework for learning robot policies from demonstrations: manipulation datasets, offline algorithms and evaluation tools. In the paper’s simulated Square task, an arm grasps a square nut and places it over a peg. This is a paper walkthrough, not a BotsTest experiment.

Official simulation diagram of an arm aligning a square nut and peg

Author simulation: Square task, cropped from Figure 1(d) of Mandlekar et al., CoRL, PMLR 164 (2022), CC BY 4.0.

Same setup, three ways to choose a checkpoint

Table 29 compares checkpoints under one policy type (BC-RNN), demonstration source (PH, an experienced teleoperator) and input type (low-dimensional state, not images). The lowest offline validation-loss checkpoint reached 7.3 ± 5.0% simulated task success; the final checkpoint 74.0 ± 7.5%; and the best observed with hindsight 84.0 ± 0.0%. The authors report the mean and spread across three training seeds, with 50 simulator rollouts per evaluated checkpoint. The 7.3% is task success, not validation loss.

Paper plot showing relative changes from checkpoint-selection criteria across four settings

Author plot: “Effect of Policy Selection Criteria,” p. 6, left, cropped from the published paper, CC BY 4.0. Relative changes across four settings, not the absolute Table 29 numbers.

Three checkpoint-selection outcomes in the same Square simulation setup

BotsTest redraw of the authors’ results from Table 29; no new measurements.

Best observed does not mean independently verified

Offline validation loss measures how well the policy fits held-out demonstration actions. In a rollout, the robot has to complete the whole task while dealing with the states its own earlier actions create. The metrics disagree here, but this row does not explain why. The 84.0% winner was picked after observing those rollout results; a reported spread of 0.0 does not remove uncertainty. These are not hardware results and do not show that longer training always wins.

Split a small simulation budget into two tests

  1. Define the shortlist and pass rule. Try lowest loss, last and a predeclared milestone. Deduplicate and lock the simulator version, timeout, initial conditions and definition of success.
  2. Select on matched starts. Try each of three candidates on the same ten simulation starts (up to 30 trials). Log success counts and recurring failures. A small gap can be inconclusive.
  3. Confirm on untouched starts. Freeze the selected checkpoint. For example, run it on 20 new starts or seeds; picking again on this batch would make it a selection batch too. These counts illustrate a budget, not a guarantee of adequate sample size. Report simulation and physical hardware separately.

Two batches: matched simulator starts for selection, unused starts for confirmation

BotsTest illustration of selection and confirmation; sample counts are illustrative.

Sources

BotsTest resource · Official documentation · Published paper

Turn the shortlist into a decision

Write down the task, operating conditions, success criteria and total ownership costs. Ask shortlisted suppliers to demonstrate your representative task, including failure recovery. Use the same acceptance test for each candidate.

Explore the full robotics directory →