The drawer is still open

Open the drawer. Push a block inside. Pick an object back out, stack the blocks, then close the drawer. This is one of the instruction chains in Figure 5 of the CALVIN paper, not footage or a test run by BotsTest. Pulling a drawer open is one skill. Finishing the sequence means every move has to leave the scene in a state the next instruction can use.

Five drawer actions in order: open, push a block inside, retrieve an object, stack, close.

A five-step example based on CALVIN Figure 5; original illustration.

Two scores, two questions

CALVIN is a simulated benchmark for robot policies that follow language. In the single-task test, the authors initialize each trial from a valid demonstration start. The long-horizon test instead uses 1,000 feasible five-instruction chains assembled from the same 34-task pool. The policy receives one instruction at a time; the evaluator advances only after the simulator registers success. The scene is not restored between successful steps. This tests execution and transitions, not whether a model can invent a five-step plan from one goal. See the evaluation protocol and official implementation.

The isolated design resets before each task; the chain carries the physical scene forward.

Reset-based and consecutive evaluation answer different questions.

The chain falls apart before step five

In Figure 8, a 2022 multi-context imitation learning (MCIL) baseline using a fixed RGB camera, trained and tested in environment D, scored 53.9% in the separate 34-task setting. The percentages for completing at least the first one, two, three, four or five instructions were 48.9%, 12.9%, 2.6%, 0.5% and 0.08%, respectively, as printed. Each threshold uses all started chains as the denominator, not only runs that reached it. The largest drop in this row is from the first to the first two instructions together. This does not identify a specific drawer action as the cause. These are historical simulations for one setup, not a current policy ranking or a real-robot failure rate.

Fixed RGB, D to D MCIL: 53.9% isolated; five thresholds 48.9%, 12.9%, 2.6%, 0.5% and 0.08%.

CALVIN Figure 8, one 2022 baseline row; author measurement in simulation.

Run a check that can locate the bottleneck

Record the model version and test split, then freeze the chains, starts and timeouts before comparing policies. For every run, log the first unsuccessful step and the state it inherited. Report both isolated success and cumulative counts over all starts: at least one, two, three, four and all five. If the second step is a bottleneck, replay it from an approved clean start and from the state actually left by step one. That contrast can narrow the search without proving a cause. Report hardware and simulation separately.

Checklist for inputs, cumulative completion and the first unsuccessful transition.

BotsTest reporting suggestions, not a new experimental result.

Explore CALVIN in the BotsTest directory. Study: Mees et al., 2022, CALVIN v4. Drawings and reporting template are original illustrations, not simulator captures.

Turn the shortlist into a decision

Write down the task, operating conditions, success criteria and total ownership costs. Ask shortlisted suppliers to demonstrate your representative task, including failure recovery. Use the same acceptance test for each candidate.

Explore the full robotics directory →