If you could add exactly 7,362 robot demonstrations, would you repeat the busiest scenes or cover more scenes? DROID's authors compared two samples of the same size. The more varied sample performed better in their out-of-distribution comparison. The question was which examples to collect, not how many.

Two DROID samples each have 7,362 demonstrations; one concentrates on 20 scenes while the other spans more.

Original illustration. The icons show the difference in selection, not the number of scenes.

Equal counts, different source slices

The concentrated sample takes the 20 scenes with the largest demonstration counts, yielding 7,362 trajectories. The broader sample draws the same number of eligible successful demonstrations uniformly at random from the DROID training pool. That means sampling demonstrations, not assigning every scene an equal quota. It does not mean the sample covers all DROID scenes.

Both training runs use 50 percent task-local data and 50 percent DROID source data, with equal source-slice sizes.

Both runs use the same batch mix. The source-data selection is different.

What the paper found

The DROID paper, Section V-C and Figure 8 compares the two samples across three out-of-distribution tasks. The broader sample performed better in the authors’ comparison. A shift can be an additional distractor or a novel target, not necessarily a new lab. This is a paper result, not a BotsTest replication.

A directional summary of the DROID authors’ out-of-distribution comparison, without estimated scores.

Authors’ results in the arXiv version, Figure 8. Do not read this as cross-hardware transfer.

A fair slicing test for your next collection

  1. Hold out complete scenes or collection sessions. Avoid scattering neighboring clips across train and test. Declare success, time limits and how to record interventions.
  2. Build two equal-size training slices. Keep the local examples, mixing weights, model configuration, training steps and seed set the same. Log differences in objects, tasks and operators too. If those differ, attribute the result to the selection strategy, not scene count alone.
  3. Report successes over attempts by condition. Use matched starting conditions, the same hardware and scoring. Compare familiar and shifted cases separately, and decide up front how much familiar-condition performance you are willing to trade.

Hold out scenes, match samples and training settings, and report successes over attempts for each shift.

BotsTest proposed acceptance checklist, not experimental data.

The comparison does not show transfer across hardware or remove the need for local examples. Before spending the next collection budget on repeats, check whether the missing cases are really new scenes. See the BotsTest DROID resource and the DROID research project.

Turn the shortlist into a decision

Write down the task, operating conditions, success criteria and total ownership costs. Ask shortlisted suppliers to demonstrate your representative task, including failure recovery. Use the same acceptance test for each candidate.

Explore the full robotics directory →