In session 674, two policies both completed a task involving a computer mouse. The evaluator chose B because it finished on the first try. In their account, A appeared to drop the mouse after lifting it, then picked it up again and completed the task. Both received the same progress score. See RoboArena v2, Appendix D, Table 3. That is one evaluator’s observation, not a mechanical diagnosis or a BotsTest test.

Two illustrated A/B policies complete a computer-mouse task; A may have dropped and re-grasped, B finishes on its first try.

Session 674, based on the evaluator’s account, not footage or a BotsTest test.

Completion and preference answer different questions

A progress score asks how much of the task got done. A preference asks which performance the observer would rather have and why. The first-try finish mattered in this pair, but a person might prefer something different in another task. A preference does not tell you an exact time or establish which policy is better overall.

What makes it a fair pair?

RoboArena hides policy identities and matches the task and initial setup as closely as possible within each pair. Later pairs can use other scenes and tasks. Each run ends at completion or its preset timeout. The authors’ protocol, Section 3.1.

Two policies have matching start positions within a pair; other pairs can use different layouts.

RoboArena’s paired protocol, illustrated.

The scope and the denominator

The authors report 612 real-world comparisons, across 7 universities and 7 policies on DROID hardware. The 612 counts pairs, not individual rollouts or repeats of the mouse task. The authors report preference and progress feedback disagreed in 11% of A/B evaluations, including equal-progress pairs with a preference and cases in which the two signals pointed in opposite directions. It is not the fraction of pairs in which both finished.

612 pairs, seven policies and seven universities, using DROID hardware.

Reported study scope, not a cross-hardware finding.

Run an A/B test on your own bench

  1. Define success and the process metrics. Specify retries, recovery, intervention and timeout. Define start and stop events if you measure time.
  2. Match the pair, vary the next pair. Reset the scene, use the same instructions, hide identities, and document the trial order. Change an object or target in later pairs.
  3. Keep both measures. Log successes and valid attempts, and separately report A wins, B wins, ties and reasons. Do not generalize a single evaluator’s choice to another hardware platform.

A test template separates completion, retries, recovery, time, intervention and preference.

A proposed scoring sheet, not the authors’ data.

Explore RoboArena in the BotsTest directory.

Turn the shortlist into a decision

Write down the task, operating conditions, success criteria and total ownership costs. Ask shortlisted suppliers to demonstrate your representative task, including failure recovery. Use the same acceptance test for each candidate.

Explore the full robotics directory →