Evaluating Robot Task Progress and Quality

Summary & Methodology Analysis

The PRM-as-a-Judge 1.5 toolkit shifts away from traditional binary pass-fail benchmarks to evaluate robotic trajectories through process assessment. It utilizes a Process Reward Model (PRM), a system that assigns a scalar value to partial sequence steps to estimate progress, to generate progress curves from rollout videos. By mapping the Case ID, task instructions, and video data, the system creates a detailed report that visualizes how the model advances toward task completion. This methodology integrates the existing Outcome-Process-Diagnosis (OPD) framework to extract specific performance indicators like FNS, DRR, and SQS, allowing for a more nuanced analysis of execution quality and failure modes.

The benchmark suite covers a variety of models including GR00T-N1.7, GalaxeaVLA, InternVLA-A1, pi0, pi0.5, Spirit v1.5, StarVLA-alpha, X-VLA, Xiaomi-Robotics-0/1, Hy-Embodied-0.5-VLA, and several WAMs such as Spatial Forcing, X-WAM, GigaWorld-Policy-0, AHA-WAM, and Fast-WAM. Evaluations conducted on RoboDojo, RoboDojo-Sim, and RoboDojo-RealWorld show that VLAs generally hold top-tier rankings compared to WAMs. The assessment process is supported by specific utilities like RoboPulse++, LDA-1B, Robo-Dopamine, RoboReward, RoboMeter, and RynnValue.

Despite these advancements, the framework faces notable limitations. The progress judges currently struggle to detect regression in trajectories more than they detect positive progress. Furthermore, there is only a weak correlation between performance measured in simulation and actual performance in real-world environments. This gap is most pronounced in tasks requiring high-precision alignment or complex physical contact, where models exhibit significantly larger performance degradation in real-world deployment than simulation results might suggest.