Improving Humanoid Motion Tracking Evaluation

Summary & Methodology Analysis

Current humanoid tracking evaluation relies on kinematic metrics that often misrepresent human perception, specifically regarding physical artifacts like sliding or unstable contacts. To address this, the researchers established a unified evaluation framework using the MuJoCo simulator. This framework enforces identical reference indexing and rollout accounting across all tested tracking methods, ensuring consistency. They categorized motion data into four families: Daily, Highly Dynamic, Interaction, and Ground, and utilized a randomized interface to gather human preference comparisons for these trajectories. The researchers then developed HumanScore, a reward model based on a temporal Transformer, which is a neural network architecture that processes sequential data by weighing the importance of different temporal parts of an input to predict human preferences. By applying Bradley-Terry loss, a method used to estimate the probability of one item being preferred over another, and equality constraints, they trained the model on strict preference pairs and similar judgment pairs. This model achieves an alignment rate of 0.9083 with human preferences, compared to 0.8049 for MPJPE and 0.7882 for Foot Contact Accuracy. In practice, models like Humanoid-GPT excelled at Daily and Highly Dynamic motions, whereas SONIC performed best on Interaction and Ground categories. Despite these gains, the research has notable limitations. The benchmark is restricted to a single 29-degree-of-freedom humanoid embodiment in MuJoCo. Furthermore, HumanScore relies on simulator state features that may not be observable on real-world hardware, and optimizing the score as a reward function might lead to models exploiting internal imperfections rather than improving true performance.