Training Robots to Predict Failed Actions
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 5 concepts
Key Takeaways
- FACT overcomes the bias of models that only learn from successful demonstrations by including failure rollouts in training.
- The model separates action generation from future prediction, preventing the hallucination of positive outcomes for bad actions.
- Prediction quality on failure scenarios improved significantly, with PSNR rising from 19.51 to 25.92.
- A two-stage inference process enables ranking of action candidates using learned progress estimates.
- Implementation requires an additional scoring pass if using the optional best-of-N selection for action candidates.
Summary & Methodology Analysis
Existing world-action models often suffer from a selection bias because they are trained primarily on successful agent demonstrations. This leads to hallucinations where the model incorrectly predicts successful future states even when given poor input actions. To address this, the authors propose FACT (Failure-Aware Causal Training), which treats failure rollouts as valid data for supervision. The architecture uses a teacher-forced action-conditioned mask, which is a mechanism that forces the model to learn a specific dependency by gating the flow of information between action generation and future state prediction. By training with a joint denoising loss (a process that learns to reconstruct data from noise), the model learns the consequences of both good and bad actions simultaneously without treating failed actions as targets for imitation.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem this paper solves?
Current models often hallucinate success for bad actions because they are only trained on successful demonstrations.
Q2. How does FACT improve robotic prediction?
It uses failure rollouts to explicitly train the model on the outcomes of failed actions alongside successful ones.
Q3. What does the model achieve in terms of performance?
It improves prediction quality on failure rollouts, increasing PSNR from 19.51 to 25.92.
Q4. How does the two-stage inference process work?
Stage 1 denoises an action chunk, and Stage 2 denoises value and future-video conditioned on that action.
Q5. Does this method hurt performance on normal tasks?
No, the paper reports that prediction quality for normal futures is not degraded.
Q6. What is the trade-off for using best-of-N selection?
It trades computation for reliability, as it requires an additional scoring pass during inference.
Q7. What datasets or baselines are mentioned in the paper?
The paper references various models including RoboTwin, Cosmos, π 0, X-VLA, π 0.5, Gigaworld-Policy, Motus, WAN2.2-5B, BagelVLA, DreamZero, Rt-1, OpenVLA, and Rt-2.
Q8. Are there limitations regarding the scale of the model?
The paper notes that scaling FACT to broader robot and human-interaction data may further improve performance.
Q9. Is the computational cost for training specified?
The paper does not specify the training cost or resource requirements.