Evaluating Video Models as World Simulators
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- No model among the eleven tested consistently matches physical reference probabilities or covers the range of valid future outcomes across 50 scenarios.
- Current systems often fail to differentiate between genuine causal interventions and non-causal visual cues, resulting in inaccurate probability shifts.
- The PAWEval system demonstrates an 81.3% agreement rate with human judgments across 888 video samples.
- The benchmark highlights that existing models struggle to maintain reliable performance as stochastic samplers of world dynamics.
Summary & Methodology Analysis
The researchers developed PAWBench to treat video generation systems as stochastic samplers, which means models should produce a probability distribution of potential futures rather than a single deterministic output. The methodology uses fixed initial observations and action prompts across multiple rollouts, then relies on the PAWEval system to map these generated videos to terminal outcomes based on a predefined rubric. To ensure these models act as accurate world models, the benchmark measures both calibration, where the model output matches known physical probability references, and coverage, which assesses whether the model generates the full breadth of valid outcomes in a given scenario. Models including HappyHorse, Seedance 2, LTX-2.5, Cosmos 3 Super I2V, and LingBot-Video-MoE were subjected to this evaluation to observe how they handle physical transition dynamics compared to superficial visual cues.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary purpose of PAWBench?
PAWBench is designed to evaluate video generation models as stochastic samplers of world dynamics rather than as static content creators.
Q2. Do current video models perform well on this benchmark?
No, the research shows that none of the eleven tested models consistently achieve accurate probabilities or broad coverage of valid futures across the 50 scenarios.
Q3. How reliable is the PAWEval system compared to human reviewers?
When tested on 888 videos, PAWEval agreed with the decisive human label on 722 samples, resulting in an 81.3% agreement rate.
Q4. What specific models were evaluated in this study?
The study evaluated eleven systems, including HappyHorse, Seedance 2, LTX-2.5, Cosmos 3 Super I2V, and LingBot-Video-MoE.
Q5. What is the main limitation regarding how the benchmark handles trajectories?
The benchmark evaluates stochastic futures using terminal outcomes, which simplifies testing but fails to capture full trajectory level dynamics or intermediate physical processes.
Q6. How does the sampling strategy affect the evaluation results?
Evaluation is based on a finite number of rollouts, which may limit the reliability of the estimated distributions; increasing the sampling budget improves reliability but increases costs and does not correct biased model distributions.
Q7. Does the benchmark cover complex or embodied environments?
No, the benchmark currently relies on curated, visually parseable scenarios and does not address complex, long horizon, or embodied environments.
Q8. How do models behave when presented with non-causal prompts?
The models tend to change their output distribution too little when physical transitions change, yet can shift incorrectly when only non-causal visual cues are altered.
Q9. Are there specific hardware or parameter requirements mentioned for running these models?
The paper does not specify hardware requirements, parameter counts, or computational budget details for the evaluated systems.