Reproducibility Issues in Latent World Models
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 4 concepts
Key Takeaways
- Evaluated performance varies drastically from 84.0% to 14.0% based solely on whether the repo configuration or the appendix protocol is used.
- A corrected reproduction checkpoint achieved 94.0% success by adhering to specific pipeline conventions.
- One-step prediction accuracy is a poor proxy for long-horizon planning performance, as highly accurate models can still fail to reach goals.
- Four critical, undocumented implementation details were identified, including ImageNet pixel normalization and action z-scoring.
Summary & Methodology Analysis
The author audited the LeWorldModel by reimplementing its architecture, which consists of a ViT-Tiny vision encoder, a six-layer transformer predictor, and MLP projectors. By comparing 40 distinct components against the source code, the study identified four undocumented requirements: dense action gathering, programmatic action-encoder width, ImageNet pixel normalization, and action z-scoring. These conventions were essential to align the reproduction with the intended environment interactions. The analysis also addressed drifting batch normalization statistics by applying precise BN recalibration to correct validation loss inflation. Furthermore, the researchers utilized latent distance checks to confirm that the evaluation frames stayed within the training distribution, ensuring valid inference.
Performance metrics for the model were evaluated using the cross-entropy method (CEM), a planning algorithm that samples action sequences to optimize future rewards, on the TwoRoom environment. The results show that the model's learned encoder is highly effective at state representation, recovering agent position with a Pearson correlation of 0.9988 and an R2 of 0.9977. Despite this precise spatial mapping, the study found a disconnect between one-step prediction accuracy and planning success. In several cases, models that performed the best on one-step prediction actually ended up farther from the goal than a random-action policy when executing long-horizon tasks.
The findings are limited to the 18.03M-parameter configuration on the TwoRoom diagnostic task. Because the study utilized only a single seed for each configuration, it lacks data on performance variance. Additionally, while the paper's appendix suggests training for 10 epochs, the repository mentions 100 epochs, but this study adhered to the 10-epoch limit. Consequently, the results may not generalize to different model scales or environments, and the lack of multiple training seeds prevents a broader assessment of model stability.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem addressed by this paper?
The paper addresses the reproducibility of the LeWorldModel on the TwoRoom environment, specifically focusing on discrepancies between published evaluation protocols and actual source code requirements.
Q2. Does the model's one-step prediction accuracy guarantee success in planning?
No. The study found that one-step prediction accuracy fails to predict long-horizon planning success.
Q3. How much does the evaluation protocol impact reported success rates?
The evaluation protocol makes a massive difference. For example, the authors' checkpoint achieved 84.0% success using the repository's configuration, but only 14.0% using the protocol described in the appendix.
Q4. What architecture is used for the LeWorldModel?
The architecture includes a ViT-Tiny vision encoder, a six-layer transformer predictor, and MLP projectors.
Q5. What specific undocumented conventions were identified in the source code?
The study identified four undocumented conventions: dense action gathering, programmatic action-encoder width, ImageNet pixel normalization, and action z-scoring.
Q6. How did the researchers validate that the evaluation distributions were correct?
They used domain-specific precondition checks, specifically measuring the latent distance from evaluation frames to training neighbors.
Q7. What was the result of the linear probe on the learned encoder?
The learned encoder recovered agent position with a Pearson correlation of 0.9988 and an R2 of 0.9977.
Q8. Did the study train the model for 100 epochs as mentioned in the repo?
No. The study limited training runs to 10 epochs to remain consistent with the paper's appendix.
Q9. Does this paper provide a comprehensive look at how model scale affects results?
No. The study is specific to the 18.03M-parameter configuration and does not generalize to other scales.