Back to Feed
Computer Vision / Training & Fine-Tuning

Removing User Interface Overlays from Gameplay

Original: Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
By Wenxuan Shen, Dongna Jin, Dongping Chen Published 2026-08-25

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 3 concepts

Key Takeaways

  • Training world models on gameplay footage free of UI overlays yields a 6.83% improvement in VideoReward metrics.
  • GameCleaner is a mask-free model for removing UI elements that achieves an average AAR of 95.36 on synthetic test videos.
  • The Game2World engine provides 96K synthetic paired training clips and 1,079 in-the-wild clips across 303 games.
  • GameCleaner maintains 99.8% background preservation during the UI removal process.

Summary & Methodology Analysis

The researchers developed Game2World as an engine to address the noise introduced by heads-up displays and interface elements in raw gameplay data. The system generates a large-scale repository of 96K synthetic videos with precise reconstruction targets, which are then used to train GameCleaner. This model functions as a mask-free UI removal tool that leverages multimodal semantic understanding combined with video editing capabilities. To handle the underlying video processing, the team utilized Qwen3.5-9B to generate captions and fine-tuned a Wan2.1-T2V-1.3B video diffusion transformer, a model architecture that treats video generation as a sequence prediction task using attention mechanisms to process spatial and temporal dependencies.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary goal of this research?

The researchers aim to remove screen-space interface elements from gameplay videos to create cleaner training data for world models.

Q2. How does removing UI elements affect model performance?

Models trained on UI-free gameplay show a 6.83% improvement in overall VideoReward compared to those trained on data with UI overlays.

Q3. What tools did the researchers build to achieve this?

They built the Game2World engine for data preparation and the GameCleaner model for removing interface elements.

Q4. How well does GameCleaner perform on synthetic data?

GameCleaner achieves an average AAR of 95.36 on synthetic videos, which is 57.3% better than the strongest temporal mask baseline.

Q5. Does GameCleaner preserve the game background during removal?

Yes, the model achieves an in-the-wild AAR of 80.05 while maintaining 99.8% background preservation.

Q6. What specific models were used in the pipeline?

The team used Qwen3.5-9B for caption generation and fine-tuned Wan2.1-T2V-1.3B for the video generation tasks.

Q7. What are the current limitations regarding game diversity?

The current study does not yet cover the full diversity of all games, interfaces, and recording conditions.

Q8. What was the role of the downstream study in this paper?

The downstream study used a text-conditioned video generation model as a controlled proxy for world-model training rather than an action-conditioned gameplay world model.

Q9. Are there specific scenarios where GameCleaner struggles?

The model may produce imperfect reconstructions when dealing with large opaque interfaces, rapidly changing overlays, or UI elements that are visually entangled with the underlying scene.

Flag an issue

What is wrong with this summary?

Browse the archive →
Training & Fine-Tuning / Reinforcement Learning Improving AI Code Generation With Robust Testing The paper introduces a framework called RobustTests that improves AI code generation by synthesizing diverse, failure-inducing test cases to guide reinforcement learning. Training & Fine-Tuning / Reasoning Improving AI Reasoning Through Verifiable Distillation The paper introduces a method called OPDVR that aligns reinforcement learning signal with task success during model distillation to improve reasoning performance. Training & Fine-Tuning / Benchmarks & Evals Automated Self-Improvement for LLM Judges RecurSE enables LLM-based judges to improve their evaluation performance by creating a bounded, self-correcting feedback loop that eliminates the need for external gold standard rewards.