Video as World and Event Stream

Summary & Methodology Analysis

The paper addresses the problem of enabling a general-purpose pretraining objective for native-streaming generation that transfers across various real-time tasks. To achieve this, the authors reframed video as a world plus an event stream, separating persistent world context from time-varying events. They defined a general-purpose pretraining task to predict how the world moves, changes, and responds in real time, given a world and incoming input. The persistent world context for real-time full-duplex audio-visual interaction is instantiated with scene, character, ambient-sound, and voice timbre settings. Furthermore, the agent's behavior channel is widened to open-vocabulary actions described in natural language by adopting a role-play chat format that interleaves spoken words with parenthesized behavior directives in the language output.

The system maps streaming multimodal user input to language-form speech and behavior actions using a vision-language-action-like multimodal understanding process. User and agent text, audio, and video are modeled on one causal timeline with a single Transformer, which is a deep learning architecture using attention mechanisms to process sequential data. This learns perception, response timing, speaking, visible listening, and synchronized video as one behavior inherited from v0.1. A Ulysses-style context-parallel performer handles expensive latent generation, inherited from v0.2. The world context is tokenized, meaning text or media is converted into numeric tokens for model ingestion, and prefilled once before streaming. Language, audio, and video inputs and outputs share a causal timeline coordinated by block-causal attention. Discrete parts of the event, including language and behavior directives, are optimized with next-token prediction. Continuous audio and video latents are generated with conditional flow matching under the same clean context, coupling speech, motion, appearance, and scene evolution. Finally, synchronized audio-video generation is conditioned based on predicted tokens using causal decoders.

The key result of this approach is that behavior directives in the language stream are realized as coherent facial expressions, posture changes, responses to sounds, and interactions with nearby objects, remaining synchronized with speech, grounded in the configured scene, and consistent with the character's identity and appearance. Regarding limitations, this paper exclusively studies the real-time full-duplex audio-visual interaction specialization and defers systematic adaptation and evaluation of other potential applications to future work.