Efficient Memory for Real-time Video Synthesis

Summary & Methodology Analysis

The NSTM architecture addresses the instability and performance bottlenecks of standard streaming video synthesis by decoupling the frequency of memory updates from the synthesis process. Instead of forcing parameter updates at every frame, which triggers excessive overhead and instability, the system performs periodic updates. To reconcile motion misalignment between these snapshots and the current incoming video, the model employs cross-view attention, a mechanism that allows the system to weigh the relevance of data from different perspectives to correctly synthesize a coherent scene. The system further stabilizes the parameter space by using a Memory Caching strategy that aggregates historical checkpoints and applies Newton-Schulz orthonormalization (Muon), a technique that enforces mathematical constraints on weight matrices to ensure stability during updates. To further ensure reliable reconstruction, an auxiliary Memory Loss objective is applied during training to force the memory module to reconstruct scene data independently of external inputs, while an L2 objective replaces the standard dot-product loss to prevent magnitude inflation in the weights. The system achieves a memorization step of 58.14ms and a synthesis step of 27.01ms on an H100 GPU, demonstrating its capability to handle long-horizon tasks, maintaining an mPSNR above 20 dB, whereas competing models like LaCT-NVS drop by 5.47 dB and Token-Mem by 1.56 dB over the same duration. Despite these gains, the system faces limitations including a fixed memory capacity and a bias toward early-stage memorization. Events occurring between periodic updates may be missed, and the model struggles with complex long-range camera and subject motion within the compressed memory state.