Enhancing Video Reasoning in Multimodal Models

Summary & Methodology Analysis

Multimodal large language models typically process videos through sparse uniform sampling to control visual-token and attention costs, which are the computational weights determining how different parts of an input relate to each other. This sparse approach discards critical transitions between sampled frames and limits reasoning about object movement, collisions, and causal interactions. To fix this, the authors introduce a method that recovers dense point trajectories from the video using a frozen point tracker, which is a system that follows specific pixels across video frames without updating its own weights.

The system then compensates for global camera motion using a global similarity transformation and estimates frame-wise motion energy based on speed, acceleration, and curvature dimensions. It selects motion-informative frames using uniform anchors combined with greedy non-maximum suppression, a filtering algorithm that removes redundant overlapping predictions, alongside an importance-sampling fallback. Finally, it selects moving points and renders inter-frame trajectory segments as visual markers onto later sampled frames for the frozen multimodal large language model, which is an artificial intelligence system that processes both text and visual inputs.

Regarding results and limitations, the paper notes that on GPT-5.5, this approach achieves an accuracy of 80.0% on SSv2, outperforming uniform sampling by 8.9% and the strongest baseline, FOCUS, by 2.4%. However, an exception appears in the moving-direction task on GPT-5.5, where SoM achieves the highest score of 84.5% versus 75.5% for this method because object-centric identity cues provide less benefit for other tasks. The paper does not specify other computational costs, latencies, or hardware requirements.