Benchmarking Video Models for Physical Reasoning

Summary & Methodology Analysis

The paper addresses a significant gap in current video generation pipelines where models are often assessed purely on the final frame appearance rather than the underlying physical logic. To solve this, the authors created the Apple-pi benchmark, which utilizes the Orchard dataset. This dataset covers ten classical mechanics tasks. The benchmarking protocol forces models to engage in a structured three-stage pipeline consisting of Perception, Formulation, and Deduction. This approach moves beyond simple visual similarity by providing an infographic-style annotated first frame alongside a sequence of frames to evaluate if the model follows the expected physical chain of logic. For the assessment, the authors implemented a hybrid evaluation suite that merges subjective scores from MLLMs with objective, physics-law-based measures to verify that the output adheres to physical reality. Models tested include Seedance 2.0, GPT Image 2, Nano Banana 2, Wan2.2, HunyuanVideo-1.5, VBVR-Wan2.2, Veo 3.1, BAGEL, OmniGen2, SenseNova-U1-8B-MoT, and SenseNova-U1-8B-MoT-Think. Results indicate that GPT Image 2 leads the group with a score of 0.704, while Nano Banana 2 follows at 0.699, and Seedance 2.0 reaches 0.473. Despite these efforts, the current scope remains limited. The dataset only uses basic geometric primitives like spheres, cubes, cylinders, and cones. Furthermore, the evaluation environment relies on a single fixed camera angle, meaning the benchmark does not assess the model's ability to handle multi-view consistency or novel-view generation. The paper does not specify the computational cost or hardware requirements for running these evaluations.