Evaluating Scientific Accuracy in Video Generation

Summary & Methodology Analysis

Sci-VBench addresses the limitation that many video generation models prioritize visual plausibility over factual scientific accuracy. To measure performance, researchers curated 1,253 tasks across 60 scientific subjects, requiring models to infer complex mechanisms from minimal prompts specifying only the initial setup and desired objective. The architecture relies on an automated scoring pipeline that uses rubric-conditioned MLLM-as-judge, which functions as an LLM that evaluates output based on specific provided criteria, to grade performance across four dimensions including prompt grounding, scientific and causal correctness, and spatiotemporal consistency. Low-level perceptual fidelity is assessed separately using vision tools from VBench. The evaluation was applied to 16 models including Sora-2, Veo-3.1, and various open-source architectures like CogVideoX1.5-5B and LTX-2.3. The results reveal that while models are comparable in perceptual quality, proprietary models hold a clear edge in reasoning-intensive tasks such as causal correctness and prompt grounding. Current limitations are notable for engineers, specifically that generating the full benchmark using commercial APIs is prohibitively expensive, and MLLM-as-judge systems still struggle to reliably detect fine-grained visual and temporal artifacts.