Agentic Evaluation for Video Generation Models

Summary & Methodology Analysis

VideoArgus addresses the fragmentation and rigidity of existing video evaluation pipelines by implementing an agentic, rubric grounded approach. Instead of relying on fixed benchmarks, the system generates a unique, output blind rubric for every input. This rubric serves as a blueprint for the evaluation, explicitly defining criteria, scoring rules, and evidence plans. By freezing this rubric, the system ensures that multiple candidate models are held to identical requirements, eliminating the inconsistency inherent in traditional, static testing suites.

The core execution logic utilizes an agentic workflow that selectively triggers Visual Language Model (VLM) based Question Answering and specialized computer vision tools. Depending on the rubric evidence plan, the agent may deploy specific utilities for tracking, OCR, depth estimation, or visual similarity analysis. Once these tools return data, the system aggregates the findings into criterion level scores, detailed rationales, and diagnostic reports. To prevent skewed results, it applies importance weighting and implements hard criterion failure caps to ensure that catastrophic model errors are penalized appropriately.

While the framework consistently outperforms previous benchmarks across five tasks, it is not without practical trade-offs. The authors note that while the evaluation backbones demonstrate consistent rankings, the reliance on external APIs to generate the initial instance specific rubrics introduces an additional cost per evaluation. The paper does not specify the exact latency or financial cost per API call, nor does it provide a comparison of the inference time versus previous static benchmarks.