Agentic Optimization for Image-to-Video Adherence

Summary & Methodology Analysis

Modern black-box Image-to-Video models suffer from stochasticity and a lack of fine-grained control, forcing users into inefficient, brute-force trial-and-error workflows to achieve precise adherence to creative briefs. To solve this, the authors introduce an agentic optimization framework that coordinates multiple evaluation and refinement steps. The pipeline relies on iterative prompt refinement using an mLLM, which stands for multimodal large language model that processes both text and images, to improve the user initial input. It then applies automated evaluation using Davidsonian Scene Graph queries to ensure semantic adherence, alongside Common Mistake Questions to detect visual artifacts and temporal inconsistencies. A novel contribution of this work is the Bayesian optimization of model hyperparameters, specifically targeting random seeds and classifier free guidance scales, which is abbreviated as CFG scales. To tie the loop together, the framework applies a Video-Text Adherence metric, known as VTA, to rank outputs and guide the optimization loop.

Under the hood, the system coordinates several models and tools. It utilizes Veo 2.0 alongside Gemini 2.5 Pro, and evaluates performance using V-Bench, RAHF, UVQ, and Vizier. In experimental evaluations, the framework achieved human preference win rates of up to 69 percent against a baseline random search. Additionally, Gemini 2.5 Pro achieved 92 percent accuracy on Davidsonian Scene Graph questions and 82 percent accuracy on Common Mistake Questions when validating its reliability as an evaluator. These components work together to automate what was previously a manual prompt engineering process, shifting the burden from the human operator to the agentic loop.

Despite these performance gains, the approach has notable constraints. The framework involves a significant computational cost compared to unguided manual iteration. Furthermore, the mLLM-guided approach is constrained by current limitations in video understanding, specifically regarding nuanced motion and temporal dynamics. The paper does not specify the exact hardware infrastructure required to run this loop, nor does it detail the absolute latency per optimization step. Engineers adopting this pattern must weigh the higher human preference win rates against the increased compute overhead.