Testing Active Vision in Multimodal Models

Summary & Methodology Analysis

The paper investigates whether current multimodal large language models, which are models that process multiple data modalities such as text and images, exercise active observation. Active observation is the iterative, hypothesis-driven looking loop essential for human vision. Standard and existing mechanics mean that multimodal large language models encode images once as a fixed sequence of visual tokens during autoregressive reasoning, which is the process of generating text token by token, without an explicit perception-action loop. Classical computer vision scripts use primitives such as findContours, Canny, and template matching. The paper introduces a novel contribution called ActiveVision, a benchmark comprising 17 tasks across three cognitive categories. These categories are Distributed Scanning, Sequential Traversal, and Visual Attribute Transfer, and they are designed to force repeated visual perception rather than a single static description. The tasks are constructed via a two-stage pipeline where procedural scaffolds specify exact geometry and GPT-image-2 re-renders them into photorealistic images while preserving positions, counts, labels, and topology. Evaluated models include GPT-5.5, Claude Fable 5, Gemini 3.1 Pro, and others alongside benchmarks like MMMU-Pro, CharXiv, and MMVP.

Key results show that the highest-scoring model evaluated, GPT-5.5 at its highest reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks. Claude Fable 5 solves just 3.5% of ActiveVision items. In contrast, three human participants average 96.1% accuracy on the benchmark, and autonomous coding agents built on frontier models reach between 24.7% and 50.6% accuracy. Other models and datasets mentioned in the evaluation scope include Codex, Claude Code, Claude Opus 4.8, Gemini 3.5 Flash, MMStar, CV-Bench, BLINK, BlindTest, ArtQA, BabyVision, ChildBench, KidGym, OSWorld, and BenchCAD. The paper does not specify compute infrastructure, exact latency, or dollar cost figures beyond these evaluation accuracy percentages.

Limitations of the approach include the fact that generated images are synthetic even with photorealistic re-rendering, produced from a controlled prompt rather than natural-image distribution samples. Additionally, as models improve at describing images, the benchmark property that no short language description carries the answer can erode. The paper does not specify further limitations or hardware requirements for running the evaluations.