Auditing Visual Tool-Use in Multimodal Models

Summary & Methodology Analysis

The paper addresses the marginal or negative accuracy gains often observed when multimodal large language models use active visual operations, specifically the thinking-with-images paradigm involving crop-and-zoom, and investigates how returned visual evidence causally influences final model predictions. To analyze this, the authors formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. They implement a three-level intervention protocol consisting of policy, trajectory by corrupting all observations during rollout, and step using counterfactual observation replacement. Furthermore, they define Visual Evidence Gain as the step-level estimand using a counterfactual intervention to isolate the Natural Indirect Effect along the tool-action-observation-answer path, and they categorize trajectory rollouts into four diagnostic groups including No-call, Mode 1: Calling Without Looking, Mode 2: Looking Without Planning, and Calibrated, before decomposing policy-level accuracy improvement into per-group contributions.

The experimental evaluation spans models and benchmarks including DeepEyes, Pixel Reasoner, Mini-o3, Qwen3-VL-4B, Qwen3-VL-8B, Thyme, V*, HR-Bench, VisualProbe, and MME-RealWorld-Lite. The key finding from this analysis is that accuracy improvements from visual tool-use are primarily driven by a calibrated minority of rollouts, revealing a significant miscalibration in how these multimodal systems deploy visual tools during inference. The paper does not specify precise execution latency, memory consumption, or dollar cost metrics for these evaluations.

Several limitations are noted regarding the scope and assumptions of the work. The study is restricted to open-source models, and conclusions may not extend to closed-source models like OpenAI o3 or o4-mini which lack access to internal probability or intermediate observation interventions. Additionally, the causal framework was tested specifically using crop-and-zoom operations, meaning other visual tools might exhibit different behaviors. Finally, the attribution of policy miscalibration to outcome-only reinforcement learning, which optimizes models based only on final outcomes rather than intermediate steps, is a hypothesis rather than an experimentally established causal fact.