Tool Calls Function Without Image Feedback

Summary & Methodology Analysis

The authors investigate the Tool Call Scaffold Hypothesis, which posits that the structured text generated during a tool call acts as the primary reasoning signal for vision language models (VLMs), effectively rendering the visual feedback from tools redundant. To test this, they introduced TextCall, an intervention that replaces all tool returned images with a specific text placeholder, [Image output skipped]. This approach was rigorously evaluated using Qwen2.5-VL-7B-Instruct, Qwen2-VL, Qwen3-VL, and DeepEyes V2 across various configurations including LoRA (a parameter efficient fine-tuning technique that adapts models by updating only a small subset of weights) and reinforcement learning (a training paradigm where models learn via trial and error to maximize a reward). By performing a factorial data audit on 1,000 training trajectories, the researchers isolated the specific contributions of visual data versus scaffold text, confirming that the structural logic provided by the text is the core driver of reasoning success. Component specificity was further analyzed by decomposing the scaffold into reasoning text and spatial code to verify their individual impact on performance. Results indicate that TextCall consistently matches or exceeds the accuracy of traditional image based approaches while achieving a latency reduction of 29 to 46 percent. Furthermore, the study identifies that reinforcement learning models trained with TextCall maintain effective tool usage, whereas standard models trained with image feedback often collapse into simple direct answering strategies. These findings suggest significant potential for optimizing agent architectures by bypassing image processing overhead during tool invocation. The authors explicitly note that these claims are scoped to current training distributions and evaluation benchmarks, such as V*Bench, HR-Bench-4K, and BLINK. They do not claim universal sample efficiency across all systems, emphasizing that future, more complex tasks might eventually require actual visual data for reasoning. Consequently, the findings remain constrained by the current state of these specific datasets.