Improving Visual Tool Use for Models

Summary & Methodology Analysis

OpenVisTool addresses the issue that large multimodal models often lose fine-grained details during image encoding, which limits their ability to process complex visual tasks. The authors propose a training recipe that explicitly teaches models to selectively use tools. The process begins with difficulty screening to identify queries that a base probe model cannot answer reliably on its own. They then perform domain-specific trajectory synthesis where a teacher model guides the collection of evidence for tasks like chart series isolation, table alignment, or web rendering. Finally, the training data is filtered by verifying that tool usage provides causal utility, meaning the model's reliability is demonstrably improved by the tool call rather than just arriving at the correct final result by chance. This approach forces the model to learn the functional necessity of each tool interaction. The authors validated the method across four backbones ranging from 4B to 27B parameters, achieving an average gain of 10.7 points on the OpenVisTool-Bench. The models consistently outperformed no-tool baselines on benchmarks like Agentic-MME and VTC-Bench. Despite these gains, the paper notes that cross-domain transfer can be asymmetric, where strategies optimized for one domain may clash with others, such as the conflict between frequent image cropping and the global full-page reasoning required for some tasks. Furthermore, in the GUI domain, specialized single-domain training may sometimes outperform the general multi-domain mixture.