Testing Visual Grounding in Vision Language Models

Summary & Methodology Analysis

The researchers evaluated various vision language models (VLMs), which are systems that process both images and text, by implementing quadrotor and ground vehicle controllers within highway env and gym pybullet drones environments. To verify if these models actually ground their actions in visual data, the team performed an input ablation battery (a testing process where specific inputs are systematically removed or manipulated). This included blind image controls, repeated inputs, lane axis reflections, and non visual baselines, alongside an exploratory VLM MPC hierarchy that attempts to recover high level visual intent through low level optimization using model predictive control (MPC), which is an algorithm that computes future trajectories based on a system model. The study utilized a range of models including LLaVA 1.5 7B, the Qwen series, the Gemma series, and the MiniCPM series. The team also implemented a renderer calibrated image only positive control to isolate geometric estimation capabilities. To address hazard assessment reliability, they created a post hoc symmetry consensus guardian, a mechanism that aggregates predictions from multiple models and reflected views to filter outputs. This guardian performed well, reaching a balanced accuracy of 0.954 on 272 held out frames, and the positive control measured lead gaps with 0.090 m mean absolute error (MAE). The analysis highlights significant limitations, notably that no tested VLM could concurrently satisfy both longitudinal and lateral grounding requirements. Furthermore, performance proved highly sensitive to prompt formulation, making null results potentially inconclusive. The study also cautions that MPC cannot recover missing geometric intent if the high level VLM output is collapsed or lacks sufficient detail.