Optimizing Vision Language Models for Driving

Summary & Methodology Analysis

The researchers investigate how planning-related computations are distributed across the depth of a vision-language-action (VLA) model, a transformer architecture that processes visual input to output physical actions. To understand internal representations, the authors probed the planning token (a specialized internal vector representing future movement) at every intermediate decoder layer. They evaluated these tokens using native generative planners to measure trajectory compatibility, while also applying multinomial logistic regression to assess the linear decodability of navigation commands. Additionally, they trained auxiliary readouts like MLP adapters (small neural network modules used to project features between spaces) to map early layer representations to the final-layer token space, confirming that planning information is available earlier than traditionally assumed. The methodology culminates in a pruning strategy where decoder layers are ranked by the angular deviation they induce on the planning token. Layers with high input-output cosine similarity (indicating they contribute less to internal state changes) are removed to trace an accuracy-latency frontier. This provides a systematic way to reduce computational overhead without immediately collapsing the model's performance on navigation tasks. The study notes that navigation intent is identifiable at 97.7 percent accuracy from the first layer, significantly higher than the 16.7 percent chance baseline. However, open-loop performance in terms of Avg-L2 error improves steadily as more layers are kept, bottoming out at 2.11 m at the final layer. The findings are limited by the specific model checkpoint and evaluation benchmark used. Furthermore, the paper explicitly notes that pruning results are based on open-loop evaluation and do not confirm closed-loop safety or readiness for deployment in real world scenarios. Finally, the researchers acknowledge that their analysis of per-family pruning effects is statistically underpowered for formal equivalence testing.