Learning Visual Bottlenecks for Robot Policies

Summary & Methodology Analysis

The researchers developed a method to create data-efficient visuomotor policies by implementing task-conditioned visual bottlenecks. They utilize frozen DINOv3 patch features as a base, upon which a Seeker readout module builds. This module takes a task embedding and the robot proprioceptive state to initialize a query. This query then iterates through gated cross-attention (a mechanism that allows one sequence of data to selectively weigh information from another sequence) across the DINOv3 features to isolate relevant spatial cues. This process is further refined using a diffusion-based action prediction loss during pretraining to ensure the visual focus aligns with task success, ultimately converting the attention map into a bounding box or mask for downstream processing. The extracted region of interest is then used to filter inputs for RGB cropping and point-cloud filtering. The methodology results in a significant performance boost in both simulated and real-world robot tasks. By training the system to prioritize only the visual features that matter to the robot state and task, the system becomes more robust to environmental changes like varying lighting or backgrounds. However, the system is not free of operational costs. Incorporating Seeker requires a training overhead of 1.5 to 1.8 times compared to training the policy alone, and it adds 12ms of latency to the inference loop, which must be accounted for in time-sensitive robotics applications. Additionally, the approach relies on the current distribution of demonstrations, and significant shifts in task layouts may require further retraining or additional data to maintain performance.