Connecting Vision Language Models To Robots

Summary & Methodology Analysis

This paper addresses how vision language models trained on internet scale data can be incorporated directly into end to end robotic control to boost generalization and enable emergent semantic reasoning. The authors build upon standard mechanics from RT-1, utilizing a base action space comprising 6-DoF positional and rotational displacement of the robot end-effector, the level of extension of the robot gripper, and a special discrete command for terminating the episode. Continuous action dimensions are discretized into 256 bins uniformly, representing the robot action using ordinals of the discrete bins as 8 integer numbers. The novel contribution is expressing these robot actions as text tokens and incorporating them directly into the training set of the vision language model in the same way as natural language tokens, forming vision language action models. Robot data is converted to be suitable for vision language model fine-tuning where inputs include a robot camera image and a textual task description using the standard VQA format, and the output is formatted as a string of numbers representing a robot action. These vision language models are co-fine-tuned on both robotic trajectory data and internet scale vision language tasks such as visual question answering while balancing sampling weights. During decoding, the output vocabulary is constrained by only sampling valid action tokens when prompted with a robot action task. Over the course of 6k robotic evaluations, RT-2 enables significant improvements to generalization over objects, scenes, and instructions, and exhibits a breadth of emergent capabilities inherited from web scale vision language pretraining. However, the model has clear limitations. The robot does not acquire any ability to perform new motions by virtue of including web scale pretraining, meaning the model's physical skills are still limited to the distribution of skills seen in the robot data. Additionally, the computation cost of large vision language models is high, and as these methods are applied to settings that demand high frequency control, real time inference may become a major bottleneck.