Building a General Purpose Visual Assistant

Summary & Methodology Analysis

The paper addresses the challenge of extending instruction tuning, the process of training a model on prompt-response pairs to follow user commands, into the language-image multimodal space. To achieve this, the authors connected a pre-trained CLIP visual encoder, a machine learning model that aligns images and text, with the Vicuna language decoder using a linear layer. They pre-trained this model on filtered image-text pairs from CC3M while keeping the visual encoder and large language model weights frozen, training only the projection matrix for feature alignment. The novel contribution of the paper is leveraging language-only GPT-4 or ChatGPT with symbolic representations, specifically captions and bounding boxes, to automatically generate a large-scale multimodal instruction-following dataset called LLaVA-Instruct-158K. Following this, the authors fine-tuned both the projection layer and large language model weights end-to-end on the generated multimodal instruction data across conversations, detailed descriptions, and complex reasoning tasks.

In terms of architecture and results, the system leverages the synergy between visual feature extraction and large language decoders. When evaluated, LLaVA yields an 85.1 percent relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. Furthermore, when fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53 percent, while using LLaVA alone on ScienceQA yields 90.92 percent accuracy. The model also achieves an impressive 81.7 percent performance on complex reasoning questions on LLaVA-Bench in the wild, with an overall score of 67.3 percent. The models and datasets involved in this ecosystem include LLaVA, GPT-4, ChatGPT, LLaMA, Alpaca, Vicuna, GPT-4-LLM, CLIP, CC3M, COCO, Science QA, Flamingo, BLIP-2, FROMAGe, KOSMOS-1, PaLM-E, OpenFlamingo, LLaMA-Adapter, T5, PaLM, OPT, InstructGPT, FLAN-T5, FLAN-PaLM, OPT-IML, LLaVA-Bench, CC-595K, LLaVA-Instruct-158K, and CC-Val-2014.

Despite strong performance, the paper notes several important limitations. LLaVA might generate outputs that are not grounded in facts or input data, raising concerns in critical applications like medical use. Additionally, bias can be transferred from the base vision encoder and language decoder models to LLaVA, potentially leading to biased outcomes or unfair representations. Finally, the model occasionally perceives the image as a bag of patches, failing to grasp complex semantics within the image. The paper does not specify precise hardware infrastructure costs, latency numbers in milliseconds, or exact memory footprints beyond these architectural constraints.