Flamingo Visual Language Model

Summary & Methodology Analysis

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. To address this, the authors introduce Flamingo, a system trained by pretraining the vision encoder using a contrastive objective on datasets of image and text pairs, using the output of the final stage's 2D spatial grid of features flattened to a 1D sequence. A Perceiver Resampler module receives spatio-temporal features from the vision encoder and outputs a fixed number of visual tokens. Additionally, the architecture inserts freshly initialised gated cross-attention dense layers, which allow a neural network to focus on specific parts of input data, between interleaved pretrained and frozen text-only language model blocks. The model applies an image-causal modeling cross-attention masking scheme where the model attends to the visual tokens of the single image that appeared just before the text token. Training is performed by minimizing a weighted sum of per-dataset expected negative log-likelihoods of text given the visual inputs across a mixture of interleaved image-text data, image-text pairs, and video-text pairs.

Flamingo sets a new few-shot state of the art across 16 multimodal language and image or video understanding tasks, outperforming models fine-tuned on thousands of times more task-specific data. Fine-tuning, the process of adapting a pre-trained machine learning model to a specific task by training it further on task-specific data, is bypassed or outperformed because on 6 of the 16 tasks considered, Flamingo outperforms the fine-tuned state of the art using only 32 task-specific examples. The models or datasets involved span across Flamingo, Flamingo-3B, Flamingo-9B, Flamingo-80B, Chinchilla, Perceiver, DETR, NFNet, M3W, ALIGN, LTIP, VTP, COCO, OKVQA, VQAv2, MSVDQA, VATEX, VizWiz, Flick30K, MSRVTTQA, iVQA, YouCook2, STAR, VisDial, TextVQA, NextQA, HatefulMemes, RareAct, ImageNet, Kinetics700, LAION-400M, C4, and MassiveText.

Despite the strong results, Flamingo models inherit the weaknesses of the pretrained language models they are built upon, such as language model priors potentially causing occasional hallucinations and ungrounded guesses. Furthermore, language models generalize poorly to sequences longer than those seen during training and suffer from poor sample efficiency during pretraining. The paper does not specify the exact latency, deployment infrastructure, or dollar costs required to run these models in production environments.