Using Transformers for Image Recognition

Summary & Methodology Analysis

The paper investigates whether a pure Transformer architecture, which is standard for natural language processing, can perform well on image classification tasks without relying on convolutional networks. The methodology starts by splitting an input image into a sequence of fixed-size, non-overlapping two-dimensional patches. Each patch is flattened and mapped to a constant latent vector size D using a trainable linear projection to obtain patch embeddings. A learnable classification embedding, similar to BERT's class token, is prepended to the sequence of patch embeddings, and standard learnable one-dimensional position embeddings are added to retain positional information. This resulting sequence of embedding vectors is fed to a standard Transformer encoder, which consists of alternating layers of multiheaded self-attention and multilayer perceptron blocks, with LayerNorm applied before every block and residual connections after every block. The state of the classification token at the output is used as the image representation for classification by attaching a classification head. For fine-tuning on higher-resolution images, the patch size remains the same to result in a larger effective sequence length, and two-dimensional interpolation of the pre-trained position embeddings is performed based on their original image location.

Regarding performance, the Vision Transformer attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. Models pre-trained on the JFT-300M dataset outperform ResNet-based baselines on all datasets while taking substantially less computational resources to pre-train. With self-supervised pre-training, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, representing a significant improvement of 2% over training from scratch, though it remains 4% behind supervised pre-training. When pre-trained on mid-sized datasets like ImageNet without strong regularization, Vision Transformer models yield modest accuracies a few percentage points below ResNets of comparable size, and Vision Transformers overfit more than ResNets with comparable computational cost on smaller datasets. Additionally, when pre-trained on the smallest dataset, ImageNet, ViT-Large models underperform compared to ViT-Base models even with moderate regularization.

The approach faces several limitations. Transformers lack some of the inductive biases inherent to convolutional networks, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data. There is also a large gap between self-supervised and large-scale supervised pre-training performance for Vision Transformers. Furthermore, applying Vision Transformers to other computer vision tasks, such as detection and segmentation, remains a challenge. The paper does not specify precise dollar costs or exact latency figures for deployment, noting only resource consumption trends during training and pre-training.