Back to Feed
Computer Vision

Using Transformers for Computer Vision Tasks

Original: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
By Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby Published 2020-10-22

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • The Vision Transformer treats images as sequences of patches to leverage the Transformer encoder architecture.
  • This approach achieves state of the art results on several benchmarks.
  • The model requires less computational training resources than comparable ResNet models.
  • Vision Transformers are less effective on smaller datasets due to a lack of image specific inductive biases.

Summary & Methodology Analysis

The researchers adapted the standard Transformer encoder, which utilizes multiheaded self attention (a mechanism that allows the model to weigh the importance of different parts of the input data), for image recognition. Instead of processing pixels directly, they split images into fixed size square patches. Each patch is flattened into a vector and mapped to a latent space using a trainable linear projection. A learnable classification token is prepended to the sequence, and learnable 1D position embeddings are added to provide spatial context before the sequence is processed by the Transformer encoder blocks.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the core contribution of this paper?

The paper demonstrates that applying a standard Transformer architecture directly to image patches allows for competitive or superior image recognition performance compared to traditional convolutional networks.

Q2. Does this model work for all image datasets?

No. The paper notes that Vision Transformers perform poorly on smaller datasets compared to CNNs.

Q3. How does the performance compare to existing models?

The Vision Transformer achieves state of the art results on several benchmarks while requiring less computational training resources than comparable ResNet models.

Q4. How are images represented for the model?

Images are split into fixed size square patches, which are then flattened and mapped to a latent vector dimension using a trainable linear projection.

Q5. What role does the classification token play?

A learnable classification token is prepended to the sequence of patch embeddings, and the final classification head is attached to this specific token output.

Q6. What is the standard architecture used?

The model uses the original Transformer encoder, which consists of alternating layers of multiheaded self attention and MLP blocks with Layernorm and residual connections.

Q7. How are positions handled in the images?

The model uses standard learnable 1D position embeddings added to the patch embeddings.

Q8. How does self supervised pre-training compare for this model?

The paper states that self supervised pre-training using masked patch prediction for ViT-B/16 currently lags behind supervised pre-training.

Q9. Does the paper specify the exact hardware requirements?

The paper does not specify the exact hardware requirements, only that it requires less computational training resources than comparable ResNet based models.