Zero Shot Text to Image Generation

Summary & Methodology Analysis

Traditional text to image generation has relied on finding better modeling assumptions for training on a fixed dataset using complex architectures, auxiliary losses, or side information. However, these current approaches often suffer from severe artifacts and limited dataset and model sizes. To address this, the authors introduce a two stage method. In the first stage, they train a discrete variational autoencoder, a neural network that compresses high dimensional data into a compact latent space, to compress each 256 by 256 RGB image into a 32 by 32 grid of image tokens with a vocabulary size of 8192. This is done by maximizing a relaxed evidence lower bound, a statistical optimization objective, using the gumbel softmax relaxation, a technique for making categorical sampling differentiable. In the second stage, up to 256 BPE encoded text tokens, meaning words split into subword units, are concatenated with the 1024 image tokens. The system then trains an autoregressive 12 billion parameter sparse transformer, a neural network architecture that processes sequences by predicting the next token while selectively attending to subsets of inputs, to model the joint distribution over text and image tokens as a single stream of data. Finally, samples drawn from the transformer are reranked using a pretrained contrastive model, which scores how well an image matches a caption.

Despite strong human evaluation results where samples were chosen as best matching the shared caption 93.3 percent of the time against DF-GAN on MS-COCO captions, the model has notable limitations. Details such as the texture of a cat's fur, writing on a storefront, and thin lines in illustrations are sometimes lost or distorted during dVAE reconstruction. The heavy compression from the dVAE encoder renders the model unable to produce high frequency details. Furthermore, generated samples can suffer from severe artifacts, including object distortion, illogical object placement, or unnatural blending of foreground and background elements. The model also performs inconsistently on tasks requiring variable binding, such as walking a dog, sometimes drawing both animals with Christmas sweaters or a hedgehog walking a smaller hedgehog.

The paper utilizes several datasets and baseline models for training and evaluation, including MS-COCO, Conceptual Captions, JFT-300M, YFCC100M, ImageNet, CUB, CUB-200, DRAW, AttnGAN, DM-GAN, and DF-GAN. The paper does not specify precise hardware infrastructure costs, exact training durations, or inference latency numbers beyond the model parameter count and architectural details.