Fast and Memory Efficient Exact Attention

Summary & Methodology Analysis

Transformers are slow and memory-hungry on long sequences because the time and memory complexity of self-attention is quadratic in sequence length. To solve this, the authors developed FlashAttention, which restructures the attention computation to split the input into blocks and makes several passes over input blocks to incrementally perform the softmax reduction. It stores the softmax normalization factor from the forward pass to quickly recompute attention on-chip in the backward pass. Furthermore, the authors implement FlashAttention in CUDA to achieve fine-grained control over memory access and fuse all attention operations into one GPU kernel, and they extend it to block-sparse attention by skipping zero blocks based on a predefined block sparsity mask.

The method yields significant performance improvements across multiple models and tasks. FlashAttention achieves a 15% end-to-end wall-clock speedup on BERT-large with a sequence length of 512 compared to the MLPerf 1.1 training speed record. It yields a 0.7 improvement in perplexity on GPT-2, 6.4 points of lift on long-document classification, and 61.4% accuracy on the Path-X challenge with a sequence length of 16K. Additionally, block-sparse FlashAttention achieves 63.1% accuracy on Path-256 with a sequence length of 64K. The models and datasets used in the research include BERT-large, GPT-2, BERT, Linformer, OpenWebtext, ListOps, Text, Retrieval, Image, Pathfinder, Performer, Reformer, Smyrf, RoBERTa, MIMIC-III, ECtHR, Path-X, Path-256, and Path-64.

Despite its strong performance, the approach has limitations. Writing a new CUDA kernel for each new attention implementation requires significant engineering effort and a considerably lower-level language than PyTorch. Furthermore, CUDA implementations may not be transferrable across GPU architectures. The paper does not specify any other limitations beyond these two engineering constraints.