8-bit Matrix Multiplication for Large Language Models

Summary & Methodology Analysis

Large language models require significant GPU memory for inference, and existing 8-bit quantization methods degrade performance, require post-training tuning, and have only been studied for models under 350M parameters. The paper introduces LLM.int8() to solve this memory issue without performance degradation. The method works by loading a 16-bit or 32-bit checkpoint for transformer models, which are neural network architectures using attention mechanisms, and applying vector-wise quantization using separate normalization constants for each inner product in the matrix multiplication to quantize most features. To prevent performance loss, it isolates emergent outlier feature dimensions into a 16-bit matrix multiplication via a mixed-precision decomposition scheme while multiplying more than 99.9% of values in 8-bit. It then dequantizes 8-bit matrix multiplication outputs by the outer product of column and row normalization constants, and accumulates outputs from both outlier and regular matrix multiplications in 16-bit floating-point outputs for immediate inference without performance degradation. At the 6.7B scale, 150,000 outliers occur per sequence, concentrated in 6 feature dimensions. Furthermore, setting outlier feature dimensions to zero decreases top-1 attention softmax probability mass by more than 20% and degrades validation perplexity by 600-1000% despite making up about 0.1% of all input features.

The research evaluates models and datasets including OPT-175B, BLOOM, BLOOM-176B, GPT-3, OPT, WinoGrande, HellaSwag, PIQA, LAMBADA, C4, Books, English Wikipedia, CC-News, OpenWebText, CC-Stories, English CC100, GPT-2, GPT-J, RoBERTa, GLUE, WMT14, and WMT16. The paper demonstrates that LLM.int8() enables zero-degradation inference for language models up to 175B parameters.

Despite its effectiveness, the paper notes several limitations. The analysis is solely focused on the Int8 data type and does not study 8-bit floating-point data types. The study is restricted to models with up to 175B parameters, and additional emergent properties might disrupt the quantization methods at larger scales. Additionally, Int8 multiplication is not used for the attention function, and the work focuses on inference and does not study training or finetuning.