Reducing Matrix Multiplication for LLM Inference

Summary & Methodology Analysis

Transformer-based language models incur substantial inference costs due to repeated high-dimensional matrix multiplications, raising the question of whether all indices along shared multiplication axes must be evaluated for every input or if computation can be reduced adaptively without modifying model weights. To address this, the paper presents Reduced Matrix Multiplication. The method assigns each feature dimension an importance score equal to the L2 norm of the corresponding column of the activation matrix, selects a deterministic index set using TopK selection over the feature scores based on a user-controlled retention ratio, and executes the matrix multiplication using only the gathered columns and rows. For attention blocks, which are a core architectural mechanism for weighing relationships between tokens in transformers, it computes feature scores per head on queries, selects top dimensions, and evaluates reduced attention scores. It can also optionally sparsify the attention-value product over the token dimension by computing token scores from attention weights, selecting top token positions, and evaluating the reduced output.

In terms of empirical performance, on LLaMA 3.1 8B at a retention ratio of 0.5, Reduced Matrix Multiplication achieves an average zero-shot QA accuracy of 59.8 percent, outperforming SparseGPT at 56.1 percent, Wanda at 52.7 percent, SliceGPT at 37.0 percent, and Magnitude pruning at 39.3 percent. On CNN/DailyMail summarization using LLaMA 3.1 8B at a retention ratio of 0.5, it achieves a ROUGE-1 score of 34.2, ROUGE-2 of 13.6, ROUGE-L of 22.0, ROUGE-Lsum of 28.7, and a BERTScore of 85.8. The evaluation spans a wide range of models and datasets, including LLaMA 3.1 70B, LLaMA 3.1 8B, Llama 3.2 3B, Llama 3.2 1.5B, Qwen3 32B, Qwen 3.1 7B, Qwen2.5-VL-7B-Instruct, Copa, PiQA, CommonsenseQA, ARC-Easy, ARC-Challenge, MMLU, WikiText, BookCorpus, GSM8K, HumanEval, Ruler-CWE, Ruler-Hotpot, CNN/DailyMail, POPE, Blink Art Style, Blink Forensic Detection, and Blink Counting.

Despite these strong results, the approach comes with specific limitations. Reduction tolerance depends on the model family, task, component, and retention ratio. Furthermore, MLP components, which are feed-forward neural network layers operating on the hidden representations, are much more sensitive to reduction compared to attention-side computations. Finally, the authors did not explore extensions beyond the training-free setting or conduct a detailed study of how dynamic reduction behaves across different components of vision-language models.