Generalizing Transformer Attention with Power Laws

Summary & Methodology Analysis

The paper replaces the traditional fixed bilinear form of scaled dot-product attention with a learned, input-conditioned operator called G_LM. This operator is constructed by mapping the query Gram operator through a residual metric learner to create a positive tensor, which is then refined using an elementwise power law with learned exponents. The final score operator is formed by superposing this potential tensor, allowing the model to adapt its attention mechanism based on input data rather than relying on a static mathematical structure. Standard transformers are recovered as a special case when this learned operator simplifies to an identity matrix. The model architecture also exposes intermediate tensors, which can be utilized for spectral analysis and regularization. While the approach provides a more flexible attention mechanism, the empirical results show that the learned operators become effectively invariant during inference. This collapse results in constant operator behavior, with measured relative fluctuations ranging between 10^-6 and 10^-11. The paper uses this stability to quantify the performance of the model on benchmarks including TruthfulQA, WikiText-2, ARC-Easy, ARC-Challenge, HellaSwag, PIQA, Social-IQa, WinoGrande, OpenBookQA, and Lean 4. Limitations include the fact that the inference-collapse is a measured hypothesis rather than a proven theorem. Furthermore, the mathematical treatment of self-organized criticality remains a phenomenological framework. The current perturbation bounds provide insight into the stability of cached inference but do not provide formal certification.