Fixing ALiBi Positional Encoding Numerical Errors
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- ALiBi's linear bias causes floating-point underflow at long ranges, making attention heads go blind.
- At a distance of 2048 tokens, 36.6 percent of entries in the attention matrix experience underflow.
- Applying bias clamping and log-scaled distances significantly improves retrieval performance, raising AUC from 0.08 to 0.79.
- ALiBi remains a strong baseline for needle-in-a-haystack tasks despite this inherent failure mode.
Summary & Methodology Analysis
The researchers analyzed the ALiBi positional encoding, a mechanism that injects a linear bias into the attention matrix (a grid determining how tokens relate to one another) based on their distance. The team mathematically derived the conditions under which standard floating-point formats like fp32 and bf16 fail, finding that the growing bias values lead to underflow. They confirmed this effect by probing existing models like BLOOM, Falcon-RW, and MPT using associative retrieval benchmarks, including Passkey and Needle in a Haystack. To study the issue directly, they trained 148M-parameter decoder models, specifically isolating the slope configurations that trigger attention blindness.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the core issue with ALiBi?
ALiBi uses a linear bias that increases with token distance, which can cause floating-point underflow and disable attention heads.
Q2. Does this affect all models using ALiBi?
The paper demonstrates the failure mode exists in pre-trained models like BLOOM, Falcon-RW, and MPT.
Q3. Can this be fixed?
Yes, the paper proposes mitigations including clamping the bias, using robust slopes, log-scaled distances, or soft capping on attention logits.
Q4. How severe is the underflow?
At a token distance of 2048, 36.6 percent of the attention matrix entries have underflowed.
Q5. Did the authors evaluate these fixes on large models?
The paper does not specify results for large models, as experiments were limited to 148M-parameter decoders.
Q6. What is the impact of combining clamping and log-scaled distances?
This combination improved out-of-context passkey retrieval AUC from 0.08 to 0.79.
Q7. Are there limitations to the findings?
The findings might not generalize to larger models, and the study held the architecture and training corpus fixed.
Q8. How do ALiBi slopes compare to other methods like RoPE?
The paper identifies ALiBi as a strong baseline for needle-in-a-haystack retrieval despite the identified failure mode, but it does not provide a direct empirical comparison to RoPE in this context.
Q9. Does the paper suggest changing the training corpus?
No, the paper does not suggest changing the corpus; it kept the training corpus fixed to isolate the effects.