Detecting LLM Hallucinations via Attention Path Fragility

Summary & Methodology Analysis

The ASMI methodology addresses the problem of overconfident incorrect outputs by treating attention path fragility as a signal for uncertainty. The process begins by performing a single greedy decoding pass to generate a response, while simultaneously caching the prefix activations up to a target transformer layer. Instead of performing full model re-inference, the system generates multiple Monte Carlo subnetworks by applying random Bernoulli masks with a mask rate of 0.15 to the attention head outputs. The suffix of the model is then evaluated under each mask using the cached activations, which provides an efficient way to sample predictive distributions without repeating the entire compute chain.

The system calculates the BALD mutual information between these subnetworks to measure fragility, then applies a semantic agreement score based on cosine similarities of output projections. By weighting the token level mutual information using the inverse of this semantic agreement, the method discounts surface level variations in wording that do not reflect true uncertainty. An optional adaptive gate, known as Adapt-ASMI, can further modulate these weights based on the diversity of the stochastically sampled responses, providing an additional layer of control over the uncertainty signal.

Regarding performance and limitations, ASMI excels in grounded question answering, frequently outperforming Semantic Entropy on benchmarks like BabiQA, SQuAD, and CoQA. However, the paper reports that in parametric settings like TriviaQA, the ASMI variants perform below the baseline of Maximum Sequence Probability. The paper does not specify the exact memory footprint or inference latency increase in milliseconds, though it notes that the caching strategy serves to reduce the inference costs associated with sampling.