Detecting Poisoned RAG Content via Attention
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- D-SCAN effectively detects poisoned content by analyzing document-level attention metrics rather than relying on unreliable output-side uncertainty scores.
- The method achieves strong detection performance with AUC scores of 0.9337 on 2Wiki, 0.8330 on HotpotQA, and 0.9060 on Musique.
- It works by calculating attention density and entropy to catch the characteristic narrowing of attention distribution known as attention collapse.
- A simple linear classifier uses features like variance, entropy, and density to identify malicious injections during inference.
Summary & Methodology Analysis
Retrieval-augmented generation (RAG) systems often fall victim to poisoning attacks where adversarial documents manipulate model outputs. Current detection strategies typically rely on output-level uncertainty, which is frequently ineffective because poisoning attacks induce false confidence in the model. The authors introduce D-SCAN to address this by focusing on internal model behavior rather than raw output scores. The architecture operates by calculating document-level attention density, which aggregates token-level attention weights for each document normalized by document length. By evaluating attention entropy, the system identifies the narrowing of attention distribution that characterizes attention collapse, a state where the model inappropriately focuses on specific malicious tokens.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem addressed by this paper?
The paper addresses RAG poisoning attacks, where adversarial documents are injected to manipulate model outputs, bypassing existing detection methods that rely on output-level uncertainty.
Q2. What is D-SCAN?
D-SCAN is a detection system that identifies poisoned content in RAG pipelines by calculating document-level attention density and entropy to detect attention collapse.
Q3. Does this method require special training or complex hardware?
The paper does not specify hardware requirements, but the method uses a linear classifier trained on attention metrics like entropy, variance, and density.
Q4. How does D-SCAN quantify attention behavior?
It aggregates token-level attention weights for each retrieved document, normalizes them by document length to find density, and computes entropy to measure distribution narrowing.
Q5. Which models and benchmarks were used in this study?
The study used Llama-3.1-8B-Instruct, Qwen2.5-7B, Qwen2.5-14B, and Qwen3-30B models, testing against 2Wiki, HotpotQA, and Musique benchmarks.
Q6. Is multi-path sampling required for this method to work?
No, while multi-path sampling can improve detection by smoothing noise, it is not a requirement for the D-SCAN method.
Q7. How did the model perform across the evaluated benchmarks?
D-SCAN achieved an AUC of 0.9337 on 2Wiki, 0.8330 on HotpotQA, and 0.9060 on Musique.
Q8. Why are existing detection methods based on output uncertainty failing?
Existing methods fail because poisoning attacks often induce false confidence in the model, making the output appear standard even when it is manipulated.
Q9. What components are used in the linear classifier for detection?
The classifier uses features derived from token-level and document-level attention metrics, specifically entropy, variance, and density.