Testing Agent Memory Against Poisoning Attacks
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 4 concepts
Key Takeaways
- Injecting poisoned memory at 1.2 percent of the corpus reduces LongMemEval accuracy from 0.850 to 0.300.
- A four-stage write-time content screening pipeline failed to detect any of the 360 poisoned memory entries.
- Correcting provenance-weighting settings improves some accuracy but results in total retrieval failure for untrusted evidence, dropping accuracy to 0.0417.
- The tested attack is non-adaptive, meaning it serves as a lower bound on potential attacker effectiveness.
Summary & Methodology Analysis
The research evaluates the robustness of agent memory systems using three distinct corpora: Dolly-15k for instruction-following text, a templated set for memory-like entries, and NotInject for evaluating potential over-defense against trigger-word-laden text. The methodology tests two common defense layers: a four-stage write-time content screening pipeline and a read-time ranking approach that adjusts retrieval scores based on data provenance. The screening pipeline demonstrates high utility in other contexts, reaching 0.832 recall on indirect prompt injection while flagging only 1.5 percent of trigger-word-laden benign text, yet it proved ineffective against the specifically designed poisoned memories in this study.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of this research?
The paper investigates the vulnerability of agent memory systems to adversarial poisoning and evaluates whether current content screening and provenance ranking defenses are effective.
Q2. Did the content screening pipeline work?
No, the four-stage write-time screening pipeline failed to identify any of the 360 poisoned memory entries tested.
Q3. What happens when evidence is deemed untrusted?
Retrieval fails completely, as no evidence memory survives the ranking process across 120 questions, leading to an accuracy of 0.0417.
Q4. How does the poison injection rate affect accuracy?
At 1.2 percent of the corpus, the attack reduces accuracy on LongMemEval from 0.850 to 0.300.
Q5. Are the attacks adaptive?
No, the adversary is non-adaptive, meaning the attack is generated in a single pass without gradient access, search over phrasings, or iteration against system feedback.
Q6. What benchmarks were used?
The primary benchmark is LongMemEval, which covers information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention across 500 questions.
Q7. Has the provenance-based defense been fully implemented?
No, the proposed remedy of using provenance as a bounded occupancy constraint remains theoretical and has not been implemented or evaluated.
Q8. What is the NotInject dataset?
NotInject is an over-defense stress test consisting of benign sentences that are deliberately seeded with trigger words typically used by injection detectors.
Q9. How many poisoned memories were used in the evaluation?
The study utilized 360 poisoned memory entries.