Empowering Small LLM Agents With Hierarchical Memory
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 4 concepts
Key Takeaways
- Agent Memory Distillation achieves average accuracy gains of 27.2%p on AppWorld, 11.2%p on BFCL V3, and 3.4%p on ToolSandbox.
- The method constructs hierarchical memories (Workflow, Subtask, and Function) offline from successful trajectories of a larger teacher agent using GPT-5-mini.
- At inference time, relevant workflow and subtask insights are injected proactively into the system prompt, while function error hints are injected reactively.
- Current limitations include being evaluated only on text-based tool-use benchmarks, using a frozen offline memory that cannot update from student successes, and depending heavily on teacher-student compatibility.
Summary & Methodology Analysis
Memory systems show promise for improving agent performance, but their potential remains largely unexplored for small language models that struggle to generate sufficient successful trajectories on their own. Agent Memory Distillation solves this by leveraging a larger teacher model, specifically utilizing GPT-5-mini as the teacher, to generate successful execution trajectories. From these trajectories, the system constructs a hierarchical memory structure consisting of Workflow memory with high-level insights, Subtask memory segmented into coherent steps, and Function memory capturing specific tool invocations and API usage. All memory entries are encoded using a pretrained text-embedding model, which represents text as numeric vectors to enable semantic similarity searches. At inference time, the system uses proactive injection to supply relevant workflow insights and subtask labels into the system prompt, and reactive injection to supply function error hints if a tool call fails. This setup consistently outperforms existing memory-based baselines across multiple text-based tool-use benchmarks.
Despite the strong performance gains, Agent Memory Distillation has notable limitations that affect its deployment and adaptability in production environments. First, the method is evaluated only on text-based tool-use benchmarks that involve Python APIs or structured function calls, leaving its generalization to less structured agentic settings unverified. Second, the memory is constructed offline from a fixed set of teacher trajectories and remains frozen at inference time. This prevents the student agent from incorporating its own successes and failures back into the shared memory store during execution. Finally, the framework depends heavily on the quality and suitability of the teacher trajectories. The paper notes that a stronger teacher does not always yield larger student gains, because the overall benefit hinges on teacher-student compatibility.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem this paper addresses?
The paper tackles the challenge that memory systems for improving agent performance remain largely unexplored for small language models, which struggle to generate enough successful trajectories on their own.
Q2. How does Agent Memory Distillation help small language models?
It uses a larger teacher model to construct hierarchical memories from successful execution trajectories, which are then injected into the small model during inference to guide its actions.
Q3. What kind of accuracy gains does the method achieve?
It achieves average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V3, and ToolSandbox, respectively.
Q4. Which specific model is used as the teacher in the paper?
GPT-5-mini is used as the teacher model.
Q5. What are the three layers of the hierarchical memory structure?
The memory structure consists of Workflow memory, Subtask memory, and Function memory.
Q6. How are memories retrieved and injected during inference?
Workflow and subtask entries are retrieved using the task instruction and injected proactively into the system prompt, while function memories are retrieved reactively as hints when a tool call fails.
Q7. What are the main limitations regarding how the memory is updated?
The memory is constructed offline from a fixed set of teacher trajectories and remains frozen at inference time, preventing the incorporation of the student's own successes and failures.
Q8. Does a stronger teacher always result in better student performance?
No, a stronger teacher does not always yield larger student gains because the benefit also hinges on teacher-student compatibility.
Q9. What types of benchmarks were used to evaluate the method?
The method was evaluated on text-based tool-use benchmarks involving Python APIs or structured function calls, specifically AppWorld, BFCL V3, and ToolSandbox.