Deterministic Activity Compilation for Agent Memory
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Replaces non-deterministic LLM summarization with a deterministic compilation process that produces cacheable and auditable episodic memory.
- Implements a novel segmentation strategy based on dwell time, session gaps, and flicker merging to bound screen activity.
- Achieved 98.4% accuracy on question-answering benchmarks using the compiled activity frames.
- Measured a Routine Overhead Ratio of 60x for guarded operational skill plans.
Summary & Methodology Analysis
The system addresses the memory inefficiency of current computer-use agents that rely on expensive, non-deterministic LLM (a transformer model architecture using self-attention mechanisms to predict sequences) summarization. By shifting away from raw, unorganized data streams, the paper introduces a deterministic compilation method. This pipeline begins with passive screen capture, followed by a segmentation process that defines activity frames based on dwell time, session gaps, and flicker merging. These frames are processed using deterministic URL parsing rules to assign entity types, and they utilize a two-tier schema that separates measured facts from optional inferred labels.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem this paper solves?
Current computer-use agents struggle with memory because they rely on non-deterministic, expensive LLM summarization or ingest raw, unorganized streams of data.
Q2. How does this method improve agent performance?
It provides an efficient, cacheable, and mechanically auditable episodic memory structure, resulting in 98.4% accuracy on question-answering benchmarks.
Q3. Does this require special hardware?
The paper does not specify hardware requirements.
Q4. How are activity frames defined?
They are defined by segmenting screen activity streams based on dwell time, session gaps, and flicker merging.
Q5. What is the Routine Overhead Ratio observed?
The Routine Overhead Ratio for the guarded operational skill plan was 60x.
Q6. What were the recurrence rates for routines?
The delegable routine recurrence rate was 9.0% in-sample and 7.7% out-of-sample.
Q7. What are the limitations of the data used?
The empirical data is derived from a single-user corpus, which may not generalize to different roles or platforms.
Q8. Is screen activity a perfect indicator of human focus?
No, the system measures screen tenure and presence, which acts as an imperfect proxy for human attention.
Q9. What specific models or datasets are mentioned?
The paper references Microsoft Recall, OpenAI Chronicle, MemGPT, Zep, E-mem, MIRIX, FOCAL, ProAgentBench, SummAct, OmniQuery, TaskTracer, SWISH, ActivityWatch, Agent Workflow Memory, SkillWeaver, Agent Skill Induction, PreAct, Explorer, AgentTrek, Watch Learn, cotomi Act, FrugalGPT, Claude Sonnet 4.5, and Claude Opus 4.5.