Optimizing Agent Memory via Reduced Utility States

Summary & Methodology Analysis

RoMeRL addresses the memory-reward trap, where agents struggle to map long-term outcomes to specific, relevant memories in large state spaces. Instead of using traditional trajectory-indexed utility learning, which leads to diluted feedback, the framework shifts to a fixed-dimensional state space. This state space is factorized into four distinct semantic coordinates: Positive Consolidated Coordinate, Positive Adaptive Coordinate, Negative Consolidated Coordinate, and Negative Adaptive Coordinate. These coordinates categorize memories based on outcome polarity and dynamics, allowing the system to selectively retain optimal successful trajectories, initial successes after failures, and informative failure experiences while maintaining an exponential moving average of utility for each coordinate. By grounding retrieval in both semantic similarity and these learned utilities, the agent avoids reward misattribution. The authors evaluate this architecture on LifelongAgentBench, ALFWorld, and AppWorld, demonstrating that the system significantly optimizes resource usage while maintaining high task performance. The framework maintains a 84.4% reduction in memory size and increases feedback density by 6.0x, effectively mitigating issues with sparse reward signals. A notable limitation remains in the reliance on outcome-level rewards, as the framework does not fully resolve causal credit assignment, which is the process of attributing specific rewards to individual actions within a sequence.