Improving Agent Decision Making via Recursive Distillation

Summary & Methodology Analysis

Agentic reinforcement learning typically suffers from sparse trajectory-level rewards that make it difficult for models to understand which specific turns led to success. AgentOPSD addresses this by aggregating log-probability gaps between teacher and student models into turn-level evidence. This allows the system to assign credit more accurately than uniform advantage broadcasting, which often fails in multi-turn environments where the sequence of actions is complex and long-horizon. The architecture initializes a Bayesian belief state based on empirical group success rates and updates it recursively in log-odds space, applying a decay factor to track progress over time. By calculating the marginal revision of this belief state at each turn, the model reshapes the advantage signal, effectively pinpointing the importance of specific decisions within a long interaction sequence. The paper focuses on these algorithmic improvements rather than specific hardware-level architectural changes. Evaluations were conducted across a wide range of tasks including ALFWorld, WebShop, and various QA datasets like Search-QA, NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle. Performance was validated using Qwen2.5-3B and Qwen2.5-7B models. The paper does not specify memory usage or latency overhead associated with the recursive calculation process. No limitations were reported in the paper.