Teaching LLM Agents to Explore Better

Summary & Methodology Analysis

Language model agents frequently exhibit rigid behavioral patterns and fail to strategize over extended horizons in multi-turn interactive tasks. This paper attributes these shortcomings to hindsight bias in supervised fine-tuning data and exploration collapse during reinforcement learning. To solve this, the authors introduce a method to instill and refine proactive exploration capabilities, allowing agents to autonomously expand their action trajectories beyond local optima. The methodology begins with exploratory data construction, which synthesizes exploration-rich trajectories using a tree-structured context model and a powerful LLM expert. A length-penalized reward and performance filter then produces the base supervised fine-tuning dataset to initialize the agent. Supervised fine-tuning, a process of adapting a pre-trained model on a specific dataset, is used for this initialisation phase. Next, the method applies reinforcement learning, a training paradigm where agents learn by trial and error using rewards, with contrastive signal guidance. This step constructs contrastive preference pairs between student-generated candidate actions and teacher reference actions at identical states. It uses Monte Carlo reward estimation, an evaluation technique that simulates future paths to estimate returns, via a forward-only policy trained on direct success paths to estimate future returns for actions. Finally, direct preference optimization, a technique for aligning models without complex reinforcement learning loops by directly optimizing on preference pairs, is employed to calibrate the student agent decision boundary between exploration and execution based on estimated future rewards. The paper evaluates these techniques across benchmarks including WebShop, InterCode-SQL, ScienceWorld, Meta-Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and GPT-4o. Despite its strong performance, the approach has limitations. The agent proactive exploration capabilities rely on supervised fine-tuning using teacher-distilled trajectories, which may be bounded by the prior knowledge of the teacher model. The paper does not specify hardware requirements, exact training costs, or latency numbers.