LLM Agents Learn from Environment Exploration

Summary & Methodology Analysis

Traditional methods for training LLM agents often require external tasks and supervision, such as expert demonstrations or reinforcement learning on human-defined tasks with specific verifiers. This approach can limit how scalable and diverse the training process is. State2State proposes a novel 'environment-derived mid-training' method. It works by first using an exploration policy for the agent to interact with the environment. The key innovation is that it then filters and samples learnable target observations directly from this exploration. These sampled observations are paired with initial environment configurations to create state-reaching tasks. The success of these tasks is verified through simple rule-based state matching between the agent's current observation and the target observation, which then generates rule-based rewards. This process bypasses the need for task-specific test cases or handcrafted verifiers. The paper evaluates State2State on environments like ALFWorld and MobileWorld, reporting strong performance gains. For instance, when using a Qwen3-8B base model on ALFWorld, State2State achieved a 97.45% average task success rate, with 97.14% on in-distribution tasks and 97.76% on out-of-distribution tasks. On the MobileWorld GUI-only subset, it improved the MAI-UI-8B model's score from 0.275 to 0.308 after 80 training steps.

The core of the State2State method involves converting explored states into training objectives. This contrasts with standard approaches where an exploration policy interacts with an environment, and then reinforcement learning algorithms like GRPO (Proximal Policy Optimization, adapted for dynamic sampling) are applied. State2State integrates with these RL algorithms but fundamentally changes the objective generation. It leverages rule-based state matching for verification, which means success is determined by whether the agent's current observation matches a predefined target observation. This yields rewards based on these rules, rather than requiring external, task-specific validation logic. The models utilized in the experiments include Qwen3-4B, Qwen3-8B, and MAI-UI-8B, tested on benchmarks such as ALFWorld and MobileWorld. The paper notes limitations, stating that experiments focused on controlled online RL comparisons and did not exhaustively explore scaling. Evaluations were performed on specific Qwen3 models and did not include frontier-scale models. Further systematic scaling analysis is recommended to understand how performance changes with model size and training compute, and the current evaluation is confined to environments with reproducible states and verifiable success conditions.