Improving AI Agent Execution Through Live Steering

Summary & Methodology Analysis

The PILOT system introduces a supervisor-worker harness that maintains a live, two-way communication channel during long-horizon tasks. Unlike standard agent setups that run sequentially or post-hoc, PILOT allows the supervisor to inspect worker trajectories in real time, offering corrective guidance or aborting sub-optimal runs. This setup creates a feedback loop where the agent learns from its own execution history by distillating, a process that extracts core procedures and failure modes from experience into a persistent memory library that guides subsequent sessions. By updating the harness state dynamically, the system ensures that learned improvements are immediately available to future worker instances.

The system was evaluated using GLM-5.1 and Kimi-K2.6 as backbones across three benchmarks: Terminal-Bench 2.0, SWE-bench Multilingual, and SWE-bench Pro. Quantitative results show that PILOT is highly effective at increasing successful evaluations per million tokens, rising by 110.3% for GLM-5.1 and 134.0% for Kimi-K2.6. Furthermore, the system demonstrates substantial gains in overall pass rates, achieving 14.6 and 12.4 points of improvement respectively for the two models. This performance is achieved while simultaneously reducing mean output tokens by 42.9% and 47.4%, suggesting that the supervisor effectively prunes wasteful reasoning paths during complex tasks.

Despite these performance gains, the current iteration has notable constraints. The iterative self-improvement process is computationally expensive, requiring the repeat of tasks across many iterations, which limits the current scope of evaluation to only three benchmarks and two backbones. Additionally, the supervisor and worker currently share the same backbone model. This limitation means the potential benefits of using heterogeneous pairings (such as a more capable model supervising a smaller, faster worker) and the trade-offs between oversight quality and latency remain unexplored.