Automating Agent Harness Optimization

Summary & Methodology Analysis

AutoSaddler automates the improvement of agent harnesses by treating the optimization process as a supervised learning problem. The framework utilizes three specific agents, namely a Diagnosis-Patch Agent, a Reflection Agent, and an Evolution Agent, which are implemented using the Claude Agent SDK. These agents iteratively analyze task execution traces to identify performance bottlenecks and generate structural updates to the harness code. By leveraging a training set of tasks paired with gold answers and success metrics, the system automatically discovers improvements that generalize beyond the training batch, resulting in performance gains of 9.0 percent on GAIA2, 9.6 percent on SWE-Bench Pro, and 10.0 percent on Terminal-Bench 2.0 compared to base harnesses. This represents a clear improvement over existing automated baselines like GEPA and Meta-Harness. The framework exhibits high efficiency in its optimization loop, achieving peak performance on GAIA2 after consuming 147 traces, which is approximately 10 times fewer traces than the 1,400 traces required by Meta-Harness. It reaches 72.3 percent development accuracy within approximately 1,000 total task executions, whereas other baselines like GEPA and Meta-Harness saturate at 64.6 percent and 61.5 percent accuracy despite requiring approximately 2,800 executions. Despite these gains, the framework has notable limitations. It relies on the availability of a training set of tasks with ground-truth answers and clear success signals, which may be costly or unavailable in many real-world production environments. Furthermore, the current scope is restricted to stateless and independent tasks, leaving stateful settings, memory integration, and broad LLM family validation as areas for future research.