Improving Agent Optimizer Reliability Over Time

Summary & Methodology Analysis

The researchers investigate whether agent harness optimizers (like prompt and code editors) can compound improvements over time without suffering from overfitting. They baseline against a fixed agent using GPT-5.5 on the Harbor framework and compare it against GEPA, which uses evolutionary search to mutate prompts, and Meta Harness, which uses Codex to edit harness code. The evaluation uses a phased protocol where agents are first optimized on a set of tasks (T1) and then tested on a union of those and new tasks (T1 U T2). The study finds that Meta Harness, while functional in Phase 1, experiences a performance drop to 59.1% in Phase 2 because it lacks a mechanism to preserve prior gains while adapting to new requirements. To address this, the authors propose RELAI-VCL, which acts as a regression-aware continual-learning optimizer. It integrates a constraint within the search loop that explicitly rejects any candidate harness edits that improve performance on new tasks if they result in performance loss on previously solved tasks. This approach successfully maintains higher overall pass rates, reaching 76.4% compared to 66.0% for GEPA and 64.6% for Meta Harness. The study is limited by the fact that Terminal-Bench 2.0 tasks are largely independent, which simplifies the challenge compared to real-world production environments where failures are often correlated. Additionally, the evaluation assumes the availability of reliable verifiers for repeated task execution, a constraint that may not exist in every production deployment setting. The paper does not specify the computational overhead or latency introduced by the RELAI-VCL constraint loop.