Back to Feed
Agents / Benchmarks & Evals

Can Language Models Improve Agent Harnesses?

Original: Evo-Bench: Can Language Models Improve Agent Harness?
By Lisheng Huang, Chen Yang, Hao Zhou, Huatong Song, Zongchao Chen, Ran Le, Yang Song, Wayne Xin Zhao, Tao Zhang Published 2026-08-10

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 3 concepts

Key Takeaways

  • The Evo-Bench framework tests how well models can act as engineers to evolve and refine executable policy harnesses for agent tasks.
  • Top performing models achieved a gain of 16.6 points over the seed harness, nearing the human-engineered baseline of 47.5 points.
  • Models like GPT-5.6-Sol and Qwen3.7-Max outperformed the Artificial harness baseline in general tasks.
  • Performance remains inconsistent in office-related tasks, where models often fail to surpass the baseline.

Summary & Methodology Analysis

The Evo-Bench framework facilitates a controlled long-horizon evolutionary process where models are tasked with revising an executable policy harness. The methodology relies on harness-guided benchmark construction, using auxiliary-task evolution to pinpoint sensitive tasks, followed by sensitivity-aware stratified splitting to ensure the results generalize across different evaluation suites. Models operate as autonomous engineers, iteratively applying changes to the harness code, which is then validated on a held-out test set to measure genuine evolutionary progress. The core objective is to determine if current large language models possess the intrinsic capability to perform code-centric autonomous harness evolution.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary goal of this research?

The researchers aimed to benchmark the intrinsic ability of large language models to perform long-horizon and code-centric autonomous harness evolution.

Q2. Does this work provide a tool for developers?

Yes, it provides the Evo-Bench benchmark to evaluate how models handle tasks requiring the iterative revision of executable policy harnesses.

Q3. Which models were tested?

The study evaluated several models, including GPT-5.6-Sol, Claude Opus 4.8, GLM-5.2, Qwen3.7-Max, Minimax-M3, Deepseek V4 Pro, Kimi K2.7 Code, Qwen3.6-27b, and Gemma-4-31B.

Q4. How do model-evolved harnesses compare to human-engineered ones?

The top-performing models achieved a gain of 16.6 points over the seed harness, bringing them closer to the human-engineered baseline of 47.5 points.

Q5. Are there specific domains where these models fail?

The models struggle significantly with office tasks, where they generally fail to consistently outperform the Artificial harness baseline.

Q6. What are the limitations regarding the quality of the evolution?

Evolutionary behavior tends to saturate early, with models often introducing detrimental modifications after initial successes, and they rely on localized tweaks rather than deep system-level optimizations.

Q7. How do the models approach failure analysis?

The research notes that models react superficially to aggregate scores instead of actively distilling causal failure modes within the code.

Q8. Does the study provide specific hardware or compute requirements?

The paper does not specify hardware requirements, compute costs, or latency figures for these models.

Q9. Which specific datasets or environments are covered in Evo-Bench?

Evo-Bench incorporates BrowseComp, HLE, GDPval, APEX-Agents, Claw-Eval, and CodeAct.