Reducing Compute Costs for Agent Harness Optimization
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 1 concepts
Key Takeaways
- Task-CoEvolve reduces total search costs by 67-80% on Terminal-Bench 2.1 compared to traditional full-set evaluation methods.
- The method achieves performance comparable to full-set search while using only 20% of the total evaluations.
- It employs a variance-weighted selection strategy that focuses on tasks with the highest discriminative power at the agent capability frontier.
- The system supports multiple task domains including law, symptom diagnosis, and chemical reaction prediction.
Summary & Methodology Analysis
Task-CoEvolve is designed to solve the inefficiency of iterating on LLM agent harnesses using fixed, full validation sets. Instead of exhaustive testing, it uses a three-phase approach to optimize the search process. Phase 0 initializes the system by establishing baseline success rates for two starting harnesses. In Phase 1, the system performs variance-weighted task selection, where it dynamically calculates the Bernoulli variance of historical outcomes to identify which tasks effectively distinguish between candidate capabilities at the current frontier. Phase 2 then estimates total performance using inclusion probabilities to account for the partial subset of tasks evaluated.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem this paper addresses?
The paper addresses the high computational cost of evaluating LLM agent harnesses on large, static validation sets that become less effective as the agents improve.
Q2. Does this method sacrifice performance for speed?
No, the method achieves performance comparable to full-set search while significantly reducing search costs.
Q3. How much does this method reduce search costs?
On Terminal-Bench 2.1, it reduces total search costs by 67-80%.
Q4. How does the system decide which tasks to run?
It selects a subset of tasks dynamically by calculating the Bernoulli variance of historical outcomes to focus on tasks with high discriminative power near the agent's capability frontier.
Q5. What happens if a candidate harness is underperforming?
A current limitation is that the method cannot stop early on underperforming candidates.
Q6. Can the system increase evaluation depth if results are ambiguous?
No, the system fixes the number of tasks per candidate before evaluation begins and cannot increase depth for hard-to-rank candidates.
Q7. What datasets were used to validate this approach?
The authors used Terminal-Bench 2.1, LawBench, Symptom2Disease, and USPTO-50k.
Q8. How are results from partial subsets made comparable?
The method uses inclusion probabilities to estimate the full-set performance score from partial evaluations.
Q9. Is the number of tasks evaluated per candidate fixed or variable?
It is fixed before the evaluation begins.