Systematic Evaluation of AI Research Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- Proposed a three-part evaluation framework tracking Solution Framing, Execution, and Feedback Control.
- Claude-Opus-4.7 emerged as the top performer with an average score of 0.739 across tasks.
- Execution reliability remains high across models, with C2 scores ranging between 0.880 and 0.967.
- Current agents rarely produce novel research, as only 1.2 percent of successful solutions were classified as novel.
- Significant reliability gaps exist between models, even when comparing their best possible outcomes.
Summary & Methodology Analysis
The paper shifts evaluation from final performance to a process-oriented approach. It decomposes the research lifecycle into three components: Solution Framing (C1), Execution (C2), and Feedback Control (C3). This allows for a granular assessment using deterministic metrics based on verifier outcomes and trajectory signals. By applying these metrics, the researchers compared seven models, including Claude-Opus-4.7, GPT-5.5, Gemini-3.1-Pro, GLM-5.2, Kimi-K2.7-Code, DeepSeek-V4-Pro, and LongCat-2.0, across diverse harness designs like AutoLab, Claude Code, OpenCode, Codex CLI, and Kimi Code CLI. The methodology also incorporates an outer optimization loop for automated harness evolution to test agent capabilities beyond static test sets.
Performance analysis shows that while models are generally proficient at executing code, they struggle to generate novel methodological approaches. Specifically, Claude-Opus-4.7 demonstrated the highest avg@3 score of 0.739 and best@3 score of 0.790. A critical insight is the discrepancy in reliability: the gap between the strongest and weakest models is 0.237 on average performance, but narrows to 0.122 on best-seed performance. This indicates that while weaker models can occasionally succeed, they lack the consistent logic necessary for reliable long-horizon tasks. The use of LLM judges to classify novelty revealed that only 3 of 252 best-seed solutions contained original research ideas.
Limitations of this work include the nature of the metrics, which serve as proxies and do not capture internal semantic quality or the nuance of agent reasoning. Furthermore, findings regarding self-improvement are contingent on specific task selection and the interventions applied during testing. The paper does not specify the exact hardware requirements or compute costs associated with running these evaluations, noting that absolute financial costs depend on provider-specific pricing models and current infrastructure settings.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. Why is evaluating AI research agents by their final scores insufficient?
Relying only on final results hides the agent internal process and experience, making it impossible to diagnose why an agent succeeds or fails.
Q2. What core research capabilities were measured in this study?
The study measured Solution Framing, Execution, and Feedback Control.
Q3. Which model performed the best in this systematic evaluation?
Claude-Opus-4.7 achieved the highest average and best performance scores among the seven models tested.
Q4. What is the difference between average and best-seed performance in the context of this paper?
The average score measures general consistency, whereas the best-seed score highlights the model potential when it succeeds, revealing a reliability gap between different agent architectures.
Q5. How did the researchers measure the novelty of the research produced by these agents?
They utilized LLM judges to classify the submitted solutions and determine if the methodological approach was truly novel.
Q6. Did the study specify the exact hardware or GPU requirements for these evaluations?
No, the paper does not specify hardware requirements.
Q7. Are the performance results affected by the pricing models of the model providers?
Yes, the paper notes that absolute cost results are dependent on provider-specific pricing and infrastructure conditions.
Q8. What role does the harness design play in these evaluations?
The researchers compared performance across different harness designs and used an outer optimization loop to enable automated harness evolution.
Q9. What are the limitations regarding the metrics used?
The C1-C3 metrics act as proxies and may not capture internal agent reasoning or the semantic quality of the output.