Benchmarking AI Agents on Code Refactoring
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- SWE-Bench ProMax provides 170 instances across seven programming languages, averaging 11.4 modified files and 261.6 lines of code per instance.
- Open-weight models show high efficiency, with GLM-5 achieving a 36.5 percent resolve rate at a cost of $0.24 per instance.
- Current AI agents frequently struggle with multi-file refactoring, often getting stuck in unproductive edit-revert cycles.
- The benchmark addresses data quality issues found in existing evaluations by using expert-led instance filtering and precise problem statement rewriting.
Summary & Methodology Analysis
The methodology focuses on high-fidelity, real-world refactoring scenarios. Researchers automated data collection from GitHub commits using a refactor keyword filter alongside specific repository-level criteria. Each instance runs in a Docker container representing the pre-refactoring state. To ensure evaluation integrity, experts manually reviewed test suites to prune overly narrow or broad tests and rewrote problem statements to provide precise specifications. Agents were evaluated using mini-swe-agent and OpenHands scaffolds, capped at a 300-step limit and a $10 budget per instance. Success is defined strictly by the repository passing all tests in the suite after the agent finishes its work.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary purpose of SWE-Bench ProMax?
It serves as a standardized, expert-curated benchmark to evaluate AI coding agents on large-scale, multilingual code refactoring tasks.
Q2. How many tasks are included in this benchmark?
The benchmark includes 170 instances covering seven different programming languages.
Q3. Does this benchmark focus on small or large code changes?
It focuses on large-scale refactoring, with an average of 11.4 files modified and 261.6 lines of code changed per instance.
Q4. What happens if an agent enters an edit-revert cycle?
The agent consumes excessive resources and often fails to complete the refactoring task effectively.
Q5. How does the benchmark define a successful refactor?
Success is defined by the repository state passing all tests in the associated test suite.
Q6. What are the common failures identified in current AI agents?
Agents struggle with incomplete refactoring, specifically failing to apply necessary changes across all peripheral files.
Q7. How do open-weight models perform on this benchmark?
They show competitive results, such as GLM-5 achieving a 36.5 percent resolve rate at a cost of $0.24 per instance.
Q8. What are the primary limitations of existing coding benchmarks?
Existing benchmarks often suffer from data contamination, flawed tests, and imprecise problem descriptions.
Q9. Are the agents provided with unlimited time or compute?
No, agents are limited to 300 steps and a $10 cost limit per instance.