Evaluating Coding Agents on Scientific Software
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 4 concepts
Key Takeaways
- Created SWE bench Science, a repository-level benchmark consisting of 119 tasks from 98 GitHub repositories spanning 20 scientific domains.
- Organized tasks into three distinct paradigms: Issue-driven, Expert-exploratory, and Engineering-integration.
- Found that the impact of scientific guidance is non-uniform, as well-grounded information improves performance while poorly aligned guidance induces anchoring.
- Identified four recurring failure mechanisms in coding agents: knowledge deficits, poor exploration, incomplete repairs, and generalization issues.
Summary & Methodology Analysis
Existing evaluations of coding agents focus mostly on aggregate task success and fail to provide sufficient insight into why these agents fail when repairing scientific software. To address this, the authors introduce SWE bench Science, a repository-level benchmark comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. These tasks are organized into three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. The research evaluates coding agents on repository-level scientific engineering tasks and uses a paired ablation analysis, which is a controlled experiment that tests the effect of removing a specific component like explicit scientific guidance while preserving repository and executable engineering context.
The findings show that scientific knowledge is not uniformly beneficial. Well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring, meaning the agent gets stuck on suboptimal assumptions, and does not necessarily improve exact repair success. The paper notes that coding agents exhibit specific failure mechanisms when handling these scientific tasks, including deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in the analysis.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is SWE-bench Science?
It is a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains.
Q2. Why did the authors create this benchmark?
Existing evaluations of coding agents focus on aggregate task success and do not provide sufficient insight into why these agents fail when repairing scientific software.
Q3. What are the three task paradigms included in the benchmark?
The three paradigms are Issue-driven, Expert-exploratory, and Engineering-integration.
Q4. How were the tasks evaluated in terms of scientific guidance?
The authors conducted a paired ablation analysis by removing explicit scientific guidance while preserving repository and executable engineering context.
Q5. Is scientific guidance always beneficial for coding agents?
No, scientific guidance impact is non-uniform because well-grounded information can improve performance while poorly aligned guidance may induce anchoring.
Q6. What happens when agents receive well-grounded scientific information?
It can constrain repair and improve average performance and token efficiency.
Q7. What happens when agents receive poorly aligned scientific guidance?
It can induce anchoring and does not necessarily improve exact repair success.
Q8. What are the recurring failure mechanisms identified in the coding agents?
The paper identifies four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases.
Q9. Does the paper specify exact execution time, hardware requirements, or dollar costs?
The paper does not specify these metrics.