Evaluating Language Models on Real Software Issues
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 4 concepts
Key Takeaways
- Existing coding benchmarks rely on small, self-contained problems, leaving a gap in evaluating real-world software engineering capabilities.
- SWE-bench consists of 2,294 task instances sourced from real GitHub issues and pull requests across popular Python repositories.
- Evaluation requires models to generate a patch file that successfully applies and passes associated unit and system tests.
- The authors fine-tuned SWE-Llama 7b and 13b models using a newly constructed dataset called SWE-bench-train.
- Experimental results show that performance drops as context length increases because models struggle to localize problematic code in larger contexts.
Summary & Methodology Analysis
Language models have outpaced the ability to evaluate them effectively, necessitating challenging benchmarks that accurately reflect real-world applications. Traditional coding benchmarks mostly involve self-contained problems solvable in a few lines of code. In contrast, real-world software engineering requires navigating large repositories, understanding the interplay between functions in different files, and processing long contexts. To address this, the authors constructed SWE-bench by sourcing task instances from real GitHub issues and corresponding pull requests in popular Python repositories using a 3-stage pipeline involving repository selection, attribute-based filtering, and execution-based filtering. This resulted in 2,294 task instances where a successful resolution requires the patch to apply cleanly and all associated unit and system tests to pass.
To establish baselines, the authors constructed the SWE-bench-train dataset comprising 19,000 non-testing task instances from 37 repositories. They fine-tuned SWE-Llama 7b and 13b models based on CodeLlama-Python models using this dataset, specifically fine-tuning only the weights of the attention sublayer using LoRA, a parameter-efficient fine-tuning method, for memory efficiency. For inference, they utilized a retrieval-based approach to select relevant files for model context, using sparse retrieval with BM25 or oracle retrieval that provides the exact files edited by the reference patch. Model inputs combined task instructions, issue text, retrieved files, an example patch file, and a prompt, using greedy decoding to generate a single patch file per instance.
Despite these efforts, several limitations emerged. Relying solely on execution-based code testing is insufficient to guarantee reliable performance, as automated code generations can frequently be less comprehensive, efficient, or readable compared to human solutions. As total context length increases, model performance drops considerably, indicating struggles with localizing problematic code in larger contexts. Furthermore, fine-tuned models like SWE-Llama performed poorly with BM25 retrieved context due to a distribution shift from the oracle retrieval context used during training. Models also tended to write primitive Python code, failed to leverage existing third-party libraries, and struggled with understanding how changes to one function might affect other dependent parts of the codebase.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem addressed in the paper?
Language models have outpaced current evaluation methods, and existing coding benchmarks only test small, self-contained problems rather than real-world software engineering tasks.
Q2. What is SWE-bench?
SWE-bench is a benchmark constructed from real GitHub issues and corresponding pull requests in popular Python repositories to evaluate language models on software engineering tasks.
Q3. How are model solutions evaluated in this benchmark?
Proposed solutions are evaluated by applying the generated patch using Unix's patch program and then executing associated unit and system tests, requiring all tests to pass for a successful resolution.
Q4. How many task instances are included in the SWE-bench dataset?
The SWE-bench dataset contains 2,294 task instances.
Q5. What is SWE-bench-train?
SWE-bench-train is a dataset comprising 19,000 non-testing task instances from 37 repositories, collected without requiring pull requests to contribute test changes.
Q6. Which models were fine-tuned by the authors for this study?
The authors fine-tuned SWE-Llama 7b and 13b models based on CodeLlama-Python models.
Q7. How are relevant files selected to fit into the model context?
A retrieval-based approach is used, either through sparse retrieval using BM25 to retrieve files fitting specified context limits, or oracle retrieval that provides the files edited by the reference patch.
Q8. What causes the performance drop when model context length increases?
Models struggle with localizing problematic code in larger contexts, meaning that simply increasing context size or using BM25 retrieval does not improve performance.
Q9. What are some behavioral weaknesses observed in the models when generating code?
Models tend to write primitive Python code, fail to leverage existing third-party libraries, take a greedy approach without regard for code style or logical constraints, and struggle with understanding how changes affect dependent parts of the codebase.