Improving AI Agents for Software Engineering

Summary & Methodology Analysis

The paper establishes a structured bridge between software engineering and software security, which have historically been evaluated in isolation. By synthesizing representative literature, the authors propose an assurance centered framework that treats evaluation configurations as a tuple encompassing artifacts, context, model selection, tool integration, and human intervention. This shift addresses the fragmentation where agents are tested on disparate benchmarks without unified reporting protocols, making it difficult to isolate whether performance gains stem from the model architecture, the context provided, or the harness configuration. The authors emphasize that reported scores currently vary too widely to be pooled into meta-analysis because of these heterogeneous evaluation environments.

Models capable of reasoning about source code, such as GPT-4, CodeLlama, StarCoder, and DeepSeek-Coder, are reviewed in the context of their ability to resolve issues at repository scale. The paper highlights the use of SWE-bench for evaluating repository level issue resolution and notes the adoption of fine-tuned encoder models, such as CodeBERT, for function level classification using datasets like Big-Vul and Devign. These models demonstrate the current reliance on both large generative models and specialized encoders to bridge the gap between general code understanding and specific security or engineering task performance.

Limitations of the current research landscape are significant, particularly concerning the comparability of results. The authors acknowledge that their work is a structured narrative review rather than a protocol driven systematic literature review, and they caution that the findings may over-represent English language papers. Because researchers employ different retry budgets, model versions, and benchmark variants, the authors conclude that reported scores should be treated as study specific observations. Furthermore, they admit they do not claim exhaustive coverage of all relevant literature, focusing instead on defining a research agenda that brings consistency to the field.