Scrouting for Cost-Efficient Coding Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- The system achieves a 159 out of 266 solve rate on SWE-bench Pro, matching Claude Opus 4.6 performance.
- Operating costs are reduced to approximately one-fifth of current benchmarks, moving from $1.274 to $0.230 per task.
- The search phase is extremely lightweight, adding less than half a cent of GPU compute time per task.
- A new engage-then-route architecture allows for adding new model fixers to the pool without requiring costly model retraining.
Summary & Methodology Analysis
The system architecture centers on a two-phase process: a search phase followed by a verified routing decision. In the search phase, a 7B parameter model (SuperScout-7B) traverses the repository to identify relevant files, capture notes, and generate a candidate test for reproduction. Before the actual fix is attempted, a sandbox verification gate executes the candidate test to discard false claims, ensuring the router only proceeds with verified context. The router itself is resume-based, meaning it evaluates the task text alongside the searcher's hidden states, which are the internal activation vectors used by the model to process input.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem with current autonomous coding agents?
Current systems are expensive to run and often select models based solely on task text without any context regarding the codebase structure.
Q2. How does SuperScout reduce costs?
It performs an initial inexpensive search of the repository to ensure only necessary data is sent to higher-cost, more capable models.
Q3. Does this require retraining when adding new models?
No, the resume-based router allows for the integration of new fixers into the existing pool without requiring additional training.
Q4. What happens during the verification gate?
The sandbox gate replays the reproduction tests provided by the searcher and strips away any claims that do not result in a genuine failure.
Q5. Which specific models were used for testing and implementation?
The study utilized SuperScout-7B, Qwen2.5-Coder, Claude Opus 4.6, GPT-5.2, Gemini 3 Flash, Kimi K2.5, and Qwen3-Embedding-0.6B.
Q6. What are the limitations regarding the verification gate?
The paper does not perform a pass-through experiment, so the specific contribution of the verification gate to final performance is inferred rather than directly measured.
Q7. What datasets were utilized in this research?
The research used SWE-bench Pro, SWE-bench Verified, SWE-bench Multilingual, Open-SWE-Traces, SWE-rebench-openhands, and SWE-Hero.
Q8. Are there limitations to the localization analysis?
Yes, the analysis for TypeScript and C++ is based on a small sample size and is considered indicative only.
Q9. How much does the scouting phase cost in terms of compute?
The searcher adds less than half a cent of GPU time per task.