Cost Aware Evaluation for Security Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- Most security agent evaluations ignore the operational costs of model inference and priced tool usage.
- The authors developed a cost-success harness using the Inspect framework to track spending versus task success.
- GPT-5.5 reached a 94.1% success rate on Cybench at a cost of $1.16 per challenge.
- Claude Opus 4.8 proved most cost-efficient on BOTS v1, scoring 93.9% at a cost of $2.98 per 1,000 points.
- Performance results can be inflated without decontamination controls that isolate baseline agent ability from tool assistance.
Summary & Methodology Analysis
The researchers implemented a cost-success evaluation harness utilizing the Inspect framework, which orchestrates agents using ReAct, a prompt engineering technique that structures interaction through a cycle of reasoning and tool execution. To capture real world operational expenses, the team categorized costs into two buckets: model inference spend and priced tool usage, such as fees from Brave Search and WhoisXMLAPI. They applied fixed per sample cost budgets to agent runs, allowing for retrospective budget replay analysis to quantify how scaling spend directly correlates with task completion rates.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem with existing security agent evaluations?
Current evaluations focus on peak performance metrics without accounting for the operational costs and economic trade-offs required in real world security operations.
Q2. What does this paper contribute?
The authors propose a cost aware evaluation framework to measure the economic efficiency and operational suitability of both offensive and defensive security agents.
Q3. Does this paper provide a universal ranking of security agents?
No, the experiments are observational rather than prospectively randomized, making them better suited for analyzing specific operating points than for establishing a universal ranking.
Q4. How did the researchers measure offensive performance?
They used Cybench, a series of Capture The Flag challenges, which provided the agents access to both shell and Python tools.
Q5. How were defensive agent capabilities tested?
The authors utilized the Splunk Boss of the SOC v1 dataset, testing agents against an expanded suite of telemetry, enrichment, and search tools.
Q6. What is the significance of the no-tools contamination control?
It helps evaluate the baseline performance of the model without external assistance, which is necessary because high scores on benchmarks like BOTS v1 can occur even without tool usage.
Q7. Were there any constraints on the model inputs during testing?
Yes, high volume models were potentially penalized during BOTS v1 evaluations due to a 250 message context limit.
Q8. What was the highest cost efficiency reported for defensive tasks?
Claude Opus 4.8 achieved the best cost efficiency at $2.98 per 1,000 points on the BOTS v1 benchmark.
Q9. Does the paper specify the exact hardware requirements for these agents?
No, the paper does not specify the hardware requirements for these agents.