Benchmarking Agent Harness Safety Lifecycles
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- The HarnessRisk benchmark evaluates three agent harnesses, six language models, and 14 distinct system configurations.
- Across all tests, attack success rates fluctuate significantly between 12.6% and 80.9%, while system utility remains high at 75.0% to 97.6%.
- The Harness Configuration phase is consistently the most vulnerable area, allowing attackers to compromise workflows by modifying security sensitive parameters.
- Current security configurations show a surprising disconnect, as some setups detect over 90% of risks while still allowing successful attacks.
Summary & Methodology Analysis
HarnessRisk organizes agent harness safety into six specific operational phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. This lifecycle approach allows researchers to isolate security failures during different stages of an agent execution, ranging from initial credential initialization and skill permissioning to high impact actions like OAuth grants and payments. The benchmark uses a standardized test environment where every execution starts from a fresh state, and all evaluations are performed within isolated sandboxes using simulated assets to prevent actual side effects.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of HarnessRisk?
It serves as a lifecycle oriented benchmark to identify and compare safety failures across various stages of agent harness operations.
Q2. Why is it difficult to compare agent safety currently?
Existing benchmarks often focus only on individual attack mechanisms or limited operational settings, making it hard to see how failures emerge across broader harness responsibilities.
Q3. Does this benchmark use real world services?
No, all evaluations use simulated resources and services within isolated sandboxes.
Q4. Which agent harnesses were evaluated?
The researchers evaluated OpenClaw, Hermes, and Nanobot.
Q5. What language models were included in the evaluation?
The models tested were DeepSeek-V4-Pro, GLM-5.2, Kimi K2.6, MiniMax M3, GPT-5.5, and Claude Opus 4.7.
Q6. Which phase of the agent lifecycle is most vulnerable?
The Harness Configuration phase is the most vulnerable across all tested harnesses.
Q7. Are training time attacks included in this benchmark?
No, the researchers explicitly exclude training time compromises and direct attacks on underlying infrastructure.
Q8. Is external network access permitted during testing?
No, real external network access is disabled during all benchmark executions.
Q9. What is the range of attack success found by the researchers?
Attack success rates range from 12.6% to 80.9% across the 14 tested configurations.