Benchmark for Detecting Malicious Agent Skills

Summary & Methodology Analysis

MaliciousSkillBench addresses the fragmentation in agent capability security by consolidating 13 public datasets into a structured registry. The methodology involves deterministic deduplication to create unique identities and grouping them into operational structural families. By excluding cross-label conflicts and implementing conservative artifact eligibility, the researchers established a core malicious pool alongside a main benign pool, facilitating evaluation across different distribution shifts including random, malicious-structural-disjoint, and source-disjoint scenarios. The benchmark supports standardized testing through tools like Cisco-local-behavioral and SkillSpector-static, which functions as a native block gate without utilizing large language models. The primary objective is to enable more rigorous, traceable assessment of malicious artifacts compared to the inconsistent labeling found in existing fragmented sources. Despite its consolidation, the framework remains constrained by its reliance on static primary Skill artifacts. Because it excludes runtime or package-level behaviors, it may fail to capture dynamic threats that manifest only during execution. Furthermore, the reliance on frozen public sources means the benchmark is bounded by the specific set of 13 sources curated at the time of creation. Finally, the design choice to prioritize label confidence for the benign pool means that benign source coverage is intentionally narrower than the malicious side, which may impact generalizability to diverse, real-world benign production environments.