Evaluating Open-Weight LLMs for Threat Data Generation
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- Evaluated 11 open-weight LLMs across contextless, STIX-guided, and dynamic few-shot prompting strategies under four temperature settings.
- Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping.
- A multi-agent setup using Gemma-4-31B and Codestral-22B achieved F1 scores of 0.91 for SDOs and 0.43 for SROs.
- The top 10 MITRE ATT&CK techniques account for 227 out of 294 mappings, covering 77.21 percent of all technique mappings.
Summary & Methodology Analysis
The paper addresses the automated conversion of unstructured Connected and Autonomous Vehicle vulnerability descriptions from Common Vulnerabilities and Exposures into structured threat information. Researchers built a domain-specific keyword list using ChatGPT, Copilot, Gemini, and Perplexity, followed by filtering by researchers. They searched the National Vulnerability Database via REST API to collect 445 candidate records and removed duplicates to retain 368 records. They applied inclusion and exclusion criteria via annotators to filter records, achieving 94 percent inter-rater agreement and retaining 183 unique records. They manually annotated STIX Domain Objects and STIX Relationship Objects for each retained CVE according to the STIX 2.1 specification, and mapped each CVE with corresponding CWE and MITRE ATT&CK technique mappings. Finally, they programmatically converted validated CSV annotations into ground-truth STIX 2.1 JSON bundles.
The methodology evaluates 11 open-weight LLMs across contextless, STIX-guided, and dynamic few-shot prompting strategies under four temperature settings using micro-precision, micro-recall, and micro-F1 scores. They also evaluated a multi-agent setup using Gemma-4-31B and Codestral-22B for role-specific decomposition, and analyzed CWE frequencies and MITRE ATT&CK co-occurrences using FP-Growth and association rules. Single-model configurations achieved F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping. In the multi-agent setup, Gemma-4-31B and Codestral-22B achieved F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively. The search returned 445 candidate records, and after removing duplicates and relevance filtering, 183 unique records were retained. Furthermore, the top 10 MITRE ATT&CK techniques accounted for 227 out of 294 mappings, covering 77.21 percent of all technique mappings.
Despite strong results in specific areas, the paper notes key limitations. Complete MITRE ATT and CK mapping remains challenging for LLMs. Additionally, LLM-generated outputs can include incomplete objects, incorrect relationships, invalid structures, and hallucinated attribute values. The models evaluated include Gemma-3-4B, Gemma-4-31B, Microsoft Phi-4, Qwen-3.5-9B, Lily-CyberSecurity-7B, CyberSec-Qwen3-DeepSeekv1, Codestral-22B, Qwen3-Coder-30B, GPT-OSS-20B, GPT-OSS-120B, and LLaMA-3.3-70B.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What problem does this paper address?
The paper addresses the automated conversion of unstructured Connected and Autonomous Vehicle vulnerability descriptions from Common Vulnerabilities and Exposures into structured threat information.
Q2. What is the main goal of the evaluation?
The paper evaluates whether open-weight Large Language Models can reliably generate Structured Threat Information Expression bundles.
Q3. What dataset was created or used in this research?
The paper uses datasets and benchmarks including CAV-STIXGen, Common Vulnerabilities and Exposures, National Vulnerability Database, CTIBench, CVE-LLM, Text2Weak, STIXnet, and CTI-GEN.
Q4. How many candidate records were initially collected from the National Vulnerability Database?
The search returned 445 candidate CVE records.
Q5. How many unique records were retained after duplicate removal and relevance filtering?
After removing duplicates and relevance filtering, 183 unique records were retained.
Q6. What were the performance scores for single-model configurations?
Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping.
Q7. How did the multi-agent setup perform?
In a multi-agent setup, Gemma-4-31B and Codestral-22B achieve F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively.
Q8. What are the primary limitations identified by the researchers?
Complete MITRE ATT and CK mapping remains challenging for LLMs, and LLM-generated outputs can include incomplete objects, incorrect relationships, invalid structures, and hallucinated attribute values.
Q9. What specific evaluation metrics and prompting strategies were used?
The paper evaluated 11 open-weight LLMs across contextless, STIX-guided, and dynamic few-shot prompting strategies under four temperature settings using micro-precision, micro-recall, and micro-F1 scores.