Validating AI Actions for Industrial Safety

Summary & Methodology Analysis

ADMITBench addresses the gap between LLM diagnostic capabilities and industrial operational reality by treating the proposed action, rather than the raw output, as the primary unit of evaluation. The architecture represents model responses as structured action records containing the advisory, diagnosis, evidence, and traceability information. These records are then subjected to a pipeline of non-compensatory eligibility gates, ranging from T0 to T4, which verify integrity, state validity, hazard awareness, and physical consequence thresholds. Only once an action survives these gates does the system perform utility ranking at the T5 level. The entire process culminates in a T6 audit trace that provides the necessary context and justification for each decision, facilitating transparency and reproducibility. The framework relies on versioned plant profiles, known as cartridges, that incorporate system graphs, safety-case definitions, and procedure cases to define the operational boundary conditions. The current release, 0.1.0, provides specific profiles for cstr_alpha and column_alpha. One notable demonstration within the column_alpha profile is the D03 case, which illustrates an authority mismatch failure that necessitates human escalation. The system does not claim to replace existing functional-safety certifications or grant deployment authorization. Furthermore, the current implementation lacks broad industrial generality and independent portability, as the profiles depend on human-authored engineering judgments.