Evaluating LLM Reliability in SPICE Netlist Tasks

Summary & Methodology Analysis

The paper addresses the lack of understanding regarding whether large language models can correctly recognize and manipulate netlists as structured circuit artifacts. To solve this, the authors build a benchmark pipeline using source SPICE netlists derived from AnalogGenie and ALIGN. They apply family-specific transformation rules, syntax perturbations, and prompt templates in a deterministic, template-driven pipeline to construct the benchmark instances. The resulting artifact is NetlistBench, which contains 2,342 cases across 24 task families.

To evaluate the models objectively, the method parses both model outputs and reference targets into a canonical circuit representation, known as an intermediate representation or IR, which lists devices, terminal nodes, parameters, directives, and subcircuit interfaces. It then employs a deterministic structure-aware oracle that checks structural equivalence under semantic-preserving normalizations, ensuring that variations in syntax that do not change the circuit meaning are correctly handled during evaluation.

The paper notes several limitations regarding the scope of the benchmark. NetlistBench evaluates bounded, circuit-block-level netlists rather than industrial-scale post-layout decks. It does not assess long-context retrieval, hierarchical partitioning, or direct processing of extracted netlists containing millions of device and parasitic statements. Furthermore, the current release covers a restricted circuit and syntax domain and should not be assumed to transfer unchanged to RF, power, digital, or mixed-signal netlists, while task instructions generated from deterministic templates do not capture the full linguistic variability or design intent of real designer assistant interactions.