Structuring LLM Rubrics as Evaluation Graphs

Summary & Methodology Analysis

Graph-Structured Rubrics (GSR) addresses the issue where LLM judges treat rubrics as unstructured prompt text, which leads to procedural instability. Instead, the authors compile rubrics into a directed acyclic graph (DAG). This graph uses criterion nodes for evaluation, alongside deterministic operator nodes including Transform, Reduce, and Gate. By defining the evaluation as a graph, the system allows for static validation before any model inference occurs. This validation checks for properties such as acyclicity, sink reachability, arity, and type compatibility, ensuring the evaluation flow is technically sound and predictable. The execution logic relies on processing criterion-level judgments in topological order through these deterministic operators. Once the graph execution concludes, a task-specific Readout is applied at the unique sink node, which serves as the final contract to produce scores or preferences. This approach moves the evaluation logic out of the prompt and into a compiled execution structure. Evaluation results across four pointwise datasets show that GSR achieves exact score agreement improvements of 0.62 to 6.75 percentage points compared to Prometheus-style scoring methods. The performance of GSR is not backbone-invariant, meaning the final results depend on the underlying LLM judge used to traverse the graph. Furthermore, the framework does not perform calibration across different rubrics or judge models, which remains a limitation for production deployments requiring cross-model consistency.