Back to Feed
Benchmarks & Evals / Reasoning

Improving Enterprise SQL Generation Reliability

Original: Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL

Listen to the summary

Uses a voice available on your device

Audio options
On this page 5 sections
Related concepts 1 concepts

Key Takeaways

  • The Semantic Path Compilation (SPC) system significantly outperforms direct DDL-to-SQL baselines in enterprise settings.
  • SPC achieved a 97.4% success rate on an adjudicated 38-question set, compared to 55.3% for the baseline.
  • The approach effectively eliminates wrong-but-executed SQL queries by replacing direct generation with a governed compilation process.
  • Performance is validated using GPT-5.4 and Gemini-3.6-Flash on the ACME insurance benchmark.

Summary & Methodology Analysis

Enterprise text-to-SQL tasks often fail because models generate queries that execute without errors but produce incorrect data due to misaligned relationship roles or aggregation grains. The paper addresses this by implementing a Semantic Path Compilation (SPC) system. Instead of generating SQL directly, the system employs a multi-turn planner that grounds question phrases into specific, governed options. This abstraction layer ensures that the underlying logic remains consistent with enterprise data rules before any SQL code is constructed.

The architecture relies on a series of deterministic steps following the planning phase. These include graph traversal, applying role predicates, and grain lowering, which is a process to adjust aggregation levels to match the schema. The SPC then generates the SQL through a deterministic compilation process rather than relying on the model to predict raw SQL strings. This methodology allows for hard-coded checks that verify the structure of the query against the schema constraints, effectively preventing the production of invalid logical paths that would otherwise pass database execution checks.

Evaluation conducted on the ACME insurance benchmark using GPT-5.4 and Gemini-3.6-Flash demonstrates that SPC produced only one refusal and zero wrong-but-executed runs across 114 outcomes, whereas the baseline generated 29 adjudicated wrong runs. The authors note a key limitation: the performance gains cannot be exclusively attributed to the compilation process itself, as the system relies on pre-defined, governed semantic artifacts that are not available to the baseline model.

Interactive System Flowchart

Click diagram to expand and zoom

Illustrative Implementation

A short sketch of the paper's core idea, not the authors' own code.

# Illustrative sketch (not from the paper)
import torch
# Placeholder model (e.g., GPT-5.4) loaded via PyTorch
model = torch.hub.load('pytorch/transformers', 'gpt5_4', pretrained=True)

def plan_semantic_path(question):
    # 1. Ground phrases and select governed options
    grounded = model.generate(question)  # mock grounding
    # 2. Build semantic graph from grounded tokens
    graph = build_graph(grounded)
    # 3. Traverse graph applying role predicates and grain lowering
    path = traverse(graph, role_predicates=True, grain_lower=True)
    # 4. Compile deterministic SQL from path
    sql = compile_sql(path)
    # 5. Deterministic check: ensure no ambiguous roles
    assert deterministic_check(sql), "Non‑deterministic plan"
    return sql

def build_graph(tokens):
    # placeholder graph construction
    return {"nodes": tokens, "edges": []}

def traverse(g, role_predicates, grain_lower):
    # placeholder traversal returning ordered tokens
    return g["nodes"]

def compile_sql(path):
    # simplistic concatenation for illustration
    return "SELECT * FROM table WHERE " + " AND ".join(path)

def deterministic_check(sql):
    # ensure the SQL contains only allowed predicates
    return "JOIN" not in sql

# Example usage
sql_query = plan_semantic_path("How many policies expired last month?")

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the core problem with existing text-to-SQL models?

Current models often generate SQL queries that execute successfully but return incorrect data due to misunderstandings of relationship roles or aggregation grains.

Q2. What is the Semantic Path Compilation system?

It is a system that uses a multi-turn planner and deterministic code-based checks to ensure generated SQL adheres to enterprise data constraints.

Q3. How does SPC compare to traditional direct generation?

SPC achieved a 97.4% success rate on the test set, while the direct DDL-to-SQL baseline only reached 55.3%.

Q4. What models were used to test this system?

The paper evaluated the system using GPT-5.4 and Gemini-3.6-Flash.

Q5. Does the system produce incorrect SQL that executes?

Across 114 outcomes, the SPC system produced zero adjudicated wrong-but-executed runs, whereas the baseline produced 29.

Q6. What does the multi-turn planner do?

It grounds phrases from the user question and selects them from specific, governed options to ensure the query remains within valid enterprise logic.

Q7. What is the role of grain lowering in this system?

Grain lowering is a step within the SPC system used to adjust aggregation levels to ensure they align correctly with the schema.

Q8. What is the main limitation of the performance results?

The performance gains cannot be attributed solely to compilation because the SPC relies on governed semantic artifacts that the baseline model does not have access to.

Q9. What dataset was used for this research?

The paper utilized the ACME insurance benchmark for testing.

Flag an issue

What is wrong with this summary?

What is wrong?