Turning LLM Repairs into Reusable Memory
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 1 concepts
Key Takeaways
- Storing verified query corrections improves held-out first-attempt accuracy by 4.34 percentage points on the BIRD dataset.
- Increasing the retrieval depth from one to ten memory cards provides an additional 3.18 percentage point gain in transfer accuracy.
- The system separates performance gains into distinct buckets: exact-query replay, cross-question retention, and same-database held-out transfer.
- The approach uses a novel memory card format and per-database memory banks to avoid revealing gold SQL strings.
Summary & Methodology Analysis
The researchers tackle the crystallization problem in text-to-SQL, which involves leveraging successful test-time repairs to build persistent knowledge for future queries. The system begins with a single-shot LLM solver that generates an initial SQL query. When the solver fails, an execution-guided repair process runs to produce a verified, functional query. Instead of discarding these successful episodes, the method constructs memory cards in a novel format and admits them into per-database memory banks. When a new question arrives, the system retrieves the top-k memory cards using question-to-question cosine similarity, which measures how closely two vectors represent the semantic intent of the natural language queries.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the crystallization problem?
It is the challenge of determining how verified repair episodes can be stored and reused to improve LLM performance on recurring and unseen questions for the same database.
Q2. Does this method require exposing the gold SQL?
No, the method acquires verified episodes without revealing the gold SQL strings.
Q3. Does this approach work for unseen schemas?
The paper does not focus on cross-database generalization to unseen schemas, noting it is a different deployment setting.
Q4. How does the system decide which queries to store?
The paper does not claim the model can safely decide what to store on its own without external verification.
Q5. What is the impact of increasing the number of retrieved memory cards?
Increasing retrieval from one to ten cards added 3.18 percentage points to transfer accuracy.
Q6. Which datasets were used in the study?
The researchers used the BIRD and Spider datasets.
Q7. How does the evaluation methodology treat different types of performance gains?
The system explicitly separates performance into three categories: exact-query replay, cross-question retention, and same-database held-out transfer.
Q8. Does the 70/30 protocol reflect real-world workloads?
No, the protocol measures conditional values in distinct settings rather than the actual prevalence of these settings in a real-world production workload.
Q9. Which models were utilized in the experiments?
The models included Qwen3.5-27B, Qwen3.5-9B, Gemma4-E4B, Gemma4-31B, gpt-oss-20b, Hunyuan-A13B, and Qwen3-Embedding-0.6B.