Building Knowledge Bases Using Only LLMs
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 1 concepts
Key Takeaways
- REAP enables closed-book knowledge base construction by using relation-specific prompts to extract structured facts from LLMs.
- The system achieves a macro-F1 score of 0.62 on the AKBC Shared Task 2026 test set.
- Performance is granular, with specific relation scores like 0.95 for countryLandBordersCountry and 0.73 for companyTradesAtStockExchange.
- The method relies on a multi-stage process involving evidence elicitation, deterministic parsing, and data normalization.
Summary & Methodology Analysis
The REAP architecture operates in a closed-book, parameter-constrained environment. Stage 1 focuses on evidence elicitation using relation-specific prompts, query decomposition, and Chain-of-Thought reasoning, which is a technique where the model generates intermediate logical steps to reach a conclusion. An empty-set gate is utilized during multi-pass querying to manage variable-cardinality answer sets. This ensures that the system handles complex queries without requiring fine-tuning, which is the process of adjusting a pre-trained model on a specific dataset.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of the REAP system?
It aims to construct knowledge bases directly from LLMs without fine-tuning, specifically handling variable-cardinality answer sets.
Q2. Does this system require training?
No, the system operates in a closed-book, parameter-constrained setting without fine-tuning.
Q3. What is the overall performance of REAP?
It achieved a macro-F1 score of 0.62 on the official AKBC Shared Task 2026 test set.
Q4. How does the system handle data serialization?
Stage 2 serializes extracted evidence into JSON arrays using deterministic regex parsing for most records, with LLM-based extraction as a fallback.
Q5. Which specific models were used in the evaluation?
The system used Mistral-Small-24B-Instruct-2501, Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Qwen3.5-9B.
Q6. What post-processing steps are applied to the data?
Normalization is achieved through numeric extraction, title filtering, parenthetical filtering, and case-insensitive deduplication.
Q7. What are the known limitations regarding entity coverage?
The parametric knowledge within the LLMs is incomplete for long-tail entities.
Q8. Are there issues with output consistency?
Yes, stochastic sampling and non-deterministic computation on TPU hardware lead to slight run-to-run variation in outputs.
Q9. What datasets were utilized for this study?
The study utilized the AKBC Shared Task 2026, Wikidata, and the LM-KBC challenge.