Reliable Fact Checking Using Adaptive Retrieval
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Key Takeaways
- Reduces hallucinations by dynamically routing queries to web search when internal model confidence is low.
- Provides formal mathematical guarantees to keep the False Discovery Rate below a user-defined threshold.
- Improves coverage on NQ-Open from 7 percent to 82 percent for the Qwen3-8B model.
- Uses Clopper-Pearson upper confidence bounds to calibrate decision thresholds.
Summary & Methodology Analysis
The system architecture operates through two primary evaluation modes to handle objective factual assessment. Mode 1 evaluates candidate answers using only the model parametric knowledge (weights acquired during pre-training). If uncertainty in Mode 1 exceeds a specific threshold, the system triggers Mode 2, which performs a web search to gather external evidence and then re-evaluates the answer. This routing mechanism ensures that the system only consumes the computational cost of retrieval when strictly necessary, balancing accuracy against performance overhead.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the core problem this paper addresses?
It addresses the tendency of LLMs to hallucinate confidently and the lack of reliable, risk-controlled methods for evaluating factual accuracy.
Q2. How does the system decide whether to search the web?
It uses an uncertainty threshold calibrated during a pre-deployment phase to determine if the model internal knowledge is sufficient or if external retrieval is required.
Q3. Does this method provide any safety guarantees?
Yes, it uses Clopper-Pearson upper confidence bounds to mathematically bound the False Discovery Rate below a target level.
Q4. What happens if the system is still uncertain after searching the web?
The system abstains from providing a verdict, ensuring that unreliable outputs are not presented as factual.
Q5. How did the model performance improve on NQ-Open?
The Qwen3-8B judge saw an increase in coverage from 7 percent to 82 percent compared to using only parametric knowledge.
Q6. Which models and benchmarks were used in the study?
The study utilized Qwen3-8B and LLaMA-3.1-70B models, evaluated across TriviaQA, Natural Questions, HotpotQA, and PopQA.
Q7. What are the limitations regarding the stability of the system?
The system relies on a non-stationary web index, meaning search results may change over time, which can impact the validity of the error rate guarantees.
Q8. What happens if I change the candidate model after calibration?
Any change to the task, the candidate model, or the judge model violates the exchangeability assumption and requires recalibration.
Q9. Is the performance improvement consistent across different models?
The paper shows that on HotpotQA, coverage for Llama-8B increases from 40 percent to 85 percent.