Interpretable Agentic Retrieval for Complex Documents

Summary & Methodology Analysis

The researchers replaced traditional chunk-and-embed retrieval with Read, a Model Context Protocol server that enables deterministic agentic operations. This approach addresses the failures of dense embedding models, which often lose layout-dependent context such as fiscal-year headers or units when breaking text into chunks. By implementing a table-aware chunker, the system propagates these critical headers across document boundaries, significantly increasing the reliability of numeric data retrieval. The agentic loop follows a cycle of hypothesis, search, outline-based inspection, and specific line-range reads, concluding with arithmetic calculation and mechanical verification of groundedness against the raw text. Performance testing on the Gujarat Finance Accounts 2024-25, Volume I showed that while a dense baseline with 2,000-character chunks and k=16 achieved 35.3% accuracy, the Read system exceeded this by 23.5 percentage points. The table-aware chunker proved highly effective, reducing the prevalence of unitless numeric chunks from 18.0% to 0.3% at the same chunk scale. The study highlights that the performance of the system relies on the specific PDF-to-Markdown converter employed, as conversion artifacts can impact results. Limitations include the reliance on a single financial document and one backbone model, namely gemini-2.5-pro, meaning results may not generalize across different domains or model architectures. Furthermore, the benchmark is small, consisting of only 51 questions, which prevents definitive statistical resolution when comparing Read against other methods like BM25.