How Corpus Scale Changes Retrieval Systems
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- A controlled study scaled corpus size across 28 tiers ranging from 1,144 to 511,959 documents to test the accuracy and cost of various retrieval paradigms.
- Traditional BM25 keyword search outperformed complex agents and dense retrieval at scale, overtaking them around a crossover point of 10 million corpus tokens.
- Graph-based retrieval pipelines hit severe construction walls, failing to build at larger corpus scales due to extreme computational token requirements.
- Combining a retrieval agent with BM25 achieved the highest overall retrieval hit rate at full scale.
Summary & Methodology Analysis
The paper presents a controlled study that varies corpus size along a 28 tier nested ladder while holding questions, readers, and judges fixed. To test different ways of finding information, the researchers evaluated seven native pipelines across four paradigms. These included traditional keyword search using an inverted index without a language model, dense retrieval using chunk embeddings, file system agents using tools over raw per source files, and several graph based systems that map relationships between entities. They used a fixed reader model at temperature zero along with a shared embedding model, and measured every language model call to track token consumption during both building and querying phases.
At smaller scales, file system agents and traditional keyword search performed similarly, but a scale dependent crossover occurred around 10 million corpus tokens. Beyond this point, traditional keyword search led at every larger shared tier, with the performance gap reaching nearly 20 points at full scale. Meanwhile, graph based builders hit severe construction walls, stopping entirely at intermediate document scales because their resource requirements extrapolated to billions of tokens and weeks or years of instance time. The study also controlled for various factors, such as using alternative judges, binary scoring protocols, and different prompt budgets, confirming that traditional keyword search maintains a strong advantage in efficiency and scaling.
Despite its thorough methodology, the study has certain limitations. Each system was run only once per question rather than through stochastic reruns, relying instead on resampling questions to quantify uncertainty. Graph pipelines stopped building early, meaning their large scale results reflect construction feasibility rather than accuracy. Furthermore, the experiments relied on a single reader model, a single embedding model, and a single enterprise style dataset, meaning findings on specific question types and scaling limits may not automatically generalize to every other corpus type or domain.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main goal of this paper?
The paper studies how different retrieval systems perform as document collections grow larger, comparing their accuracy and cost across 28 distinct corpus sizes.
Q2. Which retrieval method won at large scales?
Traditional BM25 keyword search won at large scales, overtaking other systems around 10 million corpus tokens and leading by nearly 20 points at full scale.
Q3. What happened to graph-based retrieval systems as the corpus grew?
Graph builders hit construction walls and stopped working at larger corpus tiers because their computational token requirements became too high.
Q4. How many documents and tokens were included in the largest test tier?
The largest test tier contained 511,959 documents, which amounted to 601 million tokens.
Q5. What reader and embedding models were used for the main pipelines?
The researchers used Qwen3.6-27B as the reader model and Qwen3-Embedding-0.6B as the shared embedding model.
Q6. How did the file system agent perform in terms of token usage and budget exhaustion?
File system agent query tokens grew from 226K at the bedrock to 343K at a larger tier, and budget exhaustion rose from less than 7 percent at smaller tiers to 31 percent at full scale.
Q7. What robustness checks did the authors perform on their scoring?
The authors checked robustness using an independent judge, a binary protocol, and re-scoring sessions to verify that overall family rankings were preserved.
Q8. What are some limitations regarding the graph pipelines in the study?
Construction stopped early for graph pipelines, meaning their large scale results concern construction feasibility rather than extrapolated accuracy, and some graph models did not log retrieved evidence identifiers.
Q9. Does this study test multiple different reader models?
No, the study uses a single reader model and a single embedding model.