Traceable Scientific Question Answering Benchmark

Summary & Methodology Analysis

LitTraceQA addresses a major reliability issue in retrieval-augmented generation systems where models fail to trace their output back to verifiable evidence. The benchmark pipeline forces a strict sequence: retrieving relevant paper identifiers from a metadata pool, locating specific evidence types such as tables, figures, text spans, equations, or citation contexts, and finally generating an answer. To ensure the benchmark genuinely tests retrieval-augmented reasoning, the authors used a closed-book challenger model filtering process during dataset construction to eliminate questions that could be answered by model weights alone without external evidence. This forces the system to prove its work by grounding its reasoning in the provided research literature. The evaluation framework separates performance into three distinct metrics for retrieval, grounding, and answer generation. The benchmark enforces a strict joint success criterion, meaning a system must succeed in all three stages to count as correct. This prevents systems from getting partial credit for correct answers that were produced by hallucinating or guessing without valid citations. The dataset is currently concentrated in specific scientific domains from venues such as ICCV, NAACL, and ACL. A current technical hurdle is the need for normalization of the flexible evidence locator schema to fully automate the validation process. The paper does not specify the exact latency or computational cost of running these evaluations.