Back to Feed
Benchmarks & Evals / Reasoning

Benchmarking LLMs on Evolving Regulatory Documents

Original: Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding
By Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury, Md Adnan Arefeen, Farig Sadeque, Md. Faizul Bari, Swakkhar Shatabda Published 2026-08-09

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 3 concepts

Key Takeaways

  • The study evaluates nine recent large language models on their ability to handle temporally evolving regulations like tax codes and parliamentary acts.
  • Models struggle significantly with version resolution, achieving a best macro-averaged accuracy of only 68.5%.
  • Identifying that a specific version of a document does not govern a query remains a major failure point, with an accuracy rate of just 26.7%.
  • The benchmark uses a curated dataset of 644 official documents spanning 1969 to 2025 to test reasoning grounded in temporal data.

Summary & Methodology Analysis

The researchers constructed a specialized dataset of 644 official PDFs, including Acts of Parliament and statutory orders, to build a version-sensitive benchmark. They utilized Gemini 3.1 Pro for initial data extraction, implementing a schema to preserve document hierarchies and metadata. Following a manual correction phase by thirteen volunteers, the team employed a multi-pass entity-concept extraction pipeline. They used greedy set covers to pack entities into bounded judge units and applied union-find closures to group document elements into clusters. These clusters formed the basis for generating regulatory threads, which serve as the input for eight distinct question-answering task types.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary problem this paper addresses?

The paper addresses the difficulty large language models face when processing documents that change over time, such as laws or tax codes where the correct answer depends on which version is currently in force.

Q2. What kind of data does this benchmark include?

The benchmark includes 644 official documents, including 3 Acts of Parliament, 20 General Orders, 609 Statutory Regulatory Orders, and 12 Rules ranging from 1969 to 2025.

Q3. How did the researchers ensure the accuracy of their QA pairs?

The researchers ensured accuracy by having each pair independently verified by two subject-matter experts using a three-point Likert scale.

Q4. Which models were evaluated in the study?

The study evaluated Qwen3.6-35B, GPT-OSS-120B, Gemma-4-31B, Kimi-K2.5, GLM-4.6, DeepSeek-V4-Flash, GPT-5.2, Gemini-3.5-Flash, and Claude-Sonnet-4.5.

Q5. How well do these models perform at resolving versions from an implicit date?

Models reach an accuracy of 59.7% when resolving a version based on an implicit date.

Q6. What are the limitations of the retrieval-augmented generation (RAG) setting here?

The retriever used is purely semantic and lacks temporal re-ranking, meaning the reported RAG scores represent a lower bound rather than optimal performance.

Q7. Does the paper test generalization across different languages or domains?

No. The benchmark focuses solely on Bangladesh customs in a specific language pair and calendar system, leaving cross-domain generalization untested.

Q8. What was the word error rate of the initial automated extraction?

Gemini achieved a word error rate of 0.07-0.08% during the information extraction process.

Q9. Why is it difficult to interpret the results from the parametric setting?

The parametric setting relies on unverifiable assumptions because the model providers do not disclose their training datasets.