Back to Feed
Benchmarks & Evals / Reasoning

Benchmarking Organizational Reasoning for Enterprise Systems

Original: ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering
By Akrin Zheng, Alexander Wu, Alaia Liu Published 2026-08-11

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 1 concepts

Key Takeaways

  • Most systems struggle with latent organizational reasoning, leaving 30.4% of queries unanswered even when gold documents are provided.
  • GraphRAG outperformed other retrieval methods with a 36.2% mean accuracy on latent organizational tasks.
  • The benchmark challenges systems across three paradigms: explicit lookup, cross source composition, and latent organizational reasoning.
  • The provided dataset release maintains group level comparisons but is less reliable for fine grained ranking.

Summary & Methodology Analysis

The researchers developed EntLORE to address the gap in enterprise question answering where queries require implicit relationship extraction. They first reconstructed an audited raw graph from heterogeneous sources, including internal documents, organizational tables, and operational records. After defining specific organizational conventions to certify these relations, they compiled them into a truth graph. The team projected this source world into an anonymized release while keeping the sensitive structure and truth graph hidden. Questions were then compiled into executable graph programs that compute answers and verify that the necessary information is absent from the public corpus, ensuring the benchmark tests true reasoning rather than simple retrieval.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary purpose of EntLORE?

It serves as a benchmark for testing how well AI models can reason about hidden organizational relationships in an enterprise environment.

Q2. Why is enterprise question answering difficult?

These systems often struggle to resolve queries that require recovering implicit organizational relations that are not explicitly documented.

Q3. How does EntLORE measure system performance?

It evaluates systems across three paradigms: explicit lookup, cross source composition, and latent organizational reasoning.

Q4. How did GraphRAG compare to other methods?

GraphRAG achieved the highest accuracy at 36.2%, followed by BM25 at 34.0%, LLM wiki at 27.7%, Flat RAG at 18.9%, and agentic retrieval at 13.5%.

Q5. What happens when models try to solve latent organizational questions?

Across 56 configurations, 30.4% of these questions remain unanswered even with access to gold documents, which is significantly higher than the 6.2% failure rate for compositional questions.

Q6. Are there any limitations to the benchmark?

The paper notes that the corpus release fidelity is suitable for group level comparisons but is less reliable for fine grained ranking.

Q7. Which models were used in the evaluation?

Models included GPT-5.4, GPT-5.4-mini, Claude-Sonnet-4.6, Qwen3.5-397B-A17B, GLM-5.2, Kimi-K2.6, DeepSeek-V4-Pro, and DeepSeek-V4-Flash.

Q8. Does the paper specify the hardware requirements for these benchmarks?

The paper does not specify hardware requirements.

Q9. What other datasets were involved in this research context?

The paper references various datasets including TechQA, doc2dial, MultiDoc2Dial, EKRAG, WixQA, EnterpriseRAG-Bench, FinQA, TAT-QA, MultiHiertt, ARQA, MuDABench, HotpotQA, 2WikiMultiHopQA, IIRC, MuSiQue, KILT, KQA Pro, ProofWriter, and QO-Bench.