Back to Feed
Agents / Benchmarks & Evals

Benchmarking AI Agents for Data Analytics

Original: DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces
By Boyan Li, Zhuowen Liang, Yupeng Xie, Xiaotian Lin, Tianqi Luo, Xinyu Liu, Yizhang Zhu, Zhangyang Peng, Yuan Li, Zhengxuan Zhang, Jiayi Zhang, Nan Tang, Guoliang Li, Yuyu Luo Published 2026-08-04

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • DataSpace addresses the lack of unified evaluation for analytical agents by testing across databases, long documents, and multimedia.
  • The benchmark uses a rigorous methodology including constraint-aware sampling and deterministic evaluation to ensure accurate, repeatable results.
  • Current AI backbones struggle with multimodal integration and table joins, with the best performing agent achieving only 66.34 percent accuracy.
  • The choice of agent harness, which is the framework that manages how an AI agent executes tasks, impacts final accuracy by over 15 points.

Summary & Methodology Analysis

DataSpace provides a framework for testing data agents through a pipeline that starts with cross-language transformation, ensuring that questions and their corresponding SQL queries remain semantically identical when migrated. The methodology employs constraint-aware relational sampling, which creates smaller, task-specific datasets while maintaining schema integrity and query safeguards. The benchmark then routes tasks to appropriate renderers for various formats, including CSV, JSON, SQLite, Markdown, and PDF, and generates video-based artifacts to simulate realistic workspaces. This pipeline was verified by a panel of 11 domain experts who reviewed tasks to establish canonical evaluation configurations.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is DataSpace?

DataSpace is a benchmark designed to evaluate AI data agents on complex, verifiable analytical tasks that span heterogeneous workspaces.

Q2. Why is a new benchmark needed?

Existing benchmarks often focus on isolated tasks like structured querying or retrieval but fail to represent the full scope of real-world data workspaces, which include mixed data types like multimedia and documents.

Q3. What kind of tasks does DataSpace test?

It tests agents on tasks requiring evidence discovery across diverse sources, generating complete tabular outputs, and producing results that can be verified deterministically.

Q4. How does the evaluation process ensure accuracy?

The system uses deterministic evaluation, which includes header-invariant column alignment, type and precision-aware normalization, and row comparison using a canonicalized result view.

Q5. How much does the choice of agent harness matter?

When keeping the underlying model fixed, the choice of agent harness creates a 15.36-point spread in accuracy.

Q6. What are the primary performance hurdles for current models?

Models consistently show lower accuracy when they must perform multimodal evidence integration and complex joins.

Q7. Does the benchmark include video processing?

Yes, DataSpace generates query-conditioned video artifacts to test an agent's ability to interpret multimedia data.

Q8. Are all data agents compatible with this benchmark?

No, some specialized agents are excluded because their native interfaces do not support the required video inputs or the benchmark's tabular-output contract.

Q9. Is the performance of current agents considered optimal?

No, the benchmark remains unsaturated, indicating there is significant room for improvement in data agent performance.