Benchmarking AI Agents for Data Science

Summary & Methodology Analysis

The researchers extended the existing OSWorld framework to support a realistic data science stack, including Jupyter Notebook, SQLite, and external data connectors for platforms like Kaggle and OpenML. The environment simulates an Ubuntu virtual machine where agents are evaluated on their ability to navigate file systems and manage tool coordination. Unlike previous benchmarks that rely on surface-level code execution, this system uses a deterministic, execution-based evaluation to verify the actual analytical correctness of the generated outputs. This methodology forces agents to perform iterative analytical reasoning rather than simply predicting isolated code snippets. Agents interact with the desktop via a GUI action space, utilizing both visual screenshots and accessibility trees as primary inputs for decision-making. The evaluation focuses on end-to-end workflows that mirror actual data science tasks. The current performance leaders are proprietary models, specifically Claude-4.6-Sonnet, which hit a 56.70% success rate under the combined screenshot and accessibility tree setting. In contrast, the open-source landscape remains nascent for this specific application, with all evaluated open-source models failing to exceed a 1% success rate. The study acknowledges significant limitations regarding open-source agents, as they currently lack support for accessibility tree observations and are restricted to screenshot-only inputs. Furthermore, the qualitative error analysis is confined to a subset of trajectories, which may exclude rare failure patterns in the agent workflows.