Benchmarking LLM Performance With Noisy Tools

Summary & Methodology Analysis

The paper addresses the common assumption that tool-augmented dialogue systems operate with perfect data integrity. To evaluate real-world scenarios where tools provide noisy or unreliable predictions, the authors constructed PredAct-Bench. This framework supports multi-turn dialogue, moving beyond static evaluations by analyzing how models guide users through decision-making processes using real assessment data from datasets like OULAD and synthetic score trajectories in PREDACT-CS. The methodology relies on introducing measurable noise into predictors that guide user behavior, effectively simulating environments where automated assistance is imperfect. The authors extended standard reliability metrics, specifically Relative AI-Reliance (RAIR) and Relative self-reliance (RSR), to quantify how models manage human trust during these sessions. By applying these metrics across 13 state-of-the-art LLMs, the study tracks how effectively models communicate their own uncertainty or validity when presenting tool outputs. A core component of the methodology involved human studies with instructors and teaching assistants, providing ground truth for how well these models supported actual decision-making tasks. The evaluation shows that current top-tier models frequently lack the necessary visibility to signal tool noise, causing users to mistakenly trust incorrect outputs. The study is limited by its focus on the education domain, which serves as a highly structured testbed where interventions and outcomes are well-defined. Because this domain allows for precise tracking of assessment trajectories, it is currently the only environment where the benchmark is applied. The paper does not provide data on latency, memory usage, or compute cost per inference request for the tested models.