Benchmarking Mobile Agent Planning Capabilities

Summary & Methodology Analysis

The researchers developed MobilePA-Bench to provide an interactive, stateful, and tool-centric environment for evaluating agent performance in mobile scenarios. Unlike existing frameworks like AndroidWorld or OSWorld, which focus on Vision-Language Models (VLMs) that inspect screenshots and predict pixel-level coordinates, this benchmark maps mobile operations into executable function signatures. This approach builds upon the paradigm used in general-domain benchmarks like the Berkeley Function Calling Leaderboard and ToolBench, while incorporating specific mobile requirements seen in datasets like DroidCall and AppBench. The evaluation framework supports multi-turn user interactions, mirroring transactional workflows similar to those found in TAU-Bench.

Baseline evaluations across 13 models reveal that performance remains far from the requirements for reliable autonomous deployment. The strongest agent, Claude-Opus-5, achieves an overall weighted score of 75.52%. A persistent challenge across these models is memory management, where performance averages only 50.98%, with scores as low as 33.78% and as high as 64.63%. The platform also evaluates how agents combine basic actions into executable code abstractions, utilizing concepts similar to those in VOYAGER or SkillBench, and examines task scheduling akin to environments like OpenCLAW.

The research identifies a critical reliability gap, as even the most capable agents exhibit a failure rate of at least 24.48%. A major contributor to this is the inability of models to handle runtime exceptions, such as PermissionDenied errors. When models encounter capability limits, ambiguous constraints, or execution exceptions, they tend to issue premature, hallucinated tool calls rather than asking for clarification or adjusting their strategies based on feedback. This indicates that current frontier models lack the robust planning logic needed for dynamic mobile environments.