Why Routing Web Agent Observations Fails

Summary & Methodology Analysis

Web agents typically select a single observation mode, such as text, pixels, or both, at design time for all tasks, ignoring potential per-task performance gains. To investigate whether dynamically routing tasks to different observation modes improves efficiency, the authors measure success rates and failure patterns of six distinct observation modes across eight site-model cells. They perform replicated runs for select conditions to estimate a rerun-based noise floor for evaluating routing gains, and establish an outcome oracle ceiling based on a six-arm union and a rerun-controlled cost ceiling. Building on this measurement, the study implements and tests five routing policies: which-mode selection, learned triage of spend, a zero-token rule based on task text, a confidence cascade, and pooled cost tiers. These policies are compared against trivial fixed-policy baselines.

The empirical results show that rerunning the same mode on the same tasks changes 12 to 14 percent of outcomes. Furthermore, routing only the never-solved tasks to the cheapest mode cuts cost by 9.5 to 30.6 percent in all eight cells without changing success rates. However, the five tested routing policies failed to robustly outperform fixing a single well-chosen mode. The study also finds that the label supply for a router and the tasks where routing is valuable are both correlated with agent success rate at 0.95 across cells. The evaluation leverages VisualWebArena, WebArena, and Mind2Web datasets, alongside Qwen3-VL-235B, Qwen3-VL-4B, and Gemma-3-4B models.

Despite its rigorous evaluation across multiple models and benchmarks, the study has several limitations. The paper lacks a third workload, relying only on classifieds and reddit task sets. Additionally, rerun noise bands were only measured in two cells and imported into the other six, assuming invariance. Finally, cascade outcomes were estimated via offline splicing rather than observing stateful site interaction, and the paper does not specify other potential confounding variables.