Scaling Local Computer Use AI Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Providing adequate history length improves single agent accuracy, whereas increasing history beyond four steps adds cost without improving accuracy.
- Increasing maximum decoding steps yields little to no improvement in task success across most models.
- The two-stage framework consistently underperforms the corresponding single-agent baselines.
- Single-agent configurations achieve higher accuracy at substantially lower token costs than two-stage configurations.
Summary & Methodology Analysis
This paper addresses whether inference time scaling techniques, which use additional computation during execution to improve frontier proprietary computer use agents, are effective for resource constrained local models. The authors evaluate both single agent frameworks and two stage agent frameworks on the OSWorld benchmark using an A100 GPU and the vLLM engine for precise measurement. The single agent framework uses a multimodal model taking instructions, screenshots, and history to generate actions step by step. The two stage framework decouples inference into a planning stage that generates candidate plans using history and screenshots, followed by a judging process and a specialized grounding model that translates the selected plan into specific coordinates. The authors define three compute scaling dimensions, namely temporal scaling, contextual scaling, and parallel scaling, and track metrics like steps per task and prompt token usage.
The findings reveal that contextual scaling improves performance up to a point, as single agent accuracy rises from about 18 percent with no history to over 25 percent with one step of history, and peaks at an optimal history length of four steps. However, pushing history further adds cost without helping accuracy and shifts dominant failure patterns from stalls to premature false successes. In contrast, temporal scaling by increasing maximum decoding steps fails to raise the success ceiling, instead extending erroneous trajectories because local models lack sufficient reasoning capacity. Furthermore, structural decomposition into a two stage agent actually harms performance by introducing formatting errors and planning overhead, while parallel scaling yields sub-linear gains compared to its sharp increase in token usage.
Several core limitations constrain local computer use agents based on these findings. Local models have limited planning capability, which can cause generated plans to be incomplete or poorly aligned with visual states. Additionally, two stage decomposition creates formatting dependencies where planners omit required fields, causing parsing failures. The paper notes that agentic frameworks designed for stronger proprietary models do not transfer directly to local models, meaning fully local systems require lightweight alternatives to large model recovery mechanisms since escalating to stronger models is not an option.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What problem does the paper address?
The paper addresses whether inference time scaling techniques are effective for resource constrained local computer use agents and identifies the failure patterns that limit further gains under strict hardware constraints.
Q2. What hardware and software engines were used for the experiments?
All experiments were run on an A100 80GB GPU using the vLLM engine for model inference and precise token usage measurement.
Q3. What is the main takeaway regarding single agents versus two-stage agents?
Single agent configurations achieve higher accuracy at substantially lower token costs than two stage configurations, while the two stage framework consistently underperforms single agent baselines.
Q4. How does contextual scaling affect agent performance?
Accuracy surges from about 18 percent with no history to over 25 percent with one step of history, with four steps offering the best accuracy cost tradeoff before accuracy saturates and shifts failures toward premature false successes.
Q5. What happens when maximum decoding steps are increased for temporal scaling?
Increasing maximum decoding steps yields little to no improvement in task success across most models, often extending erroneous trajectories rather than correcting them due to the limited reasoning capacity of local models.
Q6. How does parallel scaling impact two-stage agents?
Increasing the number of multiple candidate plans from one to four partially recovers task success and reduces format errors, but token usage increases sharply, yielding sub-linear gains relative to computation.
Q7. What benchmark and tasks were used in the evaluation?
The paper evaluated models on the OSWorld benchmark using 361 real world Ubuntu tasks, with eight Google Drive tasks excluded due to environment constraints.
Q8. What specific models were evaluated as single agents?
The evaluated single agents include Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct, UI-TARS-1.5-7B, and OpenCUA-7B.
Q9. What are the limitations of the agent perception capabilities according to the paper?
Agents operate solely from screenshots without access to auxiliary interfaces such as accessibility trees or visual prompting techniques like Set-of-Mark.