Improving GUI Agent Accuracy Through Reflection
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 7 concepts
Key Takeaways
- LookAgain addresses the limitations of one-shot grounding by implementing a predict, look, and refine loop.
- The 8B parameter version achieves 73.0 overall accuracy on OSWorld-G, outperforming the base Qwen3-VL-8B by 21.7 points.
- LookAgain improves grounding performance on dense layouts and small targets, showing gains of 15.5 points on UIVision and 13.1 points on ScreenSpot-Pro.
- The training pipeline combines supervised fine-tuning with reinforcement learning using Group Relative Policy Optimisation to improve grounding accuracy.
Summary & Methodology Analysis
The LookAgain framework addresses the performance decay found in existing GUI grounders when handling small targets, dense layouts, or out-of-distribution interfaces. Rather than treating coordinate prediction as a single pass, the system utilizes a multi-turn approach. The agent uses a locate primitive to propose a coordinate and render a red marker at the center. It then generates a 512x512 local patch of the target area, which is appended to the dialogue history, allowing the model to perform post-hoc visual inspection before deciding whether to accept or reject the prediction via a confirm primitive.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem with current GUI grounding models?
Current models fail because they treat grounding as a one-shot process without a mechanism to review their predicted coordinates, causing issues on small targets and dense layouts.
Q2. How does LookAgain fix this issue?
It implements a multi-turn process where the model predicts a coordinate, inspects a local crop of the image, and uses a confirmation step to accept or refine the result.
Q3. Does LookAgain work on standard GUI benchmarks?
Yes, it shows significant performance improvements on benchmarks like OSWorld-G, UIVision, and ScreenSpot-Pro.
Q4. What specific reinforcement learning technique does the paper use?
The paper uses Group Relative Policy Optimisation (GRPO), which optimizes the model using terminal grounding correctness as the reward.
Q5. How much better is the 8B model compared to the baseline?
LookAgain-8B reached 73.0 accuracy on OSWorld-G, which is 21.7 points higher than the base Qwen3-VL-8B model.
Q6. What happens during the two-stage training pipeline?
The process begins with supervised fine-tuning on multi-turn reflective trajectories, followed by a second stage of reinforcement learning.
Q7. Are there any known risks to using this model?
The model inherits the risk of hallucinating ungrounded text or producing biased results from the underlying foundation models.
Q8. How does the 32B model compare to other versions?
LookAgain-32B attained the best overall accuracy on OSWorld-G among models of comparable scale.
Q9. What is the specific input resolution of the local patch used for inspection?
The locate primitive creates a local 512x512 patch for the visual inspection process.