Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

What they did

1. Built a real-device mobile environment with over 100 physical Android devices and more than 150 apps, used for task design, data collection, reinforcement learning, and evaluation on actual phones
2. Unified the action space to include GUI operations, bash-based CLI commands, structured API calls, and user-confirmation requests, and lets the model emit batched sequences of actions in one decision step
3. Scaled verifier-guided online reinforcement learning to trajectories exceeding 100 interaction steps, running roughly 10,000 simulated environments concurrently to speed up training
4. Built an AutoResearch-style data flywheel where agents generate tasks, diagnose failures, and plan the next training iteration with less human effort
5. Built a proactive service harness that detects phone notifications such as a flight cancellation and proposes an actionable plan for user approval rather than waiting for explicit instructions

Findings

• On MobileWorld-Real (a real-device benchmark with over 400 tasks), Qwen-UI-Agent reaches 92.2% success, beating Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 6.0, 7.5, 6.8, and 3.5 percentage points respectively
• It reaches 97.5% on the real-device AndroidDaily benchmark and 82.1% on the simulated MobileWorld benchmark, surpassing Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 14.6, 12.0, and 8.9 points there
• On OSWorld-Verified (computer use) it scores 79.5%, ahead of GPT-5.5, Gemini 3.1 Pro, and Seed 2.1 Pro; on the harder OSWorld-v2 it gets a 40.0% partial-progress score, exceeding MiniMax M3 and Qwen 3.7 Plus by 17.7 and 18.5 points while using 58.4% and 21.7% fewer steps
• On WebArena (browser use) it scores 73.6%, ahead of Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro by 1.7, 4.1, and 8.3 points, and on DeepSearch it scores 64.1% (BrowseComp) and 75.0% (BrowseComp-ZH), surpassing Qwen3.5-397B-A17B and UI-TARS-2
• Analysis of OSWorld trajectories shows CLI actions make up 40.7% to 55.1% of all actions, batched actions make up 39.6% to 41.6%, and batches average 3.1 primitive actions each

Where it can be used

• Designing proactive mobile assistants that detect notifications or state changes and propose alternatives for user approval instead of waiting for instructions
• Building desktop automation that combines CLI execution with GUI interaction for file processing or structured data tasks where pure GUI clicking is inefficient
• Designing cross-platform workflows that gather information across multiple apps and hand off results to another device such as a computer
• Referencing real-device failure patterns like pop-ups, placeholder text misreading, and slider control when designing training data for GUI agents

Limits and open work

• Reported real-device results cover only two benchmarks, MobileWorld-Real and AndroidDaily, so generalization to other languages or device ecosystems is not established
• The detailed failure analysis targets trajectories from a comparison model, Qwen 3.7 Plus, rather than presenting a full breakdown of Qwen-UI-Agent's own remaining failure modes
• On the harder OSWorld-v2 setting, Qwen-UI-Agent ranks second on binary success rate (13.9%) and third on partial-progress score, so it is not the top performer under the most difficult condition
• The proactive service harness's long-term self-improvement from user responses is illustrated through example scenarios rather than quantitative evaluation
• Comparisons are against specific frontier models such as Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol at the time of writing, so relative standing may shift as those models are updated

Why it matters

GUI agents typically look strong on simulated benchmarks but often fail on physical devices, and this report directly targets that gap by building and testing on real phones and desktops. The reported numbers and system structure give practitioners a reference point for designing real-device automation, cross-device workflows, and notification-triggered proactive assistants.

Terms in this paper

• GUI agent · an AI that operates apps on behalf of a user by perceiving the screen and performing actions like taps and typing
• CLI · command-line interface, executing tasks via typed commands instead of screen interaction
• batched action · issuing several actions together in one decision step instead of one action per model turn
• online reinforcement learning · training where the model keeps interacting with an environment and updates based on task success signals
• harness layer · a supporting layer that connects the core agent to real-world context so it can start services proactively

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3