Figure 1: Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings.
Table 1: Action Space in Qwen-UI-Agent.
Action
Definition
GUI Actions
click
Clicks at coordinates (x,y).
double_click
Double-clicks at coordinates (x,y).
long_press
Long-presses at coordinates (x,y).
type
Types the specified text content.
open
Opens the specified app.
drag
Drags from start coordinates (x1,y1) to end coordinates (x2,y2).
system_button
Presses a system button, selected from back, home, menu, and enter.
wait
Waits for a specified duration in seconds.
CLI Actions
cli_command
Executes a bash command in the active CLI environment.
API Actions
api_call
Invokes an API with specified arguments.
Interaction and Control Actions
ask_user
Interact with the user to complete the task.
terminate
Ends the task with a final answer and a status of success or failed.
Figure 2: An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left panel summarizes the system capabilities underlying the trajectory, including cross-platform execution, a hybrid action space spanning GUI, CLI, and API operations, and environments ranging from large-scale sandboxes to real devices. The right panel illustrates a travel-recovery scenario triggered by a flight-cancellation notification. After identifying the affected tasks and commitments, the agent searches for alternative flights through an API, requests user approval before rebooking via a mobile GUI, updates the affected meeting schedule through desktop GUI and CLI actions, and sends the revised artifact to the relevant recipients.
Table 2: Performance comparison on the GUI-only subset of MobileWorld (Kong et al. 2026), consisting of 117 tasks. We include representative general-purpose VLMs and specialized GUI models. The best result is highlighted in bold, and the second-best result is underlined.
Model
Access / Size
Success Rate (%)
General-purpose VLMs
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
73.2
GPT-5.6 Sol (OpenAI 2026b)
Closed-source
70.1
Claude Opus 4.8 (Anthropic 2026)
Closed-source
67.5
Seed 2.0 Pro (ByteDance Seed 2026a)
Closed-source
63.2
Qwen 3.7 Plus (Qwen Team 2026a)
397B-A17B
62.3
Gemini 3.1 Pro (Google 2026a)
Closed-source
58.1
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
55.6
Specialized GUI Models
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
43.9
MAI-UI-235B-A22B (Zhou et al. 2025a)
235B-A22B
39.7
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
30B-A3B
17.1
Ours
Qwen-UI-Agent
27B
82.1
Qwen-UI-Agent
35B-A3B
65.0
Figure 3: The environment infrastructure of Qwen-UI-Agent. (a) Scalable sandbox environments spanning mobile-use, computer-use, browser-use, and DeepSearch provide controllability and repeatable evaluation for data synthesis and training. (b) A sim-to-real bridge extends agents to real devices with real applications, networks, and account states, supporting user takeover for login, payment, permission, and confirmation. (c) A hybrid GUI+CLI action space interleaves graphical operations with direct command execution. (d) A unified interface standardizes the thinking–action–observation loop across heterogeneous environments.
Table 3: Performance comparison on real-device mobile benchmarks. MobileWorld-Real is our proposed Chinese real-device mobile GUI benchmark (Section 3.2), and AndroidDaily (Sui et al. 2026) covers high-frequency daily scenarios on real Android devices.
Model
Access / Size
MobileWorld-Real
AndroidDaily
Baselines
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
88.7
95.2
Gemini 3.1 Pro (Google 2026a)
Closed-source
86.2
93.8
GPT-5.6 Sol (OpenAI 2026b)
Closed-source
85.4
92.6
Claude Opus 4.8 (Anthropic 2026)
Closed-source
84.7
93.0
Qwen 3.7 Plus (Qwen Team 2026a)
Closed-source
72.7
79.8
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
62.6
67.6
PhoneBuddy-4B (Tang et al. 2026)
4B
53.5
69.0
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
30B-A3B
33.0
61.7
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
32.4
60.9
GELab-Zero-4B-preview (Yan et al. 2025)
4B
31.3
73.4
Ours
Qwen-UI-Agent
27B
92.2
97.5
Qwen-UI-Agent
35B-A3B
87.4
93.9
Figure 4: Real-device mobile runtime with closed-loop environment governance. The health-aware scheduler routes each task to an eligible phone, App/account, and display; unhealthy targets remain blacklisted until repair and revalidation. Virtual displays allow one phone to run multiple Apps concurrently. Evidence-based review separates task success, model failure, and environment failure from the complete trajectory, and confirmed environment issues are fed back to the scheduler.
Table 4: Performance comparison on OSWorld-Verified (Xie et al. 2024).
Model
Access / Size
Success Rate (%)
Baselines
Claude Opus 4.8 (Anthropic 2026)
Closed-source
83.4
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
78.8
GPT-5.5 (OpenAI 2026a)
Closed-source
78.7
Gemini 3.5 Flash (Google 2026b)
Closed-source
78.4
Gemini 3.1 Pro (Google 2026a)
Closed-source
76.2
Qwen 3.7 Plus (Qwen Team 2026a)
Closed-source
73.3
MiniMax M3 (MiniMax 2026)
428B-A23B
75.2
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
73.1
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
56.5
Ours
Qwen-UI-Agent
27B
79.5
Figure 5: The data flywheel of Qwen-UI-Agent. Domain capability bootstrapping initializes training, iterative refinement loop identifies capability weaknesses and generate targeted tasks, and the resulting data improve the next training iteration.
Table 6: Performance comparison on WebArena (Zhou et al. 2024). Baseline results marked with ∗ are obtained from our own evaluation under the same experimental setup.
Model
Access / Size
Success Rate (%)
Baselines
Claude Opus 4.8∗ (Anthropic 2026)
Closed-source
71.9
GPT-5.5∗ (OpenAI 2026a)
Closed-source
69.5
Gemini 3.1 Pro∗ (DeepMind 2025)
Closed-source
65.3
Qwen 3.7 Plus∗ (Qwen Team 2026a)
Closed-source
59.0
CUA-GYM-A17B (Wang et al. 2026)
397B-A17B
56.0
Kimi K2.6∗ (Moonshot AI 2026)
1T-A32B
55.8
Qwen3.5-397B-A17B (Qwen Team 2026b)
397B-A17B
54.0
GUI-Owl-1.5-32B-Thinking (Xu et al. 2026)
32B
48.4
Qwen3.5-27B (Qwen Team 2026b)
27B
41.5
Qwen3.5-35B-A3B (Qwen Team 2026b)
35B-A3B
40.8
Ours
Qwen-UI-Agent
27B
73.6
Qwen-UI-Agent
35A3B
69.2
Human Performance (Zhou et al. 2024)
–
78.2
Figure 6: Overview of our harness for proactive service initiation and cross-platform execution. (I) Without the harness, users must manually interpret notifications, retrieve related context, and provide instructions for the agent. The proactive harness instead detects a flight cancellation, reasons over relevant affairs, proactively implement flight rebooking, railway alternatives, and meeting-conflict checks, and report an actionable plan. (II) The harness also maintains task state and support cross-platform execution. In the restaurant-selection example, Qwen-UI-Agent discovers candidates on mobile, organizes them in a desktop spreadsheet, sends the artifact for approval, and saves the selected locations back to mobile.
Table 7: Performance comparison on DeepSearch benchmarks: BrowseComp (BC) and BrowseComp-ZH (BC-ZH). “–” indicates the result is not reported.
Model
Access / Size
BC (%)
BC-ZH (%)
Baselines
GPT-5.5 (OpenAI 2026a)
Closed-source
90.1
–
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
86.2
–
Gemini 3.1 Pro (Google 2026a)
Closed-source
85.9
–
Claude Opus 4.8 (Anthropic 2026)
Closed-source
84.3
–
UI-TARS-2 (Wang et al. 2025a)
Closed-source
29.6
50.5
Qwen3.5-397B-A17B (Qwen Team 2026b)
397B-A17B
78.6
70.3
Apodex-1.0-mini (Apodex Team 2026)
35B-A3B
71.5
80.6
Qwen3.5-27B (Qwen Team 2026b)
27B
61.0
62.1
GLM-4.7 (Z.ai 2025)
358B
52.0
66.6
DeepSeek-V3.2 (DeepSeek-AI 2025)
685B
51.4
65.0
Tongyi-DR-30B (Tongyi DeepResearch 2025)
30B-A3B
43.4
46.7
Ours
Qwen-UI-Agent
27B
64.1
75.0
Figure 7: Overview of MobileWorld-Real, a real-device benchmark with human-written tasks that reflect the breadth of everyday mobile use. Representative examples and aggregate statistics show broad domain coverage and a long-tailed App distribution. Matched-model results further show lower success rates and longer trajectories than on AndroidDaily, highlighting the challenge of real-world mobile interaction.
Table 8: Performance comparison on GUI grounding benchmarks. ScreenSpot-Pro reports no-zoom results, with zoom-in results shown in parentheses when available. Baseline results marked with ∗ are obtained from our own evaluation.
Models
Grounding Benchmarks
SS-Pro(zoom in)
SS-V2
MM-GUI-L2
OSW-G-R
UI-Vision
Baselines
Qwen 3.7 Plus* (Qwen Team 2026a)
68.9 (79.0)
96.6
90.5
78.2
68.0
Seed 2.1 Pro* (ByteDance Seed 2026b)
65.3 (80.7)
96.6
90.9
78.0
62.0
Qwen3.5-4B* (Qwen Team 2026b)
59.1 (60.3)
94.3
83.4
62.0
32.6
Qwen3.5-35B-A3B* (Qwen Team 2026b)
64.5 (68.6)
95.2
87.5
63.6
52.3
Qwen3.5-27B* (Qwen Team 2026b)
68.4 (70.3)
96.1
89.1
67.9
46.6
GUI-Owl-1.5-2B-Instruct (Xu et al. 2026)
57.8 (70.4)
89.7
72.1
62.6
–
GUI-Owl-1.5-4B-Instruct (Xu et al. 2026)
66.8 (75.6)
93.2
83.2
68.4
–
GUI-Owl-1.5-8B-Instruct (Xu et al. 2026)
71.1 (77.8)
93.7
82.5
69.3
–
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
72.9 (80.3)
95.3
86.8
69.7
–
UI-Venus-1.5-2B (Gao et al. 2026)
57.7 (64.6)
92.8
80.3
59.4
44.8
UI-Venus-1.5-8B (Gao et al. 2026)
68.4 (73.9)
95.9
88.1
69.7
46.5
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
69.6 (74.8)
96.2
88.6
70.6
54.7
ZoomOnce-4B (Liu et al. 2026)
66.2
95.2
87.6
73.1
40.2
HyMobileAgent-A3B (Shen et al. 2026)
66.5
96.2
89.3
–
–
MAI-UI-2B (Zhou et al. 2025a)
57.4 (62.8)
92.5
82.6
63.5
30.3
MAI-UI-8B (Zhou et al. 2025a)
65.8 (70.9)
95.2
88.8
72.9
40.7
MAI-UI-32B (Zhou et al. 2025a)
67.9 (73.5)
96.5
91.3
75.0
47.1
GTA1-7B (Yang et al. 2025a)
50.1
92.4
78.5
67.7
–
GTA1-32B (Yang et al. 2025a)
63.6
95.2
83.4
72.2
–
UI-Ins-7B (Chen et al. 2025)
52.2
94.0
83.1
-
-
UI-Ins-32B (Chen et al. 2025)
57.0
94.9
84.9
-
-
OpenCUA-7B (Wang et al. 2025b)
50.0
92.3
–
–
29.7
OpenCUA-32B (Wang et al. 2025b)
55.3
93.4
–
–
33.3
OpenCUA-72B (Wang et al. 2025b)
60.8
92.9
–
–
37.3
Ours
Qwen-UI-Agent-4B
67.8 (74.0)
94.9
87.9
70.5
51.6
Qwen-UI-Agent-35B-A3B
76.1 (80.2)
96.7
92.0
74.6
65.9
Qwen-UI-Agent-27B
76.6 (81.5)
97.5
92.6
78.5
70.0
Figure 8: Demonstration of real-device mobile GUI execution. The trajectory is rendered as key frames with the executed action annotated beneath each frame. In this cross-app task, the agent looks up the target address on Amap, finds the most popular café nearby on Dianping, and posts a summary of the findings on RedNote.
Table 10: Failure-pattern distribution over all failed Qwen 3.7 Plus trajectories on real devices.
Dimension
Failure Pattern
Prop.
Typical Behavior
Execution Capability Limitations (40.3%)
Exploration Failure
19.5%
Fail to locate deep in-app entries
Erroneous Action Loops
14.3%
Repeat ineffective actions
Lost Execution State
6.5%
Forget finished sub-tasks
Real-world Scenario Challenges (52.0%)
UI Misreading
24.7%
Misread stateful page semantics
Pop-up Interference
18.2%
Ads, paywalls, CAPTCHAs, blank pages
Physical Widget Control
9.1%
Overshoot targets, never converge
Others
–
7.7%
Under-execution, premature stop
Figure 9: Demonstration of hybrid GUI+CLI execution in computer-use tasks. Key frames are shown with the emitted actions, where CLI commands are highlighted in blue, GUI actions in red, and each step carries a short annotation of its intent. In this example, the agent selects the target photo and filters by stitching all candidates into a single montage via CLI for one-shot visual inspection, rather than opening each image individually. It then leverages hybrid execution to combine CLI-based processing with GUI-based verification for the final output.
Table 11: GUI+CLI usage and batched-execution statistics on OSWorld-Verified and OSWorld-v2. Panel (a) reports CLI and batched-action usage at the action and task levels. Panel (b) reports the composition of batched outputs. Mean batch size counts the number of primitive actions in a batch. Differences are computed as OSWorld-v2 minus OSWorld-Verified. Differences are reported in percentage points (pp).
Statistic
Level
OSWorld-Verified
OSWorld-v2
Difference (pp)
(a) Overall CLI and batched-action usage
CLI
Action
40.7%
55.1%
+14.4
CLI
Task
92.0%
98.2%
+6.2
Batched
Action
39.6%
41.6%
+2.0
Batched
Task
62.1%
88.9%
+26.8
(b) Composition of batched outputs
GUI-only batches
75.8%
64.7%
-11.1
CLI-only batches
13.1%
15.0%
+1.9
Mixed GUI+CLI batches
11.0%
20.3%
+9.3
Mean primitive actions per batch
3.1
3.1
0.0
Figure 10: Demonstration of DeepSearch-assisted GUI execution. The upper panels show the multi-round search process, including keyword-based DeepSearch, targeted web fetches with intermediate thoughts, and the final search summary; the lower row shows the subsequent GUI trajectory. DeepSearch resolves the knowledge and reasoning problem before GUI execution, converting the vague cross-source request into an explicit target: Qwen-UI-Agent identifies the largest comeback in the World Cup knockout stage through DeepSearch, then opens RedNote and navigates directly to the highest-engagement related post from the past week without blind in-app searching.
Table 13: Performance comparison between frequent and long-tail actions.
Action Group
Original Data Proportion
Reward before Action RL
Reward after Action RL
Frequent actions
80.1%
88.3%
92.3%
Long-tail actions
19.9%
71.5%
77.9%
Figure 11: Demonstration of proactive service based on mobile notifications. The trajectory is organized into highlighted stages: proactive task initiation from a detected notification, proactive execution phases, and a decision-ready result, with the executed action annotated beneath each key frame. Flight-cancellation recovery: when the user’s next-morning flight is canceled and a 14:00 demo is at risk, the harness proactively searches alternative flights and high-speed trains, evaluates which options arrive on time, and presents a decision-ready travel recovery plan.
研究结果
在MobileWorld-Real(超过400个任务的真机基准)上,Qwen-UI-Agent达到92.2%的成功率,分别领先Gemini 3.1 Pro、Claude Opus 4.8、GPT-5.6 Sol和Seed 2.1 Pro 6.0、7.5、6.8和3.5个百分点
在真机基准AndroidDaily上达到97.5%,在模拟基准MobileWorld上达到82.1%,领先Opus 4.8、GPT-5.6 Sol和Seed 2.1 Pro 14.6、12.0和8.9个百分点
在OSWorld-Verified(电脑使用)上取得79.5%,领先GPT-5.5、Gemini 3.1 Pro和Seed 2.1 Pro;在更难的OSWorld-v2上部分进度得分为40.0%,分别领先MiniMax M3和Qwen 3.7 Plus 17.7和18.5个百分点,且所需步数分别减少58.4%和21.7%
在WebArena(浏览器使用)上取得73.6%,领先Claude Opus 4.8、GPT-5.5和Gemini 3.1 Pro 1.7、4.1和8.3个百分点;在深度搜索基准上BrowseComp得分64.1%、BrowseComp-ZH得分75.0%,超过Qwen3.5-397B-A17B和UI-TARS-2
Figure 12: Demonstration of cross-platform task execution. In this workflow, mobile subtasks run on virtual screens of the physical device, so execution does not block the user’s own actions. Parallel multi-app search: the agent searches sushi restaurants on Dianping, Meituan, and Amap concurrently through multiple virtual screens, consolidates their ratings, and summarizes the top three options in a local report opened on the computer.
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI ag
