UI-Venus-2 Technical Report

arXiv:2609.000282026-09-02

UI-Venus-2用一个模型同时操作手机、网页和桌面系统,把环境、任务和验证一起做大

UI-Venus-2是一个GUI智能体,通过观察屏幕并执行点击、输入、滚动等操作来自动完成任务,这次把覆盖范围扩展到170多个手机应用以及原生桌面操作系统。它用深度研究式流程自动生成任务指令,并用轨迹级和样本级验证器配合多模型投票,为强化学习提供更可靠的奖励信号。团队在多个GUI基准上把UI-Venus-2的27B和9B版本与挑选出的强基线模型进行了对比评测。

METAL LAB 解读图

UI-Venus-2三阶段训练流程

证据状态实测结果与计划中的工作并存

中期训练用在手机、网页、操作系统环境中收集的大规模交互轨迹先注入GUI操作知识。
分领域离线强化学习对Grounding、CAPTCHA、Mobile、Web、Computer各领域分别做基于步骤的强化学习优化。
多教师在策略蒸馏(MOPD)把各领域专门化的模型融合成一个UI-Venus-2模型,并把监督信号集中在可执行动作token上。
轨迹级与样本级验证用视觉关键点和多模型投票分别检验整条轨迹和单个动作,为强化学习提供可靠的奖励信号。

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

智能体在手机、网页、桌面三种环境中都遵循同一个闭环推理-行动框架:观察界面、判断任务状态、执行动作、再根据环境反馈调整下一步。
同时扩大三个维度:环境(170多个多语言手机应用加桌面操作系统)、任务(基于深度研究的指令生成)、验证(轨迹级和样本级评估器)。
训练分三个阶段:大规模轨迹式中期训练、按领域分别做的离线强化学习、以及把各领域专家模型融合成一个模型的多教师在策略蒸馏(MOPD)。
MOPD阶段对推理文字和可执行动作区别对待,把蒸馏信号集中在真正改变环境状态的动作token上。
系统还加入了处理登录、注册环节验证码(CAPTCHA)的能力,以及控制潜在高风险操作的安全机制。

Figure 1: Performances of UI-Venus-2 on GUI-agent benchmarks. Each panel compares UI-Venus-2-27B and UI-Venus-2-9B with some selected strong baselines. We favor standalone end-to-end systems evaluated on the closest available task subset and step budget; source-reported action scaffolds may still differ. MobileWorld uses GUI-only success rate on 117 tasks with 50 steps, WebVoyager uses the refreshed 595-task split, Odysseys uses average rubric score over 200 tasks, VenusBench-CAPTCHA uses micro Pass@1 over all 219 examples, and VenusBench-GD uses English-instruction micro-average accuracy. “*” denotes the results are reproduced by us.

Table 1: Performance comparison on various mobile GUI benchmarks. VenusBench-Mobile reports success rate on its 149-task primary pool. For MobileWorld, we report GUI-only success rate on 117 tasks under the 50-step setting; values in parentheses, when available, use 100 steps. MemGUI reports Main Results pass@1. “*” denotes the baseline results evaluated or reproduced by us.
Models	MobileGym	VenusBench-Mobile	AndroidWorld	MobileWorld	KnowUBench	MemGUI
General VLMs
Qwen3.5-9B (Qwen Team, 2026a)	9.0*	15.3*	57.8	18.0(18.0)*	33.3	6.2*
Qwen3.6-27B (Qwen Team, 2026b)	24.6*	28.0*	70.3	36.8(41.9)*	-	25.7*
Claude-Opus-4.6 (Anthropic, 2026a)	-	36.5*	-	44.5	-	-
Kimi-K2.6 (Moonshot AI, 2026a)	38.7*	31.2*	-	55.6	-	39.1
Kimi-K3 (Moonshot AI, 2026b)	-	-	-	74.4	-	-
Seed-2.0-Pro (Seed, 2026)	52.0	20.1*	-	63.2	51.6	65.6*
Seed-2.1-Pro (ByteDance Seed, 2026)	-	-	-	73.2	-	-
GPT-5.6-Sol (OpenAI, 2026)	-	-	-	70.1	-	-
GUI-specific Models
UI-Venus-1.5-8B (Team et al., 2026c)	18.4*	16.1	73.7	22.2*	26.0	3.9*
UI-Venus-1.5-30B-A3B (Team et al., 2026c)	21.5*	21.5	77.6	17.1	-	10.9*
GUI-Owl-1.5-32B-Instruct (Xu et al., 2026)	20.3*	-	69.8	43.9	-	10.9
MAI-UI-8B (Zhou et al., 2025b)	21.5*	12.7	70.7	27.5	26.0	17.2*
Qwen-UI-Agent-27B (Zhou et al., 2026)	-	-	-	82.1(85.5)	-	-
Ours
UI-Venus-2-9B	52.7	46.5	80.2	65.8(75.2)	56.5	62.6
UI-Venus-2-27B	60.5	48.7	84.0	76.1(82.9)	59.7	70.3

Figure 2: System Overview of UI-Venus-2. The figure illustrates the task generation and trajectory collection process of UI-Venus-2. Diverse tasks are constructed to form a multi-domain task pool, and interaction trajectories are collected across mobile, browser, and computer environments, covering a broad range of real-world applications, websites, and desktop software.

Table 2: Performance comparison on computer-use agent benchmarks: OSWorld-Verified and DeskCraft (left), and OSWorld 2.0 (right) under the official 150-step budget with 108 tasks. Reported OSWorld-Verified baselines use the 361-task setting in their cited source and may use model-specific action scaffolds. For DeskCraft, we report an author-evaluated aggregate over the 538-task union of the Standard and Interactive splits, which differs from the benchmark’s official split-level reporting. OSWorld 2.0 results report the official Binary Accuracy and Partial Score metrics; baselines are taken from the official leaderboard, possibly with model-specific tool settings, and the reasoning-effort setting is labeled in parentheses for models with multiple official entries. “*” indicates baseline results evaluated by us.
Models	OSWorld-Verified	DeskCraft
General VLMs
Claude-Opus-4.8 (Zhou et al., 2026)	83.4	-
Qwen3.5-9B (Qwen Team, 2026a)	41.8	14.6∗
Qwen3.6-27B (Qwen Team, 2026b)	62.0	28.7∗
Kimi-K2.6 (Moonshot AI, 2026a)	73.1	41.4∗
Seed-2.0-Pro (Seed, 2026)	62.3	40.0∗
Seed-2.1-Pro (Zhou et al., 2026)	78.8	-
GPT-5.5 (Zhou et al., 2026)	78.7	-
GUI-specific Models
GUI-Owl-1.5-32B-Instruct (Xu et al., 2026)	56.5	-
Qwen-UI-Agent-27B (Zhou et al., 2026)	79.5	-
Ours
UI-Venus-2-9B	70.8	48.0
UI-Venus-2-27B	80.5	55.5

Figure 3: The Three-Stage Pipeline of UI-Venus-2. Following the overall training recipe of UI-Venus-1.5, UI-Venus-2 starts with large-scale trajectory-based mid-training to inject GUI interaction knowledge. The resulting model is then optimized independently for each domain using step-level Offline-RL, covering Grounding, CAPTCHA, Mobile, Web, and Computer tasks. Finally, the domain-specialized models are consolidated into the final UI-Venus-2 model via multi-teacher on-policy distillation.

Table 3: Performance comparison on four live-web benchmarks: WebVoyager, Online-Mind2Web, REAL, and Odysseys. The Fara1.5 and GPT-5 (SoM) WebVoyager entries use the refreshed 595-task, 100-step robust protocol and are averaged over three runs; live-site states may vary by evaluation date. For Odysseys, we report both the averaged rubric score (Avg.) and the perfect rubric score (Perfect). "*" indicates our reproduced results.
Models	WebVoyager	Online-Mind2Web	REAL	Odysseys
Avg.	Perfect
General VLMs
Qwen3.5-9B (Qwen Team, 2026a)	46.9∗	27.3∗	18.2∗	42.6∗	13.5∗
Qwen3.5-4B (Qwen Team, 2026a)	–	–	–	42.9	10.7
Qwen3.6-27B (Qwen Team, 2026b)	84.3∗	55.3∗	27.3∗	39.5∗	18.5∗
OpenAI Operator (OpenAI, 2025)	87.0	61.3	–	–	–
GPT-5 (SoM) (Awadallah et al., 2026)	90.6	–	–	–	–
GPT-5.4 (Singh et al., 2025)	–	–	–	55.4	33.5
Seed-2.0-Pro (Seed, 2026)	85.1∗	68.5∗	74.4∗	60.2∗	30.1∗
GLM-5V-Turbo (Hong et al., 2026)	88.5	–	–	–	–
Claude-Opus-4.6 (Anthropic, 2026a)	88.0	–	–	68.9	44.5
Claude-Sonnet-4.6 (Anthropic, 2026b)	–	–	–	49.8	31.0
Kimi-K2.6 (Moonshot AI, 2026a)	76.8∗	–	74.4∗	–	–
GUI-specific Models
UI-TARS-1.5 (Seed, 2025b)	84.8	75.8	–	–	–
UI-Venus-1.5-30B-A3B (Team et al., 2026c)	76.0	–	38.0∗	–	–
GUI-Owl-1.5-32B-Thinking (Xu et al., 2026)	82.1	–	44.6∗	–	–
MolmoWeb-8B (Gupta et al., 2026)	78.2	35.3	–	–	–
Fara1.5-4B (Awadallah et al., 2026)	80.8	–	–	–	–
Fara1.5-9B (Awadallah et al., 2026)	86.6	63.4	–	–	–
Fara1.5-27B (Awadallah et al., 2026)	89.3	72.3	–	–	–
Ours
UI-Venus-2-9B	90.8	74.0	76.9	77.3	62.0
UI-Venus-2-27B	93.4	78.3	80.2	80.4	66.3

Figure 4: Examples of synthesized GUI Grounding training data. Our pipeline generates diverse and realistic interface screenshots spanning desktop (macOS, Windows), mobile (iOS), and web platforms, covering both professional software and consumer applications. All interfaces are rendered in a real headless Chromium browser via Playwright, ensuring high-fidelity visual output that closely mirrors authentic user environments.

Table 4: Performance comparison on various Grounding Benchmarks. VenusBench-GD reports English-instruction micro-average point-in-box accuracy. “*” indicates baselines evaluated or reproduced by us.
Models	Grounding Benchmarks
VenusBench-GD	ScreenSpot-Pro	OSworld-G-R	UI-Vision
General VLMs
Qwen 3.7 Plus (Qwen Team, 2026c)	75.2*	68.9	78.2	68.0
Seed 2.1 Pro (ByteDance Seed, 2026)	73.9*	65.3	78.0	62.0
Kimi-K2.6 (Moonshot AI, 2026a)	73.1*	52.0*	69.7*	51.7*
Qwen3.6-27B (Qwen Team, 2026b)	67.7*	65.2*	76.9*	58.3*
GUI-specific Models
UI-Venus-Ground-72B (Gu et al., 2025)	70.2	61.9	69.5	36.8
Holo2-30B-A3B (H-Company, 2025)	59.5*	66.1	76.1	40.9*
Step-GUI-4B (Yan et al., 2025)	54.6*	60.0	66.9	30.0*
MAI-UI-8B (Zhou et al., 2025b)	65.2*	65.8	68.6	40.7
MAI-UI-32B (Zhou et al., 2025b)	-	67.9	73.9	47.1
UI-Venus-1.5-30B-A3B (Team et al., 2026c)	75.0	69.6	76.4	54.7
Qwen-UI-Agent-27B (Zhou et al., 2026)	-	76.6	78.5	70.0
Ours
UI-Venus-2-9B	77.1	73.0	78.5	53.2
UI-Venus-2-27B	80.1	74.1	79.1	66.9

Figure 5: Overview of VenusBench-CAPTCHA. Each panel shows one complete, uncropped representative screenshot. The OCR label gives the target transcription, numbered boxes indicate the required click order, and arrows visualize annotated drag trajectories. These annotations are added for presentation only and are not part of the model input.

Table 5: Performance Comparison across CAPTCHA Benchmarks. All results are Pass@1 percentages, and higher is better. We evaluate on VenusBench-CAPTCHA and four public benchmarks: MCA-Bench Wu et al. (2026b), Spatial-CAPTCHA-Bench Kharlamova et al. (2026), NextGen-CAPTCHAs Liu et al. (2026b), and Open CaptchaWorld Luo et al. (2025b). We use 1,000 sampled MCA-Bench examples, 15 NextGen-CAPTCHAs task types, and 16 Open CaptchaWorld task types; see the appendix for selection details.
Models	VenusBench- CAPTCHA	MCA-Bench	Spatial- CAPTCHA-Bench	NextGen- CAPTCHAs	Open CaptchaWorld
General VLMs
Qwen3.5-9B (Qwen Team, 2026a)	28.3	30.4	4.9	2.8	36.4
Qwen3.6-27B (Qwen Team, 2026b)	53.0	51.7	31.0	14.1	47.7
Seed-2.0-Pro (Seed, 2026)	47.9	36.5	43.8	20.4	55.6
Kimi-K2.6 (Moonshot AI, 2026a)	39.7	38.7	24.8	7.2	47.8
Claude-Opus-4.6 (Anthropic, 2026a)	16.0	25.9	9.5	2.8	23.3
Ours
UI-Venus-2-9B	78.1	75.7	42.8	47.6	50.7
UI-Venus-2-27B	79.9	79.6	48.6	54.5	56.3

Table 6: Safety evaluation on OSHARM and OS-BLIND benchmarks. ASR denotes Attack Success Rate (%, lower is better).
Model	OSHarm (ASR ↓)	OSBlind (ASR ↓)
General VLMs
Qwen3.5-9B (Qwen Team, 2026a)	25.3	79.4
Qwen3.5-27B (Qwen Team, 2026a)	18.0	89.3
Kimi K2.6 (Moonshot AI, 2026a)	32.0	93.6
GUI-specific Models
EvoCUA-8B (Xue et al., 2026)	39.3	85.3
EvoCUA-32B (Xue et al., 2026)	33.3	90.7
UI-TARS-1.5 (Seed, 2025b)	36.0	83.3
ScaleCUA (Lv et al., 2026)	25.3	84.7
Ours
UI-Venus-2-9B	11.3	48.8
UI-Venus-2-27B	15.3	47.9

Table 7: All actions and their definitions used in UI-Venus-2. We unify the action space and map all the actions in the existing open-source dataset to this space.
Action	Definition
Shared Actions (All Platforms)
Click(point=(x, y))	Click at coordinates (x, y).
Drag(start=(x1, y1), end=(x2, y2))	Drag from (x1, y1) to (x2, y2).
Swipe(start=(x1, y1), end=(x2, y2))	Scroll by swiping from (x1, y1) to (x2, y2).
DoubleClick(point=(x, y))	Perform a double-click/tap at coordinates (x, y).
LongPress(point=(x, y))	Long press at coordinates (x, y) to trigger extra options.
Type(content=”)	Type the specified content.
Wait()	Wait for loading.
CallUser(content=”)	Request user takeover or additional information.
Finished(content=”)	Mark the task as completed, with optional information.
Mobile†† Coordinates are normalized to [0,999].
PressBack()	Press the ‘back’ button.
PressHome()	Press the ‘home’ button.
PressEnter()	Press the ‘enter’ button.
PressRecent()	Press the ‘recent’ button.
LaunchApp(app=”)	Launch the specified app.
GetScreenshot()	Take a screenshot and save it to the photo album.
Answer(content=”)	Answer the user’s questions as requested.
Desktop†† Coordinates are normalized to [0,999].
RightClick(box=(x, y))	Right-click at (x, y) to open context menus.
Hotkey(keys=[‘ctrl’, ‘c’])	Press a keyboard shortcut, e.g., Ctrl+C for copy.
Web †† Coordinates are normalized to [0,999].
Scroll(point=(x, y), direction=‘up/down/left/right’)	Scroll at (x, y) in the specified direction.
Launch(url=”)	Launch the target URL.
GetUrl()	Get the URL of the current browser tab.
TakeNote(content=”)	Record important information from the screenshot.
Hover(point=(x, y))	Move the mouse cursor to coordinates (x, y) without clicking.
Hotkey(keys=(‘ctrl’, ‘c’))	Press combination keys (up to 3).
SelectOption(index=3)	Choose an option from a native HTML select element.
PressBack()	Return to the previous page.

研究结果

Figure 1和Table 1至5报告了UI-Venus-2-27B与9B在MobileWorld、WebVoyager、Odysseys、VenusBench-CAPTCHA、VenusBench-GD等多个基准上与挑选出的强基线模型的对比结果。
VenusBench-CAPTCHA在219个样本、八种交互类型(OCR文字输入、顺序点击、图像旋转、拖拽、滑块拼图等)上以Pass@1指标进行了评测。
Table 6在OSHARM和OS-BLIND两个基准上以攻击成功率(ASR,数值越低越安全)报告了安全性评测结果。
论文报告称,MOPD阶段按动作类型条件化的蒸馏信号设计,在多个GUI领域上带来了更稳健的融合效果。

可应用场景

尝试用屏幕操作方式(而非API)自动化登录、填表等重复性手机/网页任务
构建需要处理验证码(CAPTCHA)环节的自动化工作流
参考其桌面操作系统能力(文件管理、办公软件操作等)的通用智能体设计思路
参考其轨迹级/样本级、多模型投票的强化学习奖励验证机制设计

局限与待验证事项

论文自己说明,不同基线的对比可能使用了各自来源报告的不同动作脚手架或评测协议,并非完全同条件比较。
部分基准如OSWorld-Verified,评测所用任务数量(108个)与该基准官方设置(361个)不同,限制了直接可比性。
安全机制仅在OSHARM和OS-BLIND两个基准上得到验证,未涉及更广泛真实风险场景的检验。
多语言手机应用覆盖仅限于100多个中文应用和70多个英文应用,其他语言/地区应用上的表现未经验证。
论文明确指出,实时网页基准的结果可能因评测日期不同而随网站实际状态变化。

为什么重要

GUI智能体常常只在特定基准测试中表现好,一到真实的应用、网站和桌面系统就变得脆弱,这是自动化工具走向实际使用的一大障碍。这项工作把环境覆盖、任务生成和验证可靠性一起扩大,为缩小这一差距提供了一种可参考的做法。

本文术语

GUI智能体 · 通过观察屏幕并模仿人类点击、输入等动作来操作软件的AI
离线强化学习 · 只用已经收集好的交互记录来训练,而不是实时探索环境的强化学习方式
多教师在策略蒸馏(MOPD) · 让多个教师模型对学生模型自己生成的动作打分,再据此训练学生模型的方法
轨迹 · 智能体完成一个任务过程中记录下来的全部屏幕画面和动作序列
VLM as a Judge · 用视觉语言模型充当裁判,给其他模型的输出结果打分

论文原文摘要（英文）

Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.

作者 · Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jian

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Venus Team et al., arXiv:2609.00028, arxiv-nonexclusive