METAL LAB

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

arXiv:2609.000482026-09-02

GUI世界模型能把单张画面做得很逼真,但连续走几步之后经常记不住自己到底在哪个App里

GUI-CC是一个基准测试,把GUI世界模型(预测下一个界面、并被反复循环利用的模型)当作多步骤的智能体环境来考核,而不只是考核单步预测。它包含一个沿500条真实手机轨迹回放动作的离线赛道,以及让固定智能体直接和模型生成画面交互的在线赛道(30个App、200个任务)。实验发现,单步画面看起来合理,并不代表多步连续生成后环境依然可靠,任务进度普遍很低。

METAL LAB 解读图

GUI-CC 双赛道结构

证据状态已报告实测结果

  1. 问题设定GUI世界模型通常按单张画面预测器打分,但实际用途是被反复重用作为多步骤环境的输入。
  2. 离线参考动作赛道沿着GUIOdyssey中500条真实轨迹的固定动作顺序,让模型自回归地不断生成下一步画面,检验可执行性。
  3. 在线智能体循环赛道固定的GPT-5.5智能体直接和模型生成的画面互动,在30个App、200个任务中追踪里程碑进度。
  4. 四个评测维度转换保真度、转换合理性、上下文一致性、任务进度,分别用Sele、Suse、Scp、Srap等指标打分。
  5. 核心发现单张画面可以很逼真,但连续多步后App身份、保存状态或搜索词常常出错,导致任务进度得分大幅下降。
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 背景问题:GUI世界模型通常只被当作单步预测器来评分,但它们实际的用途是被反复喂回作为下一步输入的多步骤环境。这种脱节导致一个关键问题一直没被真正检验:生成的画面在被反复重用时是否还保持上下文一致。
  2. 方法:GUI-CC设两条赛道。离线赛道从GUIOdyssey中挑出500条真实手机轨迹,按固定动作顺序让模型自回归地不断生成下一步画面;在线赛道让固定的GPT-5.5智能体自己选动作,直接和模型生成的画面互动,覆盖30个App、200个经过模拟器验证的任务。评测维度包括转换保真度、转换合理性、上下文一致性和任务进度四个方面。
  3. 结果:在离线赛道中,表现最好的模型的参考动作进度分(Srap)也只有16.7分(满分100),说明大多数连续生成的轨迹走几步就无法再支持后续动作。细粒度的元素对齐和布局完整性指标,最好的模型也不到20分,说明整体画面相似度高,但细节元素常常出错。
  4. 结果:画面看起来可用,不代表动作效果是对的。Flux.2-dev在两条赛道的画面可用性得分都很高,但任务进度几乎为零,说明它生成的界面看似合理,却没有真正体现动作带来的实际结果。
  5. 结果:给模型加上历史上下文(前几步的画面和动作)能提升上下文一致性得分,但对任务进度的帮助有限。例如GPT-5.5在在线赛道的总分从82.5提升到90.3,Code2World在两条赛道的一致性指标也有提升,但Code2World的参考动作进度只从9.1微升到9.5。
Figure 1: Representative failure cases of GUI world models under long-horizon rollouts. The examples illustrate app-context drift, action-effect lag, state-update propagation failure, and launcher-state memory loss.
Figure 1: Representative failure cases of GUI world models under long-horizon rollouts. The examples illustrate app-context drift, action-effect lag, state-update propagation failure, and launcher-state memory loss.
Table 1: Comparison of GUI-CC with prior GUI world-model evaluations. Pred. fed back: predicted states are reused as the next environment state during the evaluation rollout. Agent-in-loop: the rollout is driven by an agent acting on generated states. Direct WM eval: the headline metric is computed on the world model’s output rather than on downstream agent or search outcomes. Consistency eval: the evaluation directly tests cross-step state-reuse consistency. GUI-CC measures reference-action executability and context persistence in the offline track, and milestone progress, loops, and stagnation in the agent-loop track. GEBench reports an image-level temporal coherence score but does not test state-reuse consistency.
EvaluationReleaseDomainOutput StateMulti-StepPred. Fed BackAgent-in-LoopDirect WM EvalConsistency Eval
WMA (Chae et al., 2025)2024-10WebText✓✗✗✗✗
MobileWorldBench (Li et al., 2025)2025-12MobileText✗✗✗✓✗
gWorld (Koh et al., 2026)2026-02MobileHTML code✗✗✗✓✗
GEBench (Li et al., 2026)2026-02Mobile / desktopImage✓✗✗✓✗
Code2World (Zheng et al., 2026)2026-02MobileHTML code✗✗✗✓✗
CUWM (Guan et al., 2026)2026-02DesktopText / image✗✗✗✓✗
WebWorld (Xiao et al., 2026)2026-02WebStructured✓✓✓✓✗
MobileWorld (Xu et al., 2026)2026-05MobileCode / image / text✓✓✓✓✗
GUI-CC Offline Track2026-05MobileCode / image✓✓✗✓✓
GUI-CC Agent-Loop Track2026-05MobileCode / image✓✓✓✓✓
Figure 19: Single-step qualitative comparison for the action Clear the search bar. The figure compares the predicted next UI states from different GUI world models against the ground-truth next state.
Figure 19: Single-step qualitative comparison for the action Clear the search bar. The figure compares the predicted next UI states from different GUI world models against the ground-truth next state.
Table 2: Offline reference-action track evaluation of GUI world models. The Overall column is the unweighted mean across all ten metrics. Setting denotes the conditioning context: w/o history uses only the current UI and action, while w/ history uses multi-step historical states and actions. CC denotes Contextual Consistency. TP denotes Task Progress. The metrics evaluate UI element alignment (Sele), layout integrity (Slay), global similarity measured by SigLIP (Ssig) and DINO (Sdino), action adherence (Sad), action identifiability (Sid), GUI state usability (Suse), state and context persistence (Scp), action-controlled rollout dynamics (Srd), and reference-action progress (Srap). Within each model group, the best score per column is bold and the second best is underlined.
#World ModelSettingTransition FidelityTransition PlausibilityCCTPOverall
𝑺𝐞𝐥𝐞𝑺𝐥𝐚𝐲𝑺𝐬𝐢𝐠𝑺𝐝𝐢𝐧𝐨𝑺𝐚𝐝𝑺𝐢𝐝𝑺𝐮𝐬𝐞𝑺𝐜𝐩𝑺𝐫𝐝𝑺𝐫𝐚𝐩
Code / HTML-output world models
1Claude Opus 4.7w/ history16.815.673.447.169.579.099.665.093.816.457.6
2GPT-5.5w/ history17.816.675.049.863.468.299.459.580.915.354.6
3Claude Opus 4.7w/o history13.011.971.045.962.078.399.646.978.113.152.0
4GPT-5.5w/o history13.111.871.547.553.363.798.235.955.114.746.5
5Code2Worldw/ history9.88.368.936.049.161.596.038.058.99.543.6
6MobileWorld-html-8Bw/ history10.39.070.340.647.660.293.430.753.88.042.4
7MobileWorld-html-8Bw/o history10.28.869.437.544.057.691.028.754.39.341.1
8Code2Worldw/o history6.75.365.236.635.447.895.821.731.19.135.5
9gWorld-32Bw/ history6.96.064.039.223.267.395.08.317.83.633.1
10gWorld-8Bw/ history6.04.963.638.121.968.093.47.014.34.832.2
11gWorld-32Bw/o history5.04.262.638.419.566.091.95.017.03.531.3
12gWorld-8Bw/o history4.33.361.838.319.060.891.24.011.04.529.8
Direct image-generation world models
1GPT Image 2w/o history16.914.777.340.065.969.499.960.488.816.755.0
2Gemini 3.1 Flash Imagew/o history13.311.675.739.869.668.992.257.674.414.051.7
3Vimow/o history4.83.265.328.341.038.939.08.420.44.325.4
4Flux.2-devw/o history9.48.969.928.116.44.194.10.011.60.024.3
5Qwen-Image-Edit-2511w/o history5.85.265.824.517.51.542.80.00.00.016.3
6MobileWorld-Diffusionw/o history2.21.353.010.012.14.220.80.00.40.210.4
Figure 20: Single-step qualitative comparison for the action Open the GenZArt app. The figure compares the predicted next UI states from different GUI world models against the ground-truth next state.
Figure 20: Single-step qualitative comparison for the action Open the GenZArt app. The figure compares the predicted next UI states from different GUI world models against the ground-truth next state.
Table 3: Online agent-loop track evaluation of GUI world models. The Overall column is the unweighted mean across all six metrics. Setting denotes the conditioning context: w/o history uses only the current UI and action, while w/ history uses multi-step historical states and actions. CC denotes Contextual Consistency. TP denotes Task Progress. The metrics evaluate action adherence (Sad), action identifiability (Sid), GUI state usability (Suse), state and context persistence (Scp), action-controlled rollout dynamics (Srd), and ordered milestone progress (Smp). Within each model group, the best score per column is bold and the second best is underlined.
#World ModelSettingTransition PlausibilityCCTPOverall
SadSidSuseScpSrdSmp
Code / HTML-output world models
1GPT-5.5w/ history89.588.999.993.399.270.990.3
2GPT-5.5w/o history84.284.6100.080.095.051.282.5
3Claude Opus 4.7w/ history84.784.099.483.389.248.181.4
4Claude Opus 4.7w/o history82.088.099.457.578.341.474.4
5Code2Worldw/ history64.565.297.554.267.532.263.5
6gWorld-32Bw/ history58.971.198.744.253.324.458.4
7gWorld-8Bw/ history58.971.696.444.254.220.757.7
8MobileWorld-html-8Bw/ history61.664.191.744.254.219.956.0
9Code2Worldw/o history61.160.596.138.354.219.454.9
10MobileWorld-html-8Bw/o history60.666.195.526.750.022.653.6
11gWorld-8Bw/o history46.357.895.513.339.212.144.0
12gWorld-32Bw/o history42.563.797.411.731.715.443.7
Direct image-generation world models
1Gemini 3.1 Flash Imagew/o history85.089.195.670.885.843.778.3
2GPT Image 2w/o history82.682.2100.068.390.843.978.0
3Vimow/o history37.840.241.615.820.014.028.2
4Flux.2-devw/o history13.33.892.12.521.71.022.4
5Qwen-Image-Edit-2511w/o history14.44.942.92.50.81.411.2
6MobileWorld-Diffusionw/o history12.22.331.20.00.03.38.2
Figure 21: Qualitative comparison between GUI world model (Code2World) rollouts with and without historical context. The history-conditioned rollout preserves the task state across multiple steps and successfully reaches the expected final UI, whereas the no-history rollout deviates from the intended trajectory and fails to complete the task. This example illustrates the importance of historical conditioning for long-horizon GUI state prediction and task-progress consistency.
Figure 21: Qualitative comparison between GUI world model (Code2World) rollouts with and without historical context. The history-conditioned rollout preserves the task state across multiple steps and successfully reaches the expected final UI, whereas the no-history rollout deviates from the intended trajectory and fails to complete the task. This example illustrates the importance of historical conditioning for long-horizon GUI state prediction and task-progress consistency.
Table 4: World models evaluated in our benchmark.
ModelProviderTypeSizeAccess / CheckpointReference
Code / HTML-output world models
Claude Opus 4.7AnthropicProprietary (API)–claude-opus-4-7(Anthropic, 2026)
GPT-5.5OpenAIProprietary (API)–gpt-5.5(OpenAI, 2026a)
Code2World–Open-source8BGD-ML/Code2World(Zheng et al., 2026)
MobileWorld-html-8B–Open-source8Bxwk123/MobileWorld-html-8B(Xu et al., 2026)
gWorld-32BTrillion LabsOpen-source32Btrillionlabs/gWorld-32B(Koh et al., 2026)
gWorld-8BTrillion LabsOpen-source8Btrillionlabs/gWorld-8B(Koh et al., 2026)
Direct image-generation world models
GPT Image 2OpenAIProprietary (API)–gpt-image-2(OpenAI, 2026b)
Gemini 3.1 Flash ImageGoogle DeepMindProprietary (API)–gemini-3.1-flash-image-preview(Google DeepMind, 2026)
Qwen-Image-Edit-2511AlibabaOpen-source20BQwen/Qwen-Image-Edit-2511(Wu et al., 2025)
Flux.2-devBlack Forest LabsOpen-source32Bblack-forest-labs/FLUX.2-dev(Labs, 2025)
Vimo–Open-source–ai-agents-2030/ViMo (GitHub)(Luo et al., 2025)
MobileWorld-Diffusion–Open-source20Bxwk123/MobileWorld-Diffusion(Xu et al., 2026)
(b) Without history. The model loses long-horizon contextual consistency, leading to an incorrect trajectory and a failed final UI.
(b) Without history. The model loses long-horizon contextual consistency, leading to an incorrect trajectory and a failed final UI.

研究结果

  • 离线赛道中表现最好的模型,参考动作进度(Srap)也只有16.7分(满分100)。
  • 离线细粒度指标——元素对齐和布局完整性,最好的模型得分也低于20分。
  • Flux.2-dev在两条赛道的界面可用性(Suse)得分都很高,但任务进度几乎为零。
  • 加入历史上下文后,GPT-5.5在线赛道总分从82.5升到90.3,Code2World在两条赛道的一致性指标也有提升,但Code2World的参考动作进度只从9.1变为9.5。
  • 对GPT-5.5和GPT Image 2的200条失败轨迹人工复核发现,约42%的失败源于缺乏App专属或安卓层面的转换知识,约33%源于早期小偏差被反复重用后逐步放大的错误累积,约25%源于App身份丢失、保存状态消失等上下文不一致。
Figure 22: Offline rollout comparison under the same long-horizon cross-app action sequence. Green and orange borders indicate successful and failed step-level transitions, respectively. Code2World fails in most steps, while GPT Image 2 preserves early states but drifts later.
Figure 22: Offline rollout comparison under the same long-horizon cross-app action sequence. Green and orange borders indicate successful and failed step-level transitions, respectively. Code2World fails in most steps, while GPT Image 2 preserves early states but drifts later.

可应用场景

  • 在用GUI世界模型替代真实设备、低成本训练或评测GUI智能体之前,先检验该世界模型是否值得信赖
  • 作为基准测试,帮助开发者检验新的GUI世界模型在多步生成中的上下文一致性,而不只是单张画面的质量
  • 用来区分某项改进(比如加入历史上下文)是真的提高了任务完成率,还是只是让一致性分数看起来更好
(b) GPT Image 2. Early steps succeed, but later steps drift.
(b) GPT Image 2. Early steps succeed, but later steps drift.

局限与待验证事项

  • 目前基准只覆盖手机端GUI任务,尚未扩展到网页、桌面或多设备环境。
  • 在线赛道只用了一个固定的探测智能体(GPT-5.5),规划能力或恢复策略不同的智能体可能会得到不同结果。
  • 评分依赖视觉语言模型(VLM)裁判,论文自己也指出,当画面局部看起来合理时,裁判可能会低估选错项目、保留过期搜索词等细粒度任务状态错误。
  • 细粒度保真度指标只和单一的标准答案下一步画面比较,可能会不公平地惩罚另一种同样合理且可执行的下一步画面。
  • 作者提到未来版本计划加入人工验证的检查、结构化的界面状态探测和可执行验证,这些在当前版本中尚未实现。
(c) Ground truth. Reference trajectory.
(c) Ground truth. Reference trajectory.

为什么重要

如果想让GUI智能体不依赖真实设备就能低成本训练和评测,充当替代环境的世界模型就必须在多步交互中保持一致,而这篇论文用具体数字说明目前的模型还做不到这一点。这提醒开发者不能只看单张画面像不像真的就判断模型好坏,也为后续改进方向(持久状态维护、App专属转换知识)提供了依据。

本文术语

  • GUI世界模型 · 输入当前界面和一个动作、预测下一个界面的模型,目的是替代真实设备作为智能体的交互环境。
  • 离线参考动作赛道 · 按真实记录的动作顺序回放,同时让模型不断自己生成下一步界面的评测方式。
  • 在线智能体循环赛道 · 让一个固定的智能体看着模型生成的界面、自己决定下一步动作的评测方式。
  • 上下文一致性 · 在多步生成过程中,App身份、已输入文字、已选中项目等任务相关状态是否保持连贯。
  • 参考动作进度(Srap) · 衡量模型生成的连续画面能支持多少个连续参考动作而不崩坏的指标。

论文原文摘要（英文）

GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI agents. This mismatch leaves a key requirement under-tested: generated states must remain contextually consistent when they are repeatedly reused for future interaction. We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments rather than isolated next-screen predictors. GUI-CC contains two complementary tracks: an offline reference-action track that rolls models along real mobile GUI trajectories, and an online agent-loop track that lets fixed probing agents interact with model-generated UIs. We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps. GUI-CC evaluates transition fidelity, transition plausibility, contextual consistency, and task progress. Experiments show that plausible single-step generation does not guarantee reliable environment simulation: current models often produce usable-looking screens while failing to preserve task-relevant context or support executable multi-step rollouts.

作者 · Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Lin Fu et al., arXiv:2609.00048, CC BY 4.0