GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

What they did

1. Motivation: GUI world models are usually graded on producing one plausible next screen, but they're actually deployed as multi-step environments where generated screens get fed back in repeatedly. This mismatch means nobody has properly tested whether generated states stay contextually consistent under repeated reuse.
2. Method: GUI-CC has two tracks. The offline track autoregressively rolls each model along fixed reference action sequences from 500 real GUIOdyssey trajectories. The online track lets a frozen GPT-5.5 probing agent choose its own actions on model-generated screens across 200 emulator-verified tasks spanning 30 mobile apps. Four evaluation dimensions are used: transition fidelity, transition plausibility, contextual consistency, and task progress.
3. Result: Even the best offline model only reached a reference-action progress score of 16.7 out of 100, meaning most rollouts broke down after just a few steps and could no longer support the next scripted action. Fine-grained element alignment and layout integrity scores stayed below 20 for the best models even though coarse visual similarity scores were relatively high.
4. Result: Looking usable is not the same as behaving correctly. Flux.2-dev scored near the top on GUI state usability in both tracks but achieved near-zero task progress, showing it produced readable interfaces without actually modeling the consequences of the action taken.
5. Result: Feeding models a short history of previous screens and actions improved contextual-consistency scores but not task progress by much. GPT-5.5's online overall score rose from 82.5 to 90.3 with history, and Code2World improved on consistency metrics too, but Code2World's reference-action progress barely moved, from 9.1 to 9.5.

Findings

• On the offline track, even the best model's reference-action progress (Srap) reached only 16.7 out of 100.
• Fine-grained offline metrics for element alignment and layout integrity stayed below 20 for the best models.
• Flux.2-dev scored high on GUI state usability (Suse) in both tracks but near-zero on task progress.
• Adding history raised GPT-5.5's online overall score from 82.5 to 90.3, and improved Code2World on both tracks, but Code2World's reference-action progress only moved from 9.1 to 9.5.
• A manual review of 200 failed rollouts from GPT-5.5 and GPT Image 2 attributed about 42% of failures to missing app or Android-level transition knowledge, about 33% to error accumulation from small early deviations, and about 25% to context inconsistency such as lost app identity or forgotten saved state.

Where it can be used

• Checking how trustworthy a GUI world model is before using it as a low-cost, device-free environment for training or evaluating GUI agents
• Serving as a benchmark for developers building new GUI world models, to test multi-step contextual consistency rather than only single-screen quality
• Distinguishing whether a fix like adding history conditioning actually improves task completion or just makes consistency scores look better

Limits and open work

• The benchmark currently covers only mobile GUI tasks; extension to web, desktop, or multi-device environments has not been done yet.
• The online track uses only one fixed probing agent (GPT-5.5), so results may differ with agents that have different planning or recovery strategies.
• Scoring relies on VLM judges, which the paper notes can under-penalize fine-grained task-state errors like selecting the wrong item or keeping a stale search query when the screen still looks locally plausible.
• Fine-grained fidelity metrics compare only against a single ground-truth next screen, which may unfairly penalize a different but equally valid and executable next state.
• The authors note future versions could add human-validated checks, structured UI-state probes, and executable verification for finer task states, which are not yet in this version.

Why it matters

If GUI agents are ever going to be trained and tested cheaply without live devices, the world models standing in for those devices need to stay coherent over many steps, and this paper shows current models mostly don't. It's a concrete warning against judging a world model by how good its next screen looks, and points toward what actually needs fixing: persistent state tracking and app-specific transition knowledge rather than better images alone.

Terms in this paper

• GUI world model · A model that takes the current screen plus an action and predicts the next screen, meant to stand in for a real device as an agent's environment.
• Offline reference-action track · An evaluation that replays a real recorded action sequence while the model keeps regenerating the next screen on its own.
• Online agent-loop track · An evaluation where a fixed agent picks its own next action by looking at the model's previously generated screen.
• Contextual consistency · Whether task-relevant state such as app identity, typed text, or selected items stays coherent across many generated steps.
• Reference-action progress (Srap) · A score measuring how many consecutive reference actions a model's rollout can still support before breaking.

Latest papers

• UI-Venus-2 Technical ReportUI-Venus-2 is a single GUI agent that operates across mobile, web, and desktop, scaling environments, tasks, and verification togetherarXiv:2609.000282026-09-02
• SHAPE of Chain-of-Thought in Math ReasoningA framework that tags each step of an AI's math 'thinking' with what mathematical move it's making and what interpretation it's stuck in, revealing why some reasoning succeeds and why RL training narrows a model's strategiesarXiv:2608.286002026-09-01
• CoVA-SFT: A Large-Scale Dataset for Chain of Visual AbstractionsA new 52K-example dataset teaches multimodal AI to 'draw while thinking' instead of writing everything in wordsarXiv:2608.289582026-09-01
• CNeo-Bench: Diagnosing Large Language Models on Chinese NeologismsLLMs can often explain what Chinese internet slang like 886, yyds, or 彳亍 means, but many fail to reconstruct the original form behind itarXiv:2608.280532026-08-31
• FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial OutcomesA dataset that finally teaches AI what biology, chemistry, and physics peer reviewers actually argue about, not just CS reviewersarXiv:2608.261292026-08-28
• JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness EvolutionAn AI system that writes a custom 'operating scaffold' for other AI agents on the spot, for every new taskarXiv:2608.255932026-08-27
• The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling PipelineAI language models still charge a hidden 'dialect tax' on AAVE and other non-standard English at every stage, not just tokenizationarXiv:2608.249522026-08-27
• Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal BayesiansA math model shows that even a perfectly rational person can be talked into delusion by a chatbot that keeps agreeing with themarXiv:2602.191412026-08-26

Latest from METAL LAB

• AI Models Skip Words to Talk Faster — A Fields Medalist Builds the Bridge Between Them
• ATV Big Air Tour cuts inventory work from three days to three hours with ChatGPT Work
• Inworld AI Launches Realtime TTS-2, Reigniting Race for Top Voice Synthesis Spot
• Google opens cyber defense AI to governments and enterprises through Fairwind program
• Perplexity Cited 215,000 Pages From Three AI-Targeted Sites