OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

What they did

1. Reusing existing benchmarks' leftover trajectories mixes in quality problems and noisy labels, so the researchers built fresh cross-platform infrastructure themselves, on web, mobile, Ubuntu, and Windows, populated with real applications, logged-in accounts, real files, and distractor content
2. Agents from four model families (Claude, Gemini, Kimi, Qwen) executed human-verified instructions, producing 1019 trajectories with real successes and failures; each was labeled independently by three human annotators, with disagreements escalated to senior meta-review, costing roughly 800 human hours in total
3. This gold set was split into the full OSReward set, a difficulty-concentrated OSReward-Hard subset of 284 trajectories (rebalanced to 30/70 success/fail), and an OSReward-Multi subset of 440 successful trajectories rated for alignment and efficiency, then used to test 27 VLM judges under one fixed protocol
4. The best judge (Claude-Opus-4-8) reached 89.7% accuracy on the full set but collapsed to 69.7% on OSReward-Hard, with the average judge falling to 52%, and two-thirds of all errors were 'over-accepting an incomplete task' as success
5. To close this gap, the team distilled over 300K judge instances into a 100K-sample corpus (OS-Shepherd-100K) and trained OS-Shepherd models in two stages (supervised fine-tuning then reinforcement learning), achieving accuracy near commercial frontier judges at 30-60x lower cost

Findings

• The best judge, Claude-Opus-4-8, scored 89.7% accuracy on the full OSReward set, but every judge's accuracy dropped 20-43 percentage points on OSReward-Hard, with the top model reaching only 69.7%
• About two-thirds of all judge errors were over-accepting an incomplete task as a success, and this was the dominant error mode for every judge tested, accounting for at least 48% of each model's mistakes
• OS-Shepherd-9B and 35B matched commercial frontier judges' accuracy at 30-60x lower cost under the identical evaluation protocol, and reinforcement learning raised validation accuracy from about 70% to about 77%
• On an external benchmark (OSWorld), 88% of judge errors were false positives, and for trajectories beyond 16 steps, accuracy fell from 0.76 to 0.57 while the false-positive rate rose from 0.20 to 0.37

Where it can be used

• Using OS-Shepherd as a low-cost reward signal to score large volumes of agent trajectories during reinforcement learning training
• Pre-filtering large batches of agent runs for likely success or failure before human review in data collection pipelines
• Screening a new VLM's suitability as a CUA judge against a hard-case set like OSReward-Hard before deploying it at scale

Limits and open work

• The benchmark covers only web, mobile, Ubuntu, and Windows, not macOS (the macOS share in the training corpus comes from a separate reused public dataset)
• The judging protocol was tested under a fixed setting using the last five screenshots by default; reliability under other input forms, such as tool access or step-level supervision, was not established here
• The authors suggest that explicitly prompting a judge to verify task completion might reduce the leniency bias more effectively than the ensembling they tested, but this was left for future work
• OS-Shepherd's reinforcement-learning data is skewed toward desktop cases because false successes concentrate there, leaving mobile and web underrepresented in that training set

Why it matters

Training and evaluating AI agents at scale depends on another AI judging whether each run truly succeeded, and if that judge's reliability is unknown, the resulting training data and evaluation scores could be systematically skewed. This work is the first to measure that reliability in a standardized way across platforms and offers an open, affordable alternative, which matters directly for anyone building or evaluating computer-using agents.

Terms in this paper

• CUA (Computer-Using Agent) · An AI agent that operates a computer directly by viewing the screen and taking actions like clicking or typing
• VLM (vision-language model) · An AI model that understands both images (like screenshots) and text together
• false success · When an agent claims a task is done but actually failed to achieve the goal
• reward model · A model that scores another AI's output, used to guide reinforcement learning or filter training data
• GRPO · The policy optimization algorithm used in OS-Shepherd's reinforcement learning stage

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3