GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

arXiv:2608.218332026-08-25

AI编程智能体擅长从零做出一款能玩的游戏,但不擅长自己发现漏洞、也难在修改中保住原有功能

研究团队分析了真实的人类与AI编程智能体开发游戏的完整过程,发现开发流程可分为三个阶段:初始生成、漏洞诊断与修复、以及多轮优化。他们据此构建了GameXpert-Bench评测套件,包含97个生成任务、100个修复任务和17条优化链条,通过实际运行并操作生成的游戏,测试了15到17个编程智能体模型。结果显示,这些智能体在做出一版能玩的游戏方面表现稳定,但在自主发现隐藏漏洞、验证运行时行为、以及在改动过程中保住原有功能这几方面明显更弱。

METAL LAB 解读图

GameXpert-Bench三条评测赛道的运作方式

证据状态已报告实测结果

第一阶段 GameGen仅凭一条自然语言需求和一个空工作区,要求智能体从零做出完整游戏;97个任务覆盖11种类型
第二阶段 GameFix在50个经人工核验的关卡中各植入19到27个漏洞,分别在告知或隐藏条件下测试,共100个修复任务
第三阶段 GameOpt基于真实用户历史重放六轮共102条请求,检验智能体能否持续改进游戏而不破坏已有功能
共同评分原则只有当实际运行的游戏展示出预期行为时才给分,而不是代码看起来合理就给分
观察到的不对称在三条赛道中,智能体在初始构建方面都强于自主发现漏洞、验证运行时行为和保留原有功能

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

研究团队分析了人类与AI智能体之间完整的游戏开发对话记录,发现开发过程反复出现三个阶段:初始生成、漏洞诊断与修复、多轮优化。
他们据此设计了三条对应的评测赛道(GameGen、GameFix、GameOpt),评分方式是实际在浏览器中运行并操作生成的游戏,而不只是检查源代码。
GameGen只给智能体一条自然语言需求和一个空工作区,要求其从零做出完整游戏,共97个任务、覆盖11种游戏类型,其中44个为3D游戏。
GameFix在50个经人工核验的正常游戏关卡中各植入19到27个可还原的漏洞,分别在明确告知漏洞和需要智能体自行发现两种条件下测试,共100个修复任务。
GameOpt基于真实用户与智能体的开发历史,重放六轮共102条请求(17条链条),检验智能体能否在持续响应新需求的同时保住此前实现的功能。

Figure 1: Overview of GameXpert-Bench and its three evaluation tracks. GameGen evaluates game creation from natural-language requirements, GameFix evaluates the repair of corrupted reference games, and GameOpt evaluates multi-turn game optimization based on human feedback.

Table 1: Coverage of representative benchmarks across the three stages studied in this paper. Prior work evaluates several constituent capabilities, while GameXpert-Bench places game creation, verified repair, and cumulative optimization within one benchmark suite. Text labels indicate partial coverage or the source of feedback.
Benchmark	Domain	Creation or implementation	Diagnosis and repair	Iterative refinement
GameDevBench [5]	Game	✓	✗	✗
GameEngineBench [15]	Game	✓	✗	✗
OpenGame-Bench [11]	Game	✓	✗	✗
WebGameBench [24]	Game	✓	✗	✗
GameCraft-Bench [16]	Game	✓	✗	✗
GBQA [10]	Game	✗	Discovery only	✗
PlayCoder [19]	GUI apps	Function-level	Explicit issue repair	Agent refiner
SWE-Together [22]	General software	✗	Explicit issue repair	Session simulator
GameXpert (ours)	Game	✓	Explicit issue +	Human–agent
self-discovery repair	co-development

Figure 3: Artifact growth across complete coding-agent game development trajectories. Filled markers denote artifact-changing requests and colors indicate their lifecycle stage; hollow markers denote continuation requests. Each trajectory retains or increases its source size across stages. Game names are reported as broad categories.

Table 2: Main results on GameGen. All scores are reported out of 100, with the Overall score computed as the average of Completeness, Richness, Player Experience, and Visual Quality. Values are rounded to one decimal place. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	79.7	94.4	72.0	72.4	80.4
2	Claude-Fable-5	75.8	89.2	62.8	74.4	76.8
3	Kimi-K3	71.3	85.2	56.0	70.4	73.6
4	Claude-Opus-4.8	69.9	84.4	49.6	72.0	74.0
5	GPT5.6-sol	63.4	79.6	42.0	63.2	68.8
6	GLM5.2	63.4	76.8	44.8	62.4	69.6
7	DeepSeek-V4-Flash	62.9	81.6	48.4	57.6	64.0
8	Hy3	62.1	77.2	41.6	62.0	68.0
9	Claude-Opus-4.7	62.0	76.0	42.8	61.6	67.6
10	MiniMax-M3	59.6	73.2	40.8	56.4	68.0
11	Gemini-3.5-flash	59.1	73.2	46.0	53.2	64.0
12	GPT5.5	59.0	74.0	36.8	60.4	65.2
13	GLM5.1	58.0	72.8	43.2	56.4	60.0
14	Qwen3.7-Max	54.4	66.4	34.0	55.2	61.6
15	Seed-2.1-pro	48.7	58.4	30.8	44.8	60.8

Figure 4: Functional coverage across all 15 models. Average completeness and richness are reported on [0,10], and event pass rate is multiplied by 10 for comparison. Models follow the main-leaderboard order.

Table 3: GameFix leaderboard, sorted by the primary Strict metric (mean height of the average@3 survival curve over the band τ∈[90,100]). Cliff is the drop in macro average@3 repair score from Explicit Issue to Self-Discovery (we report this difference only, not the two absolute scores; see §5.3). All values are 3-run average@3. In each score column bold marks the best value and underline the worst (Cliff is lower-is-better, so bold is the smallest cliff and underline the largest). † models are analysed at the trajectory level in Appendix B.
#	Model	Harness	Effort	Strict↑	Cliff↓
1	Claude Opus 5†	Claude Code	max	39.0	7.6
2	Claude Fable 5	Claude Code	max	33.7	8.4
3	GPT-5.6-sol†	Codex	xhigh	29.1	11.3
4	GPT-5.6-sol	Claude Code	xhigh	26.7	10.3
5	Claude Opus 4.8†	Claude Code	max	18.0	17.9
6	Claude Opus 4.7	Claude Code	max	17.2	22.2
7	GPT-5.5	Codex	xhigh	16.3	13.3
8	GPT-5.5	Claude Code	xhigh	16.1	10.5
9	DeepSeek V4 Flash†	Claude Code	max	15.2	13.6
10	Kimi K3†	Claude Code	max	14.0	19.6
11	GLM 5.2†	Claude Code	xhigh	13.4	20.2
12	Gemini 3.5 Flash	Claude Code	high	12.2	31.1
13	MiniMax-M3	Claude Code	on	10.3	30.5
14	Seed-2.1-pro	Claude Code	high	8.1	15.3
15	GLM 5.1	Claude Code	xhigh	7.0	21.7
16	Hy3†	Claude Code	high	6.0	32.8
17	Qwen3.7-Max	Claude Code	max	5.5	30.3

Figure 5: Composition of event-level outcomes across all 15 models, including checklist events that are missing from the generated report. Each stacked bar sums to 100%, and models follow the main-leaderboard order.

Table 4: Composition of the current GameOpt evaluation collection.
	Count
Games (chains)	17
Turns	102
Acceptance criteria	701
Requirement	392
Challenge	212
Regression	97

Figure 6: Performance gaps between the 2D and 3D subsets of GameGen. (a) Difference in overall score for each model, computed as the 3D score minus the 2D score. The dashed line marks the cross-model mean gap. (b) Cross-model mean gaps across the four evaluation dimensions. All scores are reported on a [0,100] scale, and negative values indicate lower performance on the 3D subset. Models follow the main-leaderboard order.

Table 5: Overall and six-dimensional GameOpt scores for 15 model variants on 17 JavaScript games. All columns use the pooled discrimination-weighted result summary.
Rank	Model	Overall	Gameplay	Level	Balance	Art	UI	Audio
1	Claude-Opus-5	93.96	91.3	94.5	80.8	100.0	97.0	98.5
2	Claude-Fable-5	89.31	88.7	85.2	80.9	95.8	90.3	96.2
3	Kimi-K3	84.66	86.1	75.9	81.1	91.6	83.6	93.9
4	GPT-5.6-sol	83.68	82.8	89.7	83.2	87.7	67.8	97.0
5	Claude-Opus-4.8	82.53	75.3	78.3	77.8	77.9	95.1	93.5
6	Claude-Opus-4.7	82.36	87.7	83.6	89.4	79.6	66.8	94.8
7	DeepSeek-V4-Flash	78.34	71.9	67.3	70.2	87.2	90.5	83.9
8	GPT-5.5	74.24	70.5	67.0	86.4	66.5	78.7	83.7
9	GLM-5.2	72.98	74.5	68.3	63.1	67.8	79.1	88.3
10	Hy3	68.87	71.1	62.0	58.8	66.4	75.5	82.1
11	Qwen3.7-Max	64.77	67.7	55.6	54.4	65.1	72.0	76.0
12	GLM-5.1	60.67	64.3	49.2	50.1	63.7	68.4	69.8
13	MiniMax-M3	59.87	71.4	49.8	53.9	48.8	70.2	64.7
14	Gemini-3.5-Flash	47.20	52.7	36.6	33.1	53.9	50.4	58.2
15	Seed-2.1-Pro	35.89	46.8	32.6	28.6	28.7	32.8	49.0

Figure 7: Player-facing and runtime failure diagnostics. (a) Share of model–game runs whose evaluation reports contain evidence of UI overlap or misalignment. (b) Share of events recognized as implemented in code but assigned a runtime outcome of Fail or Partial. (c) Attribution of the 2,293 implementation-gap events.

Table 6: Category distribution of the GameGen corpus. The Share column is computed over all 97 games.
Category	2D	3D	Total	Share (%)
Puzzle	5	5	10	10.3
Strategy	5	5	10	10.3
RPG	5	5	10	10.3
Shooter	5	4	9	9.3
Roguelike	3	3	6	6.2
Simulation and Management	5	0	5	5.2
Action	5	5	10	10.3
Narrative	5	3	8	8.2
Idle	5	5	10	10.3
Sandbox and Building	5	5	10	10.3
Card	5	4	9	9.3
Total	53	44	97	100.0

Figure 8: Model-level association between automated functional evaluation and human-perceived quality. Each point represents one of the 15 evaluated models. The automated functional score is the mean of Completeness and Richness, while the human perceptual score is the mean of Player Experience and Visual Quality; all component scores are on a [0,100] scale. The solid line denotes the ordinary least-squares fit, and the shaded region denotes its 95% confidence band for the mean response. GPT5.6-sol and GLM5.2 have identical aggregate coordinates and therefore overlap.

Table 7: Representative full prompts from GameGen (Part I of III).
Category	Dim.	Full Prompt
Puzzle	2D	Create a 2048 game controlled with the arrow keys. Tiles with the same value should merge, and a new tile with value 2 or 4 should be spawned at random after every valid move. Include sliding animations during movement and merging, display the current score, show a victory message when the player reaches 2048, and show a game-over message when no valid move remains.
Puzzle	3D	Create a minimalist, wood-textured 3D maze game viewed from above. The game should support tilting the maze to guide a ball through a wooden map containing walls and an exit. The player tilts the maze to roll the ball, avoids traps and dead ends, and guides the ball to the exit. The level fails if the ball falls out of the maze or time expires. Include at least two basic levels: a single-route maze and a simple branching maze.
Strategy	2D	Create a hand-painted watercolor-style 2D insect-themed real-time strategy game viewed from above. Include an economy based on collecting honeydew and cultivating fungus, an ant queen that produces workers with different roles, and switching between surface and underground maps. The player controls an ant colony, expands an underground nest and its tunnels, sends worker ants to collect food, and deploys soldier ants against invading spiders or rival ant colonies.
Strategy	3D	Create a post-apocalyptic science-fiction 3D first-person tower-defense game. Include defensive structures and a weapon-upgrade system. The player builds defenses, upgrades weapon systems, and withstands waves of monsters. Enemies should approach the base along different routes. Include at least two of the following defensive structures: a machine-gun turret, a laser turret, and a freezing turret.
RPG	2D	Create a retro-JRPG-style 2D pixel-art turn-based role-playing game viewed from above. Include companion recruitment and skill-combination systems, with an open map containing both a town and a dungeon. The player can recruit companions, combine skills, and trigger side quests. NPC attitudes should change according to the player’s choices. Include at least three character classes: warrior, mage, and rogue.
RPG	3D	Create a third-person 3D action RPG in an ink-wash wuxia style that fuses ukiyo-e and Chinese ink-painting aesthetics. The environments should include a bamboo forest, a snowy mountain, and a ruined shrine. The core mechanic is a block-and-parry system: a perfect parry should inflict substantial posture damage, and breaking an enemy’s posture should enable a finishing move. Allow the player to switch among three prosthetic tools—shuriken, firecrackers, and an axe—and combine light and heavy attacks into different combos. Enemies should include a samurai general whose AI switches between high and low perilous attacks, and a vengeful spirit surrounded by ghost fire that can spread to and burn the player.
Shooter	2D	Create a 2D top-down aircraft shooter. The player can move the aircraft to dodge enemy bullet patterns, use multiple firing modes and screen-clearing bombs, and fight enemies. Defeating elite enemies should drop firepower upgrades, bombs, and auxiliary weapons, while defeating a boss should award an extra life. Firepower upgrades, extra lives, auxiliary weapons, and bombs must carry over between stages. Their effects and upgrade rules should follow the conventions of the Raiden series.
Shooter	3D	Create a simple cartoon-style 3D first-person shooter set in a small enclosed indoor map. Include firearms and AI-controlled enemies. The player should be able to move, aim, and fire. Enemies should patrol along fixed routes. Include at least two weapons: a pistol and a rifle.

Figure 9: Survival curves of the average@3 pass rate versus the all-solved threshold τ for all 17 models (eight highlighted); each run contributes its own curve and the three are averaged. The shaded band τ∈[90,100] is the region whose mean curve height defines Strict.

Table 8: Representative full prompts from GameGen (Part II of III).
Category	Dim.	Full Prompt
Roguelike	2D	Create a pixel-art 2D top-down dungeon-exploration roguelike with randomly generated rooms, permadeath, and relic collection. Each run should generate a new room layout, enemy distribution, and set of treasures. The character is permanently lost upon death, but soul stones can be carried back to upgrade global talents. Place one boss every three floors. Include at least two classes: warrior and mage.
Roguelike	3D	Create a 3D zombie-survival roguelike set in a ruined city, with randomized weapons, ammunition management, and an infection meter. The player searches through zombie hordes for randomized weapons, including pistols, rifles, and machine guns. After each wave, the player can upgrade a held weapon by increasing reserve ammunition or damage, or heal the character to reduce the infection meter. Zombie bites increase infection; when the meter is full, the player turns into a zombie and the run ends. Include at least two zombie types: normal zombies and sprinting zombies.
Simulation and Management	2D	Create a farm-management game. Customers should arrive automatically to buy crops, while the player plants and harvests produce and hires workers to help with watering and fertilizing. Customers pay coins after completing a purchase. Coins can be used to upgrade the farm, buy new seeds, and unlock animals. Allow the player to change farm decorations dynamically. Crop hybridization should have a random chance of failure, and hired workers should have a fatigue system.
Action	2D	Create a pixel-art 2D side-scrolling action game set in a gothic castle. Include a combo system, dodge rolls, and a special attack. The player can combine light and heavy attacks into combos, and spend energy to unleash a full-screen special attack. Dodge rolls should provide invincibility frames that can avoid incoming attacks. Include at least two weapon types: a sword and a whip.
Action	3D	Create a high-speed 3D character-action hack-and-slash game inspired by Devil May Cry. Its core mechanics should include real-time switching among four combat styles, aerial combos, enemy-step jumps that cancel action recovery, and a chargeable demonic state that changes move properties and adds damage. Levels should contain hidden challenge rooms; clearing one within the time limit should unlock an additional temporary attribute enhancement.
Narrative	2D	Create an urban rule-horror text game. The player must avoid supernatural corruption, inspect blurred and crossed-out clauses on an elevator notice and blood-red writing on office notes, investigate strange shadows in surveillance blind spots, identify false rules mixed among genuine ones, and escape the out-of-control office building safely before midnight.
Narrative	3D	Create a 3D text-driven treasure-hunting game set in an ancient desert city. The vast three-dimensional desert should feel open and desolate. The player traverses dunes and the Gobi, explores three-dimensional ancient-city ruins, deciphers writing on old stone walls, avoids sandstorm hazards, breaks the city’s ancient seals, and searches for a legendary treasure. Include underground-palace mechanism puzzles and sandstorm-survival gameplay.

Figure 10: Geometric meaning of Strict. Left: for the highlighted models, the survival curve over the τ∈[90,100] band; Strict is the mean height of that curve across the band (dashed line), equivalently the shaded area divided by the band width. Right: Strict for all 17 models, sorted descending.

Table 9: Representative full prompts from GameGen (Part III of III).
Category	Dim.	Full Prompt
Idle	2D	Create a retro pixel-art 2D idle game viewed from above and set in a hero camp. Include offline earnings and a skill tree. The player clicks monsters to deal damage and spends coins to purchase units that attack automatically. The hero can upgrade skills. Include at least two automated unit types: archers and mages.
Idle	3D	Create a 3D retro-steampunk idle game built around three core mechanics: a mechanical factory that automatically forges devices, steam-powered automata that automatically leave to work, and a power core that automatically stores and accumulates energy. Offline idling should mass-produce industrial items. Mechanical modification should carry a risk of malfunction; the player can allocate repair fuel to maintain stable operation and trade precision blueprints through a steam-powered merchant guild to build advanced mechanical creations.
Sandbox and Building	2D	Create a pixel-art 2D top-down sandbox-building game with terrain editing, resource gathering, and construction in a procedurally generated infinite world. The player can mine and place multiple block types—soil, stone, wood, and water—and build houses, farms, and defensive structures. Include at least two biomes: grassland and desert.
Sandbox and Building	3D	Create a low-poly 3D first-person sandbox-building game with block placement and removal, a day–night cycle, and hostile creatures on a procedurally generated continent. The player gathers wood and stone and builds houses and castles. Zombies and skeletons should spawn at night. Include at least two biomes: forest and mountain.
Card	2D	Create a dark-fantasy 2D pixel-art roguelike card game viewed from above, with a randomized room-based dungeon map and deckbuilding mechanics. The player can collect cards, construct a deck, and trigger combo effects. Enemies should patrol rooms and attack proactively. Include at least three card types: attack, defense, and utility cards.
Card	3D	Create a 3D medieval knight battle-card game set in retro castle and field environments, with an action-oriented main storyline about a kingdom at war. Allow the player to recruit heavy knights, ranger scouts, and royal mages as combat cards. Include unit-type counters and formation-charge mechanics. The player deploys cards to arrange offensive and defensive formations and engages in card-based duels between lords. Pacifying the conflict should unlock additional chapters of the kingdom campaign.

Figure 11: The Explicit Issue→Self-Discovery Cliff. Left: each model’s macro average@3 under Explicit Issue versus Self-Discovery; a steeper drop is weaker self-discovery. Right (inset): the Cliff shrinks as Strict rises.

Table 10: Completeness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Score increases after a merge	Add the value of the newly merged tile to the score and update the visible score display.
2	An invalid move leaves the state unchanged	If no tile can move or merge in the requested direction, do not spawn a tile or change the score.
3	Game over when no move remains	End the game when the board is full and no adjacent equal-valued tiles can be merged.
4	High-score update and persistence	Update the historical best score and retain it after restarting or refreshing the page.
5	Game start and initialization	Construct the board, reset the score and state, and spawn the initial tiles when the page loads.
6	Equal-valued tiles merge	Merge two adjacent tiles with the same value along the movement direction into one tile with twice the value.
7	A new tile appears after a valid move	After any move that changes the board, spawn a new tile in an empty cell.
8	Reaching 2048 triggers victory	Display a victory state when a tile with value 2048 is created.
9	Restarting begins a fresh game	The new-game control clears the board, resets the current score and overlays, and creates a new initial state.
10	Tiles visibly slide between cells	Use a smooth positional transition during movement rather than an instantaneous jump.

Figure 12: Six-dimensional capability profiles. Rows denote the six optimization dimensions and columns denote models. Each cell reports the final-product rubric score; black outlines mark the best-performing model in each dimension. The right column reports means over all 15 models, while the bottom row reports the leaderboard Overall score.

Table 11: Richness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Continue after reaching 2048	Allow the player to dismiss the victory overlay and continue playing beyond 2048.
2	Touch-swipe controls	Infer a direction from a sufficiently long touch gesture and execute the corresponding move.
3	WASD controls	Map the WASD keys to the same four movement directions as the arrow keys.
4	Spawn and merge tween effects	Animate newly spawned tiles and tiles created by merging with distinct appearance or pop effects.
5	Styles for tiles above 2048	Provide a visually distinct high-tier style for values greater than 2048.
6	Multiple merges in one move	Correctly merge four equal-valued tiles in a row into two doubled tiles during a single move.
7	Pop-in animation for created tiles	Apply a scale or appearance animation to newly spawned or merged tiles.
8	Resume state after reloading	Restore the board, score, and relevant game state from persistent browser storage.
9	Input locking during animation	Ignore additional movement commands while a movement animation is in progress.
10	Merge-bounce animation	Apply a short scaling or bouncing animation to a tile created by merging.
11	On-screen directional controls	Provide clickable directional buttons as an alternative input method.
12	Pointer-drag gesture controls	Detect a pointer drag beyond a distance threshold and move in the corresponding direction.

Table 12: Results on the 53-game 2D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	80.4	96.0	73.2	73.6	78.8
2	Claude-Fable-5	76.8	91.6	63.2	75.6	77.2
3	Claude-Opus-4.8	71.0	86.4	52.0	71.2	74.0
4	Kimi-K3	70.5	84.0	54.4	70.4	73.2
5	Claude-Opus-4.7	67.9	84.8	49.2	67.6	70.0
6	DeepSeek-V4-Flash	66.6	86.0	52.4	60.8	67.2
7	GLM5.2	65.7	79.2	46.4	65.2	72.0
8	GPT5.6-sol	65.2	80.0	44.0	67.2	69.6
9	Hy3	64.9	80.4	44.8	65.6	68.8
10	MiniMax-M3	64.0	82.0	43.6	60.0	70.4
11	Gemini-3.5-flash	62.2	77.6	49.2	56.4	65.6
12	GPT5.5	61.9	78.0	38.0	64.4	67.6
13	GLM5.1	58.9	73.2	45.2	57.6	59.2
14	Qwen3.7-Max	58.5	77.2	40.8	55.2	61.2
15	Seed-2.1-pro	54.2	66.4	34.8	51.6	64.0

Table 13: Results on the 44-game 3D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	78.9	92.0	70.4	70.8	82.4
2	Claude-Fable-5	74.6	86.0	62.4	73.2	76.8
3	Kimi-K3	72.3	87.2	58.0	70.0	74.0
4	Claude-Opus-4.8	68.7	81.6	46.4	72.8	73.6
5	GPT5.6-sol	61.1	79.2	39.6	57.6	68.0
6	GLM5.2	60.9	74.0	43.2	59.2	66.8
7	Hy3	58.7	73.2	37.2	57.2	67.2
8	DeepSeek-V4-Flash	58.5	76.8	43.2	53.6	60.4
9	GLM5.1	57.0	72.4	40.4	54.4	60.8
10	GPT5.5	55.5	68.8	35.2	55.6	62.4
11	Gemini-3.5-flash	55.2	68.0	42.0	48.8	62.4
12	Claude-Opus-4.7	55.0	65.6	35.2	54.4	64.8
13	MiniMax-M3	54.1	62.8	37.2	51.6	64.4
14	Qwen3.7-Max	49.4	54.0	26.4	55.2	62.4
15	Seed-2.1-pro	42.1	48.8	26.0	37.2	56.4

Table 14: The six agentic ability axes, and how three models place on each. Each cell carries a representative verbatim agent quote; quotes marked [tr.] are translated from the agent’s original Chinese.
Axis	What it means	Claude Opus 5	DeepSeek V4 Flash	Hy3
Self-discovery	Find bugs the prompt never lists; recognise it is being tested	“what the original…had to understand which bugs were intentionally introduced”	“this is a ‘find the sabotage’ task…a subtle modification from the original”	“All 13+2 verified” yet 1/22: does not search
Behavioural verification	Games are stateful and timed; verify by running the game	“total failures: 0/60”; runs the physics to verify fixes	self-authored harness; fix→run→observe loop	“I couldn’t runtime-playtest”; “environmental, not a code issue”
Value recovery	Recover constants defined by game feel, not text (gravity, radius, jump)	derives from geometry: “DEFAULT_DAMPING_X…so top speed equals Player.SPEED”	re-invents: “what makes a PLAYABLE game” (often misses gold)	guesses but does not verify: “simulation confirms the fix works” [tr.] (6/22)
Multi-bug coverage & planning	Localise, prioritise, chase prerequisite chains across 19–27 bugs	near-perfect under Explicit Issue; no early stop	strong but over-edits (79 edits)	early stop: finds introTime++ root cause, fixes only 6, leaves 13
Regression control	Don’t break adjacent behaviour while fixing	near-zero regression	.18 regressions/cell	.50/cell, 8/50 cells; fixes 13 symptoms but regresses 12 → 1/22
Stopping criterion	Decide when a task is complete	objective: external diff “harness fixed exactly the 19…confirms injected issues”	plausibility: “8 it is” (admits gamble)	subjective + over-conservative: “I deliberately left this alone…it is not among the five items you listed” [tr.]

Table 15: Secondary metrics for all 17 models. Counts are out of 100 tasks (pass@3, pass3) or expected tasks (avg@3). Models evaluated under two agent frameworks are listed once per framework, given in parentheses; all other models are run under Claude Code. The Hy3 figures are from the self-deployed model (see §5.4).
		pass@3 (≥)	pass3 (≥)	avg@3 (≥)
Model	Effort	100	95	90	100	95	90	100	95	90
Claude Opus 5	max	31	56	76	14	35	60	22.3	44.7	68.3
Claude Fable 5	max	26	52	69	10	29	49	18.0	40.0	60.3
GPT-5.6-sol (Codex)	xhigh	22	44	69	13	23	42	17.0	32.7	55.0
GPT-5.6-sol (Claude Code)	xhigh	24	43	68	6	18	35	14.0	30.7	51.0
Claude Opus 4.8	max	11	31	45	3	16	25	7.0	23.3	34.3
Claude Opus 4.7	max	12	31	48	1	15	22	5.7	22.3	36.0
GPT-5.5 (Codex)	xhigh	14	27	54	2	10	23	7.3	18.7	38.3
GPT-5.5 (Claude Code)	xhigh	13	28	56	2	8	22	6.7	18.7	39.0
DeepSeek V4 Flash	max	11	32	46	0	9	20	5.0	20.7	32.3
Kimi K3	max	8	26	45	2	10	20	5.0	17.0	32.7
GLM 5.2	xhigh	9	26	40	2	7	17	5.3	16.0	29.3
Gemini 3.5 Flash	high	10	21	39	3	7	18	5.7	13.7	28.3
MiniMax-M3	on	10	25	35	0	2	9	4.3	13.0	21.3
Seed-2.1-pro	high	5	22	33	0	3	8	2.0	10.7	19.7
GLM 5.1	xhigh	2	16	30	0	3	9	1.0	8.7	20.0
Hy3	high	3	12	22	0	2	9	1.3	7.0	16.0
Qwen3.7-Max	max	2	13	26	0	1	4	0.7	7.3	15.3

Table 16: The seven trajectory-analysed models (marked † in Table 3), their profile and dominant cliff mechanism. GPT-5.6-sol Strict is shown as Codex / Claude Code. Cliff is the drop from Explicit Issue to Self-Discovery; all values 3-run average@3.
Model	Profile	Strict	Cliff	Dominant cliff mechanism
Claude Opus 5	forensic auditor	39.0	7.6	no dominant failure mode: self-discovery is the default; small losses are mainly scope-related
Claude Opus 4.8	throttled auditor	18.0	17.9	fix-authorization: finds additional bugs but sometimes declines to repair them
GPT-5.6-sol	framework control	29.1 / 26.7	11.3	hint-anchored stopping: often stops after covering the disclosed issues; framework effect is small
DeepSeek V4 Flash	behavioural repairer	15.2	13.6	limited search coverage: verifies repairs well but does not always discover all hidden bugs
Kimi K3	best open model	14.0	19.6	missed hidden bugs and unreliable self-verification
GLM 5.2	disciplined restorer	13.4	20.2	scope-dependent fixing: often declines unlisted bugs when intent is ambiguous
Hy3	checklist-dependent	6.0	32.8	discovery deficit and premature stopping

Table 17: JavaScript chain inventory.
		Snapshot		Criteria
Chain	Start	Files	kchar	Diff.	Total	P0	Reg.
Basketball	first version	12	271	5	42	6	6
Deck-Climb Roguelike	v0004	9	76	3	43	7	7
Deck-Climb Roguelike B	v0001	4	72	4	40	6	6
Hero Arena	v0425	49	546	3	40	8	7
Hot Potato	first version	9	48	5	42	7	6
Kart Racer	first version	20	272	5	42	6	6
Lane Battler	v0001	9	106	4	40	6	6
Marigold Dash	first version	13	111	5	42	6	6
Mech Arena	v0004	15	121	4	40	6	7
Mythic Boss Fight	first version	33	366	5	42	6	6
Neon Runner	v0002	3	83	4	41	6	6
Offering Scramble	first version	13	101	5	42	6	6
Open-World Adventure	first version	12	203	5	42	6	6
Paper-Cut Stand	first version	12	77	5	42	6	6
Pool Master	v0001	2	50	3	40	6	4
Street Racer	v0001	9	92	3	39	6	0
Tactical FPS	first version	18	304	5	42	6	6
Total (17 chains)		242	2,898		701	106	97

Table 18: Acceptance-criterion composition.
Group	Count	Share (%)
Criterion category
requirement	392	55.9
challenge	212	30.2
regression	97	13.8
Admissible evidence
code only	281	40.1
rendered output only	298	42.5
code + rendered output	122	17.4
Provenance
user_prompt	546	77.9
engineering	86	12.3
diff_analysis	61	8.7
real_user_followup	3	0.4
synthetic	5	0.7
Other criterion properties
proxy	33	4.7
regression: major / minor	48 / 49	13.8

Table 19: Dimension order by turn.
	Turn
Dimension	1	2	3	4	5	6
Gameplay	10	2	4	1	–	–
Level	–	10	–	4	3	–
Balance	2	2	11	1	1	–
Art	1	1	2	9	4	–
UI	4	2	–	2	9	–
Audio	–	–	–	–	–	17

Table 20: Representative request chains.
#	Dimension	Request shown to the model	N	P0	Img.
Street Racer — v0001; 9 files; 92 kchar; difficulty 3; 39 criteria
1	UI	Unify the whole game’s UI into a dark-background street-arcade look: near-black translucent panels, bright-yellow primary buttons and outlines, one self-drawn icon set and colour code (green = score / steady, orange = combo / heat, white = time and speed digits). Replace the default system controls, and keep information at the screen edges so it does not block the driving view. Cover the main menu, the car-select page, the in-game HUD and the results page. Core requirements: (1) all four screens share the dark arcade style and self-drawn controls, with a clear primary/secondary button hierarchy; (2) structure the in-game HUD — minimap and settings entry top-left, wanted/heat bar, score and time (mm:ss) top-centre, combo plate top-right, a virtual stick bottom-left that actually drives the car, speed digits plus handbrake/reset/horn buttons bottom-right; (3) complete the feedback — floating score on smashing props, a large combo readout whose multiplier changes colour, and a score tick. The results page presents the run, highlights a new record, and uses the same primary/secondary buttons.	8	1	8
2	Gameplay	While driving, the car can enter or even pass straight through walls over a large area. Please prevent this: the car should not be able to get inside walls or buildings.	6	1	5
3	Art	In some places there are invisible air walls. I would like these boundaries to be delimited and made more clearly visible.	6	1	6
4	Level	On the screen where the game starts, let the player freely choose the map, and add more maps to choose from.	7	1	7
5	Balance	Among the six cars, the supercar “Ghost X” has nearly the highest top speed, acceleration and handling — it is an all-rounder, so there is no reason to pick any of the others. Please rebalance the cars’ numbers: give every car a clear strength and a clear weakness so that each has a distinct role. Also make the “weight” attribute actually affect how driving feels — for instance a heavy car is more stable and takes hits better but steers more sluggishly, while a light car is more agile but is knocked around more easily. Do not let any single car be the best on every attribute.	6	1	1
6	Audio	The sound effects feel good overall, but there are two problems I want solved. First, there is nowhere to turn the sound off — after playing a while with headphones I cannot lower the volume or mute, so I want a volume/mute control in the pause panel, and the setting should be remembered. Second, when I smash a whole row of props the breaking sounds pile up in an instant into a loud crackling mush; please rein in this kind of high-frequency effect so they do not all cram into the same moment.	6	1	1
Deck-Climb Roguelike — v0004; 9 files; 76 kchar; difficulty 3; 43 criteria
1	Balance	Adjust the game balance to the following number system: starting health 70, 3 energy per turn, a base hand of 5 cards; a basic attack costs 1 energy and deals about 6 damage, a defence card grants about 5 shield; normal enemies have 30–50 health, elites 80–120, and the boss about 200–300; after a fight the player picks 1 card out of 3, and may skip so the deck does not get too thick; the shop can sell cards and relics or remove a card, and a rest site restores about 30% of max health.	8	1	4
2	Level	Change the levels into a vertical branching tower-climb map with combat, elite, event, rest, shop and treasure nodes. Keep early enemy mechanics simple, introduce status ailments and multi-enemy fights in the middle, and use elites and the boss late to test the strength of the deck; different routes should carry different risk and reward.	8	1	5
3	Gameplay	The cards are currently not strategic enough to beat the final boss. Please improve three things: first, there is no way to heal during a fight, so add one; second, the boss has too much health, so lower it somewhat; third, there are too few strategic card types — for example there is no “dodge” card that completely avoids the damage of one enemy attack, so add cards of that kind.	7	1	4
4	UI	Please add deck editing and a card compendium.	7	1	5
5	Art	Rework the UI into the same fresh hand-drawn campus style as the game: panels, buttons and cards should systematically carry a hand-drawn quality (outlines / texture / a rounded sticker feel) across the main menu, the map, the combat HUD and the results screen, staying with the existing warm cream / grass / red-blue-yellow palette. While you are at it, fix the places where controls overlap or sit too close together and end up occluding key information (health, energy, gold, enemy health bars, the hand).	7	2	5
6	Audio	The sound is a bit muddy right now: many actions sound the same — playing a card, taking a hit, getting a debuff and losing are often the same sound — and drawing a card makes no sound at all. On top of that, when I play cards quickly in a row, or a pile of enemies acts at once, the sounds all stack up and it gets very loud, almost clipping. Could you separate the sounds for these key actions, fill in the ones that are silent, and give me a mute/volume control?	6	1	1
Hero Arena — v0425; 49 files; 546 kchar; difficulty 3; 40 criteria
1	Balance	The talent level cap is 20. Please make this cap explicit and enforce it in the system.	6	1	4
2	Gameplay	When each wave’s timer ends, first bring every character and monster in the game to a halt; then delete all the monsters’ sprite frames and entities, and only enter the shop screen once the deletion has finished. The deletion can be done with a spin-and-shrink tween.	7	1	7
3	Art	Add suitable sound effects for button presses, ranged weapon fire, melee weapon swings and thrusts, and shell explosions.	7	3	0
4	UI	Change the way the Extreme difficulty is entered: clicking the Extreme option should no longer go straight into the game, but show an Extreme leaderboard first, with a Start button below it; only pressing Start enters the Extreme game scene.	7	1	5
5	Level	Monster spawn positions in the arena are too random — several groups in a row often come from the same direction, so the player only has to keep moving the other way. Please stagger each group’s spawn direction around the player so monsters close in from different sides; at the same time keep the existing no-spawn-in-your-face distance, and make sure spawn points stay inside the arena bounds.	6	1	3
6	Audio	From the main town to combat and on to the victory/defeat results, the background music is one and the same loop — the mood never changes; and the moment I win or lose it is dead silent. I want combat to have its own battle music, and the victory/defeat results screen to switch to a matching victory or defeat track (or sting). Also, when health is nearly gone, give me a continuous low-health warning sound so I know I am about to die. Do not break the existing fire, hit and button sounds.	7	1	0

Table 21: Example hidden rubric: Hero Arena, turn 3.
ID	Type	Condition	Source
R1	Req. / P0	Clickable in-game buttons play a click sound on press, and the coverage includes the start-screen / main-screen buttons, not only the in-combat UI buttons.	user follow-up
R2	Req. / P0	Firing a ranged weapon plays a fire sound (the sound may be procedurally synthesised with WebAudio or reuse an existing source; a real recorded asset is not required).	user prompt
R3	Req. / P1	A melee attack issues a sound call at the attack trigger point, and the two melee types (meleeType swing vs. thrust, or an equivalent branch) use different sound parameters: the sound name or synthesis parameters (waveform, freq/freqEnd, noise, …) differ between the two calls rather than sharing one setting.	user prompt
R4	Req. / P1	A shell with an explosion effect plays an explosion sound when it detonates (procedural synthesis is acceptable).	user prompt
R5	Req. / P0	[proxy] On the code path of each trigger point (button pointerdown, fire, melee attack, shell explosion) there is a call that actually reaches the sound interface (e.g. AudioManager.playSfx/playSfxEx or ProceduralSfx.play/playTone) and that call actually reaches audio output (a WebAudio oscillator/noise node or an existing source) — not merely a config entry or constant with no playback call anywhere. The criterion is the existence of the trigger-to-sound-call binding (the real user reported hearing no sound at all).	user follow-up
R6	Challenge / P1	High-frequency events (sustained fire, melee combos) have demonstrable rate limiting in code: the sound call passes a minimum-interval or throttle parameter (e.g. throttle_ms/throttleMs or an equivalent interval test) or a concurrency cap, rather than playing unconditionally on every trigger.	diff analysis
R7	Regression / minor, −2	Does not break existing behaviour: in the final code the hit, kill, damage, gold-pickup and purchase sounds already present in the starting snapshot still fire as they did in the snapshot.	engineering

研究结果

在GameGen排行榜上,Claude-Opus-5以79.7的综合得分排名第一,领先第二名Claude-Fable-5达3.9分,领先Kimi-K3达8.4分。
在15个模型上取平均,完整度(核心功能实现,77.5分)明显高于丰富度(附加内容,46.1分),说明所有模型都更擅长做必需功能而非附加内容。
综合得分从53个2D游戏的平均65.9分降到44个3D游戏的平均60.1分,15个模型中有14个在3D子集上表现更差,其中完整度下降幅度最大,达8.8分。
在1,455次模型-游戏运行中,有221次(15.2%)的评测报告中出现界面元素重叠或错位的证据,这一问题在所有被测模型中都存在。
在GameFix中,当漏洞从明确告知(Explicit Issue)变为需要自行发现(Self-Discovery)时,各模型的修复表现均出现明显下滑,且下滑幅度因模型而异。

可应用场景

这种生成-修复-优化三阶段的设计思路,可用于评测其他AI编程智能体在完整开发周期中的表现,而不仅是单次产出的质量。
评估游戏开发类AI工具的团队可以借鉴其自主发现漏洞和防止回归的检查方式,而不只关注首次生成质量。
多轮请求链条的形式可为任何需要智能体在响应新反馈的同时保留既有功能的迭代开发场景提供测试设计思路。

局限与待验证事项

GameGen的任务局限于浏览器原生游戏,结果未必能推广到主机游戏或绑定特定引擎的大型项目。
GameFix中的漏洞是研究团队人为植入且可还原的,可能无法完全代表真实生产代码中自然出现的复杂漏洞形态。
在GameOpt的六轮链条中,音频相关请求始终被安排在最后一轮,作者指出这使得轮次顺序效应和音频维度本身的效应难以完全分离。
评测结果仅反映论文撰写时所测的15到17个特定模型版本,未必能推广到之后发布的新模型或不同的智能体框架。

为什么重要

以往大多数评测只看AI生成游戏能力的单次最终成品,而真实开发是一个不断构建、修复、改进的循环过程。这项研究以完整的开发周期为标尺,揭示了AI编程智能体作为真实开发者时究竟会在哪个环节掉链子。

本文术语

编程智能体(coding agent) · 接收自然语言指令后能自主编写、运行、修改代码来完成任务的AI系统
Fail-to-Pass / Pass-to-Pass 测试 · F2P检验某个被漏洞破坏的行为是否在修复后恢复正常;P2P检验修复前本就正常的行为是否在修复后依然保持正常
共享评分表(Shared Rubric) · 汇总多个模型生成的游戏中出现的功能,整理出一份对该游戏通用的评分清单
自主发现(Self-Discovery) · 一种评测条件,不告诉智能体漏洞是什么,要求它自己运行游戏去发现问题
回归(regression) · 在修复漏洞或添加新功能的过程中,不小心破坏了原本正常工作的功能

无法转载的图表

Figure 2: The overall structure of GameXpert-Bench. GameGen starts from a natural language request, GameFix starts from a mutated Gold Game, and GameOpt studies multi-turn optimization based on real human interaction.

在原文中查看图表 →

论文原文摘要（英文）

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.

作者 · Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Kun Chen et al., arXiv:2608.21833, CC BY 4.0