GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

arXiv:2608.218332026-08-25

Coding agents are good at building a game from scratch but struggle to find their own bugs and keep fixes from breaking things

Researchers studied real human-agent game development sessions and found the process splits into three stages: initial generation, bug diagnosis and repair, and multi-turn optimization. They built GameXpert-Bench, a suite with 97 generation tasks, 100 repair tasks, and 17 optimization chains, to test 15 to 17 coding-agent models across these stages by actually running and interacting with the resulting games. Agents were reliably good at producing playable first drafts, but noticeably weaker at discovering hidden defects, verifying runtime behavior, and preserving earlier functionality while making changes.

METAL LAB explanatory visual

How GameXpert-Bench's Three Tracks Work

Evidence statusMeasured results reported

Stage 1: GameGenAgent builds a full game from just a natural-language brief in an empty workspace; 97 tasks across 11 genres
Stage 2: GameFix19-27 bugs are injected into each of 50 human-verified game levels, reported or hidden, for 100 repair tasks
Stage 3: GameOptSix-turn chains of 102 real user-derived requests test whether an agent keeps improving the game without breaking it
Shared scoring principleCredit is given only when the actual running game demonstrates the intended behavior, not just when code looks plausible
Observed asymmetryAcross all three tracks, agents were stronger at initial construction than at self-discovering bugs, verifying runtime behavior, and preserving prior functionality

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The team analyzed complete human-agent game development trajectories and identified three recurring lifecycle stages: initial generation, bug diagnosis and repair, and multi-turn optimization.
They built three matching evaluation tracks (GameGen, GameFix, GameOpt) that grade agents by actually running the resulting game in a browser and interacting with it, not just reading source code.
GameGen gives an agent a single natural-language brief in an empty workspace and asks it to build a complete game from scratch; it spans 97 tasks across 11 genres, including 44 3D games.
GameFix injects 19 to 27 reversible bugs into each of 50 human-verified game levels, testing agents both when bugs are explicitly reported and when they must be discovered on their own, for 100 total repair tasks.
GameOpt replays six-turn request chains (102 requests across 17 chains) seeded from real user-agent development history, checking whether the agent keeps earlier features working while making the requested improvements.

Figure 1: Overview of GameXpert-Bench and its three evaluation tracks. GameGen evaluates game creation from natural-language requirements, GameFix evaluates the repair of corrupted reference games, and GameOpt evaluates multi-turn game optimization based on human feedback.

Table 1: Coverage of representative benchmarks across the three stages studied in this paper. Prior work evaluates several constituent capabilities, while GameXpert-Bench places game creation, verified repair, and cumulative optimization within one benchmark suite. Text labels indicate partial coverage or the source of feedback.
Benchmark	Domain	Creation or implementation	Diagnosis and repair	Iterative refinement
GameDevBench [5]	Game	✓	✗	✗
GameEngineBench [15]	Game	✓	✗	✗
OpenGame-Bench [11]	Game	✓	✗	✗
WebGameBench [24]	Game	✓	✗	✗
GameCraft-Bench [16]	Game	✓	✗	✗
GBQA [10]	Game	✗	Discovery only	✗
PlayCoder [19]	GUI apps	Function-level	Explicit issue repair	Agent refiner
SWE-Together [22]	General software	✗	Explicit issue repair	Session simulator
GameXpert (ours)	Game	✓	Explicit issue +	Human–agent
self-discovery repair	co-development

Figure 3: Artifact growth across complete coding-agent game development trajectories. Filled markers denote artifact-changing requests and colors indicate their lifecycle stage; hollow markers denote continuation requests. Each trajectory retains or increases its source size across stages. Game names are reported as broad categories.

Table 2: Main results on GameGen. All scores are reported out of 100, with the Overall score computed as the average of Completeness, Richness, Player Experience, and Visual Quality. Values are rounded to one decimal place. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	79.7	94.4	72.0	72.4	80.4
2	Claude-Fable-5	75.8	89.2	62.8	74.4	76.8
3	Kimi-K3	71.3	85.2	56.0	70.4	73.6
4	Claude-Opus-4.8	69.9	84.4	49.6	72.0	74.0
5	GPT5.6-sol	63.4	79.6	42.0	63.2	68.8
6	GLM5.2	63.4	76.8	44.8	62.4	69.6
7	DeepSeek-V4-Flash	62.9	81.6	48.4	57.6	64.0
8	Hy3	62.1	77.2	41.6	62.0	68.0
9	Claude-Opus-4.7	62.0	76.0	42.8	61.6	67.6
10	MiniMax-M3	59.6	73.2	40.8	56.4	68.0
11	Gemini-3.5-flash	59.1	73.2	46.0	53.2	64.0
12	GPT5.5	59.0	74.0	36.8	60.4	65.2
13	GLM5.1	58.0	72.8	43.2	56.4	60.0
14	Qwen3.7-Max	54.4	66.4	34.0	55.2	61.6
15	Seed-2.1-pro	48.7	58.4	30.8	44.8	60.8

Figure 4: Functional coverage across all 15 models. Average completeness and richness are reported on [0,10], and event pass rate is multiplied by 10 for comparison. Models follow the main-leaderboard order.

Table 3: GameFix leaderboard, sorted by the primary Strict metric (mean height of the average@3 survival curve over the band τ∈[90,100]). Cliff is the drop in macro average@3 repair score from Explicit Issue to Self-Discovery (we report this difference only, not the two absolute scores; see §5.3). All values are 3-run average@3. In each score column bold marks the best value and underline the worst (Cliff is lower-is-better, so bold is the smallest cliff and underline the largest). † models are analysed at the trajectory level in Appendix B.
#	Model	Harness	Effort	Strict↑	Cliff↓
1	Claude Opus 5†	Claude Code	max	39.0	7.6
2	Claude Fable 5	Claude Code	max	33.7	8.4
3	GPT-5.6-sol†	Codex	xhigh	29.1	11.3
4	GPT-5.6-sol	Claude Code	xhigh	26.7	10.3
5	Claude Opus 4.8†	Claude Code	max	18.0	17.9
6	Claude Opus 4.7	Claude Code	max	17.2	22.2
7	GPT-5.5	Codex	xhigh	16.3	13.3
8	GPT-5.5	Claude Code	xhigh	16.1	10.5
9	DeepSeek V4 Flash†	Claude Code	max	15.2	13.6
10	Kimi K3†	Claude Code	max	14.0	19.6
11	GLM 5.2†	Claude Code	xhigh	13.4	20.2
12	Gemini 3.5 Flash	Claude Code	high	12.2	31.1
13	MiniMax-M3	Claude Code	on	10.3	30.5
14	Seed-2.1-pro	Claude Code	high	8.1	15.3
15	GLM 5.1	Claude Code	xhigh	7.0	21.7
16	Hy3†	Claude Code	high	6.0	32.8
17	Qwen3.7-Max	Claude Code	max	5.5	30.3

Figure 5: Composition of event-level outcomes across all 15 models, including checklist events that are missing from the generated report. Each stacked bar sums to 100%, and models follow the main-leaderboard order.

Table 4: Composition of the current GameOpt evaluation collection.
	Count
Games (chains)	17
Turns	102
Acceptance criteria	701
Requirement	392
Challenge	212
Regression	97

Figure 6: Performance gaps between the 2D and 3D subsets of GameGen. (a) Difference in overall score for each model, computed as the 3D score minus the 2D score. The dashed line marks the cross-model mean gap. (b) Cross-model mean gaps across the four evaluation dimensions. All scores are reported on a [0,100] scale, and negative values indicate lower performance on the 3D subset. Models follow the main-leaderboard order.

Table 5: Overall and six-dimensional GameOpt scores for 15 model variants on 17 JavaScript games. All columns use the pooled discrimination-weighted result summary.
Rank	Model	Overall	Gameplay	Level	Balance	Art	UI	Audio
1	Claude-Opus-5	93.96	91.3	94.5	80.8	100.0	97.0	98.5
2	Claude-Fable-5	89.31	88.7	85.2	80.9	95.8	90.3	96.2
3	Kimi-K3	84.66	86.1	75.9	81.1	91.6	83.6	93.9
4	GPT-5.6-sol	83.68	82.8	89.7	83.2	87.7	67.8	97.0
5	Claude-Opus-4.8	82.53	75.3	78.3	77.8	77.9	95.1	93.5
6	Claude-Opus-4.7	82.36	87.7	83.6	89.4	79.6	66.8	94.8
7	DeepSeek-V4-Flash	78.34	71.9	67.3	70.2	87.2	90.5	83.9
8	GPT-5.5	74.24	70.5	67.0	86.4	66.5	78.7	83.7
9	GLM-5.2	72.98	74.5	68.3	63.1	67.8	79.1	88.3
10	Hy3	68.87	71.1	62.0	58.8	66.4	75.5	82.1
11	Qwen3.7-Max	64.77	67.7	55.6	54.4	65.1	72.0	76.0
12	GLM-5.1	60.67	64.3	49.2	50.1	63.7	68.4	69.8
13	MiniMax-M3	59.87	71.4	49.8	53.9	48.8	70.2	64.7
14	Gemini-3.5-Flash	47.20	52.7	36.6	33.1	53.9	50.4	58.2
15	Seed-2.1-Pro	35.89	46.8	32.6	28.6	28.7	32.8	49.0

Figure 7: Player-facing and runtime failure diagnostics. (a) Share of model–game runs whose evaluation reports contain evidence of UI overlap or misalignment. (b) Share of events recognized as implemented in code but assigned a runtime outcome of Fail or Partial. (c) Attribution of the 2,293 implementation-gap events.

Table 6: Category distribution of the GameGen corpus. The Share column is computed over all 97 games.
Category	2D	3D	Total	Share (%)
Puzzle	5	5	10	10.3
Strategy	5	5	10	10.3
RPG	5	5	10	10.3
Shooter	5	4	9	9.3
Roguelike	3	3	6	6.2
Simulation and Management	5	0	5	5.2
Action	5	5	10	10.3
Narrative	5	3	8	8.2
Idle	5	5	10	10.3
Sandbox and Building	5	5	10	10.3
Card	5	4	9	9.3
Total	53	44	97	100.0

Figure 8: Model-level association between automated functional evaluation and human-perceived quality. Each point represents one of the 15 evaluated models. The automated functional score is the mean of Completeness and Richness, while the human perceptual score is the mean of Player Experience and Visual Quality; all component scores are on a [0,100] scale. The solid line denotes the ordinary least-squares fit, and the shaded region denotes its 95% confidence band for the mean response. GPT5.6-sol and GLM5.2 have identical aggregate coordinates and therefore overlap.

Table 7: Representative full prompts from GameGen (Part I of III).
Category	Dim.	Full Prompt
Puzzle	2D	Create a 2048 game controlled with the arrow keys. Tiles with the same value should merge, and a new tile with value 2 or 4 should be spawned at random after every valid move. Include sliding animations during movement and merging, display the current score, show a victory message when the player reaches 2048, and show a game-over message when no valid move remains.
Puzzle	3D	Create a minimalist, wood-textured 3D maze game viewed from above. The game should support tilting the maze to guide a ball through a wooden map containing walls and an exit. The player tilts the maze to roll the ball, avoids traps and dead ends, and guides the ball to the exit. The level fails if the ball falls out of the maze or time expires. Include at least two basic levels: a single-route maze and a simple branching maze.
Strategy	2D	Create a hand-painted watercolor-style 2D insect-themed real-time strategy game viewed from above. Include an economy based on collecting honeydew and cultivating fungus, an ant queen that produces workers with different roles, and switching between surface and underground maps. The player controls an ant colony, expands an underground nest and its tunnels, sends worker ants to collect food, and deploys soldier ants against invading spiders or rival ant colonies.
Strategy	3D	Create a post-apocalyptic science-fiction 3D first-person tower-defense game. Include defensive structures and a weapon-upgrade system. The player builds defenses, upgrades weapon systems, and withstands waves of monsters. Enemies should approach the base along different routes. Include at least two of the following defensive structures: a machine-gun turret, a laser turret, and a freezing turret.
RPG	2D	Create a retro-JRPG-style 2D pixel-art turn-based role-playing game viewed from above. Include companion recruitment and skill-combination systems, with an open map containing both a town and a dungeon. The player can recruit companions, combine skills, and trigger side quests. NPC attitudes should change according to the player’s choices. Include at least three character classes: warrior, mage, and rogue.
RPG	3D	Create a third-person 3D action RPG in an ink-wash wuxia style that fuses ukiyo-e and Chinese ink-painting aesthetics. The environments should include a bamboo forest, a snowy mountain, and a ruined shrine. The core mechanic is a block-and-parry system: a perfect parry should inflict substantial posture damage, and breaking an enemy’s posture should enable a finishing move. Allow the player to switch among three prosthetic tools—shuriken, firecrackers, and an axe—and combine light and heavy attacks into different combos. Enemies should include a samurai general whose AI switches between high and low perilous attacks, and a vengeful spirit surrounded by ghost fire that can spread to and burn the player.
Shooter	2D	Create a 2D top-down aircraft shooter. The player can move the aircraft to dodge enemy bullet patterns, use multiple firing modes and screen-clearing bombs, and fight enemies. Defeating elite enemies should drop firepower upgrades, bombs, and auxiliary weapons, while defeating a boss should award an extra life. Firepower upgrades, extra lives, auxiliary weapons, and bombs must carry over between stages. Their effects and upgrade rules should follow the conventions of the Raiden series.
Shooter	3D	Create a simple cartoon-style 3D first-person shooter set in a small enclosed indoor map. Include firearms and AI-controlled enemies. The player should be able to move, aim, and fire. Enemies should patrol along fixed routes. Include at least two weapons: a pistol and a rifle.

Figure 9: Survival curves of the average@3 pass rate versus the all-solved threshold τ for all 17 models (eight highlighted); each run contributes its own curve and the three are averaged. The shaded band τ∈[90,100] is the region whose mean curve height defines Strict.

Table 8: Representative full prompts from GameGen (Part II of III).
Category	Dim.	Full Prompt
Roguelike	2D	Create a pixel-art 2D top-down dungeon-exploration roguelike with randomly generated rooms, permadeath, and relic collection. Each run should generate a new room layout, enemy distribution, and set of treasures. The character is permanently lost upon death, but soul stones can be carried back to upgrade global talents. Place one boss every three floors. Include at least two classes: warrior and mage.
Roguelike	3D	Create a 3D zombie-survival roguelike set in a ruined city, with randomized weapons, ammunition management, and an infection meter. The player searches through zombie hordes for randomized weapons, including pistols, rifles, and machine guns. After each wave, the player can upgrade a held weapon by increasing reserve ammunition or damage, or heal the character to reduce the infection meter. Zombie bites increase infection; when the meter is full, the player turns into a zombie and the run ends. Include at least two zombie types: normal zombies and sprinting zombies.
Simulation and Management	2D	Create a farm-management game. Customers should arrive automatically to buy crops, while the player plants and harvests produce and hires workers to help with watering and fertilizing. Customers pay coins after completing a purchase. Coins can be used to upgrade the farm, buy new seeds, and unlock animals. Allow the player to change farm decorations dynamically. Crop hybridization should have a random chance of failure, and hired workers should have a fatigue system.
Action	2D	Create a pixel-art 2D side-scrolling action game set in a gothic castle. Include a combo system, dodge rolls, and a special attack. The player can combine light and heavy attacks into combos, and spend energy to unleash a full-screen special attack. Dodge rolls should provide invincibility frames that can avoid incoming attacks. Include at least two weapon types: a sword and a whip.
Action	3D	Create a high-speed 3D character-action hack-and-slash game inspired by Devil May Cry. Its core mechanics should include real-time switching among four combat styles, aerial combos, enemy-step jumps that cancel action recovery, and a chargeable demonic state that changes move properties and adds damage. Levels should contain hidden challenge rooms; clearing one within the time limit should unlock an additional temporary attribute enhancement.
Narrative	2D	Create an urban rule-horror text game. The player must avoid supernatural corruption, inspect blurred and crossed-out clauses on an elevator notice and blood-red writing on office notes, investigate strange shadows in surveillance blind spots, identify false rules mixed among genuine ones, and escape the out-of-control office building safely before midnight.
Narrative	3D	Create a 3D text-driven treasure-hunting game set in an ancient desert city. The vast three-dimensional desert should feel open and desolate. The player traverses dunes and the Gobi, explores three-dimensional ancient-city ruins, deciphers writing on old stone walls, avoids sandstorm hazards, breaks the city’s ancient seals, and searches for a legendary treasure. Include underground-palace mechanism puzzles and sandstorm-survival gameplay.

Figure 10: Geometric meaning of Strict. Left: for the highlighted models, the survival curve over the τ∈[90,100] band; Strict is the mean height of that curve across the band (dashed line), equivalently the shaded area divided by the band width. Right: Strict for all 17 models, sorted descending.

Table 9: Representative full prompts from GameGen (Part III of III).
Category	Dim.	Full Prompt
Idle	2D	Create a retro pixel-art 2D idle game viewed from above and set in a hero camp. Include offline earnings and a skill tree. The player clicks monsters to deal damage and spends coins to purchase units that attack automatically. The hero can upgrade skills. Include at least two automated unit types: archers and mages.
Idle	3D	Create a 3D retro-steampunk idle game built around three core mechanics: a mechanical factory that automatically forges devices, steam-powered automata that automatically leave to work, and a power core that automatically stores and accumulates energy. Offline idling should mass-produce industrial items. Mechanical modification should carry a risk of malfunction; the player can allocate repair fuel to maintain stable operation and trade precision blueprints through a steam-powered merchant guild to build advanced mechanical creations.
Sandbox and Building	2D	Create a pixel-art 2D top-down sandbox-building game with terrain editing, resource gathering, and construction in a procedurally generated infinite world. The player can mine and place multiple block types—soil, stone, wood, and water—and build houses, farms, and defensive structures. Include at least two biomes: grassland and desert.
Sandbox and Building	3D	Create a low-poly 3D first-person sandbox-building game with block placement and removal, a day–night cycle, and hostile creatures on a procedurally generated continent. The player gathers wood and stone and builds houses and castles. Zombies and skeletons should spawn at night. Include at least two biomes: forest and mountain.
Card	2D	Create a dark-fantasy 2D pixel-art roguelike card game viewed from above, with a randomized room-based dungeon map and deckbuilding mechanics. The player can collect cards, construct a deck, and trigger combo effects. Enemies should patrol rooms and attack proactively. Include at least three card types: attack, defense, and utility cards.
Card	3D	Create a 3D medieval knight battle-card game set in retro castle and field environments, with an action-oriented main storyline about a kingdom at war. Allow the player to recruit heavy knights, ranger scouts, and royal mages as combat cards. Include unit-type counters and formation-charge mechanics. The player deploys cards to arrange offensive and defensive formations and engages in card-based duels between lords. Pacifying the conflict should unlock additional chapters of the kingdom campaign.

Figure 11: The Explicit Issue→Self-Discovery Cliff. Left: each model’s macro average@3 under Explicit Issue versus Self-Discovery; a steeper drop is weaker self-discovery. Right (inset): the Cliff shrinks as Strict rises.

Table 10: Completeness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Score increases after a merge	Add the value of the newly merged tile to the score and update the visible score display.
2	An invalid move leaves the state unchanged	If no tile can move or merge in the requested direction, do not spawn a tile or change the score.
3	Game over when no move remains	End the game when the board is full and no adjacent equal-valued tiles can be merged.
4	High-score update and persistence	Update the historical best score and retain it after restarting or refreshing the page.
5	Game start and initialization	Construct the board, reset the score and state, and spawn the initial tiles when the page loads.
6	Equal-valued tiles merge	Merge two adjacent tiles with the same value along the movement direction into one tile with twice the value.
7	A new tile appears after a valid move	After any move that changes the board, spawn a new tile in an empty cell.
8	Reaching 2048 triggers victory	Display a victory state when a tile with value 2048 is created.
9	Restarting begins a fresh game	The new-game control clears the board, resets the current score and overlays, and creates a new initial state.
10	Tiles visibly slide between cells	Use a smooth positional transition during movement rather than an instantaneous jump.

Figure 12: Six-dimensional capability profiles. Rows denote the six optimization dimensions and columns denote models. Each cell reports the final-product rubric score; black outlines mark the best-performing model in each dimension. The right column reports means over all 15 models, while the bottom row reports the leaderboard Overall score.

Table 11: Richness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Continue after reaching 2048	Allow the player to dismiss the victory overlay and continue playing beyond 2048.
2	Touch-swipe controls	Infer a direction from a sufficiently long touch gesture and execute the corresponding move.
3	WASD controls	Map the WASD keys to the same four movement directions as the arrow keys.
4	Spawn and merge tween effects	Animate newly spawned tiles and tiles created by merging with distinct appearance or pop effects.
5	Styles for tiles above 2048	Provide a visually distinct high-tier style for values greater than 2048.
6	Multiple merges in one move	Correctly merge four equal-valued tiles in a row into two doubled tiles during a single move.
7	Pop-in animation for created tiles	Apply a scale or appearance animation to newly spawned or merged tiles.
8	Resume state after reloading	Restore the board, score, and relevant game state from persistent browser storage.
9	Input locking during animation	Ignore additional movement commands while a movement animation is in progress.
10	Merge-bounce animation	Apply a short scaling or bouncing animation to a tile created by merging.
11	On-screen directional controls	Provide clickable directional buttons as an alternative input method.
12	Pointer-drag gesture controls	Detect a pointer drag beyond a distance threshold and move in the corresponding direction.

Table 12: Results on the 53-game 2D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	80.4	96.0	73.2	73.6	78.8
2	Claude-Fable-5	76.8	91.6	63.2	75.6	77.2
3	Claude-Opus-4.8	71.0	86.4	52.0	71.2	74.0
4	Kimi-K3	70.5	84.0	54.4	70.4	73.2
5	Claude-Opus-4.7	67.9	84.8	49.2	67.6	70.0
6	DeepSeek-V4-Flash	66.6	86.0	52.4	60.8	67.2
7	GLM5.2	65.7	79.2	46.4	65.2	72.0
8	GPT5.6-sol	65.2	80.0	44.0	67.2	69.6
9	Hy3	64.9	80.4	44.8	65.6	68.8
10	MiniMax-M3	64.0	82.0	43.6	60.0	70.4
11	Gemini-3.5-flash	62.2	77.6	49.2	56.4	65.6
12	GPT5.5	61.9	78.0	38.0	64.4	67.6
13	GLM5.1	58.9	73.2	45.2	57.6	59.2
14	Qwen3.7-Max	58.5	77.2	40.8	55.2	61.2
15	Seed-2.1-pro	54.2	66.4	34.8	51.6	64.0

Table 13: Results on the 44-game 3D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	78.9	92.0	70.4	70.8	82.4
2	Claude-Fable-5	74.6	86.0	62.4	73.2	76.8
3	Kimi-K3	72.3	87.2	58.0	70.0	74.0
4	Claude-Opus-4.8	68.7	81.6	46.4	72.8	73.6
5	GPT5.6-sol	61.1	79.2	39.6	57.6	68.0
6	GLM5.2	60.9	74.0	43.2	59.2	66.8
7	Hy3	58.7	73.2	37.2	57.2	67.2
8	DeepSeek-V4-Flash	58.5	76.8	43.2	53.6	60.4
9	GLM5.1	57.0	72.4	40.4	54.4	60.8
10	GPT5.5	55.5	68.8	35.2	55.6	62.4
11	Gemini-3.5-flash	55.2	68.0	42.0	48.8	62.4
12	Claude-Opus-4.7	55.0	65.6	35.2	54.4	64.8
13	MiniMax-M3	54.1	62.8	37.2	51.6	64.4
14	Qwen3.7-Max	49.4	54.0	26.4	55.2	62.4
15	Seed-2.1-pro	42.1	48.8	26.0	37.2	56.4

Table 14: The six agentic ability axes, and how three models place on each. Each cell carries a representative verbatim agent quote; quotes marked [tr.] are translated from the agent’s original Chinese.
Axis	What it means	Claude Opus 5	DeepSeek V4 Flash	Hy3
Self-discovery	Find bugs the prompt never lists; recognise it is being tested	“what the original…had to understand which bugs were intentionally introduced”	“this is a ‘find the sabotage’ task…a subtle modification from the original”	“All 13+2 verified” yet 1/22: does not search
Behavioural verification	Games are stateful and timed; verify by running the game	“total failures: 0/60”; runs the physics to verify fixes	self-authored harness; fix→run→observe loop	“I couldn’t runtime-playtest”; “environmental, not a code issue”
Value recovery	Recover constants defined by game feel, not text (gravity, radius, jump)	derives from geometry: “DEFAULT_DAMPING_X…so top speed equals Player.SPEED”	re-invents: “what makes a PLAYABLE game” (often misses gold)	guesses but does not verify: “simulation confirms the fix works” [tr.] (6/22)
Multi-bug coverage & planning	Localise, prioritise, chase prerequisite chains across 19–27 bugs	near-perfect under Explicit Issue; no early stop	strong but over-edits (79 edits)	early stop: finds introTime++ root cause, fixes only 6, leaves 13
Regression control	Don’t break adjacent behaviour while fixing	near-zero regression	.18 regressions/cell	.50/cell, 8/50 cells; fixes 13 symptoms but regresses 12 → 1/22
Stopping criterion	Decide when a task is complete	objective: external diff “harness fixed exactly the 19…confirms injected issues”	plausibility: “8 it is” (admits gamble)	subjective + over-conservative: “I deliberately left this alone…it is not among the five items you listed” [tr.]

Table 15: Secondary metrics for all 17 models. Counts are out of 100 tasks (pass@3, pass3) or expected tasks (avg@3). Models evaluated under two agent frameworks are listed once per framework, given in parentheses; all other models are run under Claude Code. The Hy3 figures are from the self-deployed model (see §5.4).
		pass@3 (≥)	pass3 (≥)	avg@3 (≥)
Model	Effort	100	95	90	100	95	90	100	95	90
Claude Opus 5	max	31	56	76	14	35	60	22.3	44.7	68.3
Claude Fable 5	max	26	52	69	10	29	49	18.0	40.0	60.3
GPT-5.6-sol (Codex)	xhigh	22	44	69	13	23	42	17.0	32.7	55.0
GPT-5.6-sol (Claude Code)	xhigh	24	43	68	6	18	35	14.0	30.7	51.0
Claude Opus 4.8	max	11	31	45	3	16	25	7.0	23.3	34.3
Claude Opus 4.7	max	12	31	48	1	15	22	5.7	22.3	36.0
GPT-5.5 (Codex)	xhigh	14	27	54	2	10	23	7.3	18.7	38.3
GPT-5.5 (Claude Code)	xhigh	13	28	56	2	8	22	6.7	18.7	39.0
DeepSeek V4 Flash	max	11	32	46	0	9	20	5.0	20.7	32.3
Kimi K3	max	8	26	45	2	10	20	5.0	17.0	32.7
GLM 5.2	xhigh	9	26	40	2	7	17	5.3	16.0	29.3
Gemini 3.5 Flash	high	10	21	39	3	7	18	5.7	13.7	28.3
MiniMax-M3	on	10	25	35	0	2	9	4.3	13.0	21.3
Seed-2.1-pro	high	5	22	33	0	3	8	2.0	10.7	19.7
GLM 5.1	xhigh	2	16	30	0	3	9	1.0	8.7	20.0
Hy3	high	3	12	22	0	2	9	1.3	7.0	16.0
Qwen3.7-Max	max	2	13	26	0	1	4	0.7	7.3	15.3

Table 16: The seven trajectory-analysed models (marked † in Table 3), their profile and dominant cliff mechanism. GPT-5.6-sol Strict is shown as Codex / Claude Code. Cliff is the drop from Explicit Issue to Self-Discovery; all values 3-run average@3.
Model	Profile	Strict	Cliff	Dominant cliff mechanism
Claude Opus 5	forensic auditor	39.0	7.6	no dominant failure mode: self-discovery is the default; small losses are mainly scope-related
Claude Opus 4.8	throttled auditor	18.0	17.9	fix-authorization: finds additional bugs but sometimes declines to repair them
GPT-5.6-sol	framework control	29.1 / 26.7	11.3	hint-anchored stopping: often stops after covering the disclosed issues; framework effect is small
DeepSeek V4 Flash	behavioural repairer	15.2	13.6	limited search coverage: verifies repairs well but does not always discover all hidden bugs
Kimi K3	best open model	14.0	19.6	missed hidden bugs and unreliable self-verification
GLM 5.2	disciplined restorer	13.4	20.2	scope-dependent fixing: often declines unlisted bugs when intent is ambiguous
Hy3	checklist-dependent	6.0	32.8	discovery deficit and premature stopping

Table 17: JavaScript chain inventory.
		Snapshot		Criteria
Chain	Start	Files	kchar	Diff.	Total	P0	Reg.
Basketball	first version	12	271	5	42	6	6
Deck-Climb Roguelike	v0004	9	76	3	43	7	7
Deck-Climb Roguelike B	v0001	4	72	4	40	6	6
Hero Arena	v0425	49	546	3	40	8	7
Hot Potato	first version	9	48	5	42	7	6
Kart Racer	first version	20	272	5	42	6	6
Lane Battler	v0001	9	106	4	40	6	6
Marigold Dash	first version	13	111	5	42	6	6
Mech Arena	v0004	15	121	4	40	6	7
Mythic Boss Fight	first version	33	366	5	42	6	6
Neon Runner	v0002	3	83	4	41	6	6
Offering Scramble	first version	13	101	5	42	6	6
Open-World Adventure	first version	12	203	5	42	6	6
Paper-Cut Stand	first version	12	77	5	42	6	6
Pool Master	v0001	2	50	3	40	6	4
Street Racer	v0001	9	92	3	39	6	0
Tactical FPS	first version	18	304	5	42	6	6
Total (17 chains)		242	2,898		701	106	97

Table 18: Acceptance-criterion composition.
Group	Count	Share (%)
Criterion category
requirement	392	55.9
challenge	212	30.2
regression	97	13.8
Admissible evidence
code only	281	40.1
rendered output only	298	42.5
code + rendered output	122	17.4
Provenance
user_prompt	546	77.9
engineering	86	12.3
diff_analysis	61	8.7
real_user_followup	3	0.4
synthetic	5	0.7
Other criterion properties
proxy	33	4.7
regression: major / minor	48 / 49	13.8

Table 19: Dimension order by turn.
	Turn
Dimension	1	2	3	4	5	6
Gameplay	10	2	4	1	–	–
Level	–	10	–	4	3	–
Balance	2	2	11	1	1	–
Art	1	1	2	9	4	–
UI	4	2	–	2	9	–
Audio	–	–	–	–	–	17

Table 20: Representative request chains.
#	Dimension	Request shown to the model	N	P0	Img.
Street Racer — v0001; 9 files; 92 kchar; difficulty 3; 39 criteria
1	UI	Unify the whole game’s UI into a dark-background street-arcade look: near-black translucent panels, bright-yellow primary buttons and outlines, one self-drawn icon set and colour code (green = score / steady, orange = combo / heat, white = time and speed digits). Replace the default system controls, and keep information at the screen edges so it does not block the driving view. Cover the main menu, the car-select page, the in-game HUD and the results page. Core requirements: (1) all four screens share the dark arcade style and self-drawn controls, with a clear primary/secondary button hierarchy; (2) structure the in-game HUD — minimap and settings entry top-left, wanted/heat bar, score and time (mm:ss) top-centre, combo plate top-right, a virtual stick bottom-left that actually drives the car, speed digits plus handbrake/reset/horn buttons bottom-right; (3) complete the feedback — floating score on smashing props, a large combo readout whose multiplier changes colour, and a score tick. The results page presents the run, highlights a new record, and uses the same primary/secondary buttons.	8	1	8
2	Gameplay	While driving, the car can enter or even pass straight through walls over a large area. Please prevent this: the car should not be able to get inside walls or buildings.	6	1	5
3	Art	In some places there are invisible air walls. I would like these boundaries to be delimited and made more clearly visible.	6	1	6
4	Level	On the screen where the game starts, let the player freely choose the map, and add more maps to choose from.	7	1	7
5	Balance	Among the six cars, the supercar “Ghost X” has nearly the highest top speed, acceleration and handling — it is an all-rounder, so there is no reason to pick any of the others. Please rebalance the cars’ numbers: give every car a clear strength and a clear weakness so that each has a distinct role. Also make the “weight” attribute actually affect how driving feels — for instance a heavy car is more stable and takes hits better but steers more sluggishly, while a light car is more agile but is knocked around more easily. Do not let any single car be the best on every attribute.	6	1	1
6	Audio	The sound effects feel good overall, but there are two problems I want solved. First, there is nowhere to turn the sound off — after playing a while with headphones I cannot lower the volume or mute, so I want a volume/mute control in the pause panel, and the setting should be remembered. Second, when I smash a whole row of props the breaking sounds pile up in an instant into a loud crackling mush; please rein in this kind of high-frequency effect so they do not all cram into the same moment.	6	1	1
Deck-Climb Roguelike — v0004; 9 files; 76 kchar; difficulty 3; 43 criteria
1	Balance	Adjust the game balance to the following number system: starting health 70, 3 energy per turn, a base hand of 5 cards; a basic attack costs 1 energy and deals about 6 damage, a defence card grants about 5 shield; normal enemies have 30–50 health, elites 80–120, and the boss about 200–300; after a fight the player picks 1 card out of 3, and may skip so the deck does not get too thick; the shop can sell cards and relics or remove a card, and a rest site restores about 30% of max health.	8	1	4
2	Level	Change the levels into a vertical branching tower-climb map with combat, elite, event, rest, shop and treasure nodes. Keep early enemy mechanics simple, introduce status ailments and multi-enemy fights in the middle, and use elites and the boss late to test the strength of the deck; different routes should carry different risk and reward.	8	1	5
3	Gameplay	The cards are currently not strategic enough to beat the final boss. Please improve three things: first, there is no way to heal during a fight, so add one; second, the boss has too much health, so lower it somewhat; third, there are too few strategic card types — for example there is no “dodge” card that completely avoids the damage of one enemy attack, so add cards of that kind.	7	1	4
4	UI	Please add deck editing and a card compendium.	7	1	5
5	Art	Rework the UI into the same fresh hand-drawn campus style as the game: panels, buttons and cards should systematically carry a hand-drawn quality (outlines / texture / a rounded sticker feel) across the main menu, the map, the combat HUD and the results screen, staying with the existing warm cream / grass / red-blue-yellow palette. While you are at it, fix the places where controls overlap or sit too close together and end up occluding key information (health, energy, gold, enemy health bars, the hand).	7	2	5
6	Audio	The sound is a bit muddy right now: many actions sound the same — playing a card, taking a hit, getting a debuff and losing are often the same sound — and drawing a card makes no sound at all. On top of that, when I play cards quickly in a row, or a pile of enemies acts at once, the sounds all stack up and it gets very loud, almost clipping. Could you separate the sounds for these key actions, fill in the ones that are silent, and give me a mute/volume control?	6	1	1
Hero Arena — v0425; 49 files; 546 kchar; difficulty 3; 40 criteria
1	Balance	The talent level cap is 20. Please make this cap explicit and enforce it in the system.	6	1	4
2	Gameplay	When each wave’s timer ends, first bring every character and monster in the game to a halt; then delete all the monsters’ sprite frames and entities, and only enter the shop screen once the deletion has finished. The deletion can be done with a spin-and-shrink tween.	7	1	7
3	Art	Add suitable sound effects for button presses, ranged weapon fire, melee weapon swings and thrusts, and shell explosions.	7	3	0
4	UI	Change the way the Extreme difficulty is entered: clicking the Extreme option should no longer go straight into the game, but show an Extreme leaderboard first, with a Start button below it; only pressing Start enters the Extreme game scene.	7	1	5
5	Level	Monster spawn positions in the arena are too random — several groups in a row often come from the same direction, so the player only has to keep moving the other way. Please stagger each group’s spawn direction around the player so monsters close in from different sides; at the same time keep the existing no-spawn-in-your-face distance, and make sure spawn points stay inside the arena bounds.	6	1	3
6	Audio	From the main town to combat and on to the victory/defeat results, the background music is one and the same loop — the mood never changes; and the moment I win or lose it is dead silent. I want combat to have its own battle music, and the victory/defeat results screen to switch to a matching victory or defeat track (or sting). Also, when health is nearly gone, give me a continuous low-health warning sound so I know I am about to die. Do not break the existing fire, hit and button sounds.	7	1	0

Table 21: Example hidden rubric: Hero Arena, turn 3.
ID	Type	Condition	Source
R1	Req. / P0	Clickable in-game buttons play a click sound on press, and the coverage includes the start-screen / main-screen buttons, not only the in-combat UI buttons.	user follow-up
R2	Req. / P0	Firing a ranged weapon plays a fire sound (the sound may be procedurally synthesised with WebAudio or reuse an existing source; a real recorded asset is not required).	user prompt
R3	Req. / P1	A melee attack issues a sound call at the attack trigger point, and the two melee types (meleeType swing vs. thrust, or an equivalent branch) use different sound parameters: the sound name or synthesis parameters (waveform, freq/freqEnd, noise, …) differ between the two calls rather than sharing one setting.	user prompt
R4	Req. / P1	A shell with an explosion effect plays an explosion sound when it detonates (procedural synthesis is acceptable).	user prompt
R5	Req. / P0	[proxy] On the code path of each trigger point (button pointerdown, fire, melee attack, shell explosion) there is a call that actually reaches the sound interface (e.g. AudioManager.playSfx/playSfxEx or ProceduralSfx.play/playTone) and that call actually reaches audio output (a WebAudio oscillator/noise node or an existing source) — not merely a config entry or constant with no playback call anywhere. The criterion is the existence of the trigger-to-sound-call binding (the real user reported hearing no sound at all).	user follow-up
R6	Challenge / P1	High-frequency events (sustained fire, melee combos) have demonstrable rate limiting in code: the sound call passes a minimum-interval or throttle parameter (e.g. throttle_ms/throttleMs or an equivalent interval test) or a concurrency cap, rather than playing unconditionally on every trigger.	diff analysis
R7	Regression / minor, −2	Does not break existing behaviour: in the final code the hit, kill, damage, gold-pickup and purchase sounds already present in the starting snapshot still fire as they did in the snapshot.	engineering

Findings

On the GameGen leaderboard, Claude-Opus-5 ranked first with an overall score of 79.7, ahead of Claude-Fable-5 by 3.9 points and Kimi-K3 by 8.4 points.
Averaged across 15 models, Completeness (core feature implementation, 77.5) was substantially higher than Richness (bonus content, 46.1), meaning every model was better at required features than at extras.
Overall scores dropped from 65.9 on 53 2D games to 60.1 on 44 3D games, with 14 of 15 models performing worse on 3D, and Completeness dropping the most (-8.8 points).
221 of 1,455 model-game runs (15.2%) showed evidence of UI overlap or misalignment in the evaluation reports, a problem present across every evaluated model.
In GameFix, performance dropped sharply from the Explicit Issue condition (bugs described) to the Self-Discovery condition (bugs hidden), with the size of this drop varying by model.

Where it can be used

The three-track design offers a template for evaluating AI coding agents across a full generation-repair-optimization cycle rather than a single output snapshot.
Teams evaluating game-development AI tools could adopt similar checks for self-discovered bug finding and regression avoidance, not just initial build quality.
The multi-turn request-chain format could inform test design for any iterative development workflow where an agent must preserve earlier features while responding to new feedback.

Limits and open work

GameGen tasks are limited to browser-native games, so results may not generalize to console titles or large projects locked to a specific game engine.
GameFix bugs are deliberately and reversibly injected by the researchers, which may not fully represent the messiness of naturally occurring bugs in real production code.
In GameOpt's six-turn chains, the audio-related request is always placed in the final turn, so the authors note that turn-order effects and the audio dimension itself cannot be fully separated.
Results reflect only the 15 to 17 specific model versions evaluated at the time of the paper and may not generalize to newer models or different agent frameworks.

Why it matters

Most prior benchmarks judge game-generation AI only by a single final output, but real development is an ongoing cycle of building, fixing, and improving. This benchmark measures that whole cycle, exposing where coding agents actually break down as real-world developers rather than just at the first demo.

Terms in this paper

coding agent · An AI system that takes natural-language instructions and autonomously writes, runs, and edits code to complete a task
Fail-to-Pass / Pass-to-Pass tests · F2P checks that a broken behavior fails before repair and passes after; P2P checks that behavior that was already correct keeps passing after repair
Shared Rubric · A checklist built by pooling the game features observed across all tested models, used as a common scoring standard for that game
Self-Discovery · An evaluation condition where the agent is not told what bugs exist and must find them by playing the game itself
regression · When a fix or new feature accidentally breaks something that used to work correctly

Figures we cannot republish

Figure 2: The overall structure of GameXpert-Bench. GameGen starts from a natural language request, GameFix starts from a mutated Gold Game, and GameOpt studies multi-turn optimization based on real human interaction.

See the figures in the original paper →

Original abstract (English)

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.

Authors · Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Kun Chen et al., arXiv:2608.21833, CC BY 4.0