AI 코딩 에이전트에게 게임 하나를 처음부터 만들게 하고, 고쳐보게 하고, 여러 번 개선까지 시켜본 결과 '만들기'는 잘해도 '고치기'와 '유지하기'는 서툴렀다

arXiv:2608.218332026-08-25

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

연구팀은 사람과 AI 코딩 에이전트가 실제로 게임을 만든 과정을 분석해, 게임 개발이 초기 제작, 버그 진단·수리, 다회차 최적화라는 세 단계로 이루어진다는 것을 확인했다. 이를 바탕으로 GameXpert-Bench라는 평가 도구를 만들어 97개 게임 제작 과제, 100개 수리 과제, 17개 연속 개선 과제로 15~17개 AI 모델을 시험했다. 결과적으로 AI들은 처음부터 그럴듯하게 돌아가는 게임을 만드는 데는 능숙했지만, 숨겨진 버그를 스스로 찾아내고 실행 중 동작을 검증하며 이전 기능을 망가뜨리지 않고 유지하는 데는 훨씬 취약했다.

METAL LAB 해설 도표

GameXpert-Bench 세 트랙의 흐름

증거 상태측정 결과가 보고됨

1단계 GameGen빈 작업공간과 자연어 한 줄 설명만으로 완전한 게임을 만들게 함, 97개 과제·11개 장르
2단계 GameFix사람이 검증한 정상 게임 50개에 19~27개 버그를 심고, 알려주거나 숨긴 채 100개 과제로 수리시킴
3단계 GameOpt실제 사용자-에이전트 개발 기록을 바탕으로 6턴 102개 요청을 순서대로 던져 지속 개선 능력을 측정
공통 채점 원리정적 코드 확인이 아니라 실제 게임을 실행·조작해 의도한 동작이 나오는지 확인해야 점수를 줌
관찰된 비대칭세 트랙 모두에서 초기 생성은 상대적으로 강하지만 버그 자가 발견, 실행 검증, 기능 보존은 취약하게 나타남

METAL LAB이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

연구팀은 사람과 AI 에이전트가 실제로 주고받은 게임 개발 대화 기록을 분석해, 개발 과정이 초기 생성, 버그 진단·수리, 다회차 최적화라는 세 단계로 나뉜다는 것을 확인했다.
이 세 단계를 각각 측정하는 세 개의 평가 트랙(GameGen, GameFix, GameOpt)을 설계해, 브라우저에서 실제로 게임을 실행시켜 클릭·조작해 보며 채점하는 방식을 채택했다.
GameGen은 빈 작업공간에서 자연어 설명 한 줄만 주고 완전한 게임을 만들게 하는 시험으로, 11개 장르에 걸친 97개 과제(3D 44개 포함)로 구성됐다.
GameFix는 사람이 검증한 정상 게임 50개에 19~27개의 버그를 의도적으로 심어 놓고, 버그를 알려준 경우와 스스로 찾아야 하는 경우로 나눠 100개 수리 과제로 고치게 했다.
GameOpt는 실제 사용자와 에이전트가 주고받은 개발 기록을 바탕으로 6번의 연속 개선 요청(총 102개 요청, 17개 연쇄)을 순서대로 던져, 이전에 만든 기능을 유지하면서 게임을 개선하는지 봤다.

Figure 1: Overview of GameXpert-Bench and its three evaluation tracks. GameGen evaluates game creation from natural-language requirements, GameFix evaluates the repair of corrupted reference games, and GameOpt evaluates multi-turn game optimization based on human feedback.

Table 1: Coverage of representative benchmarks across the three stages studied in this paper. Prior work evaluates several constituent capabilities, while GameXpert-Bench places game creation, verified repair, and cumulative optimization within one benchmark suite. Text labels indicate partial coverage or the source of feedback.
Benchmark	Domain	Creation or implementation	Diagnosis and repair	Iterative refinement
GameDevBench [5]	Game	✓	✗	✗
GameEngineBench [15]	Game	✓	✗	✗
OpenGame-Bench [11]	Game	✓	✗	✗
WebGameBench [24]	Game	✓	✗	✗
GameCraft-Bench [16]	Game	✓	✗	✗
GBQA [10]	Game	✗	Discovery only	✗
PlayCoder [19]	GUI apps	Function-level	Explicit issue repair	Agent refiner
SWE-Together [22]	General software	✗	Explicit issue repair	Session simulator
GameXpert (ours)	Game	✓	Explicit issue +	Human–agent
self-discovery repair	co-development

Figure 3: Artifact growth across complete coding-agent game development trajectories. Filled markers denote artifact-changing requests and colors indicate their lifecycle stage; hollow markers denote continuation requests. Each trajectory retains or increases its source size across stages. Game names are reported as broad categories.

Table 2: Main results on GameGen. All scores are reported out of 100, with the Overall score computed as the average of Completeness, Richness, Player Experience, and Visual Quality. Values are rounded to one decimal place. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	79.7	94.4	72.0	72.4	80.4
2	Claude-Fable-5	75.8	89.2	62.8	74.4	76.8
3	Kimi-K3	71.3	85.2	56.0	70.4	73.6
4	Claude-Opus-4.8	69.9	84.4	49.6	72.0	74.0
5	GPT5.6-sol	63.4	79.6	42.0	63.2	68.8
6	GLM5.2	63.4	76.8	44.8	62.4	69.6
7	DeepSeek-V4-Flash	62.9	81.6	48.4	57.6	64.0
8	Hy3	62.1	77.2	41.6	62.0	68.0
9	Claude-Opus-4.7	62.0	76.0	42.8	61.6	67.6
10	MiniMax-M3	59.6	73.2	40.8	56.4	68.0
11	Gemini-3.5-flash	59.1	73.2	46.0	53.2	64.0
12	GPT5.5	59.0	74.0	36.8	60.4	65.2
13	GLM5.1	58.0	72.8	43.2	56.4	60.0
14	Qwen3.7-Max	54.4	66.4	34.0	55.2	61.6
15	Seed-2.1-pro	48.7	58.4	30.8	44.8	60.8

Figure 4: Functional coverage across all 15 models. Average completeness and richness are reported on [0,10], and event pass rate is multiplied by 10 for comparison. Models follow the main-leaderboard order.

Table 3: GameFix leaderboard, sorted by the primary Strict metric (mean height of the average@3 survival curve over the band τ∈[90,100]). Cliff is the drop in macro average@3 repair score from Explicit Issue to Self-Discovery (we report this difference only, not the two absolute scores; see §5.3). All values are 3-run average@3. In each score column bold marks the best value and underline the worst (Cliff is lower-is-better, so bold is the smallest cliff and underline the largest). † models are analysed at the trajectory level in Appendix B.
#	Model	Harness	Effort	Strict↑	Cliff↓
1	Claude Opus 5†	Claude Code	max	39.0	7.6
2	Claude Fable 5	Claude Code	max	33.7	8.4
3	GPT-5.6-sol†	Codex	xhigh	29.1	11.3
4	GPT-5.6-sol	Claude Code	xhigh	26.7	10.3
5	Claude Opus 4.8†	Claude Code	max	18.0	17.9
6	Claude Opus 4.7	Claude Code	max	17.2	22.2
7	GPT-5.5	Codex	xhigh	16.3	13.3
8	GPT-5.5	Claude Code	xhigh	16.1	10.5
9	DeepSeek V4 Flash†	Claude Code	max	15.2	13.6
10	Kimi K3†	Claude Code	max	14.0	19.6
11	GLM 5.2†	Claude Code	xhigh	13.4	20.2
12	Gemini 3.5 Flash	Claude Code	high	12.2	31.1
13	MiniMax-M3	Claude Code	on	10.3	30.5
14	Seed-2.1-pro	Claude Code	high	8.1	15.3
15	GLM 5.1	Claude Code	xhigh	7.0	21.7
16	Hy3†	Claude Code	high	6.0	32.8
17	Qwen3.7-Max	Claude Code	max	5.5	30.3

Figure 5: Composition of event-level outcomes across all 15 models, including checklist events that are missing from the generated report. Each stacked bar sums to 100%, and models follow the main-leaderboard order.

Table 4: Composition of the current GameOpt evaluation collection.
	Count
Games (chains)	17
Turns	102
Acceptance criteria	701
Requirement	392
Challenge	212
Regression	97

Figure 6: Performance gaps between the 2D and 3D subsets of GameGen. (a) Difference in overall score for each model, computed as the 3D score minus the 2D score. The dashed line marks the cross-model mean gap. (b) Cross-model mean gaps across the four evaluation dimensions. All scores are reported on a [0,100] scale, and negative values indicate lower performance on the 3D subset. Models follow the main-leaderboard order.

Table 5: Overall and six-dimensional GameOpt scores for 15 model variants on 17 JavaScript games. All columns use the pooled discrimination-weighted result summary.
Rank	Model	Overall	Gameplay	Level	Balance	Art	UI	Audio
1	Claude-Opus-5	93.96	91.3	94.5	80.8	100.0	97.0	98.5
2	Claude-Fable-5	89.31	88.7	85.2	80.9	95.8	90.3	96.2
3	Kimi-K3	84.66	86.1	75.9	81.1	91.6	83.6	93.9
4	GPT-5.6-sol	83.68	82.8	89.7	83.2	87.7	67.8	97.0
5	Claude-Opus-4.8	82.53	75.3	78.3	77.8	77.9	95.1	93.5
6	Claude-Opus-4.7	82.36	87.7	83.6	89.4	79.6	66.8	94.8
7	DeepSeek-V4-Flash	78.34	71.9	67.3	70.2	87.2	90.5	83.9
8	GPT-5.5	74.24	70.5	67.0	86.4	66.5	78.7	83.7
9	GLM-5.2	72.98	74.5	68.3	63.1	67.8	79.1	88.3
10	Hy3	68.87	71.1	62.0	58.8	66.4	75.5	82.1
11	Qwen3.7-Max	64.77	67.7	55.6	54.4	65.1	72.0	76.0
12	GLM-5.1	60.67	64.3	49.2	50.1	63.7	68.4	69.8
13	MiniMax-M3	59.87	71.4	49.8	53.9	48.8	70.2	64.7
14	Gemini-3.5-Flash	47.20	52.7	36.6	33.1	53.9	50.4	58.2
15	Seed-2.1-Pro	35.89	46.8	32.6	28.6	28.7	32.8	49.0

Figure 7: Player-facing and runtime failure diagnostics. (a) Share of model–game runs whose evaluation reports contain evidence of UI overlap or misalignment. (b) Share of events recognized as implemented in code but assigned a runtime outcome of Fail or Partial. (c) Attribution of the 2,293 implementation-gap events.

Table 6: Category distribution of the GameGen corpus. The Share column is computed over all 97 games.
Category	2D	3D	Total	Share (%)
Puzzle	5	5	10	10.3
Strategy	5	5	10	10.3
RPG	5	5	10	10.3
Shooter	5	4	9	9.3
Roguelike	3	3	6	6.2
Simulation and Management	5	0	5	5.2
Action	5	5	10	10.3
Narrative	5	3	8	8.2
Idle	5	5	10	10.3
Sandbox and Building	5	5	10	10.3
Card	5	4	9	9.3
Total	53	44	97	100.0

Figure 8: Model-level association between automated functional evaluation and human-perceived quality. Each point represents one of the 15 evaluated models. The automated functional score is the mean of Completeness and Richness, while the human perceptual score is the mean of Player Experience and Visual Quality; all component scores are on a [0,100] scale. The solid line denotes the ordinary least-squares fit, and the shaded region denotes its 95% confidence band for the mean response. GPT5.6-sol and GLM5.2 have identical aggregate coordinates and therefore overlap.

Table 7: Representative full prompts from GameGen (Part I of III).
Category	Dim.	Full Prompt
Puzzle	2D	Create a 2048 game controlled with the arrow keys. Tiles with the same value should merge, and a new tile with value 2 or 4 should be spawned at random after every valid move. Include sliding animations during movement and merging, display the current score, show a victory message when the player reaches 2048, and show a game-over message when no valid move remains.
Puzzle	3D	Create a minimalist, wood-textured 3D maze game viewed from above. The game should support tilting the maze to guide a ball through a wooden map containing walls and an exit. The player tilts the maze to roll the ball, avoids traps and dead ends, and guides the ball to the exit. The level fails if the ball falls out of the maze or time expires. Include at least two basic levels: a single-route maze and a simple branching maze.
Strategy	2D	Create a hand-painted watercolor-style 2D insect-themed real-time strategy game viewed from above. Include an economy based on collecting honeydew and cultivating fungus, an ant queen that produces workers with different roles, and switching between surface and underground maps. The player controls an ant colony, expands an underground nest and its tunnels, sends worker ants to collect food, and deploys soldier ants against invading spiders or rival ant colonies.
Strategy	3D	Create a post-apocalyptic science-fiction 3D first-person tower-defense game. Include defensive structures and a weapon-upgrade system. The player builds defenses, upgrades weapon systems, and withstands waves of monsters. Enemies should approach the base along different routes. Include at least two of the following defensive structures: a machine-gun turret, a laser turret, and a freezing turret.
RPG	2D	Create a retro-JRPG-style 2D pixel-art turn-based role-playing game viewed from above. Include companion recruitment and skill-combination systems, with an open map containing both a town and a dungeon. The player can recruit companions, combine skills, and trigger side quests. NPC attitudes should change according to the player’s choices. Include at least three character classes: warrior, mage, and rogue.
RPG	3D	Create a third-person 3D action RPG in an ink-wash wuxia style that fuses ukiyo-e and Chinese ink-painting aesthetics. The environments should include a bamboo forest, a snowy mountain, and a ruined shrine. The core mechanic is a block-and-parry system: a perfect parry should inflict substantial posture damage, and breaking an enemy’s posture should enable a finishing move. Allow the player to switch among three prosthetic tools—shuriken, firecrackers, and an axe—and combine light and heavy attacks into different combos. Enemies should include a samurai general whose AI switches between high and low perilous attacks, and a vengeful spirit surrounded by ghost fire that can spread to and burn the player.
Shooter	2D	Create a 2D top-down aircraft shooter. The player can move the aircraft to dodge enemy bullet patterns, use multiple firing modes and screen-clearing bombs, and fight enemies. Defeating elite enemies should drop firepower upgrades, bombs, and auxiliary weapons, while defeating a boss should award an extra life. Firepower upgrades, extra lives, auxiliary weapons, and bombs must carry over between stages. Their effects and upgrade rules should follow the conventions of the Raiden series.
Shooter	3D	Create a simple cartoon-style 3D first-person shooter set in a small enclosed indoor map. Include firearms and AI-controlled enemies. The player should be able to move, aim, and fire. Enemies should patrol along fixed routes. Include at least two weapons: a pistol and a rifle.

Figure 9: Survival curves of the average@3 pass rate versus the all-solved threshold τ for all 17 models (eight highlighted); each run contributes its own curve and the three are averaged. The shaded band τ∈[90,100] is the region whose mean curve height defines Strict.

Table 8: Representative full prompts from GameGen (Part II of III).
Category	Dim.	Full Prompt
Roguelike	2D	Create a pixel-art 2D top-down dungeon-exploration roguelike with randomly generated rooms, permadeath, and relic collection. Each run should generate a new room layout, enemy distribution, and set of treasures. The character is permanently lost upon death, but soul stones can be carried back to upgrade global talents. Place one boss every three floors. Include at least two classes: warrior and mage.
Roguelike	3D	Create a 3D zombie-survival roguelike set in a ruined city, with randomized weapons, ammunition management, and an infection meter. The player searches through zombie hordes for randomized weapons, including pistols, rifles, and machine guns. After each wave, the player can upgrade a held weapon by increasing reserve ammunition or damage, or heal the character to reduce the infection meter. Zombie bites increase infection; when the meter is full, the player turns into a zombie and the run ends. Include at least two zombie types: normal zombies and sprinting zombies.
Simulation and Management	2D	Create a farm-management game. Customers should arrive automatically to buy crops, while the player plants and harvests produce and hires workers to help with watering and fertilizing. Customers pay coins after completing a purchase. Coins can be used to upgrade the farm, buy new seeds, and unlock animals. Allow the player to change farm decorations dynamically. Crop hybridization should have a random chance of failure, and hired workers should have a fatigue system.
Action	2D	Create a pixel-art 2D side-scrolling action game set in a gothic castle. Include a combo system, dodge rolls, and a special attack. The player can combine light and heavy attacks into combos, and spend energy to unleash a full-screen special attack. Dodge rolls should provide invincibility frames that can avoid incoming attacks. Include at least two weapon types: a sword and a whip.
Action	3D	Create a high-speed 3D character-action hack-and-slash game inspired by Devil May Cry. Its core mechanics should include real-time switching among four combat styles, aerial combos, enemy-step jumps that cancel action recovery, and a chargeable demonic state that changes move properties and adds damage. Levels should contain hidden challenge rooms; clearing one within the time limit should unlock an additional temporary attribute enhancement.
Narrative	2D	Create an urban rule-horror text game. The player must avoid supernatural corruption, inspect blurred and crossed-out clauses on an elevator notice and blood-red writing on office notes, investigate strange shadows in surveillance blind spots, identify false rules mixed among genuine ones, and escape the out-of-control office building safely before midnight.
Narrative	3D	Create a 3D text-driven treasure-hunting game set in an ancient desert city. The vast three-dimensional desert should feel open and desolate. The player traverses dunes and the Gobi, explores three-dimensional ancient-city ruins, deciphers writing on old stone walls, avoids sandstorm hazards, breaks the city’s ancient seals, and searches for a legendary treasure. Include underground-palace mechanism puzzles and sandstorm-survival gameplay.

Figure 10: Geometric meaning of Strict. Left: for the highlighted models, the survival curve over the τ∈[90,100] band; Strict is the mean height of that curve across the band (dashed line), equivalently the shaded area divided by the band width. Right: Strict for all 17 models, sorted descending.

Table 9: Representative full prompts from GameGen (Part III of III).
Category	Dim.	Full Prompt
Idle	2D	Create a retro pixel-art 2D idle game viewed from above and set in a hero camp. Include offline earnings and a skill tree. The player clicks monsters to deal damage and spends coins to purchase units that attack automatically. The hero can upgrade skills. Include at least two automated unit types: archers and mages.
Idle	3D	Create a 3D retro-steampunk idle game built around three core mechanics: a mechanical factory that automatically forges devices, steam-powered automata that automatically leave to work, and a power core that automatically stores and accumulates energy. Offline idling should mass-produce industrial items. Mechanical modification should carry a risk of malfunction; the player can allocate repair fuel to maintain stable operation and trade precision blueprints through a steam-powered merchant guild to build advanced mechanical creations.
Sandbox and Building	2D	Create a pixel-art 2D top-down sandbox-building game with terrain editing, resource gathering, and construction in a procedurally generated infinite world. The player can mine and place multiple block types—soil, stone, wood, and water—and build houses, farms, and defensive structures. Include at least two biomes: grassland and desert.
Sandbox and Building	3D	Create a low-poly 3D first-person sandbox-building game with block placement and removal, a day–night cycle, and hostile creatures on a procedurally generated continent. The player gathers wood and stone and builds houses and castles. Zombies and skeletons should spawn at night. Include at least two biomes: forest and mountain.
Card	2D	Create a dark-fantasy 2D pixel-art roguelike card game viewed from above, with a randomized room-based dungeon map and deckbuilding mechanics. The player can collect cards, construct a deck, and trigger combo effects. Enemies should patrol rooms and attack proactively. Include at least three card types: attack, defense, and utility cards.
Card	3D	Create a 3D medieval knight battle-card game set in retro castle and field environments, with an action-oriented main storyline about a kingdom at war. Allow the player to recruit heavy knights, ranger scouts, and royal mages as combat cards. Include unit-type counters and formation-charge mechanics. The player deploys cards to arrange offensive and defensive formations and engages in card-based duels between lords. Pacifying the conflict should unlock additional chapters of the kingdom campaign.

Figure 11: The Explicit Issue→Self-Discovery Cliff. Left: each model’s macro average@3 under Explicit Issue versus Self-Discovery; a steeper drop is weaker self-discovery. Right (inset): the Cliff shrinks as Strict rises.

Table 10: Completeness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Score increases after a merge	Add the value of the newly merged tile to the score and update the visible score display.
2	An invalid move leaves the state unchanged	If no tile can move or merge in the requested direction, do not spawn a tile or change the score.
3	Game over when no move remains	End the game when the board is full and no adjacent equal-valued tiles can be merged.
4	High-score update and persistence	Update the historical best score and retain it after restarting or refreshing the page.
5	Game start and initialization	Construct the board, reset the score and state, and spawn the initial tiles when the page loads.
6	Equal-valued tiles merge	Merge two adjacent tiles with the same value along the movement direction into one tile with twice the value.
7	A new tile appears after a valid move	After any move that changes the board, spawn a new tile in an empty cell.
8	Reaching 2048 triggers victory	Display a victory state when a tile with value 2048 is created.
9	Restarting begins a fresh game	The new-game control clears the board, resets the current score and overlays, and creates a new initial state.
10	Tiles visibly slide between cells	Use a smooth positional transition during movement rather than an instantaneous jump.

Figure 12: Six-dimensional capability profiles. Rows denote the six optimization dimensions and columns denote models. Each cell reports the final-product rubric score; black outlines mark the best-performing model in each dimension. The right column reports means over all 15 models, while the bottom row reports the leaderboard Overall score.

Table 11: Richness checklist for the 2048 task.
#	Checklist Item	Verification Criterion
1	Continue after reaching 2048	Allow the player to dismiss the victory overlay and continue playing beyond 2048.
2	Touch-swipe controls	Infer a direction from a sufficiently long touch gesture and execute the corresponding move.
3	WASD controls	Map the WASD keys to the same four movement directions as the arrow keys.
4	Spawn and merge tween effects	Animate newly spawned tiles and tiles created by merging with distinct appearance or pop effects.
5	Styles for tiles above 2048	Provide a visually distinct high-tier style for values greater than 2048.
6	Multiple merges in one move	Correctly merge four equal-valued tiles in a row into two doubled tiles during a single move.
7	Pop-in animation for created tiles	Apply a scale or appearance animation to newly spawned or merged tiles.
8	Resume state after reloading	Restore the board, score, and relevant game state from persistent browser storage.
9	Input locking during animation	Ignore additional movement commands while a movement animation is in progress.
10	Merge-bounce animation	Apply a short scaling or bouncing animation to a tile created by merging.
11	On-screen directional controls	Provide clickable directional buttons as an alternative input method.
12	Pointer-drag gesture controls	Detect a pointer drag beyond a distance threshold and move in the corresponding direction.

Table 12: Results on the 53-game 2D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	80.4	96.0	73.2	73.6	78.8
2	Claude-Fable-5	76.8	91.6	63.2	75.6	77.2
3	Claude-Opus-4.8	71.0	86.4	52.0	71.2	74.0
4	Kimi-K3	70.5	84.0	54.4	70.4	73.2
5	Claude-Opus-4.7	67.9	84.8	49.2	67.6	70.0
6	DeepSeek-V4-Flash	66.6	86.0	52.4	60.8	67.2
7	GLM5.2	65.7	79.2	46.4	65.2	72.0
8	GPT5.6-sol	65.2	80.0	44.0	67.2	69.6
9	Hy3	64.9	80.4	44.8	65.6	68.8
10	MiniMax-M3	64.0	82.0	43.6	60.0	70.4
11	Gemini-3.5-flash	62.2	77.6	49.2	56.4	65.6
12	GPT5.5	61.9	78.0	38.0	64.4	67.6
13	GLM5.1	58.9	73.2	45.2	57.6	59.2
14	Qwen3.7-Max	58.5	77.2	40.8	55.2	61.2
15	Seed-2.1-pro	54.2	66.4	34.8	51.6	64.0

Table 13: Results on the 44-game 3D subset of GameGen. The best and second-best results in each column are shown in bold and underlined, respectively.
#	Model	Overall	Completeness	Richness	Experience	Visual
1	Claude-Opus-5	78.9	92.0	70.4	70.8	82.4
2	Claude-Fable-5	74.6	86.0	62.4	73.2	76.8
3	Kimi-K3	72.3	87.2	58.0	70.0	74.0
4	Claude-Opus-4.8	68.7	81.6	46.4	72.8	73.6
5	GPT5.6-sol	61.1	79.2	39.6	57.6	68.0
6	GLM5.2	60.9	74.0	43.2	59.2	66.8
7	Hy3	58.7	73.2	37.2	57.2	67.2
8	DeepSeek-V4-Flash	58.5	76.8	43.2	53.6	60.4
9	GLM5.1	57.0	72.4	40.4	54.4	60.8
10	GPT5.5	55.5	68.8	35.2	55.6	62.4
11	Gemini-3.5-flash	55.2	68.0	42.0	48.8	62.4
12	Claude-Opus-4.7	55.0	65.6	35.2	54.4	64.8
13	MiniMax-M3	54.1	62.8	37.2	51.6	64.4
14	Qwen3.7-Max	49.4	54.0	26.4	55.2	62.4
15	Seed-2.1-pro	42.1	48.8	26.0	37.2	56.4

Table 14: The six agentic ability axes, and how three models place on each. Each cell carries a representative verbatim agent quote; quotes marked [tr.] are translated from the agent’s original Chinese.
Axis	What it means	Claude Opus 5	DeepSeek V4 Flash	Hy3
Self-discovery	Find bugs the prompt never lists; recognise it is being tested	“what the original…had to understand which bugs were intentionally introduced”	“this is a ‘find the sabotage’ task…a subtle modification from the original”	“All 13+2 verified” yet 1/22: does not search
Behavioural verification	Games are stateful and timed; verify by running the game	“total failures: 0/60”; runs the physics to verify fixes	self-authored harness; fix→run→observe loop	“I couldn’t runtime-playtest”; “environmental, not a code issue”
Value recovery	Recover constants defined by game feel, not text (gravity, radius, jump)	derives from geometry: “DEFAULT_DAMPING_X…so top speed equals Player.SPEED”	re-invents: “what makes a PLAYABLE game” (often misses gold)	guesses but does not verify: “simulation confirms the fix works” [tr.] (6/22)
Multi-bug coverage & planning	Localise, prioritise, chase prerequisite chains across 19–27 bugs	near-perfect under Explicit Issue; no early stop	strong but over-edits (79 edits)	early stop: finds introTime++ root cause, fixes only 6, leaves 13
Regression control	Don’t break adjacent behaviour while fixing	near-zero regression	.18 regressions/cell	.50/cell, 8/50 cells; fixes 13 symptoms but regresses 12 → 1/22
Stopping criterion	Decide when a task is complete	objective: external diff “harness fixed exactly the 19…confirms injected issues”	plausibility: “8 it is” (admits gamble)	subjective + over-conservative: “I deliberately left this alone…it is not among the five items you listed” [tr.]

Table 15: Secondary metrics for all 17 models. Counts are out of 100 tasks (pass@3, pass3) or expected tasks (avg@3). Models evaluated under two agent frameworks are listed once per framework, given in parentheses; all other models are run under Claude Code. The Hy3 figures are from the self-deployed model (see §5.4).
		pass@3 (≥)	pass3 (≥)	avg@3 (≥)
Model	Effort	100	95	90	100	95	90	100	95	90
Claude Opus 5	max	31	56	76	14	35	60	22.3	44.7	68.3
Claude Fable 5	max	26	52	69	10	29	49	18.0	40.0	60.3
GPT-5.6-sol (Codex)	xhigh	22	44	69	13	23	42	17.0	32.7	55.0
GPT-5.6-sol (Claude Code)	xhigh	24	43	68	6	18	35	14.0	30.7	51.0
Claude Opus 4.8	max	11	31	45	3	16	25	7.0	23.3	34.3
Claude Opus 4.7	max	12	31	48	1	15	22	5.7	22.3	36.0
GPT-5.5 (Codex)	xhigh	14	27	54	2	10	23	7.3	18.7	38.3
GPT-5.5 (Claude Code)	xhigh	13	28	56	2	8	22	6.7	18.7	39.0
DeepSeek V4 Flash	max	11	32	46	0	9	20	5.0	20.7	32.3
Kimi K3	max	8	26	45	2	10	20	5.0	17.0	32.7
GLM 5.2	xhigh	9	26	40	2	7	17	5.3	16.0	29.3
Gemini 3.5 Flash	high	10	21	39	3	7	18	5.7	13.7	28.3
MiniMax-M3	on	10	25	35	0	2	9	4.3	13.0	21.3
Seed-2.1-pro	high	5	22	33	0	3	8	2.0	10.7	19.7
GLM 5.1	xhigh	2	16	30	0	3	9	1.0	8.7	20.0
Hy3	high	3	12	22	0	2	9	1.3	7.0	16.0
Qwen3.7-Max	max	2	13	26	0	1	4	0.7	7.3	15.3

Table 16: The seven trajectory-analysed models (marked † in Table 3), their profile and dominant cliff mechanism. GPT-5.6-sol Strict is shown as Codex / Claude Code. Cliff is the drop from Explicit Issue to Self-Discovery; all values 3-run average@3.
Model	Profile	Strict	Cliff	Dominant cliff mechanism
Claude Opus 5	forensic auditor	39.0	7.6	no dominant failure mode: self-discovery is the default; small losses are mainly scope-related
Claude Opus 4.8	throttled auditor	18.0	17.9	fix-authorization: finds additional bugs but sometimes declines to repair them
GPT-5.6-sol	framework control	29.1 / 26.7	11.3	hint-anchored stopping: often stops after covering the disclosed issues; framework effect is small
DeepSeek V4 Flash	behavioural repairer	15.2	13.6	limited search coverage: verifies repairs well but does not always discover all hidden bugs
Kimi K3	best open model	14.0	19.6	missed hidden bugs and unreliable self-verification
GLM 5.2	disciplined restorer	13.4	20.2	scope-dependent fixing: often declines unlisted bugs when intent is ambiguous
Hy3	checklist-dependent	6.0	32.8	discovery deficit and premature stopping

Table 17: JavaScript chain inventory.
		Snapshot		Criteria
Chain	Start	Files	kchar	Diff.	Total	P0	Reg.
Basketball	first version	12	271	5	42	6	6
Deck-Climb Roguelike	v0004	9	76	3	43	7	7
Deck-Climb Roguelike B	v0001	4	72	4	40	6	6
Hero Arena	v0425	49	546	3	40	8	7
Hot Potato	first version	9	48	5	42	7	6
Kart Racer	first version	20	272	5	42	6	6
Lane Battler	v0001	9	106	4	40	6	6
Marigold Dash	first version	13	111	5	42	6	6
Mech Arena	v0004	15	121	4	40	6	7
Mythic Boss Fight	first version	33	366	5	42	6	6
Neon Runner	v0002	3	83	4	41	6	6
Offering Scramble	first version	13	101	5	42	6	6
Open-World Adventure	first version	12	203	5	42	6	6
Paper-Cut Stand	first version	12	77	5	42	6	6
Pool Master	v0001	2	50	3	40	6	4
Street Racer	v0001	9	92	3	39	6	0
Tactical FPS	first version	18	304	5	42	6	6
Total (17 chains)		242	2,898		701	106	97

Table 18: Acceptance-criterion composition.
Group	Count	Share (%)
Criterion category
requirement	392	55.9
challenge	212	30.2
regression	97	13.8
Admissible evidence
code only	281	40.1
rendered output only	298	42.5
code + rendered output	122	17.4
Provenance
user_prompt	546	77.9
engineering	86	12.3
diff_analysis	61	8.7
real_user_followup	3	0.4
synthetic	5	0.7
Other criterion properties
proxy	33	4.7
regression: major / minor	48 / 49	13.8

Table 19: Dimension order by turn.
	Turn
Dimension	1	2	3	4	5	6
Gameplay	10	2	4	1	–	–
Level	–	10	–	4	3	–
Balance	2	2	11	1	1	–
Art	1	1	2	9	4	–
UI	4	2	–	2	9	–
Audio	–	–	–	–	–	17

Table 20: Representative request chains.
#	Dimension	Request shown to the model	N	P0	Img.
Street Racer — v0001; 9 files; 92 kchar; difficulty 3; 39 criteria
1	UI	Unify the whole game’s UI into a dark-background street-arcade look: near-black translucent panels, bright-yellow primary buttons and outlines, one self-drawn icon set and colour code (green = score / steady, orange = combo / heat, white = time and speed digits). Replace the default system controls, and keep information at the screen edges so it does not block the driving view. Cover the main menu, the car-select page, the in-game HUD and the results page. Core requirements: (1) all four screens share the dark arcade style and self-drawn controls, with a clear primary/secondary button hierarchy; (2) structure the in-game HUD — minimap and settings entry top-left, wanted/heat bar, score and time (mm:ss) top-centre, combo plate top-right, a virtual stick bottom-left that actually drives the car, speed digits plus handbrake/reset/horn buttons bottom-right; (3) complete the feedback — floating score on smashing props, a large combo readout whose multiplier changes colour, and a score tick. The results page presents the run, highlights a new record, and uses the same primary/secondary buttons.	8	1	8
2	Gameplay	While driving, the car can enter or even pass straight through walls over a large area. Please prevent this: the car should not be able to get inside walls or buildings.	6	1	5
3	Art	In some places there are invisible air walls. I would like these boundaries to be delimited and made more clearly visible.	6	1	6
4	Level	On the screen where the game starts, let the player freely choose the map, and add more maps to choose from.	7	1	7
5	Balance	Among the six cars, the supercar “Ghost X” has nearly the highest top speed, acceleration and handling — it is an all-rounder, so there is no reason to pick any of the others. Please rebalance the cars’ numbers: give every car a clear strength and a clear weakness so that each has a distinct role. Also make the “weight” attribute actually affect how driving feels — for instance a heavy car is more stable and takes hits better but steers more sluggishly, while a light car is more agile but is knocked around more easily. Do not let any single car be the best on every attribute.	6	1	1
6	Audio	The sound effects feel good overall, but there are two problems I want solved. First, there is nowhere to turn the sound off — after playing a while with headphones I cannot lower the volume or mute, so I want a volume/mute control in the pause panel, and the setting should be remembered. Second, when I smash a whole row of props the breaking sounds pile up in an instant into a loud crackling mush; please rein in this kind of high-frequency effect so they do not all cram into the same moment.	6	1	1
Deck-Climb Roguelike — v0004; 9 files; 76 kchar; difficulty 3; 43 criteria
1	Balance	Adjust the game balance to the following number system: starting health 70, 3 energy per turn, a base hand of 5 cards; a basic attack costs 1 energy and deals about 6 damage, a defence card grants about 5 shield; normal enemies have 30–50 health, elites 80–120, and the boss about 200–300; after a fight the player picks 1 card out of 3, and may skip so the deck does not get too thick; the shop can sell cards and relics or remove a card, and a rest site restores about 30% of max health.	8	1	4
2	Level	Change the levels into a vertical branching tower-climb map with combat, elite, event, rest, shop and treasure nodes. Keep early enemy mechanics simple, introduce status ailments and multi-enemy fights in the middle, and use elites and the boss late to test the strength of the deck; different routes should carry different risk and reward.	8	1	5
3	Gameplay	The cards are currently not strategic enough to beat the final boss. Please improve three things: first, there is no way to heal during a fight, so add one; second, the boss has too much health, so lower it somewhat; third, there are too few strategic card types — for example there is no “dodge” card that completely avoids the damage of one enemy attack, so add cards of that kind.	7	1	4
4	UI	Please add deck editing and a card compendium.	7	1	5
5	Art	Rework the UI into the same fresh hand-drawn campus style as the game: panels, buttons and cards should systematically carry a hand-drawn quality (outlines / texture / a rounded sticker feel) across the main menu, the map, the combat HUD and the results screen, staying with the existing warm cream / grass / red-blue-yellow palette. While you are at it, fix the places where controls overlap or sit too close together and end up occluding key information (health, energy, gold, enemy health bars, the hand).	7	2	5
6	Audio	The sound is a bit muddy right now: many actions sound the same — playing a card, taking a hit, getting a debuff and losing are often the same sound — and drawing a card makes no sound at all. On top of that, when I play cards quickly in a row, or a pile of enemies acts at once, the sounds all stack up and it gets very loud, almost clipping. Could you separate the sounds for these key actions, fill in the ones that are silent, and give me a mute/volume control?	6	1	1
Hero Arena — v0425; 49 files; 546 kchar; difficulty 3; 40 criteria
1	Balance	The talent level cap is 20. Please make this cap explicit and enforce it in the system.	6	1	4
2	Gameplay	When each wave’s timer ends, first bring every character and monster in the game to a halt; then delete all the monsters’ sprite frames and entities, and only enter the shop screen once the deletion has finished. The deletion can be done with a spin-and-shrink tween.	7	1	7
3	Art	Add suitable sound effects for button presses, ranged weapon fire, melee weapon swings and thrusts, and shell explosions.	7	3	0
4	UI	Change the way the Extreme difficulty is entered: clicking the Extreme option should no longer go straight into the game, but show an Extreme leaderboard first, with a Start button below it; only pressing Start enters the Extreme game scene.	7	1	5
5	Level	Monster spawn positions in the arena are too random — several groups in a row often come from the same direction, so the player only has to keep moving the other way. Please stagger each group’s spawn direction around the player so monsters close in from different sides; at the same time keep the existing no-spawn-in-your-face distance, and make sure spawn points stay inside the arena bounds.	6	1	3
6	Audio	From the main town to combat and on to the victory/defeat results, the background music is one and the same loop — the mood never changes; and the moment I win or lose it is dead silent. I want combat to have its own battle music, and the victory/defeat results screen to switch to a matching victory or defeat track (or sting). Also, when health is nearly gone, give me a continuous low-health warning sound so I know I am about to die. Do not break the existing fire, hit and button sounds.	7	1	0

Table 21: Example hidden rubric: Hero Arena, turn 3.
ID	Type	Condition	Source
R1	Req. / P0	Clickable in-game buttons play a click sound on press, and the coverage includes the start-screen / main-screen buttons, not only the in-combat UI buttons.	user follow-up
R2	Req. / P0	Firing a ranged weapon plays a fire sound (the sound may be procedurally synthesised with WebAudio or reuse an existing source; a real recorded asset is not required).	user prompt
R3	Req. / P1	A melee attack issues a sound call at the attack trigger point, and the two melee types (meleeType swing vs. thrust, or an equivalent branch) use different sound parameters: the sound name or synthesis parameters (waveform, freq/freqEnd, noise, …) differ between the two calls rather than sharing one setting.	user prompt
R4	Req. / P1	A shell with an explosion effect plays an explosion sound when it detonates (procedural synthesis is acceptable).	user prompt
R5	Req. / P0	[proxy] On the code path of each trigger point (button pointerdown, fire, melee attack, shell explosion) there is a call that actually reaches the sound interface (e.g. AudioManager.playSfx/playSfxEx or ProceduralSfx.play/playTone) and that call actually reaches audio output (a WebAudio oscillator/noise node or an existing source) — not merely a config entry or constant with no playback call anywhere. The criterion is the existence of the trigger-to-sound-call binding (the real user reported hearing no sound at all).	user follow-up
R6	Challenge / P1	High-frequency events (sustained fire, melee combos) have demonstrable rate limiting in code: the sound call passes a minimum-interval or throttle parameter (e.g. throttle_ms/throttleMs or an equivalent interval test) or a concurrency cap, rather than playing unconditionally on every trigger.	diff analysis
R7	Regression / minor, −2	Does not break existing behaviour: in the final code the hit, kill, damage, gold-pickup and purchase sounds already present in the starting snapshot still fire as they did in the snapshot.	engineering

실제로 확인된 결과

게임 생성(GameGen) 순위표에서 Claude-Opus-5가 종합 79.7점으로 1위였고, 2위 Claude-Fable-5보다 3.9점, 3위권 Kimi-K3보다 8.4점 앞섰다.
15개 모델 평균으로 완성도(핵심 기능 구현, 77.5점)가 풍부함(부가 기능, 46.1점)보다 훨씬 높아, 모든 모델이 필수 기능은 잘 넣지만 추가 콘텐츠는 덜 넣는 경향이 공통으로 나타났다.
2D 게임(53개) 평균 65.9점 대비 3D 게임(44개) 평균 60.1점으로, 15개 모델 중 14개가 3D에서 더 낮은 점수를 받았고 특히 완성도가 8.8점 떨어졌다.
1,455건의 모델-게임 실행 중 221건(15.2%)에서 UI 요소가 겹치거나 화면을 벗어나는 등 정렬 문제가 보고됐으며, 이는 모든 평가 모델에서 공통으로 나타난 문제였다.
GameFix에서는 버그가 명시적으로 제시됐을 때와 스스로 찾아야 할 때 사이에 성능 격차(Explicit Issue→Self-Discovery Cliff)가 크게 벌어졌고, 격차는 모델별로 차이가 있었다.

어디에 쓸 수 있나

AI 코딩 에이전트가 만든 게임이나 소프트웨어의 완성도를 '한 번의 결과물'만이 아니라 생성-수리-개선의 전체 주기로 나눠 평가하는 벤치마크 설계에 참고할 수 있다.
게임 개발 AI 도구를 도입하기 전에 초기 생성 능력뿐 아니라 버그 자가 발견, 회귀 방지 능력을 별도로 점검하는 QA 체크리스트를 만드는 데 활용할 수 있다.
다회차 사용자 피드백을 반영해야 하는 반복 개발 워크플로에서 AI 에이전트가 이전 기능을 잘 보존하는지 검증하는 테스트 설계 아이디어로 쓸 수 있다.

한계와 남은 검증

GameGen 과제는 브라우저에서 실행되는 게임에 한정되며, 콘솔이나 특정 엔진에 종속된 대형 프로젝트에는 결과가 그대로 적용되지 않는다.
GameFix의 버그는 연구팀이 되돌릴 수 있게 인위적으로 주입한 것으로, 실제 개발 현장에서 자연 발생하는 복잡한 버그 양상과는 차이가 있을 수 있다.
GameOpt의 6턴 연쇄에서 오디오 관련 요청이 항상 마지막 턴에 배치돼 있어, 턴의 순서 효과와 항목(오디오)의 효과를 완전히 분리하지 못했다고 저자들이 밝혔다.
평가에 사용된 모델은 논문 작성 시점에 존재한 15~17개 특정 버전에 한정되며, 이후 등장한 모델이나 다른 코딩 에이전트 프레임워크에는 결과가 그대로 일반화되지 않을 수 있다.

왜 중요한가

지금까지의 게임 생성 AI 평가는 대부분 '한 번에 만들어낸 최종 결과물'만 보았지만, 실제 개발은 만들고, 고치고, 계속 개선하는 반복 과정이다. 이 연구는 그 전체 과정을 잣대로 삼아 AI 에이전트가 실전 개발자로서 어디서 무너지는지를 드러낸다.

이 논문의 용어

코딩 에이전트 · 사람의 자연어 지시를 받아 스스로 코드를 작성·실행·수정하는 AI 시스템
Fail-to-Pass / Pass-to-Pass 테스트 · 버그가 있을 땐 실패하고 수리 후엔 통과해야 하는 검사(F2P), 원래 정상이던 기능이 수리 후에도 계속 통과해야 하는 검사(P2P)
Shared Rubric(공유 채점표) · 여러 AI 모델이 만든 게임들을 모아 공통으로 나타난 기능과 요소를 정리해 만든 평가 기준표
Self-Discovery(자가 발견) · 버그 증상을 알려주지 않고 에이전트가 직접 게임을 실행해 문제를 찾아내야 하는 평가 조건
회귀(regression) · 새 기능을 추가하거나 버그를 고치는 과정에서 이전에 잘 작동하던 기능이 다시 망가지는 현상

본문에 싣지 못한 그림

Figure 2: The overall structure of GameXpert-Bench. GameGen starts from a natural language request, GameFix starts from a mutated Gold Game, and GameOpt studies multi-turn optimization based on real human interaction.

원문에서 그림 보기 →

저자 · Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno

arXiv에서 원문 보기

METAL LAB 최신 기사

그림 출처: Kun Chen et al., arXiv:2608.21833, CC BY 4.0

AI 코딩 에이전트에게 게임 하나를 처음부터 만들게 하고, 고쳐보게 하고, 여러 번 개선까지 시켜본 결과 '만들기'는 잘해도 '고치기'와 '유지하기'는 서툴렀다

GameXpert-Bench 세 트랙의 흐름

무엇을 했나

실제로 확인된 결과

어디에 쓸 수 있나

한계와 남은 검증

왜 중요한가

이 논문의 용어

본문에 싣지 못한 그림

최신 논문

METAL LAB 최신 기사