ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models

arXiv:2608.140222026-08-14

An AI that predicts what happens next in a video game just cut its 'thinking steps' to as few as one, without losing control accuracy

ForgeWM is a video-generating AI that renders the next moment of a game scene, like Minecraft, in response to keyboard and mouse input. The researchers took a model originally built to look at whole video clips at once and retrained it in four stages so it can generate frames one at a time, using only 1, 2, or 4 computation steps per chunk instead of many. On paired test trajectories it beat comparison systems on image quality, motion accuracy, and control accuracy, and the same recipe carried over to a gamepad-controlled shooter game.

What they did

The original model looked at an entire video clip at once, which is too slow for real-time games; the team introduced a four-stage training process that converts it into a step-by-step, past-only generator.
A key challenge was keeping discrete keyboard states and continuous mouse movements correctly aligned with compressed chunks of generated video; a dedicated action module routes these two control types separately to solve this.
The four stages are: adapt the model to the game's visuals, teach it to generate frame-by-frame using true past data, compress it to run in very few steps, and finally have it learn from its own self-generated rollouts to curb error buildup.
On Minecraft trajectories, ForgeWM led compared systems in image quality, similarity to the actual motion path taken, accuracy of recognizing opposite-direction commands, and mouse-control accuracy, plus the lowest LPIPS difference from reference footage.
A 'replay-time refinement' feature lightly re-noises and polishes a saved rollout afterward, matching the quality of full four-step generation from scratch while staying about three times closer to the originally experienced scene layout.

Figure 2: Overview of ForgeWM. (A) Frame-aligned keyboard and mouse controls condition latent chunks. (B) A shared base yields a bidirectional teacher and budget-specialized causal students through four-stage training. (C) Deployment separates low-latency interaction from optional Replay-Time Refinement (Figure 6).

Figure 3: Qualitative comparison. Rollouts at frames 0, 25, 51, and 76. Rows show the reference and three models. Controls are annotated once on the reference: each overlay shows dominant WASD and accumulated mouse-look to the next frame; the final frame has no outgoing control overlay. Left/right: daytime forest stream/rainy riverbank at night.

Table 1: Training stages. Each stage changes either the temporal execution pattern, sampling objective, or history distribution.
Stage	Initialization	Training Context	Objective	Output
0	Base	Full-clip bidirectional	FM	Domain teacher
1	Base	Clean causal history	Causal FM	Causal teacher
2	Stage 1	Clean causal history	Online CD	Few-step initializer
3	Stage 2	Self-generated history	DMD	1/2/4-step students

Figure 5: Test-time step scaling. ForgeWM-1 under different denoising budgets; shading denotes bootstrap 95% confidence intervals.

Table 2: Comparison with interactive world models. Bold marks the best quality/control value in each column; efficiency entries are not bolded.
Model	Visual Quality	Temporal Quality	Action Controllability	Efficiency
IQ↑	LPIPS↓	AQ↑	Subj. Cons.↑	Flow Prof.↑	KCtrl↑	Mouse Acc.↑	Latency (ms)↓	FPS↑
Matrix-Game 2.0	0.6282	0.6443	0.4583	0.7349	0.9343	0.9156	0.7061	370.9	32.35
HY-WorldPlay	0.6133	0.6172	0.4855	0.9466	0.8288	0.9286	0.5818	2164.3	7.54
ForgeWM-1 (1-step)	0.6776	0.6529	0.4807	0.8279	0.9403	0.9545	0.7848	168.2	72.10
ForgeWM-2 (2-step)	0.6865	0.6171	0.4814	0.8349	0.9429	0.9740	0.8268	239.7	50.31
ForgeWM-4 (4-step)	0.6788	0.6168	0.4860	0.7613	0.9420	0.9740	0.8102	369.6	32.47

Figure 6: Replay-Time Refinement. The deployed student re-noises and denoises its own saved chunks under aligned actions and the refined causal prefix; each refined chunk is committed before the next is processed.

Figure 7: Replay refinement versus direct generation. Each row shares the same initial frame and controls. (A) Saved ForgeWM-1 draft; (B) replay-refined result; (C) direct ForgeWM-4 generation from noise.

Table F1: Per-stage optimization for the reported Minecraft ForgeWM lineage. “Blk.” is the number of latent frames per attention block (21 = full-clip bidirectional; 3 = one causal chunk). “EMA” is the generator exponential-moving-average decay and the step it starts (“–” when disabled). Global batch is the effective batch after data-parallel accumulation. All learning rates are constant.
	Stage 0	Stage 1	Stage 2	Stage 3
Objective	ℒFM	ℒ1	ℒ2	ℒ3
Trainer	FM	causal FM	consist. distill.	DMD
Attention	bidir.	causal	causal	causal
Init	base	base	Stage 1	Stage 2
Blk. (latents)	21	3	3	3
Trainer iterations	4k	20k	6k	4k
Gen. lr	2e−6	2e−5	2e−6	2e−6
Critic lr	–	–	–	4e−7
Global batch	8	8	8	8
EMA	–	–	0.99/200	0.99/200

Figure 8: CrossFPS rollouts. ForgeWM-CrossFPS on (A) Halo Infinite, (B) Modern Warfare, and (C) a science-fiction shooter. Colored overlays indicate active gamepad controls.

Figure 9: Extended qualitative rollouts from the four-step Minecraft checkpoint. Each row shows one 22 s causal rollout sampled at evenly spaced timestamps, with elapsed time marked per frame. Rows span biome and time of day and include both sustained forward motion and sustained turning.

Table F2: Stage-wise inference ablation on 1,000 paired trajectories. LPIPS uses the AlexNet backbone on 16 evaluated frames; subscripts denote trajectory-level bootstrap 95% confidence intervals. IQ, AQ, and SC are the corresponding VBench metrics. Stage 0 is a non-causal domain-teacher reference evaluated in its native full-clip regime.
Stage	Inference regime	LPIPS↓	IQ↑	AQ↑	SC↑
0	bidirectional teacher (ref.)	0.814[.809,.819]	0.455	0.463	0.677
1	teacher-forced causal	0.806[.799,.812]	0.508	0.454	0.700
2	causal consistency	0.605[.600,.610]	0.659	0.483	0.760
3	distribution matching	0.617[.613,.620]	0.716	0.489	0.760

Figure F1: Training losses across stages. Raw values are shown as light traces, with exponential-moving-average trends overlaid; the horizontal axis denotes trainer iterations. In Stage 3, the critic is updated every iteration and the generator every fifth iteration. Top-left: Stage 0 bidirectional flow matching. Top-right: Stage 1 teacher-forced causal flow matching. Bottom-left: Stage 2 online causal consistency distillation. Bottom-right: Stage 3 distribution matching, with the generator (DMD) loss on the left axis and the critic (fake-denoiser) loss on the right axis. Because the stages optimize different objectives, absolute loss magnitudes are not comparable across panels; the curves document optimization behavior within each stage only.

Table H1: Per-game CrossFPS results, 25 clips per game. Flow is the mean dense optical-flow magnitude in pixels; ratio is generated over reference, where 1.0 matches the reference motion scale.
Game	LPIPS↓	PSNR↑	Flow	Ratio
Xonotic	0.5828	11.21	13.88	1.66
Modern Warfare III	0.6352	11.61	8.15	1.35
Modern Warfare	0.6479	10.99	8.29	1.46
Warzone	0.6695	10.66	9.13	1.78
Halo Infinite	0.6730	9.75	11.23	1.46
Halo	0.6920	9.31	13.28	1.29
Call of Duty	0.6933	9.67	10.40	1.16
Macro-average	0.6562	10.46	10.62	1.45

Why it matters

Building an AI video generator that reacts instantly to a person's game controls requires solving both speed and control-accuracy at once, and this work lays out a concrete recipe for doing both. It's a practical step toward making game-like 'world models' fast and controllable enough for real interactive use, without needing an actual graphics engine.

Terms in this paper

World Model · an AI that predicts and generates the next scene based only on past frames and control inputs, without a graphics engine
Causal generation · generating future frames using only past information in order, never looking ahead
Distillation · a training technique that transfers a slow, complex model's ability into a smaller, faster one
LPIPS · a metric measuring how visually similar two videos look to a human observer; lower means more similar to the reference
Replay-Time Refinement · a post-processing method that adds a bit of noise back into a saved video draft and then re-denoises it to improve quality

Original abstract (English)

Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.

Authors · Xinye Li

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Xinye Li et al., arXiv:2608.14022, CC BY 4.0