每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models

arXiv:2608.140222026-08-14

预测游戏下一帧画面的AI,把计算步骤压到1到4步,操控准确度却没打折扣

ForgeWM是一种根据键盘鼠标输入实时生成游戏下一帧画面的AI模型,例如我的世界场景。研究者把原本需要一次看完整段视频才能作画的模型,通过四个阶段的训练改造成只依靠过去信息、每次仅用1步、2步或4步计算就能逐帧生成画面的模型。在配对轨迹测试中,ForgeWM在画质、运动轨迹一致性和操控准确度上都优于对比系统,同样的方法也成功迁移到了手柄操控的射击游戏上。

他们做了什么

  1. 原模型需要一次性看完整段视频才能作画,难以用于实时游戏,团队提出四阶段训练法,把它改造成只看过去信息、逐帧生成的模型。
  2. 关键难点是让离散的按键状态和连续的鼠标移动与被压缩的视频片段保持对齐,团队用专门的动作模块分别处理这两类控制信号来解决。
  3. 四个阶段依次是:让模型适应游戏画面、用真实历史数据教它逐帧生成、把它压缩到能用极少步数生成、最后让它学习自己生成的画面以减少误差累积。
  4. 在我的世界测试中,ForgeWM在图像质量、与真实运动轨迹的吻合度、反向操作识别准确率和鼠标控制准确率上均领先对比模型,与参考视频的差异(LPIPS)也最小。
  5. 新增的回放精修功能能对保存好的视频轻微加噪后再打磨,画质接近从零重新生成四步的效果,同时比重新生成更贴近玩家实际经历的场景布局,差距缩小约三倍。
Figure 2: Overview of ForgeWM. (A) Frame-aligned keyboard and mouse controls condition latent chunks. (B) A shared base yields a bidirectional teacher and budget-specialized causal students through four-stage training. (C) Deployment separates low-latency interaction from optional Replay-Time Refinement (Figure 6).
Figure 2: Overview of ForgeWM. (A) Frame-aligned keyboard and mouse controls condition latent chunks. (B) A shared base yields a bidirectional teacher and budget-specialized causal students through four-stage training. (C) Deployment separates low-latency interaction from optional Replay-Time Refinement (Figure 6).
Figure 3: Qualitative comparison. Rollouts at frames 0, 25, 51, and 76. Rows show the reference and three models. Controls are annotated once on the reference: each overlay shows dominant WASD and accumulated mouse-look to the next frame; the final frame has no outgoing control overlay. Left/right: daytime forest stream/rainy riverbank at night.
Figure 3: Qualitative comparison. Rollouts at frames 0, 25, 51, and 76. Rows show the reference and three models. Controls are annotated once on the reference: each overlay shows dominant WASD and accumulated mouse-look to the next frame; the final frame has no outgoing control overlay. Left/right: daytime forest stream/rainy riverbank at night.
Table 1: Training stages. Each stage changes either the temporal execution pattern, sampling objective, or history distribution.
StageInitializationTraining ContextObjectiveOutput
0BaseFull-clip bidirectionalFMDomain teacher
1BaseClean causal historyCausal FMCausal teacher
2Stage 1Clean causal historyOnline CDFew-step initializer
3Stage 2Self-generated historyDMD1/2/4-step students
Figure 4: Human preferences.
Figure 4: Human preferences.
Figure 5: Test-time step scaling. ForgeWM-1 under different denoising budgets; shading denotes bootstrap 95% confidence intervals.
Figure 5: Test-time step scaling. ForgeWM-1 under different denoising budgets; shading denotes bootstrap 95% confidence intervals.
Table 2: Comparison with interactive world models. Bold marks the best quality/control value in each column; efficiency entries are not bolded.
ModelVisual QualityTemporal QualityAction ControllabilityEfficiency
IQ↑LPIPS↓AQ↑Subj. Cons.↑Flow Prof.↑KCtrl↑Mouse Acc.↑Latency (ms)↓FPS↑
Matrix-Game 2.00.62820.64430.45830.73490.93430.91560.7061370.932.35
HY-WorldPlay0.61330.61720.48550.94660.82880.92860.58182164.37.54
ForgeWM-1 (1-step)0.67760.65290.48070.82790.94030.95450.7848168.272.10
ForgeWM-2 (2-step)0.68650.61710.48140.83490.94290.97400.8268239.750.31
ForgeWM-4 (4-step)0.67880.61680.48600.76130.94200.97400.8102369.632.47
Figure 6: Replay-Time Refinement. The deployed student re-noises and denoises its own saved chunks under aligned actions and the refined causal prefix; each refined chunk is committed before the next is processed.
Figure 6: Replay-Time Refinement. The deployed student re-noises and denoises its own saved chunks under aligned actions and the refined causal prefix; each refined chunk is committed before the next is processed.
Figure 7: Replay refinement versus direct generation. Each row shares the same initial frame and controls. (A) Saved ForgeWM-1 draft; (B) replay-refined result; (C) direct ForgeWM-4 generation from noise.
Figure 7: Replay refinement versus direct generation. Each row shares the same initial frame and controls. (A) Saved ForgeWM-1 draft; (B) replay-refined result; (C) direct ForgeWM-4 generation from noise.
Table F1: Per-stage optimization for the reported Minecraft ForgeWM lineage. “Blk.” is the number of latent frames per attention block (21 = full-clip bidirectional; 3 = one causal chunk). “EMA” is the generator exponential-moving-average decay and the step it starts (“–” when disabled). Global batch is the effective batch after data-parallel accumulation. All learning rates are constant.
Stage 0Stage 1Stage 2Stage 3
ObjectiveℒFMℒ1ℒ2ℒ3
TrainerFMcausal FMconsist. distill.DMD
Attentionbidir.causalcausalcausal
InitbasebaseStage 1Stage 2
Blk. (latents)21333
Trainer iterations4k20k6k4k
Gen. lr2​e−62​e−52​e−62​e−6
Critic lr–––4​e−7
Global batch8888
EMA––0.99/2000.99/200
Figure 8: CrossFPS rollouts. ForgeWM-CrossFPS on (A) Halo Infinite, (B) Modern Warfare, and (C) a science-fiction shooter. Colored overlays indicate active gamepad controls.
Figure 8: CrossFPS rollouts. ForgeWM-CrossFPS on (A) Halo Infinite, (B) Modern Warfare, and (C) a science-fiction shooter. Colored overlays indicate active gamepad controls.
Figure 9: Extended qualitative rollouts from the four-step Minecraft checkpoint. Each row shows one 22 s causal rollout sampled at evenly spaced timestamps, with elapsed time marked per frame. Rows span biome and time of day and include both sustained forward motion and sustained turning.
Figure 9: Extended qualitative rollouts from the four-step Minecraft checkpoint. Each row shows one 22 s causal rollout sampled at evenly spaced timestamps, with elapsed time marked per frame. Rows span biome and time of day and include both sustained forward motion and sustained turning.
Table F2: Stage-wise inference ablation on 1,000 paired trajectories. LPIPS uses the AlexNet backbone on 16 evaluated frames; subscripts denote trajectory-level bootstrap 95% confidence intervals. IQ, AQ, and SC are the corresponding VBench metrics. Stage 0 is a non-causal domain-teacher reference evaluated in its native full-clip regime.
StageInference regimeLPIPS↓IQ↑AQ↑SC↑
0bidirectional teacher (ref.)0.814[.809,.819]0.4550.4630.677
1teacher-forced causal0.806[.799,.812]0.5080.4540.700
2causal consistency0.605[.600,.610]0.6590.4830.760
3distribution matching0.617[.613,.620]0.7160.4890.760
Figure F1: Training losses across stages. Raw values are shown as light traces, with exponential-moving-average trends overlaid; the horizontal axis denotes trainer iterations. In Stage 3, the critic is updated every iteration and the generator every fifth iteration. Top-left: Stage 0 bidirectional flow matching. Top-right: Stage 1 teacher-forced causal flow matching. Bottom-left: Stage 2 online causal consistency distillation. Bottom-right: Stage 3 distribution matching, with the generator (DMD) loss on the left axis and the critic (fake-denoiser) loss on the right axis. Because the stages optimize different objectives, absolute loss magnitudes are not comparable across panels; the curves document optimization behavior within each stage only.
Figure F1: Training losses across stages. Raw values are shown as light traces, with exponential-moving-average trends overlaid; the horizontal axis denotes trainer iterations. In Stage 3, the critic is updated every iteration and the generator every fifth iteration. Top-left: Stage 0 bidirectional flow matching. Top-right: Stage 1 teacher-forced causal flow matching. Bottom-left: Stage 2 online causal consistency distillation. Bottom-right: Stage 3 distribution matching, with the generator (DMD) loss on the left axis and the critic (fake-denoiser) loss on the right axis. Because the stages optimize different objectives, absolute loss magnitudes are not comparable across panels; the curves document optimization behavior within each stage only.
Table H1: Per-game CrossFPS results, 25 clips per game. Flow is the mean dense optical-flow magnitude in pixels; ratio is generated over reference, where 1.0 matches the reference motion scale.
GameLPIPS↓PSNR↑FlowRatio
Xonotic0.582811.2113.881.66
Modern Warfare III0.635211.618.151.35
Modern Warfare0.647910.998.291.46
Warzone0.669510.669.131.78
Halo Infinite0.67309.7511.231.46
Halo0.69209.3113.281.29
Call of Duty0.69339.6710.401.16
Macro-average0.656210.4610.621.45

为什么重要

要让AI视频生成器实时响应玩家操作,必须同时解决速度和操控准确性两大难题,这项工作给出了具体可行的训练方案。这为不依赖真实图形引擎、就能实时生成可玩游戏画面的世界模型技术,向实际应用迈出了实用的一步。

本文术语

  • 世界模型(World Model) · 仅根据过去画面和操作输入预测并生成下一帧画面的AI,不依赖图形引擎
  • 因果生成(Causal generation) · 生成未来画面时只能依次使用过去信息,不能提前看到后续内容
  • 蒸馏(Distillation) · 把复杂缓慢模型的能力迁移到更轻量快速模型上的训练技巧
  • LPIPS · 衡量两段视频在人眼看来有多相似的指标,数值越低越接近参考视频
  • 回放精修(Replay-Time Refinement) · 对已保存的视频草稿轻微加噪后再重新去噪打磨,以提升画质的后处理方法

论文原文摘要（英文）

Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.

作者 · Xinye Li

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Xinye Li et al., arXiv:2608.14022, CC BY 4.0