매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models

arXiv:2608.140222026-08-14

게임 속 캐릭터 움직임을 실시간으로 예측하는 AI, 계산 단계를 1~4번으로 확 줄이면서도 조작감을 살리다

ForgeWM은 마인크래프트 같은 게임 화면을 키보드·마우스 입력에 맞춰 실시간으로 다음 장면을 생성하는 AI 모델이다. 원래 여러 프레임을 한꺼번에 보고 그림을 그리던 모델을 네 단계에 걸쳐 개조해서, 과거 정보만 보고 1번, 2번, 4번의 계산만으로도 빠르게 다음 장면을 만들어내도록 만들었다. 그 결과 화질, 카메라 조작 정확도, 움직임 정확도에서 비교 대상보다 나은 성적을 냈고, FPS 게임용 컨트롤러 조작에도 같은 방법이 통했다.

무엇을 했나

  1. 원래 모델은 영상 전체를 한꺼번에 보고 그림을 그리는 방식이라 실시간 게임에 쓰기 어려웠는데, 이를 과거 장면만 보고 순서대로 그리는 방식으로 바꾸는 네 단계 학습법을 제안했다.
  2. 키보드의 켜짐/꺼짐 신호와 마우스의 연속적인 움직임 신호를 압축된 영상 조각과 어긋나지 않게 맞추는 것이 핵심 난제였고, 이를 전용 통로로 처리하는 액션 모듈로 해결했다.
  3. 1단계는 게임 화면에 적응시키고, 2단계는 순서대로 그리는 방식을 가르치고, 3단계는 적은 계산 단계로도 그릴 수 있게 압축하고, 4단계는 모델이 스스로 만든 화면을 보고 또 배우게 해서 오차 누적을 줄였다.
  4. 마인크래프트 실험에서 이미지 화질, 실제 이동 궤적과의 유사도, 반대 방향 조작 인식 정확도, 마우스 조작 정확도에서 비교 모델들보다 앞섰고, 기준 영상과의 차이(LPIPS)도 가장 작았다.
  5. 한 번 저장해둔 영상을 나중에 다시 노이즈를 살짝 입혀 다듬는 '리플레이 정제' 기능을 넣어, 새로 4단계로 처음부터 그리는 것과 비슷한 화질을 내면서도 원래 경험한 장면 구도를 3배 더 잘 유지했다.
Figure 2: Overview of ForgeWM. (A) Frame-aligned keyboard and mouse controls condition latent chunks. (B) A shared base yields a bidirectional teacher and budget-specialized causal students through four-stage training. (C) Deployment separates low-latency interaction from optional Replay-Time Refinement (Figure 6).
Figure 2: Overview of ForgeWM. (A) Frame-aligned keyboard and mouse controls condition latent chunks. (B) A shared base yields a bidirectional teacher and budget-specialized causal students through four-stage training. (C) Deployment separates low-latency interaction from optional Replay-Time Refinement (Figure 6).
Figure 3: Qualitative comparison. Rollouts at frames 0, 25, 51, and 76. Rows show the reference and three models. Controls are annotated once on the reference: each overlay shows dominant WASD and accumulated mouse-look to the next frame; the final frame has no outgoing control overlay. Left/right: daytime forest stream/rainy riverbank at night.
Figure 3: Qualitative comparison. Rollouts at frames 0, 25, 51, and 76. Rows show the reference and three models. Controls are annotated once on the reference: each overlay shows dominant WASD and accumulated mouse-look to the next frame; the final frame has no outgoing control overlay. Left/right: daytime forest stream/rainy riverbank at night.
Table 1: Training stages. Each stage changes either the temporal execution pattern, sampling objective, or history distribution.
StageInitializationTraining ContextObjectiveOutput
0BaseFull-clip bidirectionalFMDomain teacher
1BaseClean causal historyCausal FMCausal teacher
2Stage 1Clean causal historyOnline CDFew-step initializer
3Stage 2Self-generated historyDMD1/2/4-step students
Figure 4: Human preferences.
Figure 4: Human preferences.
Figure 5: Test-time step scaling. ForgeWM-1 under different denoising budgets; shading denotes bootstrap 95% confidence intervals.
Figure 5: Test-time step scaling. ForgeWM-1 under different denoising budgets; shading denotes bootstrap 95% confidence intervals.
Table 2: Comparison with interactive world models. Bold marks the best quality/control value in each column; efficiency entries are not bolded.
ModelVisual QualityTemporal QualityAction ControllabilityEfficiency
IQ↑LPIPS↓AQ↑Subj. Cons.↑Flow Prof.↑KCtrl↑Mouse Acc.↑Latency (ms)↓FPS↑
Matrix-Game 2.00.62820.64430.45830.73490.93430.91560.7061370.932.35
HY-WorldPlay0.61330.61720.48550.94660.82880.92860.58182164.37.54
ForgeWM-1 (1-step)0.67760.65290.48070.82790.94030.95450.7848168.272.10
ForgeWM-2 (2-step)0.68650.61710.48140.83490.94290.97400.8268239.750.31
ForgeWM-4 (4-step)0.67880.61680.48600.76130.94200.97400.8102369.632.47
Figure 6: Replay-Time Refinement. The deployed student re-noises and denoises its own saved chunks under aligned actions and the refined causal prefix; each refined chunk is committed before the next is processed.
Figure 6: Replay-Time Refinement. The deployed student re-noises and denoises its own saved chunks under aligned actions and the refined causal prefix; each refined chunk is committed before the next is processed.
Figure 7: Replay refinement versus direct generation. Each row shares the same initial frame and controls. (A) Saved ForgeWM-1 draft; (B) replay-refined result; (C) direct ForgeWM-4 generation from noise.
Figure 7: Replay refinement versus direct generation. Each row shares the same initial frame and controls. (A) Saved ForgeWM-1 draft; (B) replay-refined result; (C) direct ForgeWM-4 generation from noise.
Table F1: Per-stage optimization for the reported Minecraft ForgeWM lineage. “Blk.” is the number of latent frames per attention block (21 = full-clip bidirectional; 3 = one causal chunk). “EMA” is the generator exponential-moving-average decay and the step it starts (“–” when disabled). Global batch is the effective batch after data-parallel accumulation. All learning rates are constant.
Stage 0Stage 1Stage 2Stage 3
ObjectiveℒFMℒ1ℒ2ℒ3
TrainerFMcausal FMconsist. distill.DMD
Attentionbidir.causalcausalcausal
InitbasebaseStage 1Stage 2
Blk. (latents)21333
Trainer iterations4k20k6k4k
Gen. lr2​e−62​e−52​e−62​e−6
Critic lr–––4​e−7
Global batch8888
EMA––0.99/2000.99/200
Figure 8: CrossFPS rollouts. ForgeWM-CrossFPS on (A) Halo Infinite, (B) Modern Warfare, and (C) a science-fiction shooter. Colored overlays indicate active gamepad controls.
Figure 8: CrossFPS rollouts. ForgeWM-CrossFPS on (A) Halo Infinite, (B) Modern Warfare, and (C) a science-fiction shooter. Colored overlays indicate active gamepad controls.
Figure 9: Extended qualitative rollouts from the four-step Minecraft checkpoint. Each row shows one 22 s causal rollout sampled at evenly spaced timestamps, with elapsed time marked per frame. Rows span biome and time of day and include both sustained forward motion and sustained turning.
Figure 9: Extended qualitative rollouts from the four-step Minecraft checkpoint. Each row shows one 22 s causal rollout sampled at evenly spaced timestamps, with elapsed time marked per frame. Rows span biome and time of day and include both sustained forward motion and sustained turning.
Table F2: Stage-wise inference ablation on 1,000 paired trajectories. LPIPS uses the AlexNet backbone on 16 evaluated frames; subscripts denote trajectory-level bootstrap 95% confidence intervals. IQ, AQ, and SC are the corresponding VBench metrics. Stage 0 is a non-causal domain-teacher reference evaluated in its native full-clip regime.
StageInference regimeLPIPS↓IQ↑AQ↑SC↑
0bidirectional teacher (ref.)0.814[.809,.819]0.4550.4630.677
1teacher-forced causal0.806[.799,.812]0.5080.4540.700
2causal consistency0.605[.600,.610]0.6590.4830.760
3distribution matching0.617[.613,.620]0.7160.4890.760
Figure F1: Training losses across stages. Raw values are shown as light traces, with exponential-moving-average trends overlaid; the horizontal axis denotes trainer iterations. In Stage 3, the critic is updated every iteration and the generator every fifth iteration. Top-left: Stage 0 bidirectional flow matching. Top-right: Stage 1 teacher-forced causal flow matching. Bottom-left: Stage 2 online causal consistency distillation. Bottom-right: Stage 3 distribution matching, with the generator (DMD) loss on the left axis and the critic (fake-denoiser) loss on the right axis. Because the stages optimize different objectives, absolute loss magnitudes are not comparable across panels; the curves document optimization behavior within each stage only.
Figure F1: Training losses across stages. Raw values are shown as light traces, with exponential-moving-average trends overlaid; the horizontal axis denotes trainer iterations. In Stage 3, the critic is updated every iteration and the generator every fifth iteration. Top-left: Stage 0 bidirectional flow matching. Top-right: Stage 1 teacher-forced causal flow matching. Bottom-left: Stage 2 online causal consistency distillation. Bottom-right: Stage 3 distribution matching, with the generator (DMD) loss on the left axis and the critic (fake-denoiser) loss on the right axis. Because the stages optimize different objectives, absolute loss magnitudes are not comparable across panels; the curves document optimization behavior within each stage only.
Table H1: Per-game CrossFPS results, 25 clips per game. Flow is the mean dense optical-flow magnitude in pixels; ratio is generated over reference, where 1.0 matches the reference motion scale.
GameLPIPS↓PSNR↑FlowRatio
Xonotic0.582811.2113.881.66
Modern Warfare III0.635211.618.151.35
Modern Warfare0.647910.998.291.46
Warzone0.669510.669.131.78
Halo Infinite0.67309.7511.231.46
Halo0.69209.3113.281.29
Call of Duty0.69339.6710.401.16
Macro-average0.656210.4610.621.45

왜 중요한가

게임이나 시뮬레이션 속에서 사람의 조작에 즉각 반응하는 AI 영상 생성기를 만들려면 계산 속도와 조작 정확도를 동시에 잡아야 하는데, ForgeWM은 그 두 마리 토끼를 잡는 구체적인 학습 절차를 제시한다. 이는 게임 엔진 없이도 그럴듯한 게임 화면을 실시간으로 만들어내는 '월드 모델' 기술을 실제 서비스에 가깝게 만드는 실용적 진전이다.

이 논문의 용어

  • 월드 모델(World Model) · 카메라나 그래픽 엔진 없이, 과거 화면과 조작 입력만 보고 다음 화면을 예측해 만들어내는 AI 모델
  • 인과적(Causal) 생성 · 미래 장면을 만들 때 이후 시점 정보를 보지 않고 과거 정보만 순서대로 사용하는 방식
  • 증류(Distillation) · 복잡하고 느린 모델의 성능을 더 가볍고 빠른 모델에 옮겨 담는 학습 기법
  • LPIPS · 두 영상이 사람 눈에 얼마나 비슷하게 보이는지를 수치로 재는 지표, 낮을수록 원본과 비슷함
  • 리플레이 정제(Replay-Time Refinement) · 저장해둔 영상 초안에 살짝 노이즈를 더한 뒤 다시 다듬어 화질을 높이는 후처리 방법

논문 원문 초록 (영문)

Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.

저자 · Xinye Li

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Xinye Li et al., arXiv:2608.14022, CC BY 4.0