Autoregressive Diffusion Model

An autoregressive diffusion model combines two techniques into one video generation method: it draws the next scene step by step based on previous scenes, while each individual scene is refined from a rough, blurry sketch into a clear final image.

Think of a comic artist who looks at the previous panel to decide the composition of the next one. While drawing each panel, the artist repeatedly erases and redraws the sketch, sharpening the lines each time. Deciding the next panel based on the previous one is the "autoregressive" part, and refining a single panel from blurry to sharp is the "diffusion" part. Combining the two means scenes flow continuously without breaks, while the quality of each scene stays stable.

This approach is especially useful in situations where you can't know in advance when the user will do what. Instead of pre-computing an entire movie ahead of time, the system only needs to render the very next scene based on the current camera position or user input, making it possible to create a space that can be experienced continuously for as long as desired, with no fixed runtime.

The article introduces Runway's GWM Worlds 2, which uses an autoregressive diffusion model to generate 720p, 24fps video together with 48,000Hz audio. People often assume a diffusion model produces an entire video all at once, but it's worth noting that here the model works differently: it remembers past scenes like a sliding window and generates one frame at a time in real time, which means there is effectively no limit on session length.