single-stream omni transformer

A single-stream omni transformer is a structure in which one AI generates video and audio at the same time. Until now, AI video generation typically went through three separate steps: a tool that drew the picture, a tool that generated the voice, and a tool that synced the mouth movements to that voice, all stitched together in sequence. A single-stream omni transformer merges these three tools into one.

Here's an analogy. The old way is like an actor filming a silent movie, a voice actor recording dialogue afterward, and an editor dubbing it in to match the lip movements. A single-stream omni transformer is closer to a live take where the actor sings and acts at the same time. Because the picture and sound are born together in a single process from the start, there's no need to sync them up afterward.

As a result, video made this way has speech and mouth movements that naturally line up without any post-processing lip-sync step. On the other hand, it's harder to build, since the video network and the audio network have to be merged into one.

Describing MiniMax's H3, an article explains: "At its core is a 33B-scale single-stream omni transformer. The same transformer predicts video latents and audio latents simultaneously." A common misreading is to interpret this as "an audio module bolted onto a video model." In reality, it isn't two separate models — it's a single neural network that computes video and sound together.