이미지: METAL LAB 생성
Summary
- Apple ML Research published a paper in August comparing the performance characteristics of diffusion language models (DLMs) and autoregressive language models (ARMs) through both theoretical analysis and empirical measurement
- The study found that while DLMs achieve higher computational efficiency through parallel token generation, their scalability drops as context length grows
- The researchers confirmed that applying block-wise decoding allows DLMs to scale to long contexts in the same way ARMs do
- 발행처
- Apple ML Research
- 공개 시점
- 2026년 8월
- 주요 저자
- Minseo Kim(서울대), Amir Gholami(UC버클리) 등
- 핵심 발견
- DLM은 병렬 생성으로 산술 강도는 높으나 긴 문맥 확장에 취약
- 제안 기법
- 블록 단위 디코딩으로 산술 강도와 문맥 길이를 분리
Dissecting models that don't generate in order
Apple ML Research published a paper in August that directly compares the performance differences between diffusion language models (DLMs) and autoregressive models (ARMs). Kim Minseo, a researcher at Seoul National University, along with researchers from UC Berkeley and Apple, are listed as co-authors.
The paper breaks down the trade-offs between the two approaches through theoretical analysis and empirical profiling. The conclusion is clear: DLMs can process token positions simultaneously, boosting arithmetic intensity — a metric of how efficiently a GPU uses data for computation — but this advantage disappears as context length grows. The researchers proposed block-wise decoding as an alternative. Rather than generating an entire sentence at once, processing it in blocks keeps arithmetic intensity from being constrained by context length, they explained.
Why ARM and DLM diverged
ARMs, represented by the GPT family, read all preceding words to predict the next single word. This is akin to writing one character at a time — accuracy is high, but the need to preserve order creates a structural limitation that makes it hard to run GPUs in parallel. DLMs, by contrast, start from an entire noisy sentence and progressively refine it, completing the whole sentence at once. In theory, this allows multiple positions to be computed simultaneously, yielding better hardware efficiency.
Apple's focus on this comparison is foreshadowed by another paper it released on August 11. At the time, Apple ML Research reported that it had trained a 1.7-billion-parameter flow-matching language model on 2.1 trillion tokens, and that a Categorical Flow Map created through self-distillation of that model could generate text in as few as four inference steps. Models that produce fast answers with minimal computation are especially valuable in on-device environments like the iPhone. This new performance comparison paper is close to a follow-up effort that verifies exactly where that direction pays off and where it hits its limits.
The researchers also confirmed that the results flip in batch inference — where multiple requests are bundled and processed together. ARMs gain a bigger advantage from processing multiple sequences in a batch simultaneously, putting them ahead of DLMs in throughput.
So what does this change
This paper puts the brakes on the trend of touting DLMs as the unconditional future alternative. The researchers noted that for open-source DLMs to achieve faster response times than ARMs, reducing the number of sampling steps is key. In fact, the four-step inference model Apple released around the same time can be seen as a direct result of following that direction. This analysis sharpens the emerging picture: DLMs are favorable for on-device tasks with short contexts where response speed matters, while ARMs remain advantageous for server environments that need to process long documents or handle multiple requests at once.
