이미지: METAL LAB 생성
Summary
- Apple's machine learning research team has published a paper on a 1.7-billion-parameter flow-matching language model trained on 2.1 trillion tokens
- A "Categorical Flow Map" created by self-distilling this model generates text with a minimum of 4 inference steps
- Prior work had only been validated at under 1 billion parameters, leaving scalability an open question — this is the first large-scale validation
- 발행처
- Apple ML Research, 논문 공개일 2026-08-07
- 베이스 모델 규모
- 17억 파라미터(1.7B) 흐름 모델
- 학습 데이터량
- 2.1조 토큰(2.1T tokens)
- 추론 단계
- 자기증류 후 최소 4단계(4-step)로 압축
- 저자
- Oscar Davis 등 7인, 옥스퍼드대 공동연구 포함
What was announced
A paper titled "Scaling Categorical Flow Maps," published by Apple's machine learning research team on August 7, 2026, presents results from scaling a language model using flow matching rather than the autoregressive approach. The team trained a base model of 1.7 billion parameters on 2.1 trillion tokens, then self-distilled it — a process where the model uses its own outputs as a teacher to train a faster version — to produce a compressed model called a Categorical Flow Map (CFM). The team reported that this CFM generates diverse, high-quality text in as few as 4 inference steps while maintaining token entropy — a measure of sentence diversity — close to that of real data. The researchers also proposed a likelihood calculation method that converts the model's performance into scores comparable on standard language model benchmarks, reporting that the results fall in a similar range to existing discrete diffusion approaches.
Why it matters
Most language models widely used today, such as those in the GPT family, are autoregressive: they predict text token by token, left to right, in sequence. Diffusion or flow-matching models, by contrast, are an approach that first took hold in image generation, starting from noise and refining multiple points simultaneously to arrive at a final output. Applying this approach to text allows multiple tokens to be refined in parallel rather than strictly in order, theoretically enabling faster generation in fewer steps. The challenge is that, unlike images, text is not continuous-valued but a discrete sequence of words or tokens. Recent research has therefore attempted to bridge one-hot encoded data distributions and Gaussian noise using flow matching, but until now this had only been validated at scales under 1 billion parameters, leaving open the question of whether the same results would hold at larger scale. This paper offers the first answer to that question at a scale of 1.7 billion parameters and 2.1 trillion tokens, marking a next step for research in the flow-matching family. For reference, flow-matching architectures have already gained traction in image generation — Black Forest Labs released FLUX.2-dev, a 32-billion-parameter rectified flow transformer, this past August.
Black Forest Labs releases 32-billion-parameter image model FLUX.2-dev
What changes as a result
This announcement is a research paper, not a commercial product. Even so, it adds further evidence that flow matching could serve as a viable alternative architecture at scale in a language model market where autoregressive models have become the de facto standard. The researchers also compiled practical guidance on how to weight loss functions and design time schedules, which is likely to serve as a direct reference for other research teams attempting this approach going forward. Whether 4-step generation can achieve the speed and quality needed for practical, real-world services remains something that will need to be confirmed through further benchmarking and validation.
