이미지: METAL LAB 생성
Summary
- Apple ML Research published a paper in August analyzing the 'outlier token' problem inside diffusion transformers (DiT)
- It confirmed that simply masking outlier tokens doesn't work, and the root cause is corrupted local patch information
- To address this, the team proposed a 'Dual-Stage Registers (DSR)' technique and validated it on ImageNet and large-scale text-to-image generation
- 논문 제목
- Taming Outlier Tokens in Diffusion Transformers
- 발행처·시점
- Apple ML Research, 2026년 8월
- 저자
- Xiaoyu Wu, Yifei Wang 외 (일부 Rice University 소속)
- 제안 기법
- Dual-Stage Registers (DSR)
- 검증 대상
- ImageNet, 대규모 텍스트-이미지 생성 파이프라인
What was published
Apple ML Research released a paper in August that directly examines the 'outlier token' phenomenon occurring inside Diffusion Transformers (DiT) used for image generation. The team confirmed that this phenomenon appears both in pretrained Vision Transformer (ViT) encoders and in the DiT itself within a Representation Autoencoder (RAE)-based DiT pipeline. The tokens were especially prominent in the middle layers of the DiT.
An interesting finding from the team was that simply masking or removing the high-value outlier tokens did not improve performance. This means the problem isn't caused by a few extreme values, but by the fact that the local patch information at those positions is itself corrupted. To address this, the team proposed 'Dual-Stage Registers (DSR)': using learned registers where available, and otherwise applying registers iteratively at inference time, while attaching a separate diffusion register to the denoiser.
Why 'outlier tokens' are a problem
Vision transformers split an image into small pieces (patches) and treat each one as a 'token' — similar to how language models process sentences word by word. However, as training progresses, some tokens have been reported to take on unusually large values (high-norm) while barely carrying any actual image information at their position, and instead excessively drawing the attention of other tokens. In ViT research, a known solution has been to create separate 'register' slots to absorb such tokens, but how this phenomenon operates in DiT, a generative model, had not been properly clarified until now.
DiT is the architecture that recent image-generation models such as Stable Diffusion 3 and the FLUX series have adopted in place of the traditional U-Net structure. Apple has previously pursued a series of studies on diffusion-based models, including comparing the performance of diffusion-based and autoregressive approaches, and scaling a diffusion language model up to 1.7 billion parameters. This paper can be seen as an extension of that line of research, pinpointing the root cause of a subtle flaw that degrades image quality.
What changes as a result
The outlier token problem has been cited as one cause of the occasional localized blotches or distorted patches seen in generated images. Apple's research is notable for addressing this at the level of training architecture rather than as a stopgap fix, and it could inform future design guidelines for improving the stability of DiT-based image and video generation models. That said, this result has only been validated at the level of a research paper, and it remains to be seen how — or whether — it will be incorporated into commercial models, pending further follow-up research.
