이미지: METAL LAB 생성
Summary
- Multiverse Computing has unveiled two system improvements that significantly reduce VRAM usage in knowledge distillation training
- Pre-caching the teacher model's top-K logits eliminates the need to load it into memory alongside the student model at the same time
- A new KL divergence loss function was applied that avoids constructing a full vocabulary-by-sequence-length matrix
- 논문 제목
- Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
- 발표
- Multiverse Computing, Hugging Face Blog, 2026-08-10
- 예시 모델
- gpt-oss-120b, 어휘 201,088 토큰
- 메모리 사례
- 시퀀스 32K·배치 4 기준 교사 확률 텐서 1개만 bf16으로 약 50GB
- 기존 증류 전체 소요
- 학습 1스텝 당 최대 약 250GB VRAM
Multiverse Computing has published a paper on the Hugging Face blog detailing two system improvements that lower the cost of training large language models through knowledge distillation. The paper is titled "Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss."
Knowledge distillation is a technique in which a smaller student model is trained to mimic the performance of a larger teacher model. In conventional approaches, the teacher model must be loaded into memory alongside the student at every training step to compute probability distributions over the entire vocabulary, placing a heavy burden on VRAM. For instance, the paper notes that for gpt-oss-120b, which has a vocabulary of 201,088 tokens, at a sequence length of 32K and batch size of 4, the teacher probability tensor alone reaches roughly 50GB in bf16. Adding gradients, activations, and optimizer states on top of that can push VRAM usage in a single training step up to as much as roughly 250GB.
Two Improvements
The research team pre-computed and cached the teacher model's top-K logits just once, eliminating the need to keep the teacher model loaded in memory alongside the student model during training. They also introduced a new memory-efficient KL divergence loss function that avoids directly constructing a matrix on the scale of the full vocabulary size times sequence length, which they say significantly cuts VRAM usage compared to default implementations in libraries such as PyTorch or NVIDIA's Megatron-Bridge.
The team said that combining these two changes makes it possible to run long-context recovery training on a single GPU, substantially lowering the cost of running large-scale experiments. The result comes amid a wave of increasingly massive models — such as Kimi-K3, a 2.8-trillion-parameter model requiring roughly 3TB of VRAM just to load — alongside a parallel trend of compressed model releases, including NVIDIA's Nemotron 3 Puzzle 75B and Multiverse Computing's own Hypernova 60B.
