Motif 3: Technical Report

arXiv:2608.091192026-08-10

一家韩国团队发布了3140亿参数的开源混合专家(MoE)大模型Motif 3,每个token只激活132亿参数

Motif 3是一个混合专家(MoE)语言模型,总参数3140亿,但每个token只激活132亿参数。团队用一种新的注意力机制GDLA以及多项训练稳定化技术,在约12.5万亿token上完成预训练,再通过六个专家教师模型和蒸馏过程整合成一个统一模型。已报告的结果显示,GDLA用比原方法更少的训练量达到了更低的训练损失,最终模型在多项基准测试中表现出与主流开源模型相当的竞争力。

METAL LAB 解读图

Motif 3流程:架构设计、大规模预训练、多教师蒸馏

证据状态实测结果与计划中的工作并存

稀疏MoE结构总参数3140亿,但每层384个路由专家中只选用8个,每个token仅激活132亿参数
GDLA注意力机制结合差分(抑制噪声)注意力与压缩的潜在键值表示,降低训练损失并减少缓存开销
低精度与长上下文训练系统MXFP8计算加窗口感知上下文并行技术,支持12.5万亿token预训练及最长25.6万token的上下文
七个专家教师模型六个强化学习训练的专家教师(代理任务、专业办公、长上下文与拒答、数学、代码与科学、对话)加一个监督微调的软件工程教师
多教师在线策略蒸馏将七个教师模型的能力整合进一个统一的学生模型中

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

Motif 3采用细粒度稀疏MoE结构:每层384个路由专家中每个token只选用8个,3140亿总参数里只激活132亿。
团队提出了新的注意力结构GDLA(分组差分潜在注意力),将能抑制噪声的差分注意力机制与能压缩键值缓存的多头潜在注意力(MLA)结合起来。
还引入了按专家分别学习激活函数系数的Expert-Specific PolyNorm、把残差连接扩展为多条并行流的改进版mHC,以及提升推理效率的多token预测(MTP)技术。
预训练数据约12.5万亿token,涵盖网页文档、STEM、代码、数学、多语言及专业领域语料,并借助MXFP8低精度计算和窗口感知的上下文并行技术,支持最长25.6万token的上下文训练。
后训练阶段结合了通用监督微调、六个通过强化学习训练的专家教师模型、一个通过监督微调训练的软件工程教师模型,以及将多个教师能力汇总到一个模型中的多教师在线策略蒸馏(MOPD)。

Figure 2: Attention training-loss comparison. GDLA achieves lower loss than GDA and MLA and reaches a loss of 3.2 with 9.2% fewer training tokens than MLA.

Table 1: Summary of the principal architectural configuration and model dimensions used in Motif 3, including its hybrid attention pattern, fine-grained expert structure, and long-context support.
Property	Configuration
Total parameters	Approximately 314B
Activated parameters	Approximately 13.2B per token
Transformer layers	53 (2 dense and 51 MoE)
Hidden dimension	4,096
Attention mechanism	Grouped Differential Latent Attention with output gating
Attention pattern	Hybrid GDLA (1 full / 3 sliding-window)
Query / KV heads	80 / 16
Signal / noise query heads	64 / 16
Query-key / Value head dim	192 / 128
FFN dimension (dense / expert)	12,288 / 1,280
Routed experts	384, with top-8 routing
Shared experts	1
Expert activation	Expert-Specific PolyNorm
Residual architecture	Modified manifold-constrained hyper-connections
MTP head	1 layer for self-speculative decoding
Maximum context length	262,144 tokens (256K)

Figure 3: Overall low-precision training recipe. Color encodes numerical precision: teal for MXFP8, blue for BF16, and peach for FP32. Dashed borders mark collectives that cross ranks. Only the row-wise MXFP8 weight is All-Gathered; the column-wise copy required by Dgrad is produced locally by a fused row-to-column transcode kernel. Expert activations are quantized once before EP dispatch, allowing the dispatch itself to use MXFP8. Gradient synchronization exchanges BF16 shards while performing each reduction locally in FP32.

Table 3: Principal expert-health signals monitored during Motif 3 pretraining.
Metric	Failure mode indicated
Dispatch min/median	Starvation or a dead expert caused by routing imbalance
Maximum expert-token count	Concentrated traffic and possible expert overload
Output-weight min/median	Hidden collapse in which an expert receives tokens but its output projection contributes increasingly little
Routed/shared RMS, mean, and maximum	Dominance of shared experts, routed-expert collapse, or a change in the routed-output distribution
Maximum expert-output cosine similarity	Functional collapse in which nominally distinct experts learn similar representations
Routed and shared abs-max/RMS	Activation outliers and early warning signs of numerical instability

Figure 6: MoE component training comparisons from controlled experiments using models with approximately 10 billion parameters. (a) Expert-Specific PolyNorm maintains a higher mean effective rank in the expert gate weights than SwiGLU across layers, indicating a more evenly distributed singular-value spectrum. (b) Decaying router noise reduces the maximum number of tokens assigned to an expert more rapidly and guides the routing distribution toward the median-load regime early in training.

Table 4: Evaluation results for the Motif 3 pretrained base model. CoT denotes chain-of-thought prompting.
MMLU 5-shot	MMLU-Pro 5-shot CoT	ARC-C 25-shot	WinoGrande 5-shot	HellaSwag 10-shot	PIQA 0-shot	GSM8K 8-shot CoT	MATH 4-shot CoT	HumanEval 0-shot	MBPP 3-shot
86.20	68.56	94.71	80.90	88.30	85.14	93.93	70.58	73.70	84.60

Table 5: The seven specialist teachers and their capability coverage.
Teacher	Coverage
Agentic tool use	Interactive shell and tool environments; multi-step task execution
Professional work	Open-ended occupational deliverables graded by comparison
Software engineering	Repository-level modifications verified through test execution
Long-context reasoning & abstention	Retrieval and synthesis over very long inputs; calibrated abstention
Mathematics	Competition and proof-style problems; symbolic and judged answer checking
Code and science	Program synthesis, scientific computing, and physical reasoning
Chat	Dialogue quality, instruction following, and safety

Figure 7: Mean reward over cumulative RL compute for the six GRPO-trained specialist teachers. The lighter curves show the per-update reward measurements, and the darker curves show their smoothed trends.

Table 6: Evaluation results for Motif 3. An asterisk (*) indicates that the corresponding result for Motif 3 is evaluated on the public subset only.
Benchmark	Motif 3 314B-A13B	MiniMax-3 428B-A23B	GLM-5.1 744B-A40B	Kimi-K2.6 1T-A32B	Qwen-3.7 Max	DS-v4-Pro 1.6T-A49B
[0pt][0pt] Agentic
GDPval-AA v2	38.7	44.4	37.8	34.4	39.0	40.2
τ2-Bench Telecom	94.7	88.9	97.7	95.9	94.7	96.2
τ3-Banking	35.3	15.3	13.6	23.3	12.0	30.1
ITBench-AA	51.5∗	-	40.3	31.2	42.5	38.3
[0pt][0pt] Coding
SWE-bench Verified	76.2	75.0	76.4	76.2	80.4	77.4
Terminal-Bench 2.1	74.9	65.2	61.8	65.9	75.0	64.0
SciCode	40.6	45.4	43.8	53.5	53.5	50.0
[0pt][0pt] Reasoning and Knowledge
IMO-AnswerBench	83.2	-	83.8	81.8	90.0	89.8
Apex Shortlist	75.5	-	71.1	77.4	44.5	85.8
GPQA Diamond	83.4	92.9	86.8	91.1	92.4	88.8
HLE	37.0	39.0	30.1	37.5	41.4	37.5
CritPt	6.6	3.7	4.6	8.0	11.4	12.9
AA-Omniscience Accuracy	30.1	16.7	23.7	32.6	31.0	42.9
AA-Omniscience Non-Hallucination	71.6	81.6	70.1	59.5	74.0	5.9
[0pt][0pt] Long Context and Instruction Following
AA-LCR	72.3	80.3	68.0	76.7	75.0	70.0
IFBench	78.2	82.9	76.3	76.0	79.1	76.5

研究结果

在约100亿参数规模的对照实验中,GDLA的训练损失低于GDA和MLA,达到3.2的损失值时所需训练token比MLA少9.2%。
在同等规模的对照实验中,Expert-Specific PolyNorm在专家门控权重上保持了比SwiGLU更高的有效秩(衡量奇异值分布均匀程度的指标)。
采用逐渐衰减的路由噪声后,分配给单个专家的最大token数量下降得更快,并在训练早期把负载分布引导到中位数水平附近。
在整体评测集上,Motif 3被报告在与主流开源权重模型的对比中表现出竞争力,在长周期代理任务、数学推理、科学知识以及对幻觉敏感的评测中表现较强。

可应用场景

GDLA以及mHC缩放退火、专家负载均衡等稳定化技术,可作为其他团队设计或扩展自有MoE架构时的参考方案。
该统一模型覆盖推理、编程、工具使用、专业文档撰写、长上下文理解等多种能力,可作为想把多种能力整合进单一可部署模型的团队的参考案例。
面向最长25.6万token上下文的MXFP8低精度训练方案与上下文并行策略,可为设计超长上下文或超大规模模型训练基础设施提供参考。

局限与待验证事项

与其他开源模型的性能对比主要引用了各自排行榜上公布的分数,而非在完全相同条件下对所有模型进行的统一复测。
GDLA、PolyNorm等核心架构组件仅在约100亿参数规模的对照实验中得到验证,并未在3140亿参数的完整模型上做逐项消融实验。
部分基准测试结果标注为仅在公开子集上评测,这些项目与其他模型在完整测试集上的分数不能直接比较。
部分后训练组件(如针对韩语场景设计的对话教师评分规则)主要针对韩语场景验证,在更广泛多语言场景下的泛化效果仍需进一步检验。

为什么重要

这份报告详细记录了一个规模不大的团队如何设计、训练并稳定一个3000多亿参数的MoE大模型,为其他团队在架构设计和大规模训练工程上提供了具体可参考的经验。报告中公开的低精度训练与长上下文训练方案,也为想要扩展自有系统的团队提供了实际的技术参照。

本文术语

混合专家(MoE) · 一种架构,每次只从许多小型专家网络中选用一部分进行计算,从而在总参数量很大的情况下降低每个token的计算量
GDLA · 本文提出的新注意力机制,结合了差分(抑制噪声)注意力和压缩的潜在键值表示
MXFP8 · 一种8比特低精度数值格式,用于降低内存占用和计算开销
MOPD(多教师在线策略蒸馏) · 一种将多个专家教师模型的能力蒸馏到同一个学生模型中的方法
GRPO · 用于训练各专家教师模型的一种强化学习方法

论文原文摘要（英文）

We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

作者 · Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Junghwan Lim et al., arXiv:2608.09119, arxiv-nonexclusive