Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

arXiv:2608.050002026-08-04

一项控制实验研究:语言、图像理解与图像生成在同一模型中联合训练时如何互相促进或干扰

这篇论文系统研究了在单一"统一多模态"模型中同时预训练文本、图像理解与图像生成时,三种能力之间知识如何流动。研究同时使用大规模真实数据和基于CLEVR的可控合成数据,发现这三种能力之间的迁移是不对称的,且依赖于具体概念。基于这些发现,作者提炼出数据配比与架构设计的实用方案,并通过训练135亿参数的MoE模型(2万亿token)在更大规模上进行了验证。

METAL LAB 解读图

统一多模态预训练中的知识流动方向

证据状态已报告实测结果

语言数据增加DCLM语言数据可同时提升图像理解和图像生成
图像理解数据增加理解数据大幅提升生成,但使纯语言表现略微下降
图像生成数据增加生成数据对语言和理解只带来轻微波动,无明显趋势
CLEVR概念迁移实验颜色/形状双向迁移失败;关系/大小/数量主要从理解迁移到生成
架构与时机设计(split_ffn、早期统一)共享注意力/归一化并分离前馈网络,加上尽早联合引入视觉,能产生协同效应

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

提高语言数据比例能持续改善图像理解和图像生成表现,而提高图像理解数据比例能大幅提升图像生成,但会使纯语言表现略微下降。
增加图像生成数据对语言能力或图像理解能力既没有明显帮助也没有明显损害,只带来轻微波动。
在CLEVR合成测试平台上,颜色、形状等低层属性在理解与生成之间双向都无法迁移,而空间关系、大小、数量等结构性概念能从理解迁移到生成,但反方向大多失败。
共享注意力和归一化层、仅将前馈网络按模态分离(split_ffn)的设计,能在消除完全共享(dense)架构带来的模态竞争的同时保留跨模态协同效应。
从训练早期就引入视觉数据并与语言联合训练,效果优于延迟引入视觉或按顺序训练,延迟整合会引发一种"视觉懒惰"现象,即模型过度依赖语言先验而未充分优化视觉部分。

Figure 1: Impact of scaling language data on visual understanding and generation. Increasing the language ratio universally improves both vision capabilities.

Figure 2: Impact of scaling visual understanding data. It significantly benefits visual generation but degrades pure language performance.

Table 1: Extensive grid search of data mixing ratios across three axes. We evaluate models on Language, Visual Understanding, and Visual Generation. The searches confirm that while language requires a dominant token share, visual capabilities peak at highly specific and asymmetrical ratios. The optimal configuration emerges in the "Next" sweep at a 70/25/5 split for Language, Understanding, and Generation respectively.
	Mix %	Language	Visual Understanding	Visual Generation
	L	U	G	PPL ↓	Acc ↑	Gen ↑	Know ↑	OCR ↑	V-Ctr ↑	Avg ↑	DPG ↑	GenEval ↑	CLIP-Sim ↑	DiffLoss ↓
Fix MM	10	45	45	19.27	41.89	37.9	26.9	22.9	42.9	32.7	0.326	0.148	0.256	0.2984
20	40	40	17.51	44.46	45.7	30.8	24.0	44.0	36.1	0.395	0.189	0.273	0.2802
30	35	35	16.73	45.03	43.1	29.9	24.9	42.2	35.0	0.361	0.183	0.273	0.2826
40	30	30	16.31	45.79	46.1	31.2	24.2	45.6	36.8	0.331	0.186	0.269	0.2946
50	25	25	15.98	46.59	47.2	30.9	25.4	44.6	37.0	0.385	0.219	0.274	0.2804
60	20	20	15.81	46.70	44.6	33.0	24.8	43.9	36.6	0.387	0.203	0.275	0.2883
70	15	15	15.68	46.99	48.1	32.3	25.2	46.6	38.1	0.399	0.219	0.273	0.2996
80	10	10	15.57	46.85	45.9	32.8	24.0	45.2	37.0	0.388	0.216	0.271	0.2868
90	5	5	15.48	48.08	43.7	31.8	21.4	46.3	35.8	0.336	0.204	0.273	0.2894
Fix Lan	50	5	45	16.05	45.26	43.8	29.4	23.7	43.1	35.0	0.358	0.206	0.273	0.2785
50	10	40	16.03	46.18	45.9	31.0	24.2	45.6	36.7	0.401	0.221	0.281	0.2801
50	15	35	16.06	46.42	46.6	32.1	23.4	45.4	36.9	0.370	0.193	0.273	0.2834
50	20	30	16.02	46.17	46.8	31.2	23.5	45.2	36.7	0.390	0.199	0.274	0.2909
50	25	25	15.98	46.59	47.2	30.9	25.4	44.6	37.0	0.385	0.219	0.274	0.2804
50	30	20	16.01	46.05	45.5	31.1	25.2	44.7	36.6	0.370	0.208	0.273	0.2893
50	35	15	16.05	45.85	47.0	32.5	23.8	46.0	37.3	0.399	0.199	0.275	0.2821
50	40	10	16.00	46.14	47.7	32.9	26.5	45.7	38.2	0.420	0.216	0.276	0.2874
50	45	5	16.03	46.14	46.9	32.3	25.8	46.1	37.8	0.392	0.200	0.269	0.2931
Next	70	5	25	15.67	46.55	47.0	32.9	23.9	43.5	36.8	0.375	0.204	0.271	0.2823
70	10	20	15.71	46.34	46.8	30.6	22.7	44.3	36.1	0.358	0.206	0.273	0.2855
70	15	15	15.68	46.99	48.1	32.3	25.2	46.6	38.1	0.399	0.219	0.273	0.2996
70	20	10	15.65	46.65	48.0	31.9	26.3	46.3	38.1	0.401	0.221	0.272	0.2934
70	25	5	15.68	46.86	48.3	32.7	25.8	47.1	38.5	0.450	0.237	0.275	0.2868

Figure 3: Impact of scaling visual generation data. Adding visual generation causes minor fluctuations in language and most understanding tasks.

Figure 4: Overview of the synthetic CLEVR testbed. We extend the standard CLEVR vocabulary across five conceptual axes: color, shape, spatial relation, size, and object count. To isolate knowledge flow, specific target concepts (highlighted in red) are systematically ablated from targeted modality training streams.

Table 2: Scaling results and controlled baseline comparisons. We evaluate our model against three controlled baselines to validate our main design choices: data mixture recipes (Balanced Recipe), architecture design style (Dense Model), and vision alignment strategy (Late-Fusion).
Model	Language	Visual Understanding	Visual Generation
PPL ↓	Acc ↑	Gen ↑	Know ↑	OCR ↑	V-Ctr ↑	Avg ↑	DPG ↑	GenEval ↑	CLIP-Sim ↑	DiffLoss ↓
Balanced Recipe	11.97	52.86	51.50	38.90	25.15	50.14	41.42	0.676	0.467	0.310	0.261
Dense Model	12.14	52.03	50.12	36.66	25.43	49.74	40.49	0.667	0.459	0.308	0.266
Late-Fusion	12.25	51.78	49.89	37.03	26.22	49.50	40.66	0.672	0.471	0.308	0.269
Full	11.67	54.31	53.63	40.11	27.23	51.33	43.08	0.689	0.482	0.312	0.272

Figure 5: Zero-shot concept transfer results on CLEVR. Left (Color, Shape): Low-level (Color, Shape) attributes fail to transfer in either direction. Right (Relation, Size, Count): Structural concepts exhibit an asymmetric transfer. Understanding helps zero-shot generation, whereas generation largely fails to help understanding, with a minor exception for counting.

Figure 6: Concept recovery via fine-tuning. We measure how quickly models learn a missing low-level concept. Top row: Prior exposure via visual understanding provides no acceleration for color generation, but leaves a usable prior that accelerates shape generation. Bottom row: Prior exposure via visual generation acts as a booster, accelerating visual understanding learning across both color and shape.

研究结果

将语言数据比例从0%提高到80%,使图像理解在四个评测维度(General、Knowledge、OCR & Chart、Vision-Centric)上均单调提升,同时条件与无条件的扩散损失都下降。
提高图像理解数据比例显著改善了图像生成指标和扩散损失,但纯语言基准分数和困惑度略有变差。
增加图像生成数据比例只使语言准确率/困惑度以及图像理解四个维度出现轻微波动,没有明显趋势。
在CLEVR概念消融实验中,颜色和形状在理解与生成两个方向上的零样本迁移完全失败,表现降至无接触对照组水平;而空间关系、大小、数量能从理解迁移到生成,但反方向大多失败,仅计数概念有轻微例外。
架构实验表明,共享注意力和归一化、仅分离前馈网络(split_ffn)消除了完全共享(dense)架构中的竞争同时保留了协同效应,且这一结果在RAE、Raw Pixels、CLIP+VAE、AR(UniTok)四种视觉分词方案上都成立;延长纯语言预训练阶段会使视觉理解和生成性能持续下降,联合训练在几乎所有指标上都优于顺序训练。

Figure 7: Overview of data complexity progressions. Examples of visual (top) and language (bottom) data used to evaluate the impact of task complexity on modality interactions, ranging from simple synthetic patterns to complex real-world distributions.

Figure 8: Impact of task complexity on modality interaction. Left: Escalating visual task complexity gradually turns synergy into competition. Simple visual tasks (e.g., backgrounds, noise) improve language modeling, whereas complex visual distributions (SSTK, video) degrade text perplexity. Right: Introducing language universally aids visual generation, but the simplest linguistic distribution provides the maximum synergistic boost.

可应用场景

在设计统一多模态预训练时,参考语言/理解/生成大致70/25/5的数据配比方案
考虑采用共享注意力与归一化、仅按模态分离前馈网络的split_ffn式架构
在设计训练课程时,决定何时以及如何(联合还是顺序)引入视觉数据

Figure 9: Impact of parameter sharing on cross-modal performance. Fully shared (dense) parameters force modality competition, degrading both language and vision. Decoupling solely the FFNs (split_ffn) perfectly mitigates this competition while leveraging shared attention to foster strong synergy. Decoupling attention (split_ffn_attn) or normalization (split_ffn_norm) significantly diminishes these improvements, and fully isolating all parameters (split_all) yields identical results to baselines.

Figure 10: Impact of vision encoder designs on modality synergy. Left: The impact of pairing pure background images with language across different encoder configurations on language perplexity (Δ PPL). Right: The relative change in diffusion loss (%) for conditional and unconditional generation when paired with simple language. Modality synergy consistently occurs across all four visual tokenization designs.

局限与待验证事项

主要控制实验基于一个特定的1.5B至2.3B规模骨干模型以及特定数据集(SSTK、DCLM),尚未验证是否能推广到其他架构或数据组合。
基于CLEVR得出的结论来自简化的合成环境,是否适用于更复杂的真实图像和语言分布仍需进一步检验。
大规模验证仅限于一组135亿参数MoE模型、2万亿token的训练,在其他规模或模型系列上的可复现性仍需更多实验。
生成质量评判依赖另一个模型Qwen3-VL-8B-Instruct作为自动裁判,该裁判模型本身可能存在的偏差未被单独验证。

Figure 11: Timing of unification training. The x-axis represents the number of pure language tokens consumed before visual data is introduced to the training mix. While extending the initial pure language phase yields marginal improvements in unimodal text metrics like language accuracy and perplexity, it triggers a steep and consistent decline in performance across all visual understanding and visual generation benchmarks.

Figure 12: Impact of sequential versus joint pretraining across various modality orderings. The charts display the performance of six distinct sequential training paths. Solid bars denote strict sequential training, while patterned bars indicate training with a 12.5% replay buffer of previously seen modalities. The horizontal dashed line represents the simultaneous joint training baseline. The results clearly show that joint training dominates all sequential approaches across almost every metric. Although data replay slightly mitigates catastrophic forgetting, it fails to match the cross-modal synergies.

为什么重要

随着行业转向在单一网络中同时处理文本和图像的统一模型,团队此前主要靠经验法则来决定数据配比、架构和训练顺序。这项研究用受控实验取代了这种猜测,为数据配比、架构设计和训练时机提供了具体可用的方案。

本文术语

统一多模态预训练 · 从一开始就在同一个模型中联合训练文本生成、图像理解和图像生成
早期统一(Early Unification) · 从训练最初阶段就把视觉数据与语言数据一起引入,让两种模态共同演化
视觉懒惰(Vision Laziness) · 延迟引入视觉数据导致模型未充分优化视觉部分、而过度依赖语言先验的现象
split_ffn · 共享注意力和归一化层,但为每种模态单独保留前馈网络的架构设计
CLEVR · 一种可以精确控制颜色、形状、数量、空间位置的合成三维场景数据集

论文原文摘要（英文）

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pre

作者 · Junlin Han

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Junlin Han et al., arXiv:2608.05000, arxiv-nonexclusive