Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

arXiv:2608.050002026-08-04

A controlled study of how language, image understanding, and image generation help or hurt each other when trained together in one model

This paper systematically tests how knowledge flows between text, visual understanding, and visual generation when they are pretrained jointly in a single 'unified multimodal' model. Using both large-scale real data and a controlled synthetic CLEVR benchmark, it shows that transfer between these three abilities is asymmetric and depends on the concept involved. The findings are turned into practical data-mixing and architecture recipes and validated by training 13.5B MoE models on 2T tokens.

METAL LAB explanatory visual

Directions of knowledge flow in unified multimodal pretraining

Evidence statusMeasured results reported

Language dataScaling DCLM language data improves both visual understanding and visual generation
Visual understanding dataScaling understanding data strongly boosts generation but slightly hurts pure language
Visual generation dataScaling generation data causes only minor fluctuations in language and understanding, no clear trend
CLEVR concept transferColor/shape fail to transfer either way; relation/size/count transfer mainly understanding-to-generation
Architecture & timing (split_ffn, early unification)Shared attention/norm with split FFN, plus early/joint introduction of vision, creates synergy

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

Increasing the ratio of language data steadily improves both visual understanding and visual generation, while increasing visual understanding data strongly boosts visual generation but slightly hurts pure language performance.
Adding more visual generation data does not clearly help or hurt language or understanding performance, only causing minor fluctuations.
In the CLEVR synthetic testbed, low-level attributes like color and shape fail to transfer in either direction between understanding and generation, while structural concepts like spatial relation, size, and count transfer from understanding to generation but mostly not the other way.
Sharing attention and normalization layers while keeping feed-forward networks modality-specific (split_ffn) removes the competition seen in a fully shared (dense) model while preserving cross-modal synergy.
Introducing vision early and training all modalities jointly outperforms introducing vision late or training modalities sequentially, and delayed integration leads to a 'vision laziness' effect where the model leans on language priors instead of properly learning vision.

Figure 1: Impact of scaling language data on visual understanding and generation. Increasing the language ratio universally improves both vision capabilities.

Figure 2: Impact of scaling visual understanding data. It significantly benefits visual generation but degrades pure language performance.

Table 1: Extensive grid search of data mixing ratios across three axes. We evaluate models on Language, Visual Understanding, and Visual Generation. The searches confirm that while language requires a dominant token share, visual capabilities peak at highly specific and asymmetrical ratios. The optimal configuration emerges in the "Next" sweep at a 70/25/5 split for Language, Understanding, and Generation respectively.
	Mix %	Language	Visual Understanding	Visual Generation
	L	U	G	PPL ↓	Acc ↑	Gen ↑	Know ↑	OCR ↑	V-Ctr ↑	Avg ↑	DPG ↑	GenEval ↑	CLIP-Sim ↑	DiffLoss ↓
Fix MM	10	45	45	19.27	41.89	37.9	26.9	22.9	42.9	32.7	0.326	0.148	0.256	0.2984
20	40	40	17.51	44.46	45.7	30.8	24.0	44.0	36.1	0.395	0.189	0.273	0.2802
30	35	35	16.73	45.03	43.1	29.9	24.9	42.2	35.0	0.361	0.183	0.273	0.2826
40	30	30	16.31	45.79	46.1	31.2	24.2	45.6	36.8	0.331	0.186	0.269	0.2946
50	25	25	15.98	46.59	47.2	30.9	25.4	44.6	37.0	0.385	0.219	0.274	0.2804
60	20	20	15.81	46.70	44.6	33.0	24.8	43.9	36.6	0.387	0.203	0.275	0.2883
70	15	15	15.68	46.99	48.1	32.3	25.2	46.6	38.1	0.399	0.219	0.273	0.2996
80	10	10	15.57	46.85	45.9	32.8	24.0	45.2	37.0	0.388	0.216	0.271	0.2868
90	5	5	15.48	48.08	43.7	31.8	21.4	46.3	35.8	0.336	0.204	0.273	0.2894
Fix Lan	50	5	45	16.05	45.26	43.8	29.4	23.7	43.1	35.0	0.358	0.206	0.273	0.2785
50	10	40	16.03	46.18	45.9	31.0	24.2	45.6	36.7	0.401	0.221	0.281	0.2801
50	15	35	16.06	46.42	46.6	32.1	23.4	45.4	36.9	0.370	0.193	0.273	0.2834
50	20	30	16.02	46.17	46.8	31.2	23.5	45.2	36.7	0.390	0.199	0.274	0.2909
50	25	25	15.98	46.59	47.2	30.9	25.4	44.6	37.0	0.385	0.219	0.274	0.2804
50	30	20	16.01	46.05	45.5	31.1	25.2	44.7	36.6	0.370	0.208	0.273	0.2893
50	35	15	16.05	45.85	47.0	32.5	23.8	46.0	37.3	0.399	0.199	0.275	0.2821
50	40	10	16.00	46.14	47.7	32.9	26.5	45.7	38.2	0.420	0.216	0.276	0.2874
50	45	5	16.03	46.14	46.9	32.3	25.8	46.1	37.8	0.392	0.200	0.269	0.2931
Next	70	5	25	15.67	46.55	47.0	32.9	23.9	43.5	36.8	0.375	0.204	0.271	0.2823
70	10	20	15.71	46.34	46.8	30.6	22.7	44.3	36.1	0.358	0.206	0.273	0.2855
70	15	15	15.68	46.99	48.1	32.3	25.2	46.6	38.1	0.399	0.219	0.273	0.2996
70	20	10	15.65	46.65	48.0	31.9	26.3	46.3	38.1	0.401	0.221	0.272	0.2934
70	25	5	15.68	46.86	48.3	32.7	25.8	47.1	38.5	0.450	0.237	0.275	0.2868

Figure 3: Impact of scaling visual generation data. Adding visual generation causes minor fluctuations in language and most understanding tasks.

Figure 4: Overview of the synthetic CLEVR testbed. We extend the standard CLEVR vocabulary across five conceptual axes: color, shape, spatial relation, size, and object count. To isolate knowledge flow, specific target concepts (highlighted in red) are systematically ablated from targeted modality training streams.

Table 2: Scaling results and controlled baseline comparisons. We evaluate our model against three controlled baselines to validate our main design choices: data mixture recipes (Balanced Recipe), architecture design style (Dense Model), and vision alignment strategy (Late-Fusion).
Model	Language	Visual Understanding	Visual Generation
PPL ↓	Acc ↑	Gen ↑	Know ↑	OCR ↑	V-Ctr ↑	Avg ↑	DPG ↑	GenEval ↑	CLIP-Sim ↑	DiffLoss ↓
Balanced Recipe	11.97	52.86	51.50	38.90	25.15	50.14	41.42	0.676	0.467	0.310	0.261
Dense Model	12.14	52.03	50.12	36.66	25.43	49.74	40.49	0.667	0.459	0.308	0.266
Late-Fusion	12.25	51.78	49.89	37.03	26.22	49.50	40.66	0.672	0.471	0.308	0.269
Full	11.67	54.31	53.63	40.11	27.23	51.33	43.08	0.689	0.482	0.312	0.272

Figure 5: Zero-shot concept transfer results on CLEVR. Left (Color, Shape): Low-level (Color, Shape) attributes fail to transfer in either direction. Right (Relation, Size, Count): Structural concepts exhibit an asymmetric transfer. Understanding helps zero-shot generation, whereas generation largely fails to help understanding, with a minor exception for counting.

Figure 6: Concept recovery via fine-tuning. We measure how quickly models learn a missing low-level concept. Top row: Prior exposure via visual understanding provides no acceleration for color generation, but leaves a usable prior that accelerates shape generation. Bottom row: Prior exposure via visual generation acts as a booster, accelerating visual understanding learning across both color and shape.

Findings

Raising the language ratio from 0% to 80% monotonically improved visual understanding across all four evaluation axes (General, Knowledge, OCR & Chart, Vision-Centric) and reduced both conditional and unconditional diffusion loss for visual generation.
Increasing the visual understanding data ratio markedly improved visual generation metrics and diffusion loss, while pure language benchmark scores and perplexity slightly worsened.
Scaling visual generation data caused only minor fluctuations, without a clear trend, in language accuracy/perplexity and in visual understanding across General, Knowledge, OCR & Chart, and Vision-Centric axes.
In the CLEVR concept-ablation study, color and shape showed complete zero-shot transfer failure in both directions, collapsing to zero-exposure control levels, while spatial relation, size, and count transferred from understanding to generation but largely failed in the reverse direction, with a minor exception for counting.
Architecture experiments showed that sharing attention and normalization while splitting only the FFN (split_ffn) eliminated the competition seen with fully shared (dense) parameters while preserving synergy, and this held across four visual tokenization designs (RAE, Raw Pixels, CLIP+VAE, AR/UniTok); extending the pure-language phase before introducing vision caused a steep decline in visual understanding and generation performance, and joint training outperformed sequential training across almost every metric.

Figure 7: Overview of data complexity progressions. Examples of visual (top) and language (bottom) data used to evaluate the impact of task complexity on modality interactions, ranging from simple synthetic patterns to complex real-world distributions.

Figure 8: Impact of task complexity on modality interaction. Left: Escalating visual task complexity gradually turns synergy into competition. Simple visual tasks (e.g., backgrounds, noise) improve language modeling, whereas complex visual distributions (SSTK, video) degrade text perplexity. Right: Introducing language universally aids visual generation, but the simplest linguistic distribution provides the maximum synergistic boost.

Where it can be used

Choosing data-mixing ratios (e.g., roughly 70/25/5 for language/understanding/generation) when designing unified multimodal pretraining
Considering a split_ffn-style architecture that shares attention and normalization but separates feed-forward layers per modality
Deciding when and how (jointly vs. sequentially) to introduce visual data in a training curriculum

Figure 9: Impact of parameter sharing on cross-modal performance. Fully shared (dense) parameters force modality competition, degrading both language and vision. Decoupling solely the FFNs (split_ffn) perfectly mitigates this competition while leveraging shared attention to foster strong synergy. Decoupling attention (split_ffn_attn) or normalization (split_ffn_norm) significantly diminishes these improvements, and fully isolating all parameters (split_all) yields identical results to baselines.

Figure 10: Impact of vision encoder designs on modality synergy. Left: The impact of pairing pure background images with language across different encoder configurations on language perplexity (Δ PPL). Right: The relative change in diffusion loss (%) for conditional and unconditional generation when paired with simple language. Modality synergy consistently occurs across all four visual tokenization designs.

Limits and open work

The main controlled experiments use a specific 1.5B-2.3B backbone and specific datasets (SSTK, DCLM), so generalization to other architectures or data mixes is not verified.
The CLEVR-based conclusions come from a simplified synthetic environment, and whether they hold for more complex real-world image and language distributions needs further checking.
Large-scale validation is limited to a single set of 13.5B MoE models trained on 2T tokens, so reproducibility at other scales or with other model families requires further experiments.
Generation quality judgments rely on another model, Qwen3-VL-8B-Instruct, as an automatic judge, and the potential bias of this judging model itself has not been separately verified.

Figure 11: Timing of unification training. The x-axis represents the number of pure language tokens consumed before visual data is introduced to the training mix. While extending the initial pure language phase yields marginal improvements in unimodal text metrics like language accuracy and perplexity, it triggers a steep and consistent decline in performance across all visual understanding and visual generation benchmarks.

Figure 12: Impact of sequential versus joint pretraining across various modality orderings. The charts display the performance of six distinct sequential training paths. Solid bars denote strict sequential training, while patterned bars indicate training with a 12.5% replay buffer of previously seen modalities. The horizontal dashed line represents the simultaneous joint training baseline. The results clearly show that joint training dominates all sequential approaches across almost every metric. Although data replay slightly mitigates catastrophic forgetting, it fails to match the cross-modal synergies.

Why it matters

As the field moves toward unified models that handle text and images in a single network, teams have mostly relied on intuition to decide data ratios, architecture, and training schedules. This work replaces that guesswork with controlled evidence and offers concrete recipes for data mixing, architecture design, and training timing.

Terms in this paper

Unified multimodal pretraining · Training text generation, image understanding, and image generation together in one model from the start
Early Unification · Introducing visual data alongside language data from the very beginning of training so both modalities co-evolve
Vision laziness · A phenomenon where delayed integration of vision causes the model to under-optimize its visual pathways and rely on language priors
split_ffn · An architecture where attention and normalization layers are shared but feed-forward layers are kept separate per modality
CLEVR · A synthetic dataset of rendered scenes where color, shape, count, and spatial relations can be precisely controlled

Original abstract (English)

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pre

Authors · Junlin Han

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Junlin Han et al., arXiv:2608.05000, arxiv-nonexclusive