Figure 1: Overview of the pipeline. Left: Self-Generated Data Construction. The dashed line separates two views: on the left, the real listening process, where a human hears audio x and responds; on the right, our generation surrogate. Instead of collecting human responses, we feed the paired caption c into the frozen LLM without any instruction to obtain r=g(c). The caption thus serves as a semantic surrogate for the audio, and r becomes the training target. Right: Instruction-Free Alignment-Only training. Audio x passes through a frozen encoder and a trainable projector into the same frozen LLM, again without instructions. The LLM here is identical to the one used on the left. This consistency ensures that r matches what this LLM would produce given the caption surrogate. Training the projector with cross-entropy against r therefore aligns audio representations to the LLM’s own caption-conditioned response distribution, not to external annotation.
Table 1: Comparison on all four audio-understanding benchmarks. MMAU reports test-mini / test accuracy across Sound, Music, Speech, and their average; MMAR and MMSU report the overall average; MMAU-Pro reports instruction following (IF), open-ended response, and overall average.
Model
Size
Training Data
MMAU (test-mini / test)
MMAR
MMSU
MMAU-Pro
# Samples | # Hours
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Proprietary models
GPT-4o mini Audio [62]
–
–
50.8 / 49.7
39.2 / 36.0
69.1 / 67.5
53.0 / 51.0
50.6
–
79.7
41.6
48.3
GPT-4o-Audio [62]
–
–
64.6 / 63.2
56.3 / 49.9
66.7 / 69.3
62.5 / 60.8
63.5
56.4
82.5
43.2
52.5
Gemini 2.0 Flash [30]
–
–
71.2 / 68.9
65.3 / 59.3
75.1 / 72.9
70.5 / 67.0
65.6
51.0
94.2
66.8
55.7
Gemini 2.5 Flash [31]
–
–
73.3 / 69.5
65.6 / 69.4
76.6 / 68.3
71.8 / 67.4
68.4
–
95.1
67.5
59.2
Open-source/access LALMs
SALMONN [75]
13B
2.3M | 4.4K
41.1 / 42.1
37.1 / 37.8
26.4 / 28.8
34.9 / 36.2
33.2
30.1
38.5
33.6
39.6
LTU [29]
7B
5.6M | –
20.4 / 20.7
16.0 / 15.7
15.9 / 15.3
17.4 / 17.2
19.2
22.6
–
–
–
Qwen2-Audio-Instruct [12]
7B
– | 320K
67.3 / 61.2
56.3 / 55.7
55.3 / 55.4
59.6 / 57.4
30.0
53.3
–
–
–
Qwen2.5-Omni [84]
7B
– | –
78.1 / 76.8
65.9 / 67.3
70.6 / 68.9
71.5 / 71.0
56.7
60.6
61.3
52.3
52.2
Audio-Flamingo 2 [25]
3B
5.9M | –
71.5 / 68.1
71.0 / 70.2
44.7 / 44.9
62.4 / 61.1
21.9
–
29.6
43.2
42.6
Audio-Flamingo 3 [26]
8B
26.7M | 54.4K
79.6 / 75.8
74.0 / 74.5
66.4 / 67.0
73.3 / 72.4
58.5
–
33.3
44.2
51.7
Kimi-Audio [43]
7B
– | 13.3M
75.7 / 70.7
66.8 / 65.9
62.2 / 56.6
68.2 / 64.4
–
59.3
42.3
34.5
46.6
ALARM [33]
4B
5.5M | 17K
64.0 / 59.1
54.8 / 54.2
77.2 / 73.7
65.3 / 62.4
48.7
61.3
–
–
–
Ours (AudioSet-Zipformer)
7B
576.8K | 1.6K
80.8 / 77.4
69.8 / 68.6
54.1 / 52.8
68.2 / 66.3
54.3
47.0
62.9
50.8
52.8
Ours (Whisper-large-v2)
7B
576.8K | 1.6K
75.7 / 72.5
62.9 / 61.3
60.4 / 60.1
66.3 / 64.6
52.3
50.6
72.6
55.4
48.4
Figure 2: (a) Scaling behavior of four audio encoders across increasing dataset sizes on four benchmarks. (b) Effect of caption source: comparison between ground-truth and synthetic captions.
Table 2: Effect of audio encoder. The LLM (Qwen2.5-7B-Instruct) and training data (CaptionStew 400K + 10% speech) are held fixed; only the audio encoder is swapped, with the projector downsampling rate r adjusted per encoder to maintain a post-projector token rate of 6.25–12.5 Hz.
Audio Encoder
Size
Down. r
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
7B
4
80.78
69.76
54.05
68.20
54.30
47.00
62.91
50.84
52.82
ASR-supervised pretraining
Whisper-large-v2
7B
4
75.68
62.87
60.36
66.30
52.30
50.61
72.57
55.39
48.40
Qwen3-ASR AuT encoder
7B
2
63.36
51.80
55.15
56.80
47.20
45.15
59.43
52.12
41.92
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
7B
4
68.77
50.9
45.65
55.10
48.30
43.31
67.74
53.77
45.24
Qwen3-Omni AuT encoder
7B
2
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
Table 3: Effect of LLM backbone. The audio encoder, training data, and projector configuration are held fixed; only the LLM is swapped. We compare matched and mismatched settings, where the same or different LLM is used for response generation (𝒮 generator) and alignment training.
LLM Backbone
Size
𝒮 Generator
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Qwen2.5 family
Qwen2.5-7B-Instruct
7B
Qwen2.5-7B-Instruct
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
Qwen3 family
Qwen3-8B
8B
Qwen2.5-7B-Instruct
63.44
55.18
45.43
54.70
48.10
45.84
61.09
50.32
40.54
Qwen3-8B
8B
Qwen3-8B
66.77
61.80
53.25
60.60
49.80
46.86
67.74
54.81
48.31
Table 4: Effect of targeted speech-QA SFT on the (Whisper-large-v2, Qwen2.5-7B) recipe.
Setting
MMAU (test-mini)
Sound
Music
Speech
Avg.
Alignment Only
75.68
62.87
60.36
66.30
Alignment → SFT
70.87
58.68
65.47
65.00
Table 5: Overview of the public datasets constituting CaptionStew. The table summarizes their scale, domain coverage, audio sources, and diverse captioning pipelines (from human annotation to LLM generation).
Table 6: Dataset statistics. Audio Coverage marks the audio modalities a corpus contributes: Sp = speech, So = environmental sound, Mu = music. Hours is the audio duration actually consumed in our experiments. Purpose is either Instruction-Free Alignment (IFA) or speech-QA SFT.
Table 7: Effect of projector downsampling rate r across four audio encoders. The audio-token frequency at the LLM input equals the encoder’s native frame rate divided by r.
Audio Encoder
Down. r
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
4
80.78
69.76
54.05
68.2
54.3
47.00
62.91
50.84
52.82
2
76.58
64.37
51.35
64.1
53.0
47.17
60.22
55.01
49.40
ASR-supervised pretraining
Whisper-large-v2
16
67.57
59.58
49.25
58.8
48.5
43.48
58.77
55.67
48.75
8
69.07
63.17
50.75
61.0
52.0
45.92
58.05
48.82
43.49
4
75.68
62.87
60.36
66.3
52.3
50.61
72.57
55.39
48.40
2
63.96
42.81
31.53
46.1
24.0
43.50
50.86
43.21
27.83
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
4
68.77
50.9
45.65
55.1
48.3
43.31
67.74
53.77
45.24
2
70.27
53.29
53.31
59.0
48.0
45.41
58.05
49.46
45.89
Qwen3-Omni AuT encoder
2
67.87
60.78
54.95
61.2
48.4
45.81
67.74
60.59
47.32
1
66.97
53.89
56.76
59.2
49.4
48.28
77.43
51.29
50.97
Table 8: Full scaling results across audio encoders and training-data sizes (400K, 1M, and 4M from CaptionStew, augmented with 10% speech). The LLM (Qwen2.5-7B-Instruct) and projector configuration are held fixed; the projector downsampling rate r is set per encoder to maintain a post-projector token rate of 6.25–12.5 Hz.
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we
