Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
arXiv:2608.181322026-08-20
A frozen language model plus one lightweight connector is enough to build a capable audio-understanding AI
Instead of the usual multi-stage recipe of cross-modal alignment, instruction fine-tuning, and preference optimization, the researchers froze both the audio encoder and the language model (LLM) and trained only a small connector module (a projector) between them. This connector was trained on data the LLM generated for itself, without any human-written instructions or Q&A pairs, by treating audio captions as stand-ins for the audio. The resulting model matched or beat much more heavily trained open-source audio-language models while using far less data.
What they did
Standard audio-language models are trained through alignment, instruction fine-tuning (SFT), and preference optimization, but the latter stages can erode the LLM's original general-purpose instruction-following ability -- motivating a simpler approach.
The fix keeps the audio encoder and the LLM fully frozen and trains only a lightweight two-layer projector between them; training data is built automatically by feeding audio captions (no instructions) into the LLM and using its own free-form responses as training targets.
Across four benchmarks (MMAU, MMAR, MMSU, MMAU-Pro), the version using the AudioSet-Zipformer encoder scored 80.8/77.4 on MMAU Sound, beating the previous best open-source model Audio-Flamingo 3 (79.6/75.8) despite that model training on 46x more samples and 34x more audio hours.
On MMAU-Pro's instruction-following score, the model reached 62.9/72.6, topping every open-source audio-language model listed (previous best 61.3), because keeping the LLM frozen preserved its original instruction-following skill intact.
Testing five different audio encoders and two LLMs (Qwen2.5-7B-Instruct, Qwen3-8B) showed performance is jointly capped by how much information the encoder captures and how capable the LLM is; scaling training data 10x (400K to 4M samples) barely moved multiple-choice benchmark scores but improved open-ended reasoning benchmarks.
Figure 1: Overview of the pipeline. Left: Self-Generated Data Construction. The dashed line separates two views: on the left, the real listening process, where a human hears audio x and responds; on the right, our generation surrogate. Instead of collecting human responses, we feed the paired caption c into the frozen LLM without any instruction to obtain r=g(c). The caption thus serves as a semantic surrogate for the audio, and r becomes the training target. Right: Instruction-Free Alignment-Only training. Audio x passes through a frozen encoder and a trainable projector into the same frozen LLM, again without instructions. The LLM here is identical to the one used on the left. This consistency ensures that r matches what this LLM would produce given the caption surrogate. Training the projector with cross-entropy against r therefore aligns audio representations to the LLM’s own caption-conditioned response distribution, not to external annotation.
Table 1: Comparison on all four audio-understanding benchmarks. MMAU reports test-mini / test accuracy across Sound, Music, Speech, and their average; MMAR and MMSU report the overall average; MMAU-Pro reports instruction following (IF), open-ended response, and overall average.
Model
Size
Training Data
MMAU (test-mini / test)
MMAR
MMSU
MMAU-Pro
# Samples | # Hours
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Proprietary models
GPT-4o mini Audio [62]
–
–
50.8 / 49.7
39.2 / 36.0
69.1 / 67.5
53.0 / 51.0
50.6
–
79.7
41.6
48.3
GPT-4o-Audio [62]
–
–
64.6 / 63.2
56.3 / 49.9
66.7 / 69.3
62.5 / 60.8
63.5
56.4
82.5
43.2
52.5
Gemini 2.0 Flash [30]
–
–
71.2 / 68.9
65.3 / 59.3
75.1 / 72.9
70.5 / 67.0
65.6
51.0
94.2
66.8
55.7
Gemini 2.5 Flash [31]
–
–
73.3 / 69.5
65.6 / 69.4
76.6 / 68.3
71.8 / 67.4
68.4
–
95.1
67.5
59.2
Open-source/access LALMs
SALMONN [75]
13B
2.3M | 4.4K
41.1 / 42.1
37.1 / 37.8
26.4 / 28.8
34.9 / 36.2
33.2
30.1
38.5
33.6
39.6
LTU [29]
7B
5.6M | –
20.4 / 20.7
16.0 / 15.7
15.9 / 15.3
17.4 / 17.2
19.2
22.6
–
–
–
Qwen2-Audio-Instruct [12]
7B
– | 320K
67.3 / 61.2
56.3 / 55.7
55.3 / 55.4
59.6 / 57.4
30.0
53.3
–
–
–
Qwen2.5-Omni [84]
7B
– | –
78.1 / 76.8
65.9 / 67.3
70.6 / 68.9
71.5 / 71.0
56.7
60.6
61.3
52.3
52.2
Audio-Flamingo 2 [25]
3B
5.9M | –
71.5 / 68.1
71.0 / 70.2
44.7 / 44.9
62.4 / 61.1
21.9
–
29.6
43.2
42.6
Audio-Flamingo 3 [26]
8B
26.7M | 54.4K
79.6 / 75.8
74.0 / 74.5
66.4 / 67.0
73.3 / 72.4
58.5
–
33.3
44.2
51.7
Kimi-Audio [43]
7B
– | 13.3M
75.7 / 70.7
66.8 / 65.9
62.2 / 56.6
68.2 / 64.4
–
59.3
42.3
34.5
46.6
ALARM [33]
4B
5.5M | 17K
64.0 / 59.1
54.8 / 54.2
77.2 / 73.7
65.3 / 62.4
48.7
61.3
–
–
–
Ours (AudioSet-Zipformer)
7B
576.8K | 1.6K
80.8 / 77.4
69.8 / 68.6
54.1 / 52.8
68.2 / 66.3
54.3
47.0
62.9
50.8
52.8
Ours (Whisper-large-v2)
7B
576.8K | 1.6K
75.7 / 72.5
62.9 / 61.3
60.4 / 60.1
66.3 / 64.6
52.3
50.6
72.6
55.4
48.4
Figure 2: (a) Scaling behavior of four audio encoders across increasing dataset sizes on four benchmarks. (b) Effect of caption source: comparison between ground-truth and synthetic captions.
Table 2: Effect of audio encoder. The LLM (Qwen2.5-7B-Instruct) and training data (CaptionStew 400K + 10% speech) are held fixed; only the audio encoder is swapped, with the projector downsampling rate r adjusted per encoder to maintain a post-projector token rate of 6.25–12.5 Hz.
Audio Encoder
Size
Down. r
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
7B
4
80.78
69.76
54.05
68.20
54.30
47.00
62.91
50.84
52.82
ASR-supervised pretraining
Whisper-large-v2
7B
4
75.68
62.87
60.36
66.30
52.30
50.61
72.57
55.39
48.40
Qwen3-ASR AuT encoder
7B
2
63.36
51.80
55.15
56.80
47.20
45.15
59.43
52.12
41.92
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
7B
4
68.77
50.9
45.65
55.10
48.30
43.31
67.74
53.77
45.24
Qwen3-Omni AuT encoder
7B
2
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
Table 3: Effect of LLM backbone. The audio encoder, training data, and projector configuration are held fixed; only the LLM is swapped. We compare matched and mismatched settings, where the same or different LLM is used for response generation (𝒮 generator) and alignment training.
LLM Backbone
Size
𝒮 Generator
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Qwen2.5 family
Qwen2.5-7B-Instruct
7B
Qwen2.5-7B-Instruct
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
Qwen3 family
Qwen3-8B
8B
Qwen2.5-7B-Instruct
63.44
55.18
45.43
54.70
48.10
45.84
61.09
50.32
40.54
Qwen3-8B
8B
Qwen3-8B
66.77
61.80
53.25
60.60
49.80
46.86
67.74
54.81
48.31
Table 4: Effect of targeted speech-QA SFT on the (Whisper-large-v2, Qwen2.5-7B) recipe.
Setting
MMAU (test-mini)
Sound
Music
Speech
Avg.
Alignment Only
75.68
62.87
60.36
66.30
Alignment → SFT
70.87
58.68
65.47
65.00
Table 5: Overview of the public datasets constituting CaptionStew. The table summarizes their scale, domain coverage, audio sources, and diverse captioning pipelines (from human annotation to LLM generation).
Table 6: Dataset statistics. Audio Coverage marks the audio modalities a corpus contributes: Sp = speech, So = environmental sound, Mu = music. Hours is the audio duration actually consumed in our experiments. Purpose is either Instruction-Free Alignment (IFA) or speech-QA SFT.
Table 7: Effect of projector downsampling rate r across four audio encoders. The audio-token frequency at the LLM input equals the encoder’s native frame rate divided by r.
Audio Encoder
Down. r
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
4
80.78
69.76
54.05
68.2
54.3
47.00
62.91
50.84
52.82
2
76.58
64.37
51.35
64.1
53.0
47.17
60.22
55.01
49.40
ASR-supervised pretraining
Whisper-large-v2
16
67.57
59.58
49.25
58.8
48.5
43.48
58.77
55.67
48.75
8
69.07
63.17
50.75
61.0
52.0
45.92
58.05
48.82
43.49
4
75.68
62.87
60.36
66.3
52.3
50.61
72.57
55.39
48.40
2
63.96
42.81
31.53
46.1
24.0
43.50
50.86
43.21
27.83
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
4
68.77
50.9
45.65
55.1
48.3
43.31
67.74
53.77
45.24
2
70.27
53.29
53.31
59.0
48.0
45.41
58.05
49.46
45.89
Qwen3-Omni AuT encoder
2
67.87
60.78
54.95
61.2
48.4
45.81
67.74
60.59
47.32
1
66.97
53.89
56.76
59.2
49.4
48.28
77.43
51.29
50.97
Table 8: Full scaling results across audio encoders and training-data sizes (400K, 1M, and 4M from CaptionStew, augmented with 10% speech). The LLM (Qwen2.5-7B-Instruct) and projector configuration are held fixed; the projector downsampling rate r is set per encoder to maintain a post-projector token rate of 6.25–12.5 Hz.
Audio Encoder
Training Data
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
# Samples | # Hours
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
576.8K | 1.6K
80.78
69.76
54.05
68.20
54.30
47.00
62.91
50.84
52.82
1.18M | 3.7K
80.48
67.66
53.15
67.10
57.40
54.47
67.74
58.92
51.07
4.18M | 14.2K
80.28
67.37
54.25
67.30
58.10
51.81
72.57
67.52
53.19
ASR-supervised pretraining
Whisper-large-v2
576.8K | 1.6K
75.68
62.87
60.36
66.30
52.30
50.61
72.57
55.39
48.40
1.18M | 3.7K
75.98
68.56
54.95
66.50
54.20
50.27
71.87
49.25
51.35
4.18M | 14.2K
75.98
64.37
52.55
64.30
54.90
51.56
72.88
56.27
50.29
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
576.8K | 1.6K
70.27
53.29
53.31
59.00
48.00
45.41
58.05
49.46
45.89
1.18M | 3.7K
71.07
57.78
49.75
59.50
48.30
46.36
62.91
50.40
46.39
4.18M | 14.2K
69.77
59.28
49.75
59.60
48.50
46.36
67.74
55.15
46.15
Qwen3-Omni AuT encoder
576.8K | 1.6K
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
1.18M | 3.7K
67.57
60.48
54.65
60.90
50.20
44.88
67.74
59.25
47.64
4.18M | 14.2K
69.77
61.98
52.55
61.40
53.00
42.37
62.88
62.31
48.81
Table 9: Effect of Caption Source.
Audio Encoder
Caption Source
MMAU (test-mini)
MMAR
MMSU
MMAU-Pro
Sound
Music
Speech
Avg.
Avg.
Avg.
IF
Open-ended
Avg.
Discriminative pretraining
AudioSet-Zipformer
Ground-truth
80.78
69.76
54.05
68.20
54.30
47.00
62.91
50.84
52.82
Synthetic
79.50
68.26
50.15
66.00
51.30
47.19
56.93
54.62
48.00
ASR-supervised pretraining
Whisper-large-v2
Ground-truth
69.07
63.17
50.75
61.00
52.00
45.92
58.05
48.82
43.49
Synthetic
62.76
58.98
49.55
57.10
44.10
44.26
58.51
56.68
40.20
Joint audio-language pretraining
Qwen2.5-Omni audio encoder
Ground-truth
70.27
53.29
53.31
59.00
48.00
45.41
58.05
49.46
45.89
Synthetic
50.76
46.34
41.95
46.40
36.20
48.61
77.09
40.65
39.46
Qwen3-Omni AuT encoder
Ground-truth
67.87
60.78
54.95
61.20
48.40
45.81
67.74
60.59
47.32
Synthetic
60.96
58.68
60.96
60.20
49.80
50.43
78.72
44.22
45.09
Why it matters
This offers a much cheaper way to keep audio-AI systems up to date with newer language models, since only the small connector needs retraining rather than the whole pipeline. It also shows audio-understanding models can be built without costly human-annotated question-answer datasets, since the LLM can generate its own training targets from existing captions.
Terms in this paper
Projector · a small neural network that converts audio features into a form the language model can read
Supervised Fine-Tuning (SFT) · additional training where a model is shown example questions and correct answers to learn a specific task
Instruction-Free training · training a model to respond appropriately to audio alone, without any accompanying question or instruction
Caption · a text description of what's in an audio clip, used here as a stand-in for the audio to generate training targets
Frozen parameters · model weights that are kept unchanged during training rather than being updated
Original abstract (English)
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we
