One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Represented but Ignored: A Causal Account of Prosodic Underuse in Audio-Language Models

arXiv:2608.192112026-08-21

Audio AI models can correctly hear tone of voice but still fail to use it when answering

Speech-understanding AI models often detect prosody, like whether a sentence is a question or a statement, or whether a voice sounds happy or angry, correctly inside their internal processing, yet fail to fully reflect that in their final answers. The researchers built a step-by-step diagnostic method to trace audio information from the sound input through the model's internals to its final response, showing the bottleneck is not failing to hear prosody but failing to use it. Simple, targeted edits to the model's internal state were enough to push suppressed correct answers back out.

What they did

  1. Four audio-language models (Qwen2.5-Omni, Phi-4-multimodal-instruct, Audio-Flamingo-3, DeSTA2.5) were tested with probes inserted at each stage, the audio encoder, the internal language-model layers, and the final answer, to locate where prosodic information gets lost
  2. Matched-content audio pairs (same words, different tone) were used for question/statement intonation and for happy/sad and neutral/angry emotion, isolating prosody as the only difference
  3. In most model-task combinations, prosody was well preserved by the audio encoder and still decodable from late internal states of the language model, but only partially expressed in the final answer, showing an underuse pattern in 7 of 11 tested combinations
  4. Two targeted interventions at a specific internal layer, adding a directional nudge or swapping in another clip's internal state, shifted the model's answer in the predicted direction across all 18 tested combinations, and a single such edit was often enough to recover the suppressed correct answer
  5. Breaking the internal state into a small set of sparse features (via a sparse autoencoder) showed that editing under 0.5% of these features reproduced the same recovery effect, and some emotion-related features aligned with known acoustic cues like energy variation
Figure 1: Probe ladder for localizing prosody failures in audio-LLMs. The ladder follows prosodic information from the audio pathway, through the LLM representation, to the final answer. The bottom panel maps probe outcomes to three failure modes. In F1, prosody is not perceived; in F2, prosody is internally misinterpreted; and in F3, prosody is represented correctly but underused at the decision stage. The gap between a lens-readable internal signal and behavioral performance defines the capability gap.
Figure 1: Probe ladder for localizing prosody failures in audio-LLMs. The ladder follows prosodic information from the audio pathway, through the LLM representation, to the final answer. The bottom panel maps probe outcomes to three failure modes. In F1, prosody is not perceived; in F2, prosody is internally misinterpreted; and in F3, prosody is represented correctly but underused at the decision stage. The gap between a lens-readable internal signal and behavioral performance defines the capability gap.
Table 1: Corpora used as the primary matched-content contrasts.
CorpusContrastnPatchable pairs
IViEQuestion/statement430✓
CREMA-D4-class emotion4,348✓
VESUS4-class emotion10,073✓
Figure 2: Audio-path probes for the four audio-LLMs, one panel per model. Each panel shows layer-wise probe WA for IViE Q/stmt and three emotion corpora. The x-axis is normalized encoder depth (0 = input, 1.0 = final encoder block). Markers at PROJ are the post-projector representation passed to the LLM. Solid lines are the trained encoder, dotted the random-initialized baseline (same pipeline). DeSTA2.5’s CREMA-D and ESD cells lie in its training data and are masked. Exact per-cell numbers, cross-validation folds, and held-out / training-overlap routing are in Appendix B.
Figure 2: Audio-path probes for the four audio-LLMs, one panel per model. Each panel shows layer-wise probe WA for IViE Q/stmt and three emotion corpora. The x-axis is normalized encoder depth (0 = input, 1.0 = final encoder block). Markers at PROJ are the post-projector representation passed to the LLM. Solid lines are the trained encoder, dotted the random-initialized baseline (same pipeline). DeSTA2.5’s CREMA-D and ESD cells lie in its training data and are masked. Exact per-cell numbers, cross-validation folds, and held-out / training-overlap routing are in Appendix B.
Table 2: Per-cell internal-representation and behavioral readouts across the four models. AUC = peak logit-lens AUC at layer L∗. I𝒱 = predictive V-information at L∗ in bits ± bootstrap 95% CI half-width (ceiling = label entropy: 1.0 bit for IViE, ≈2.0 bits for 4-class emotion; Appendix D.2). Verdicts follow the §4.2 taxonomy and rest on the lens and ladder. The 1.00±0.00 entry (DeSTA × IViE) reflects perfectly separable states, for which the bootstrap CI is degenerate.
CellModelAUC (L∗)I𝒱% rec.verdict
IViE Q/stmtQwen2.5-Omni0.82 (L26)0.32±0.2436%F3 underuse
Phi-4-MM0.81 (L25)0.41±0.1931%F3 underuse
AF30.83 (L24)0.23±0.2830%F3 underuse
DeSTA2.51.00 (L18)1.00±0.000%F3 (complete)
CREMA-DQwen2.5-Omni0.89 (L27)1.56±0.0648%F3 underuse
Phi-4-MM0.61 (L29)0.93±0.0923%F1 (§4.1)
AF30.99 (L25)1.75±0.0892%≈reference
DeSTA2.5–––masked (in-train)
VESUSQwen2.5-Omni0.63 (L27)0.93±0.0624%partial F2
Phi-4-MM0.56 (L28)0.62±0.0616%F1 (§4.1)
AF30.84 (L25)1.55±0.0443%F3 underuse
DeSTA2.50.81 (L32)1.49±0.0559%F3 underuse
Figure 3: Three-condition ladder schematic. The audio’s position on the recall axis sits between the text-no-cue baseline (LM lexical prior) and the text+cue reference. The reference is the model’s response when the prosodic state is given explicitly in text. The ladder measures audio’s behavioral contribution; the final F-mode diagnosis combines this ladder position with the internal lens readout.
Figure 3: Three-condition ladder schematic. The audio’s position on the recall axis sits between the text-no-cue baseline (LM lexical prior) and the text+cue reference. The reference is the model’s response when the prosodic state is given explicitly in text. The ladder measures audio’s behavioral contribution; the final F-mode diagnosis combines this ladder position with the internal lens readout.
Table 3: Direction-injection slopes at L∗ across the eighteen clean (model × contrast) cells. All slopes are positive with 95% confidence intervals excluding zero (Table 15).
ContrastQwenPhi-4AF3DeSTA
IViE Q/stmt+1.34+1.84+2.26+0.29
CREMA-D h/s+1.76+0.64+3.20masked
CREMA-D n/a+2.01+0.48+2.57masked
VESUS h/s+0.95+0.47+2.04+1.83
VESUS n/a+1.09+0.31+2.80+2.92
Figure 4: Behavioral ladders for the three main cells. Color identifies the model. Top to bottom within each block: Qwen2.5-Omni, Phi-4-MM, Audio-Flamingo-3, DeSTA2.5. Each bar spans the text-no-cue baseline to the text+cue reference; the dot marks the audio condition and the right tag its % recovery of that range. DeSTA × CREMA-D is masked (in-training). The supplementary corpus (ESD) is in Appendix D.3.
Figure 4: Behavioral ladders for the three main cells. Color identifies the model. Top to bottom within each block: Qwen2.5-Omni, Phi-4-MM, Audio-Flamingo-3, DeSTA2.5. Each bar spans the text-no-cue baseline to the text+cue reference; the dot marks the audio condition and the right tag its % recovery of that range. DeSTA × CREMA-D is masked (in-training). The supplementary corpus (ESD) is in Appendix D.3.
Table 4: Activation-patching Δ​log-odds at L∗ on VESUS. All sixteen shifts take the predicted sign with bootstrap 95% confidence intervals excluding zero, and effects are antisymmetric within each cell.
VESUSQwenPhi-4AF3DeSTA
happy/sad+0.9/−0.9+0.5/−0.6+3.3/−3.4+3.6/−3.8
neutral/angry+1.3/−1.3+0.5/−0.4+5.1/−5.2+3.6/−3.5
Figure 5: Two interventions at the lens-peak layer L∗. (A) Direction injection. A class-mean direction d=μ+−μ− is computed from positive- and negative-class L∗ states. The answer-position residual stream is then edited as hL∗last←hL∗last+α​d. (B) Activation patching. For a matched pair, we run a donor clip A to L∗. We then insert its answer-position state into the recipient clip B at the same layer and position. The rest of B’s forward pass is unchanged. Both interventions test whether changing the L∗ answer-position state changes the model’s answer-token distribution.
Figure 5: Two interventions at the lens-peak layer L∗. (A) Direction injection. A class-mean direction d=μ+−μ− is computed from positive- and negative-class L∗ states. The answer-position residual stream is then edited as hL∗last←hL∗last+α​d. (B) Activation patching. For a matched pair, we run a donor clip A to L∗. We then insert its answer-position state into the recipient clip B at the same layer and position. The rest of B’s forward pass is unchanged. Both interventions test whether changing the L∗ answer-position state changes the model’s answer-token distribution.
Table 5: Feature-level sufficiency across four models. Each entry is positive-class recall at α=0 (SAE reconstruction, no clamp) → peak under the S0.95 additive-diff clamp. DeSTA × CREMA-D is masked as in-training. Full α-sweeps in Table 20.
CellQwenPhi-4AF3DeSTA
IViE Q/stmt0.46→1.000.01→1.000.63→1.000.03→1.00
CREMA-D h/s0.87→1.000.89→1.000.98→1.00masked
CREMA-D n/a0.17→1.000.33→1.000.97→1.00masked
VESUS h/s0.39→1.000.96→0.990.68→1.000.55→1.00
VESUS n/a0.12→1.000.09→0.780.57→1.000.59→1.00
Figure 6: Acoustic correlates in sparse features. One example cell from each of three models, chosen to cover the three datasets. The cells are DeSTA on IViE question/statement, Qwen on CREMA-D neutral/angry, and AF3 on held-out VESUS happy/sad. Each panel shows the strongest class-appropriately-aligned feature among the cell’s top-3 attribution features. It is plotted against the expected acoustic cue (F0 slope for intonation, RMS-energy variation for emotion). Dots are individual clips; the line is an OLS fit; r is the Pearson correlation across the cell’s clips. Alignment is strong for the emotion cells and weak for intonation (see text); per-model, per-cell grids are in Appendix F.6.
Figure 6: Acoustic correlates in sparse features. One example cell from each of three models, chosen to cover the three datasets. The cells are DeSTA on IViE question/statement, Qwen on CREMA-D neutral/angry, and AF3 on held-out VESUS happy/sad. Each panel shows the strongest class-appropriately-aligned feature among the cell’s top-3 attribution features. It is plotted against the expected acoustic cue (F0 slope for intonation, RMS-energy variation for emotion). Dots are individual clips; the line is an OLS fit; r is the Pearson correlation across the cell’s clips. Alignment is strong for the emotion cells and weak for intonation (see text); per-model, per-cell grids are in Appendix F.6.
Table 6: All audio models used in the paper. Parameter counts are taken from the public model cards and are total architecture sizes, including text/decoder branches where applicable.
ModelHuggingFace checkpointArchitecture# ParamsRole
Probed audio-LLMs (§4.1–§4.4)
Qwen2.5-Omni-7BQwen/Qwen2.5-Omni-7BWhisper-style audio tower + 28-layer LLM7 Baudio path & LLM probed
Audio-Flamingo-3nvidia/audio-flamingo-3-hfWhisper-large-v3 + 28-layer Qwen2.5-7B LLM8.3 Baudio path & LLM probed
DeSTA2.5-AudioDeSTA-ntu/DeSTA2.5-Audio-Llama-3.1-8BWhisper-large-v3 + 32-layer Llama-3.1-8B LLM8.7 Baudio path & LLM probed
Phi-4-MMmicrosoft/Phi-4-multimodal-instructConformer audio tower + 32-layer LLM5.6 Baudio path & LLM probed
Standalone speech encoders (§4.1 calibration only)
Whisper-base.enopenai/whisper-base.enencoder–decoder transformer (English)74 Mencoder calibration
Whisper-mediumopenai/whisper-mediumencoder–decoder transformer769 Mencoder calibration
Whisper-large-v2openai/whisper-large-v2encoder–decoder transformer1.55 Bencoder calibration
Whisper-large-v3openai/whisper-large-v3encoder–decoder transformer1.55 Bencoder calibration
WavLM-basemicrosoft/wavlm-baseSSL transformer encoder95 Mencoder calibration
WavLM-largemicrosoft/wavlm-largeSSL transformer encoder317 Mencoder calibration
Figure 7: Calibration of the probing protocol on standalone speech encoders. Layer-wise weighted accuracy (speaker-disjoint LOSO) of the single-layer linear probe on a representative self-supervised encoder across the three emotion corpora and IViE Q/stmt. Solid = trained encoder, dotted = random-initialized baseline.
Figure 7: Calibration of the probing protocol on standalone speech encoders. Layer-wise weighted accuracy (speaker-disjoint LOSO) of the single-layer linear probe on a representative self-supervised encoder across the three emotion corpora and IViE Q/stmt. Solid = trained encoder, dotted = random-initialized baseline.
Table 7: Per-study dataset assignment. “main” = appears in the main-text figures and tables; “Appendix” = relegated to a supplementary appendix; “—” = not used as an analysis cell at that stage.
§4.1§4.2§4.3§4.4
Corpusaudio probelensbehaviorinjectionpatchingSAE
IViE✓mainmain✓✓✓
CREMA-D✓mainmain✓✓✓
VESUS✓mainmain✓✓✓
JL-CorpusApp. B—————
ESD-English✓Appendix D.3Appendix D.3———
Figure 8: Layer-wise lens AUC, four architectures overlaid: Qwen2.5-Omni, Phi-4-MM, Audio-Flamingo-3, DeSTA2.5. Top: IViE Q/stmt (binary AUC on the prosodic-pole tokens). Middle: CREMA-D emotion (4-class macro AUC; DeSTA masked, in-training). Bottom: held-out VESUS (4-class macro AUC). Dots mark L∗.
Figure 8: Layer-wise lens AUC, four architectures overlaid: Qwen2.5-Omni, Phi-4-MM, Audio-Flamingo-3, DeSTA2.5. Top: IViE Q/stmt (binary AUC on the prosodic-pole tokens). Middle: CREMA-D emotion (4-class macro AUC; DeSTA masked, in-training). Bottom: held-out VESUS (4-class macro AUC). Dots mark L∗.
Table 8: Per-class clip counts. IViE’s two classes are question (Q) and statement (S). All corpora are balanced in the filtered subsets.
CorpusQ/Ssadhappyangryneutral
IViE215————
CREMA-D—1,0871,0871,0871,087
VESUS—2,5172,5182,5192,519
JL-Corpus—240240240240
ESD-English—3,5003,5003,5003,500
Figure 9: TopK SAE training at L∗. The answer-position residual stream is encoded, sparsified by TopK, and decoded back to the residual-stream space. The trained feature basis is used for AtP∗ attribution and S0.95 clamp interventions.
Figure 9: TopK SAE training at L∗. The answer-position residual stream is encoded, sparsified by TopK, and decoded back to the residual-stream space. The trained feature basis is used for AtP∗ attribution and S0.95 clamp interventions.
Table 9: Binary forced-choice prompt banks (5 paraphrases per contrast), used for all §4.2–§4.4 binary cells.
ContrastIDPrompt text
Q/stmtP0Is the utterance a question or a statement? Answer in one word:
P1Did the speaker ask a question or make a statement? Answer in one word:
P2Was that a question or a statement? Answer ‘question’ or ‘statement’:
P3Is the speaker asking something, or just making a statement? In one word, ‘question’ or ‘statement’:
P4Classify the speech act: ‘question’ or ‘statement’. One word:
Happy/sadP0Does the speaker sound happy or sad? Answer in one word:
P1Is the speaker’s tone happy or sad? Reply ‘happy’ or ‘sad’:
P2Classify the speaker’s mood: ‘happy’ or ‘sad’. One word:
P3Listening to this clip, would you say the speaker is happy or sad? In one word:
P4Is the emotion in the speaker’s voice happy or sad? Answer in one word:
Neutral/angryP0Does the speaker sound angry or neutral? Answer in one word:
P1Is the speaker’s tone angry or neutral? Reply ‘angry’ or ‘neutral’:
P2Classify the speaker’s mood: ‘angry’ or ‘neutral’. One word:
P3Listening to this clip, would you say the speaker is angry or neutral? In one word:
P4Is the emotion in the speaker’s voice angry or neutral? Answer in one word:
Figure 10: Attribution concentration across model×contrast cells under the multi-corpus SAE, including the held-out VESUS cells. Features are ordered by descending |net​_​attr| and the curve is cumulative; the dotted line marks the 95% threshold defining S0.95. Qwen, Phi-4, and AF3 are shown (full per-feature attribution available); DeSTA’s |S0.95| on its clean cells (52–58) is reported in Table 24. Every cell reaches 95% within ∼100 features of dictionaries of 24,576–28,672 features.
Figure 10: Attribution concentration across model×contrast cells under the multi-corpus SAE, including the held-out VESUS cells. Features are ordered by descending |net​_​attr| and the curve is cumulative; the dotted line marks the 95% threshold defining S0.95. Qwen, Phi-4, and AF3 are shown (full per-feature attribution available); DeSTA’s |S0.95| on its clean cells (52–58) is reported in Table 24. Every cell reaches 95% within ∼100 features of dictionaries of 24,576–28,672 features.
Table 10: 4-class emotion prompt bank, used for §4.2’s 4-class emotion evaluation. The Latin-square design guarantees each emotion appears at each letter position exactly once across B1–B4.
IDPrompt text
Direct prompts (free-form generation; substring-matched)
A1What emotion is the speaker feeling? Answer in one word:
A2How does the speaker sound? Reply with one emotion: happy, sad, angry, or neutral:
A3In one word, what emotion is conveyed by the speaker’s tone?
MCQ prompts (shared instruction template, Latin-square shuffles)
Instruction template: “Listen to the audio and identify the speaker’s emotion. (A) {A} (B) {B} (C) {C} (D) {D}. Answer with just the letter A, B, C, or D:”
B1A = happy, B = sad, C = angry, D = neutral
B2A = sad, B = angry, C = neutral, D = happy
B3A = angry, B = neutral, C = happy, D = sad
B4A = neutral, B = happy, C = sad, D = angry
Figure 11: Qwen2.5-Omni: top-3 attribution features per cell vs. class-appropriate acoustic descriptors (F0 slope for Q/stmt; RMS dynamic range for emotion).
Figure 11: Qwen2.5-Omni: top-3 attribution features per cell vs. class-appropriate acoustic descriptors (F0 slope for Q/stmt; RMS dynamic range for emotion).
Table 11: Verified pole-token sets used for the first-token logit-lens analysis.
ContrastPositive poleNegative pole
Q/stmtquestion, Question, asking, query, ?statement, Statement, stating, fact, .
Happy/sadhappy, Happy, joyfulsad, Sad, upset
Neutral/angryangry, Angryneutral, Neutral
4-class direct (A1–A3)happy, sad, angry, neutral (one per class)
4-class MCQ (B1–B4)A, B, C, D (one per letter)
Figure 12: Phi-4-MM: top-3 attribution features per cell. No feature reaches |r|>0.5 with the acoustic descriptors (cf. Table 22).
Figure 12: Phi-4-MM: top-3 attribution features per cell. No feature reaches |r|>0.5 with the acoustic descriptors (cf. Table 22).
Table 12: Audio-path probe WA per (model, cell), canonical 4-class emotion (CREMA-D/ESD/VESUS) and 2-class IViE. last = final encoder layer, peak = best layer at normalized depth @, PROJ = projector output passed to the LLM, lift = PROJ trained − random-init. † masked: the corpus is in DeSTA2.5’s training data, so the cell is excluded from cross-model claims.
ModelCelllastpeak@PROJlift
Qwen2.5-OmniIViE Q/stmt0.780.790.840.81+0.26
ESD0.790.810.970.84+0.35
VESUS0.730.730.720.71+0.29
CREMA-D0.850.851.000.88+0.28
Audio-Flamingo-3IViE Q/stmt0.650.710.970.71+0.19
ESD0.790.810.910.79+0.29
VESUS0.690.710.970.69+0.31
CREMA-D0.920.921.000.92+0.35
DeSTA2.5-AudioIViE Q/stmt0.610.790.810.76+0.25
ESD†in-training data (masked)
VESUS0.710.730.910.72+0.32
CREMA-D†in-training data (masked)
Phi-4-MMIViE Q/stmt0.780.810.750.79+0.21
ESD0.630.820.500.64+0.15
VESUS0.380.520.000.42-0.01
CREMA-D0.760.840.620.54-0.06
Table 13: State-naming cues used to construct the text+cue reference. Locked per pre-registration.
ContrastPositive cueNegative cue
Q/stmt(asking a question)(making a statement)
4-class emotion (one cue per class)
(with happiness), (with sadness), (with anger), (neutrally)
Table 14: Supplementary emotion cells on ESD-English under the same protocol as the main text. In-training (model × corpus) cells are masked. Verdicts follow the §4.2 taxonomy.
CorpusModellens AUC (L∗)FloorAudioCeiling% ladderVerdict
ESD-EnglishQwen2.5-Omni0.630.1590.3710.97026%partial F2
Phi-4-MM0.530.1280.2660.96317%partial F2
AF30.900.2070.6800.99160%F3 underuse
DeSTA2.5in-training data (masked)
Table 15: Direction-injection slope per cell with 95% confidence intervals. The CI is a t-based OLS interval for the slope of the mean answer-token log-odds across the sampled α grid. All intervals exclude zero.
ModelContrastSlope [95% CI]
QwenIViE Q/stmt+1.34​[+1.23,+1.45]
QwenCREMA-D h/s+1.76​[+1.07,+2.44]
QwenCREMA-D n/a+2.01​[+1.31,+2.71]
QwenVESUS h/s+0.95​[+0.87,+1.03]
QwenVESUS n/a+1.09​[+1.03,+1.16]
Phi-4IViE Q/stmt+1.84​[+1.28,+2.41]
Phi-4CREMA-D h/s+0.64​[+0.52,+0.75]
Phi-4CREMA-D n/a+0.48​[+0.40,+0.57]
Phi-4VESUS h/s+0.47​[+0.46,+0.47]
Phi-4VESUS n/a+0.31​[+0.24,+0.39]
AF3IViE Q/stmt+2.26​[+1.71,+2.81]
AF3CREMA-D h/s+3.20​[+1.79,+4.61]
AF3CREMA-D n/a+2.57​[+0.50,+4.64]
AF3VESUS h/s+2.04​[+1.48,+2.59]
AF3VESUS n/a+2.80​[+1.29,+4.30]
DeSTAIViE Q/stmt+0.29​[+0.28,+0.30]
DeSTACREMA-D h/smasked (in-training)
DeSTACREMA-D n/amasked (in-training)
DeSTAVESUS h/s+1.83​[+1.81,+1.85]
DeSTAVESUS n/a+2.92​[+2.90,+2.93]
Table 16: Activation-patching Δ​log-odds at L∗ on CREMA-D and IViE (canonical prompt). All twenty entries take the predicted sign, with bootstrap 95% confidence intervals excluding zero.
ModelCelln−⁣→⁣++⁣→⁣−
Qwen2.5-OmniIViE Q/stmt214+1.15−1.15
Qwen2.5-OmniCREMA-D happy/sad1,087+3.23−3.17
Qwen2.5-OmniCREMA-D neutral/angry1,087+3.37−3.38
Phi-4-MMIViE Q/stmt214+1.69−1.80
Phi-4-MMCREMA-D happy/sad1,087+1.00−1.08
Phi-4-MMCREMA-D neutral/angry1,087+0.54−0.47
Audio-Flamingo-3IViE Q/stmt150+1.59−1.64
Audio-Flamingo-3CREMA-D happy/sad150+8.17−8.75
Audio-Flamingo-3CREMA-D neutral/angry150+7.95−8.34
DeSTA2.5IViE Q/stmt150+7.15−7.60
DeSTA2.5CREMA-D h/s, n/amasked (in-training)
Table 17: Positive-class recall under raw residual-stream direction injection at L∗ across the eighteen clean cells, evaluated on a 30% held-out TEST split per cell under the five-prompt majority vote (§3.3): at each α a clip’s decoded class is the majority across the five paraphrase prompts of the sign of the answer-token log-odds. The class-mean direction d=μ+−μ− is fit on the remaining 70% TRAIN split. Bold marks the per-cell working α (smallest α at which recall+ reaches 1.000); the three cells without bold do not saturate in the tested range (see text). All cells vote over the full five-prompt bank except AF3×IViE, which votes over four (the canonical prompt’s per-clip log-odds were not retained for that cell).
ModelCellα=0α=1α=2α=3α=5α=8
Qwen2.5-OmniIViE Q/stmt0.1380.7380.9541.0001.0001.000
Qwen2.5-OmniCREMA-D happy/sad0.7251.0001.0001.0001.0001.000
Qwen2.5-OmniCREMA-D neutral/angry0.3660.7591.0001.0001.0001.000
Qwen2.5-OmniVESUS happy/sad0.3880.8280.8940.9300.9951.000
Qwen2.5-OmniVESUS neutral/angry0.1680.2380.3370.5121.0001.000
Phi-4-MMIViE Q/stmt0.0150.0920.3690.8001.0001.000
Phi-4-MMCREMA-D happy/sad0.9921.0001.0001.0001.0001.000
Phi-4-MMCREMA-D neutral/angry0.4480.7090.9160.9901.0001.000
Phi-4-MMVESUS happy/sad0.9720.9740.9740.9790.9850.992
Phi-4-MMVESUS neutral/angry0.1180.1340.1550.1850.2720.647
Audio-Flamingo-3IViE Q/stmt0.3540.6921.0001.0001.0001.000
Audio-Flamingo-3CREMA-D happy/sad0.9451.0001.0001.0001.0001.000
Audio-Flamingo-3CREMA-D neutral/angry0.9581.0001.0001.0001.0001.000
Audio-Flamingo-3VESUS happy/sad0.5300.9591.0001.0001.0001.000
Audio-Flamingo-3VESUS neutral/angry0.6140.8961.0001.0001.0001.000
DeSTA2.5IViE Q/stmt0.0000.0000.0000.0000.0000.000
DeSTA2.5CREMA-D h/s, n/amasked (in-training)
DeSTA2.5VESUS happy/sad0.5500.7960.9681.0001.0001.000
DeSTA2.5VESUS neutral/angry0.5970.7440.9791.0001.0001.000
Table 18: Geometric scale of the direction-injection intervention (original Qwen/Phi-4 cells). ‖d‖ is the L2 norm of the class-mean direction d=μ+−μ−. Likewise, ‖hlast‖med is the median L2 norm of the answer-position residual stream at L∗ over the union of positive- and negative-class clips. The ratio gives the fractional displacement applied at α=1.
ModelCell‖d‖‖hlast‖med‖d‖/‖hlast‖
Qwen2.5-OmniIViE Q/stmt31.93290.097
Qwen2.5-OmniCREMA-D happy/sad61.53300.187
Qwen2.5-OmniCREMA-D neutral/angry61.83310.187
Phi-4-MMIViE Q/stmt15.51550.100
Phi-4-MMCREMA-D happy/sad11.51510.076
Phi-4-MMCREMA-D neutral/angry12.61500.084
Table 19: Activation-patching Δ​log-odds under layer and position controls. The L∗ answer-position column is the main intervention; subscripts give 95% bootstrap confidence intervals.
L=1, lastL∗, lastL∗, full
ModelCellΔn→p / Δp→nΔn→p / Δp→nΔn→p / Δp→n
Qwen2.5-OmniIViE Q/stmt−0.00[−0.01,0.00] / +0.00[−0.00,0.01]+1.15[0.96,1.35] / −1.15[−1.34,−0.96]+1.15[0.79,1.53] / −0.18[−0.50,0.13]
Qwen2.5-OmniCREMA-D happy/sad−0.00[−0.00,0.00] / +0.00[−0.00,0.01]+3.23[3.15,3.32] / −3.17[−3.25,−3.08]+1.12[0.73,1.50] / −1.52[−2.02,−1.05]
Qwen2.5-OmniCREMA-D neutral/angry+0.00[−0.00,0.01] / +0.00[−0.00,0.01]+3.37[3.27,3.48] / −3.38[−3.49,−3.28]+3.23[2.72,3.78] / −1.20[−1.57,−0.84]
Phi-4-MMIViE Q/stmt+0.00[−0.02,0.02] / +0.00[−0.01,0.02]+1.69[1.40,2.00] / −1.80[−2.12,−1.49]+3.19[2.60,3.79] / +1.65[1.07,2.21]
Phi-4-MMCREMA-D happy/sad−0.00[−0.02,0.02] / −0.00[−0.02,0.01]+1.00[0.90,1.09] / −1.08[−1.18,−0.98]−0.10[−0.36,0.14] / −2.89[−3.34,−2.46]
Phi-4-MMCREMA-D neutral/angry+0.02[−0.00,0.04] / −0.01[−0.02,0.01]+0.54[0.47,0.62] / −0.47[−0.54,−0.39]−0.37[−0.85,0.12] / −0.46[−0.74,−0.18]
Table 20: SAE-clamp recovery on the positive class, full α-sweep under the S0.95 additive-diff clamp (f+=α(μ+−μ−)), multi-corpus SAE, evaluated on held-out test clips under the five-prompt majority vote (§3.3). Bold marks the peak per cell. † = held-out VESUS. DeSTA × CREMA-D is masked as in-training. All cells vote over the full five-prompt bank except DeSTA×IViE, which votes over four (the canonical prompt’s per-clip log-odds were not retained for that cell).
ModelCellα=0α=1α=2α=3α=5α=8
Qwen2.5-OmniIViE Q/stmt0.4650.7460.8170.9301.0001.000
Qwen2.5-OmniCREMA-D happy/sad0.8720.9971.0001.0001.0001.000
Qwen2.5-OmniCREMA-D neutral/angry0.1660.5781.0001.0001.0001.000
Qwen2.5-OmniVESUS happy/sad†0.3940.8390.9260.9571.0001.000
Qwen2.5-OmniVESUS neutral/angry†0.1220.2300.2890.3930.9831.000
Phi-4-MMIViE Q/stmt0.0140.0280.2680.5630.9721.000
Phi-4-MMCREMA-D happy/sad0.8880.9671.0001.0001.0001.000
Phi-4-MMCREMA-D neutral/angry0.3300.6320.9621.0001.0001.000
Phi-4-MMVESUS happy/sad†0.9570.9700.9750.9800.9870.993
Phi-4-MMVESUS neutral/angry†0.0880.1090.1360.1640.3110.775
Audio-Flamingo-3IViE Q/stmt0.6340.7750.8870.9721.0001.000
Audio-Flamingo-3CREMA-D happy/sad0.9751.0001.0001.0001.0001.000
Audio-Flamingo-3CREMA-D neutral/angry0.9731.0001.0001.0001.0001.000
Audio-Flamingo-3VESUS happy/sad†0.6790.9841.0001.0001.0001.000
Audio-Flamingo-3VESUS neutral/angry†0.5680.8081.0001.0001.0001.000
DeSTA2.5IViE Q/stmt0.0281.0001.0001.0001.0001.000
DeSTA2.5CREMA-D h/s, n/amasked (in-training)
DeSTA2.5VESUS happy/sad†0.5520.8230.9571.0001.0001.000
DeSTA2.5VESUS neutral/angry†0.5930.7390.9681.0001.0001.000
Table 21: Qwen2.5-Omni: per-cell descriptor correlations for the top-3 attribution features (multi-corpus SAE; CREMA-D/VESUS RMS columns vs log(RMS), others raw). Bold entries mark |r|>0.5.
Cellfeat|attr|f0_slopef0_termdur_srms_meanrms_stdrms_max_min
IViE Q/stmt (|S0.95|=59)156040.44+0.30+0.29-0.20-0.05-0.07-0.09
163350.17-0.01-0.01-0.07+0.07+0.06+0.06
9640.15-0.23-0.24+0.12-0.00+0.02+0.03
CREMA-D happy/sad (|S0.95|=62)197840.44+0.11+0.10-0.11+0.02+0.03+0.03
163350.42-0.10-0.07+0.16-0.44-0.47-0.44
126010.42+0.00-0.02-0.12-0.03-0.05-0.04
CREMA-D neutral/angry (|S0.95|=51)163350.80-0.11-0.13-0.04-0.72-0.73-0.72
156040.73-0.09-0.10-0.08-0.67-0.67-0.66
197840.72-0.05-0.09-0.11-0.67-0.69-0.68
VESUS happy/sad (|S0.95|=69)197840.24+0.04+0.11-0.30-0.19-0.14-0.17
163350.14-0.02-0.03+0.07+0.10+0.12+0.13
126010.13-0.04-0.02-0.01+0.15+0.17+0.18
VESUS neutral/angry (|S0.95|=53)156040.33+0.19+0.20+0.02-0.31-0.35-0.36
163350.31+0.19+0.19+0.10-0.29-0.35-0.36
197840.28+0.20+0.25-0.07-0.35-0.39-0.40
Table 22: Phi-4-MM: per-cell descriptor correlations for the top-3 attribution features (multi-corpus SAE; CREMA-D/VESUS RMS columns vs log(RMS), others raw). Bold entries mark |r|>0.5.
Cellfeat|attr|f0_slopef0_termdur_srms_meanrms_stdrms_max_min
IViE Q/stmt (|S0.95|=33)4651.92-0.05-0.03-0.09+0.06+0.08+0.07
48451.56+0.29+0.27+0.04+0.03+0.04+0.03
153481.03-0.20-0.24+0.18+0.11+0.09+0.10
CREMA-D happy/sad (|S0.95|=59)4650.59+0.10+0.10-0.06+0.11+0.13+0.12
39020.34+0.12+0.14-0.10+0.29+0.29+0.26
153480.25-0.00+0.00-0.05-0.10-0.09-0.08
CREMA-D neutral/angry (|S0.95|=69)4650.64+0.05+0.04-0.13+0.15+0.15+0.14
127560.54+0.03+0.05-0.16+0.22+0.21+0.21
78320.14+0.08+0.06-0.02+0.26+0.25+0.26
VESUS happy/sad (|S0.95|=102)4650.19-0.03-0.09+0.25+0.14+0.12+0.14
39020.17-0.00-0.07+0.29+0.11+0.10+0.12
153480.10-0.01-0.05+0.27+0.17+0.07+0.09
VESUS neutral/angry (|S0.95|=75)4650.43+0.00-0.03+0.02+0.01+0.04+0.04
127560.23-0.06-0.08+0.02+0.13+0.10+0.10
244940.18+0.03+0.14-0.33-0.08-0.09-0.11
Table 23: Audio-Flamingo-3: per-cell descriptor correlations for the top-3 attribution features (multi-corpus SAE; CREMA-D/VESUS RMS columns vs log(RMS), others raw). Bold entries mark |r|>0.5.
Cellfeat|attr|f0_slopef0_termdur_srms_meanrms_stdrms_max_min
IViE Q/stmt (|S0.95|=37)97650.80-0.23-0.22+0.25+0.02+0.03+0.04
128090.70-0.12-0.15+0.37+0.05+0.06+0.05
77460.40-0.16-0.16+0.12+0.09+0.08+0.09
CREMA-D happy/sad (|S0.95|=47)225671.80-0.12-0.07+0.24-0.48-0.51-0.49
23631.77+0.21+0.21-0.30+0.46+0.46+0.43
97651.57-0.07-0.05+0.06-0.46-0.46-0.46
CREMA-D neutral/angry (|S0.95|=52)212442.83-0.02+0.04+0.03-0.02+0.01+0.02
97652.61-0.09-0.14-0.19-0.55-0.58-0.58
23631.55-0.03-0.08-0.22-0.10-0.14-0.16
VESUS happy/sad (|S0.95|=84)77460.93+0.03+0.04-0.05-0.04+0.01+0.00
23630.80-0.09-0.08-0.12+0.40+0.43+0.42
8280.77+0.11+0.09+0.16-0.45-0.51-0.51
VESUS neutral/angry (|S0.95|=58)97651.80+0.02+0.05-0.05-0.04-0.10-0.12
212441.80-0.03-0.04-0.10+0.07+0.19+0.19
77461.11+0.12+0.19-0.34-0.24-0.20-0.23
Table 24: DeSTA2.5: per-cell descriptor correlations for the top-3 attribution features (multi-corpus SAE; VESUS RMS columns vs log(RMS), others raw). Bold entries mark |r|>0.5. CREMA-D is masked as in-training.
Cellfeat|attr|f0_slopef0_termdur_srms_meanrms_stdrms_max_min
IViE Q/stmt (|S0.95|=54)151518.79-0.28-0.24-0.02-0.03-0.05-0.04
179906.14+0.28+0.28-0.11+0.01+0.04+0.03
159305.11+0.29+0.26+0.01+0.04+0.06+0.04
CREMA-D happy/sad, neutral/angrymasked (in-training)
VESUS happy/sad (|S0.95|=58)280702.10-0.14-0.23+0.13+0.42+0.47+0.49
230001.64+0.02-0.07+0.32-0.08-0.07-0.05
185691.63+0.08+0.13-0.09-0.20-0.22-0.23
VESUS neutral/angry (|S0.95|=52)185693.18+0.08+0.13-0.09-0.20-0.22-0.23
62782.82+0.10+0.19-0.21-0.28-0.30-0.32
308072.81-0.12-0.24+0.16+0.35+0.40+0.42

Why it matters

For voice assistants and chatbots to respond appropriately to how something is said, not just what is said, developers need to know exactly where the failure occurs to fix it. This work shows the problem is often not perception but underuse of already-available internal information, pointing toward fixes that adjust how models use existing representations rather than retraining from scratch.

Terms in this paper

  • audio-LLM · a large language model extended to take audio speech as input
  • probe · a small classifier used to test whether specific information is present at a given point inside a model
  • logit lens · a technique that reads out a model's intermediate layer as if it were final output, to see what that layer already knows
  • activation patching · an experiment where an internal state from one input is inserted into another input at the same position to test its causal effect
  • sparse autoencoder (SAE) · a tool that decomposes a complex internal representation into a small number of interpretable features

Original abstract (English)

Human speech is richly expressive, with prosody carrying linguistic and emotional information beyond the lexical content. A capable large audio-language model (audio-LLM) should therefore support expressive speech understanding, not only transcribing what was said but also interpreting how it was said. Yet behavioral evaluations alone cannot reveal why a model fails on prosodic input. An error may reflect loss of acoustic information, incorrect internal interpretation, or failure to use a representation that is already available inside the model. We introduce a stage-specific probe ladder for localizing these failure modes in audio-LLMs. Across four understanding-only audio-LLMs, prosodic information is usually preserved in the audio path and decodable in late LLM states. Yet it is only partially expressed in the model's final response. We test the causal status of this latent representation with targeted hidden-state interventions. Every intervention shifts the answer distribution in the predicted direction, and in most model--task cells a single edit at the relevant layer is sufficient to drive the model toward the suppressed prosodic decision, though this recovery is directional rather than a selective restoration of the correct class. Feature-level analysis further suggests that this recoverable signal can be expressed through a small subspace. Some of the highest-attribution features in this analysis align with acoustic cues known to carry prosodic information. Within the matched-content contrasts we test, these results locate the recurring bottleneck not in perceiving prosody but in using it. Models that hear and correctly represent a prosodic cue can still fail to express it in their answers.

Authors · Linkai Peng, Baorian Nuchged

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Linkai Peng et al., arXiv:2608.19211, CC BY 4.0