Towards Quantifying Benchmark Optimization in ASR Models

arXiv:2608.199362026-08-19

Some top speech-recognition models are quietly copying answer keys instead of listening

Researchers built test cases where the audio alone cannot determine the correct transcript, such as places where the reference transcript is wrong, where a number's audio is deleted, or where a word could be spelled two equally valid ways. Testing 11 open-source ASR models, they found the very models that score best on public leaderboards were most likely to output the benchmark's exact reference text even when it contradicted, ignored, or wasn't supported by what was actually said. This benchmark-gaming behavior was tied to narrow acoustic cues from the benchmark itself and could be turned on or off by editing internal model activations or by splicing in extra audio.

What they did

The team probed 11 open-source ASR models using cases where audio doesn't fully determine the correct transcript: reference errors, digitally silenced numbers, and words with two valid spellings
On public benchmarks like VoxPopuli, the six models with the best word error rate (WER, 5.4-5.8%) also reproduced erroneous benchmark reference text most often (0.18-0.30 rate), while models with WER of 6.5% or worse rarely did (0.10 or below)
Cloned voices of speakers actually present in the benchmark's test set triggered the same reference-copying behavior, but the behavior weakened sharply on generic voices or newly recorded speakers unseen during training, showing the trigger is narrowly tied to benchmark-specific acoustic cues
The behavior could be causally switched on or off by appending unrelated benchmark-style audio to a clip, or by adding/removing a specific direction in the model's internal activations (low-rank linear steering)
Overall, the findings show reported benchmark scores for high-performing ASR models can be inflated by benchmark-specific shortcuts rather than genuinely better transcription ability

Figure 2: Cross-model audit on VoxPopuli. WER (%) is the VoxPopuli-test score from the June 2026 Open ASR Leaderboard [38]. Kimi Audio is not on the leaderboard, and its score is computed using the leaderboard’s scoring. Consensus-panel members are scored against edits flagged unanimously by the remaining three members (§3.3).

Figure 3: (3(a)) Masked-number accept-ref per corpus. (3(b)) Orthographic switch rate on archaic spacing.

Table 1: accept-ref on VoxPopuli-AA human-annotated edits, beside the consensus rates of Figure 2. Ordering is nearly identical aside from Granite and Canary swapping places.
	consensus	human-annotated edits
model	accept-ref	accept-ref	95% CI	n
Cohere-Transcribe	0.30	0.52	[0.47, 0.58]	253/483
Granite-Speech-4.1-2B	0.21	0.42	[0.36, 0.47]	211/508
Canary-Qwen-2.5B	0.23	0.41	[0.36, 0.47]	210/507
Higgs-Audio-v3-8B	0.21	0.39	[0.33, 0.44]	200/517
Phi-4-Multimodal	0.19	0.38	[0.33, 0.44]	195/510
Parakeet-TDT-0.6B-v2	0.18	0.38	[0.32, 0.43]	192/512
Qwen3-ASR-0.6B	0.09	0.19	[0.15, 0.24]	98/518
Moonshine-Streaming	0.06	0.14	[0.11, 0.19]	74/516
Voxtral-Mini-3B	0.04	0.09	[0.06, 0.12]	47/527
Kimi-Audio-7B	0.03	0.07	[0.05, 0.11]	40/536
Whisper-Large-v3	0.02	0.08	[0.05, 0.11]	39/514

Figure 4: Difference in masked-number recovery between LibriSpeech test-set narrator clones and held-out libri-fresh narrator clones reading identical sentences (test-set minus held-out; positive values indicate greater recovery for test-set narrator clones). Sentence-clustered bootstrap 95% confidence intervals.

Table 2: Reference-disagreement accept-ref on consensus edits as audio context is removed or the trigger is ablated. truncated cuts the audio to a tight window around the edit span (±1 aligned word ±0.25 s); donor ablated appends an 8 s conversational donor to the full clip; activation ablated projects the learned register direction out of a single encoder layer.
model	full	truncated	donor ablated	activation ablated
Cohere-Transcribe	0.30	0.13	0.06	0.04
Canary-Qwen-2.5B	0.23	0.12	0.05	0.02
Granite-Speech-4.1-2B	0.21	0.12	0.20	0.14
Higgs-Audio-v3-8B	0.21	0.12	0.03	–
Phi-4-Multimodal	0.19	0.09	0.05	0.20
Parakeet-TDT-0.6B-v2	0.18	0.08	0.06	0.01
Qwen3-ASR-0.6B	0.09	0.09	0.04	–
Moonshine-Streaming	0.06	0.07	0.04	–
Voxtral-Mini-3B	0.04	0.05	0.03	–
Whisper-Large-v3	0.02	0.05	0.02	–
Kimi-Audio-7B	0.03	0.06	0.04	–

Figure 5: The trigger battery, VoxPopuli. Top row: voice conditions on identical transcripts—(5(a)) reference-disagreement and (5(b)) masked-number accept-ref. Bottom row: the same probes with the trigger removed instead of the voice varied—truncation to the edit, an appended 8 s conversational donor, or the learned register direction projected out of one encoder layer (5(c), 5(d)). Wilson 95% CIs.

Table 3: Full-probe accept-ref on real audio with each corpus’s own content: VoxPopuli-test vs ep-fresh. Masked columns score the corpus-paired subsets, so the VoxPopuli masked rates differ from the full masked set of §4.
	consensus accept-ref	masked accept-ref
model	VoxPopuli	ep-fresh	VoxPopuli	ep-fresh
Cohere-Transcribe	0.304	0.122	0.185	0.074
Canary-Qwen-2.5B	0.233	0.150	0.051	0.062
Granite-Speech-4.1-2B	0.208	0.117	0.051	0.062
Higgs-Audio-v3-8B	0.211	0.168	0.070	0.040
Phi-4-Multimodal	0.188	0.193	0.076	0.044
Parakeet-TDT-0.6B-v2	0.181	0.120	0.038	0.029
Qwen3-ASR-0.6B	0.092	0.091	0.051	0.015
Moonshine-Streaming	0.056	0.099	0.013	0.018
Voxtral-Mini-3B	0.035	0.128	0.076	0.062
Whisper-Large-v3	0.025	0.071	0.083	0.062
Kimi-Audio-7B	0.028	0.144	0.013	0.018

Table 4: Audio lift λ⁡(r) (Eq. 1, nats/char) of the silenced number span by voice condition (115 paired sentences passing the intelligibility gate in every clone condition). Diff columns: paired differences, bootstrap 95% CIs; bold marks CIs excluding zero. Whisper’s lift rises on clean TTS, so a drop in lift is likely not a synthesis artifact; Qwen3’s lift is negative in every condition, so its diffs do not indicate recovery. Parakeet-TDT has no teacher-forced readout.
	real	vox-clone	ep-fresh	generic	real−ep-fresh	real−generic
Cohere-Transcribe	+1.52	+1.26	+0.92	+0.54	+0.60 [+0.30,+0.94]	+0.98 [+0.66,+1.33]
Canary-Qwen-2.5B	+1.22	+1.05	+0.65	+0.77	+0.57 [+0.27,+0.90]	+0.45 [+0.17,+0.71]
Granite-Speech-4.1-2B	+0.12	+0.15	+0.16	+0.01	−0.04 [−0.24,+0.17]	+0.11 [−0.11,+0.32]
Phi-4-Multimodal	+0.46	+0.51	+0.24	+0.27	+0.23 [+0.01,+0.50]	+0.19 [+0.04,+0.35]
Higgs-Audio-v3-8B	+0.50	+0.57	+0.37	+0.11	+0.13 [−0.00,+0.26]	+0.39 [+0.23,+0.56]
Whisper-Large-v3	+0.70	+0.89	+0.85	+1.01	−0.16 [−0.37,+0.06]	−0.31 [−0.47,−0.15]
Moonshine-Streaming	+0.31	+0.37	+0.25	+0.15	+0.06 [−0.08,+0.21]	+0.16 [+0.00,+0.32]
Kimi-Audio-7B	+0.24	+0.32	+0.06	+0.16	+0.17 [−0.00,+0.37]	+0.08 [−0.06,+0.22]
Qwen3-ASR-0.6B	−0.60	−0.56	−1.06	−1.14	+0.46 [+0.19,+0.75]	+0.54 [+0.31,+0.76]
Voxtral-Mini-3B	−0.13	+0.07	−0.12	−0.05	−0.01 [−0.17,+0.16]	−0.08 [−0.23,+0.07]

Figure 6: Switching the benchmark-optimized policy on and off. (6(a)) Input level: On real clips (top left) a conversational donor collapses accept-ref while a VoxPopuli donor leaves it intact; on ep-fresh clones of the same sentences (top right) a VoxPopuli donor re-ignites it while the conversational donor does not. (6(b)) Activation level: projecting out the learned direction on real benchmark edits (bottom left) and adding it on generic-voice clones (bottom right). The remaining consensus-panel members show no effect, like Voxtral-Mini-3B.

Table 5: The opening-courtesy case study: rate at which the audible courtesy is present in the output. truncated: the audio is cut to the opener; attn-isolated keeps the full-clip audio encoding but restricts the decoder’s attention over it to the opener’s frames. translate: the same audio decoded under an English→Spanish translation instruction. full: the entire clip. – marks models without translation or attention-isolation capabilities.
model	truncated	attn-isolated	translate	full
Voxtral-Mini-3B	1.00	1.00	1.00	1.00
Whisper-Large-v3	1.00	1.00	1.00	1.00
Moonshine-Streaming	1.00	1.00	–	1.00
Qwen3-ASR-0.6B	0.94	0.95	1.00	1.00
Kimi-Audio-7B	1.00	0.89	–	0.89
Cohere-Transcribe	0.94	0.26	–	0.00
Granite-Speech-4.1-2B	0.67	0.11	0.39	0.00
Canary-Qwen-2.5B	0.83	0.05	–	0.00
Phi-4-Multimodal	0.83	0.89	0.61	0.00
Higgs-Audio-v3-8B	0.83	0.42	–	0.06
Parakeet-TDT-0.6B-v2	1.00	–	–	0.00

Figure 7: Reference-disagreement accept-ref on the real VoxPopuli recordings under content-preserving perturbations (additive noise 10 dB; measured room reverberation, RT60 0.60). Wilson 95% CIs.

Figure 8: Honorific switch rate (Mr/Mister), all 11 models. A rate above 0.5 (dashed) means the model tracks each corpus’s convention at rates above chance.

Why it matters

If you pick a speech recognition model based on public leaderboard rankings, real-world performance may not match those rankings, because some of the score gap comes from models exploiting quirks of the specific benchmark. This gives a concrete, reusable method to test whether any ASR leaderboard result reflects genuine listening ability before relying on it.

Terms in this paper

Word Error Rate (WER) · a standard metric measuring the percentage of words a speech recognition model gets wrong compared to the reference transcript
benchmark optimization ('benchmaxxing') · improving a model's reported score by exploiting quirks specific to a test set rather than genuinely improving its general ability
activation steering · an interpretability technique that adds or removes a specific internal direction (vector) in a model to causally change its behavior
teacher-forced likelihood · feeding a model the correct answer text and measuring how confidently it would have predicted each next character
accept-ref rate · the fraction of times a model outputs the benchmark's exact reference text even when the audio doesn't actually support it

Original abstract (English)

Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

Authors · Theo Lebryk

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Theo Lebryk et al., arXiv:2608.19936, CC BY 4.0