每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Towards Quantifying Benchmark Optimization in ASR Models

arXiv:2608.199362026-08-19

部分顶尖语音识别模型并非在听音频,而是在抄标准答案

研究者设计了一批音频本身无法唯一确定正确文本的测试场景,比如参考文本本身有错、把某个数字的声音直接静音、或者一个词存在两种同样正确的拼写方式。他们在11个开源语音识别模型上测试后发现,那些在公开排行榜上单词错误率(WER)最低、成绩最好的模型,恰恰最容易照抄基准测试的参考文本,即便音频内容与之矛盾、被遮蔽或本身模糊不清。这种投机行为只在与基准测试高度相关的特定声学线索下才会触发,而且可以通过修改模型内部激活值或拼接额外音频来人为打开或关闭。

他们做了什么

  1. 研究者利用音频无法唯一确定文本的三类场景——参考文本错误、数字音频被静音、存在两种等效拼写的词——测试了11个开源语音识别模型
  2. 在VoxPopuli等公开基准上,单词错误率最低的六个模型(5.4%-5.8%)恰恰最容易照抄错误的参考文本(照抄率0.18-0.30),而错误率6.5%以上的模型照抄率则在0.10以下
  3. 用基准测试集里真实出现过的说话人声音克隆朗读同样内容时,模型依然会照抄参考文本;但换成普通声音或训练截止后新录制的陌生说话人后,这种行为明显减弱,说明模型抓住的是基准测试特有的狭窄声学线索,而非真正听懂了内容
  4. 在音频片段后拼接一段与基准测试相关的额外音频,或者对模型内部一个特定的激活方向进行添加或消除(低秩线性引导),都可以人为地打开或关闭这种照抄行为
  5. 整体结果表明,部分高分语音识别模型的排行榜成绩可能被基准测试特有的取巧行为拉高,而非真正反映了更强的通用转录能力
Figure 2: Cross-model audit on VoxPopuli. WER (%) is the VoxPopuli-test score from the June 2026 Open ASR Leaderboard [38]. Kimi Audio is not on the leaderboard, and its score is computed using the leaderboard’s scoring. Consensus-panel members are scored against edits flagged unanimously by the remaining three members (§3.3).
Figure 2: Cross-model audit on VoxPopuli. WER (%) is the VoxPopuli-test score from the June 2026 Open ASR Leaderboard [38]. Kimi Audio is not on the leaderboard, and its score is computed using the leaderboard’s scoring. Consensus-panel members are scored against edits flagged unanimously by the remaining three members (§3.3).
Figure 3: (3(a)) Masked-number accept-ref per corpus. (3(b)) Orthographic switch rate on archaic spacing.
Figure 3: (3(a)) Masked-number accept-ref per corpus. (3(b)) Orthographic switch rate on archaic spacing.
Table 1: accept-ref on VoxPopuli-AA human-annotated edits, beside the consensus rates of Figure 2. Ordering is nearly identical aside from Granite and Canary swapping places.
consensushuman-annotated edits
modelaccept-refaccept-ref95% CIn
Cohere-Transcribe0.300.52[0.47, 0.58]253/483
Granite-Speech-4.1-2B0.210.42[0.36, 0.47]211/508
Canary-Qwen-2.5B0.230.41[0.36, 0.47]210/507
Higgs-Audio-v3-8B0.210.39[0.33, 0.44]200/517
Phi-4-Multimodal0.190.38[0.33, 0.44]195/510
Parakeet-TDT-0.6B-v20.180.38[0.32, 0.43]192/512
Qwen3-ASR-0.6B0.090.19[0.15, 0.24]98/518
Moonshine-Streaming0.060.14[0.11, 0.19]74/516
Voxtral-Mini-3B0.040.09[0.06, 0.12]47/527
Kimi-Audio-7B0.030.07[0.05, 0.11]40/536
Whisper-Large-v30.020.08[0.05, 0.11]39/514
(b)
(b)
Figure 4: Difference in masked-number recovery between LibriSpeech test-set narrator clones and held-out libri-fresh narrator clones reading identical sentences (test-set minus held-out; positive values indicate greater recovery for test-set narrator clones). Sentence-clustered bootstrap 95% confidence intervals.
Figure 4: Difference in masked-number recovery between LibriSpeech test-set narrator clones and held-out libri-fresh narrator clones reading identical sentences (test-set minus held-out; positive values indicate greater recovery for test-set narrator clones). Sentence-clustered bootstrap 95% confidence intervals.
Table 2: Reference-disagreement accept-ref on consensus edits as audio context is removed or the trigger is ablated. truncated cuts the audio to a tight window around the edit span (±1 aligned word ±0.25 s); donor ablated appends an 8 s conversational donor to the full clip; activation ablated projects the learned register direction out of a single encoder layer.
modelfulltruncateddonor ablatedactivation ablated
Cohere-Transcribe0.300.130.060.04
Canary-Qwen-2.5B0.230.120.050.02
Granite-Speech-4.1-2B0.210.120.200.14
Higgs-Audio-v3-8B0.210.120.03–
Phi-4-Multimodal0.190.090.050.20
Parakeet-TDT-0.6B-v20.180.080.060.01
Qwen3-ASR-0.6B0.090.090.04–
Moonshine-Streaming0.060.070.04–
Voxtral-Mini-3B0.040.050.03–
Whisper-Large-v30.020.050.02–
Kimi-Audio-7B0.030.060.04–
Figure 5: The trigger battery, VoxPopuli. Top row: voice conditions on identical transcripts—(5(a)) reference-disagreement and (5(b)) masked-number accept-ref. Bottom row: the same probes with the trigger removed instead of the voice varied—truncation to the edit, an appended 8 s conversational donor, or the learned register direction projected out of one encoder layer (5(c), 5(d)). Wilson 95% CIs.
Figure 5: The trigger battery, VoxPopuli. Top row: voice conditions on identical transcripts—(5(a)) reference-disagreement and (5(b)) masked-number accept-ref. Bottom row: the same probes with the trigger removed instead of the voice varied—truncation to the edit, an appended 8 s conversational donor, or the learned register direction projected out of one encoder layer (5(c), 5(d)). Wilson 95% CIs.
(b)
(b)
Table 3: Full-probe accept-ref on real audio with each corpus’s own content: VoxPopuli-test vs ep-fresh. Masked columns score the corpus-paired subsets, so the VoxPopuli masked rates differ from the full masked set of §4.
consensus accept-refmasked accept-ref
modelVoxPopuliep-freshVoxPopuliep-fresh
Cohere-Transcribe0.3040.1220.1850.074
Canary-Qwen-2.5B0.2330.1500.0510.062
Granite-Speech-4.1-2B0.2080.1170.0510.062
Higgs-Audio-v3-8B0.2110.1680.0700.040
Phi-4-Multimodal0.1880.1930.0760.044
Parakeet-TDT-0.6B-v20.1810.1200.0380.029
Qwen3-ASR-0.6B0.0920.0910.0510.015
Moonshine-Streaming0.0560.0990.0130.018
Voxtral-Mini-3B0.0350.1280.0760.062
Whisper-Large-v30.0250.0710.0830.062
Kimi-Audio-7B0.0280.1440.0130.018
(c)
(c)
(d)
(d)
Table 4: Audio lift λ⁡(r) (Eq. 1, nats/char) of the silenced number span by voice condition (115 paired sentences passing the intelligibility gate in every clone condition). Diff columns: paired differences, bootstrap 95% CIs; bold marks CIs excluding zero. Whisper’s lift rises on clean TTS, so a drop in lift is likely not a synthesis artifact; Qwen3’s lift is negative in every condition, so its diffs do not indicate recovery. Parakeet-TDT has no teacher-forced readout.
realvox-cloneep-freshgenericreal−ep-freshreal−generic
Cohere-Transcribe+1.52+1.26+0.92+0.54+0.60 [+0.30,+0.94]+0.98 [+0.66,+1.33]
Canary-Qwen-2.5B+1.22+1.05+0.65+0.77+0.57 [+0.27,+0.90]+0.45 [+0.17,+0.71]
Granite-Speech-4.1-2B+0.12+0.15+0.16+0.01−0.04 [−0.24,+0.17]+0.11 [−0.11,+0.32]
Phi-4-Multimodal+0.46+0.51+0.24+0.27+0.23 [+0.01,+0.50]+0.19 [+0.04,+0.35]
Higgs-Audio-v3-8B+0.50+0.57+0.37+0.11+0.13 [−0.00,+0.26]+0.39 [+0.23,+0.56]
Whisper-Large-v3+0.70+0.89+0.85+1.01−0.16 [−0.37,+0.06]−0.31 [−0.47,−0.15]
Moonshine-Streaming+0.31+0.37+0.25+0.15+0.06 [−0.08,+0.21]+0.16 [+0.00,+0.32]
Kimi-Audio-7B+0.24+0.32+0.06+0.16+0.17 [−0.00,+0.37]+0.08 [−0.06,+0.22]
Qwen3-ASR-0.6B−0.60−0.56−1.06−1.14+0.46 [+0.19,+0.75]+0.54 [+0.31,+0.76]
Voxtral-Mini-3B−0.13+0.07−0.12−0.05−0.01 [−0.17,+0.16]−0.08 [−0.23,+0.07]
Figure 6: Switching the benchmark-optimized policy on and off. (6(a)) Input level: On real clips (top left) a conversational donor collapses accept-ref while a VoxPopuli donor leaves it intact; on ep-fresh clones of the same sentences (top right) a VoxPopuli donor re-ignites it while the conversational donor does not. (6(b)) Activation level: projecting out the learned direction on real benchmark edits (bottom left) and adding it on generic-voice clones (bottom right). The remaining consensus-panel members show no effect, like Voxtral-Mini-3B.
Figure 6: Switching the benchmark-optimized policy on and off. (6(a)) Input level: On real clips (top left) a conversational donor collapses accept-ref while a VoxPopuli donor leaves it intact; on ep-fresh clones of the same sentences (top right) a VoxPopuli donor re-ignites it while the conversational donor does not. (6(b)) Activation level: projecting out the learned direction on real benchmark edits (bottom left) and adding it on generic-voice clones (bottom right). The remaining consensus-panel members show no effect, like Voxtral-Mini-3B.
(b)
(b)
Table 5: The opening-courtesy case study: rate at which the audible courtesy is present in the output. truncated: the audio is cut to the opener; attn-isolated keeps the full-clip audio encoding but restricts the decoder’s attention over it to the opener’s frames. translate: the same audio decoded under an English→Spanish translation instruction. full: the entire clip. – marks models without translation or attention-isolation capabilities.
modeltruncatedattn-isolatedtranslatefull
Voxtral-Mini-3B1.001.001.001.00
Whisper-Large-v31.001.001.001.00
Moonshine-Streaming1.001.00–1.00
Qwen3-ASR-0.6B0.940.951.001.00
Kimi-Audio-7B1.000.89–0.89
Cohere-Transcribe0.940.26–0.00
Granite-Speech-4.1-2B0.670.110.390.00
Canary-Qwen-2.5B0.830.05–0.00
Phi-4-Multimodal0.830.890.610.00
Higgs-Audio-v3-8B0.830.42–0.06
Parakeet-TDT-0.6B-v21.00––0.00
Figure 7: Reference-disagreement accept-ref on the real VoxPopuli recordings under content-preserving perturbations (additive noise 10 dB; measured room reverberation, RT60 0.60). Wilson 95% CIs.
Figure 7: Reference-disagreement accept-ref on the real VoxPopuli recordings under content-preserving perturbations (additive noise 10 dB; measured room reverberation, RT60 0.60). Wilson 95% CIs.
Figure 8: Honorific switch rate (Mr/Mister), all 11 models. A rate above 0.5 (dashed) means the model tracks each corpus’s convention at rates above chance.
Figure 8: Honorific switch rate (Mr/Mister), all 11 models. A rate above 0.5 (dashed) means the model tracks each corpus’s convention at rates above chance.

为什么重要

如果只根据公开排行榜的分数来选择语音识别模型,实际使用效果可能达不到排行榜显示的水平,因为部分分数来自模型对特定测试集的投机取巧而非真正的听觉理解能力。这项工作提供了一套可复用的方法,能在信任任何语音识别排行榜结果之前先检验其是否存在这种问题。

本文术语

  • 单词错误率(WER) · 衡量语音识别结果与标准文本相比错误比例的常用指标
  • 基准测试投机(benchmark optimization) · 模型通过利用某个测试集的特有规律而非真正提升通用能力来拉高报告分数的行为
  • 激活引导(activation steering) · 一种可解释性技术,通过在模型内部添加或移除特定方向的向量来因果性地改变模型行为
  • teacher-forced似然 · 强制给模型输入正确答案文本,测量模型对下一个字符预测的置信程度
  • accept-ref率 · 在音频并不支持的情况下,模型仍然输出基准测试参考文本原文的比例

论文原文摘要（英文）

Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

作者 · Theo Lebryk

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Theo Lebryk et al., arXiv:2608.19936, CC BY 4.0