이미지: @hume_ai (X) 화면 갈무리
Summary
- Hume AI, together with Hugging Face, released a study testing 11 open-source automatic speech recognition (ASR) models using three different methods
- Even when numbers were silenced out of benchmark audio, some top-ranked models still transcribed those exact numbers correctly 30-40% of the time
- The research team added a new "Benchmark Fitting" tab to the Open ASR Leaderboard to measure this phenomenon
- 연구 주체
- 흄AI(Hume AI)와 허깅페이스(Hugging Face) 공동 연구
- 테스트 대상
- 오픈소스 음성인식(ASR) 모델 11개
- 사용 데이터셋
- VoxPopuli, LibriSpeech
- 참조 오류 비율
- 분석한 VoxPopuli 클립의 40%에서 참조 전사 오류 가능성 발견
- 오류 답습 모델 수
- 11개 중 6개 모델이 'thank you' 누락 오류를 그대로 복제
- 무음 숫자 복원율
- 일부 모델이 30~40% 확률로 무음 처리된 숫자를 작성
- 표기 구분 정확도
- 'Mr.' vs 'Mister' 구분에서 일부 모델 약 90% 정확도
- 후속 조치
- Open ASR Leaderboard에 'Benchmark Fitting' 탭 신설
How did they know the silenced number?
In a joint study by Hume AI and Hugging Face, researchers ran an experiment where a number contained in a single benchmark audio clip was completely silenced. That number existed nowhere in the audio as sound. Yet several of the top-scoring automatic speech recognition (ASR) models still transcribed that exact number correctly 30-40% of the time. One model filled in the number "2011," which was entirely absent from the audio. This means the model wasn't transcribing what it heard — it had memorized the benchmark dataset itself.
ASR models are AI systems that convert human speech into text. When evaluating such models, reference transcripts are paired with public datasets like VoxPopuli or LibriSpeech, and models are scored by how accurately they match them, measured as word error rate (WER). The problem is that the same datasets can end up mixed into training data. When that happens, the model isn't listening to the sound — it's regurgitating memorized answers. The research team used a term the industry had already coined for this. As Hume AI put it in a post on X, "The community already had a name for this: benchmaxxing."
Three tests
The research team designed three scenarios using the VoxPopuli and LibriSpeech datasets where the audio itself could not actually determine the correct transcript, and ran all 11 models through them.
The first was the "reference mismatch" test. Since VoxPopuli's reference transcripts themselves contain errors, the team identified potential reference errors in 40% of the analyzed clips. In one case, the reference transcript omitted "thank you," which was clearly present in the audio — and 6 of the 11 models ignored the audio and reproduced this missing error anyway.
The second was the "number silencing" test described above. Even with only the numbers removed from the audio while everything else remained intact, the recovery rate was high on public benchmarks, but dropped sharply when the audio was re-recorded in the same domain. This suggests the models had memorized the benchmark audio itself.
The third was the "notation switching" test. VoxPopuli writes "Mr." while LibriSpeech writes "Mister," even though the sound is identical. In principle, there should be no way to tell from audio alone which notation to use, but several models exceeded the 50% coin-flip baseline, with some reaching roughly 90% accuracy. This indicates the models were identifying which test they were taking, rather than what they were hearing.
Evidence it's not a coincidence
The research team stated this behavior is not simple statistical chance. Using a method called low-rank linear intervention, they were able to switch this copying behavior on and off, and in some cases, simply appending sound to the end of an audio segment changed the behavior. This suggests specific circuits inside the models are actively detecting the benchmark.
The attached chart reveals an interesting inverse correlation: models with the lowest VoxPopuli word error rates (WER) — meaning better scores — also tended to have higher rates of reproducing reference errors.
| Model | VoxPopuli WER(%) | Reference Error Reproduction Rate |
|---|---|---|
| Cohere-Transcribe | 5.6 | 3130.5% |
| Canary-Qwen-2.5B | 5.35 | 2423.5% |
| Higgs-Audio-v3-8B | 5.55 | 2221.5% |
| Granite-Speech-4.1-2B | 5.45 | 2121% |
| Phi-4-Multimodal | 5.65 | 2019.5% |
| Parakeet-TDT-0.6B-v2 | 5.65 | 1818% |
| Qwen3-ASR-0.6B | 6.75 | 99.2% |
| Voxtral-Mini-3B | 6.75 | 43.7% |
| Moonshine-Streaming | 8.0 | 65.5% |
| Kimi-Audio-7B | 7.7 | 32.8% |
| Whisper-Large-v3 | 8.65 | 32.7% |
Among the models identifiable in the table, Whisper-Large-v3 is known to be developed by OpenAI, Qwen3-ASR-0.6B by Alibaba Qwen, and Kimi-Audio-7B by Moonshot AI, while Cohere-Transcribe is a Cohere model. Cohere-Transcribe and Canary-Qwen-2.5B, which had among the lowest error rates, also ranked among the highest in reference error reproduction — while Kimi-Audio-7B and Whisper-Large-v3, which had higher error rates, showed lower reproduction rates. Looking at benchmark scores alone could lead to the opposite conclusion.
A new tab on the leaderboard
The research team didn't stop at a one-time announcement — they built this measurement method directly into the Open ASR Leaderboard. They created a new "Benchmark Fitting" tab that scores each model based on reference error reproduction rate and notation-switching accuracy, displayed alongside the WER rankings. The study was led by Alice Baird and Theo Lebryk, with contributions from David Ayllon, Jakub Cłapa, Jens Madsen, and Panagiotis Tzirakis.
Editor's View
The ASR model market is one where rankings shift on a single decimal point of error rate. But this study shows that a significant portion of that decimal-point difference may come not from "better listening" but from "recognizing which benchmark it's facing." This is simply a familiar problem from the LLM world spreading to another modality. Suspicions that large language models inflate their scores by memorizing answer patterns from math and coding benchmark training data have persisted for years — now it's speech models' turn.
Translating this into practice yields a simple conclusion. Teams deploying ASR models into real services — like call centers or meeting transcription automation — need to reconsider the practice of choosing models based solely on public leaderboard error rate rankings. As the table shows, the model with the best error rate also had the highest reference error reproduction rate. Audio in real-world deployment involves new voices, new accents, and new background noise that weren't in the benchmark. This means the true measure of performance isn't the benchmark ranking, but results run directly against a company's own data.
In the coming months, similar verification methodologies are likely to follow in benchmarks for other modalities. If the "Benchmark Fitting" tab on the Open ASR Leaderboard sets a precedent, it could lead to similar "memorization detection" metrics being added to image and video benchmarks as well.
Comments