A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluation
arXiv:2608.193612026-08-21
Building speech recognition for Mizo, a low-resource Indian language, by fine-tuning Whisper and SraVaani on 17.62 hours of new data
Researchers collected and cleaned 17.62 hours of Mizo speech from 200 speakers in Mizoram, India, then fine-tuned Whisper models (which don't natively support Mizo) and SraVaani 1.0 (which does) on this data. Whisper-large-v3 reached the lowest conventional word error rate of 18.08%, dropping to 7.22% under a new morphology-aware metric designed for Mizo's flexible spacing. SraVaani 1.0 improved dramatically after fine-tuning, from 58.27% to 29.45% word error rate.
What they did
200 speakers read about 8,000 Mizo sentences drawn from newspapers and court judgments via a web interface, yielding a curated corpus of 8,274 sentence-level recordings totaling 17.62 hours.
Three Whisper model sizes (small, medium, large-v3) and the Indic multilingual model SraVaani 1.0, which already supports Mizo, were each fine-tuned and evaluated on speaker-independent train/validation/test splits.
Because Mizo morphological boundaries can be written either as separate words or joined together, standard WER inflates errors; the authors introduced a morphology-aware WER that allows merging up to four adjacent words before comparing sequences.
Whisper-large-v3 achieved the best results at 18.08% conventional WER and 7.22% morphology-aware WER; SraVaani 1.0 started at 58.27% WER in zero-shot testing but improved to 29.45% WER and 17.93% morphology-aware WER after Mizo-specific fine-tuning.
Error analysis showed the untuned SraVaani 1.0 sometimes output entirely wrong scripts (Meitei-mayek or Devanagari instead of Mizo), and made many name-recognition and glottal-stop errors, most of which decreased substantially after fine-tuning.
Figure 1: Histogram of duration of speech files in the corpus.
Table 1: Data split for training, validation and testing.
Speakers
Sentences
Hours
Training
184
7656
16.18
Validation
11
426
1.02
Testing
05
192
0.42
Figure 2: Schematic diagram showing the flow of the experimental protocol.
Table 2: Durational characteristics of the speech database
Sentences
8274
Total duration
17.62 hours
Minimum duration
0.63 seconds
Maximum duration
41.22 seconds
Mean duration
7.67 seconds
Median duration
6.94 seconds
Table 3: Characteristics of the multilingual ASR models evaluated in this study.
Model
Architecture
Parameters
Pretraining data
Mel bins
Whisper-small
Encoder–Decoder Transformer
244 M
680k h
80
Whisper-medium
Encoder–Decoder Transformer
769 M
680k h
80
Whisper-large-v3
Encoder–Decoder Transformer
1,550 M
∼5M h
128
SraVaani 1.0
FastConformer Hybrid RNNT/CTC
∼430 M
∼31k h
–
Table 4: Fine-tuning configurations used for the ASR models.
Model
Optimizer
LR
Effective batch size
Epochs
Selection
Whisper-small
Adafactor
5×10−6
16
20
Best val. WER
Whisper-medium
Adafactor
5×10−6
16
20
Best val. WER
Whisper-large-v3
Adafactor
5×10−6
16
20
Best val. WER
SraVaani 1.0
AdamW
1×10−4
16
20 + extended
Best val. WER
Table 5: Best epochs and corresponding validation WER of four fine-tuned models.
Model
Best epoch
Validation WER
Whisper-small Mizo–FT
15
28.99
Whisper-medium Mizo–FT
13
26.51
Whisper-large-v3 Mizo–FT
13
23.00
SraVaani 1.0 Mizo–FT
18
33.81
Table 6: Results of evaluation on five models.
Model
CER (%)
WER (%)
MA-WER (%)
Whisper-small Mizo–FT
04.83
24.00
11.49
Whisper-medium Mizo–FT
04.02
21.69
08.87
Whisper-large-v3 Mizo–FT
03.26
18.08
07.22
SraVaani 1.0
17.71
58.27
36.27
SraVaani 1.0 Mizo-FT
06.90
29.45
17.93
Table 7: Error distribution by model. Foreign-script outputs are reported as the number of utterances; all other entries indicate the number of error occurrences.
Model
Foreign script
Names
Glottal stops
Numeral transcripts
Code-mix error
< t Ω >
Whisper-small Mizo–FT
NIL
17
7
2
9
0
Whisper-medium Mizo–FT
NIL
18
6
5
3
0
Whisper-large-v3 Mizo–FT
4 sentences
8
4
3
2
0
SraVaani 1.0
21 sentences
44
9
19
49
31
SraVaani 1.0 Mizo–FT
NIL
24
6
2
19
4
Why it matters
This work delivers a publicly released speech corpus, fine-tuned ASR models, and a language-appropriate evaluation metric for Mizo, giving future developers of Mizo speech technology a concrete starting point. It also demonstrates that standard WER can overstate errors for languages with flexible morphological spacing, a lesson relevant to other Tibeto-Burman languages using Roman script.
Terms in this paper
WER (Word Error Rate) · a standard metric counting word-level substitutions, deletions, and insertions between recognized and reference text
Morphology-aware WER (MA-WER) · a new metric that allows merging up to four adjacent words before comparison, designed to tolerate Mizo's variable morpheme spacing
CER (Character Error Rate) · the same kind of error measurement as WER but computed at the character level instead of word level
Zero-shot evaluation · testing a pretrained model on a language without any additional fine-tuning on that language's data
Whisper / SraVaani 1.0 · Whisper is a multilingual transformer-based speech recognition and translation model; SraVaani 1.0 is a multilingual model covering several Indian languages
Original abstract (English)
This study reports the development of an Automatic Speech Recognition (ASR) system in Mizo, a low-resource language. The development included collecting 17.62 hours of speech data, curating it, and fine-tuning the Mizo ASR system with three Whisper multilingual models and with the SraVaani 1.0 Indic multilingual model. Whisper-large-v3 achieved the lowest conventional WER (18.08%), while morphology-aware evaluation yielded a WER of 7.22%. Zero-shot evaluation of the SraVaani 1.0 Indic multilingual model yielded a WER of 58.27%, while Mizo-specific fine-tuning reduced the conventional WER to 29.45% and the morphology-aware WER to 17.93%. The results demonstrate that the Whisper model can achieve a substantially low WER, even when adapted to an unseen language. In contrast, SraVaani 1.0 supports the Mizo language in its multilingual model; however, fine-tuning with carefully curated Mizo speech data substantially improves its performance.
