Nepali-English Code-Mixed Speech Recognition: A Practical Guide to Data Preparation and Model Adaptation

Nepali speakers frequently mix English words into everyday conversation, yet multilingual speech recognition systems often struggle with code-switching and English script. Based on a study of the SraVaani model, this article explores how data preparation and model adaptation can improve recognition of Nepali-English code-mixed speech.

Challenges of Code-Mixed Speech Recognition and the SraVaani Model

Code-mixing between Nepali and English is extremely common in spoken language, but speech recognition systems face two major difficulties: first, frequent language switching makes it hard for acoustic and language models to adapt in sync; second, English words in Nepali transcriptions are often spelled in Devanagari, which is inconsistent with standard English spelling and causes a script mismatch between training and test data.

The study uses SraVaani, a 449-million-parameter hybrid TDT-CTC model whose pretrained tokenizer does not support digits or several common symbols. Directly replacing the tokenizer or discarding recordings containing these symbols would both lose information, so the authors adopted a compromise: generate both Nepali and English readings for digits, and use zero-shot ASR output to select the more plausible transcription form. This preprocessing step alleviates the tokenizer coverage problem without changing the model architecture.

TDT-CTC Hybrid Architecture: TDT (Token-and-Duration Transducer) and CTC (Connectionist Temporal Classification) are two common approaches to output modeling in speech recognition. CTC assumes conditional independence between frame outputs and aligns audio with text by introducing a blank symbol; training is simple but alignment is coarse. TDT, on the other hand, explicitly models the duration of each token within the transducer framework, handling speech rate variation and repeated characters more flexibly. Mixing the two (e.g., sharing an encoder and jointly training CTC and TDT losses) can combine CTC's fast convergence with TDT's fine-grained alignment, and typically outperforms either structure alone in multilingual, variable-rate scenarios. SraVaani adopts this hybrid architecture, and its encoder must adapt to the acoustic characteristics of different languages simultaneously—which also explains why freezing the encoder later leads to a marked performance drop.

Tokenizer and Subword Coverage: An ASR model's tokenizer is responsible for segmenting text labels into subword units, and its vocabulary is fixed during pretraining. If the target language's digits, symbols, or specific scripts (such as English words spelled in Devanagari) are not in the vocabulary, the model cannot directly output the corresponding tokens and can only map them to unknown symbols or discard them. Replacing the tokenizer means the decoder output layer must be reinitialized, which destroys language knowledge acquired during pretraining; filtering data, meanwhile, reduces training samples. Therefore, when the vocabulary is limited, adapting to the tokenizer through transcription normalization (e.g., writing digits as their spoken forms) is a low-cost compromise.

Data Preparation: 172.8-Hour Corpus and Digit Handling Strategy

The study built a 172.8-hour corpus covering six Nepali speech domains to ensure domain diversity. To address the tokenizer's inability to handle digits and symbols, the team did not simply filter out the relevant segments; instead, they generated both Nepali and English readings for each digit, then ranked the recognition confidence of the two readings via zero-shot ASR and selected the more plausible form as the transcription label. This approach both preserves speech data containing digits and avoids the extra cost of manual annotation.

You may not have noticed that how digit segments are handled directly affects final performance: on the test set, when digit segments were excluded during training, the WER for digit speech rose from 14.3% to 21.3%, showing that retaining and properly processing digit data is crucial for recognition accuracy.

Model Adaptation: Full Fine-Tuning vs. Frozen Encoder

The study compared several fine-tuning strategies. Full fine-tuning reduced the test word error rate (WER) from 48.6% to 17.1%, and after 12 epochs of training the WER dropped further to 16.3%. For reference, the fine-tuned Whisper large-v3-turbo achieved a WER of 16.4% on this task, putting the two performances close together.

When only the decoder was fine-tuned with the encoder frozen, the WER rose to 25.5%, indicating that the encoder needs to adapt to the acoustic characteristics of Nepali code-mixed speech. This result is consistent with general experience in speech recognition: the encoder carries most of the acoustic modeling capacity, and freezing it prevents the model from fully capturing the phonetic and prosodic variation of the target language.

Out-of-Domain Evaluation: The Impact of Transcription Conventions on FLEURS

In addition to in-domain testing, the study evaluated model performance on the FLEURS dataset, comparing the effect of original reference transcriptions versus code-mixed script alternative transcriptions. The results show that transcription conventions can only explain part of the out-of-domain performance loss, not all of it. This means that out-of-domain degradation also involves acoustic domain shift, language model bias, and other factors; simply unifying the transcription script is not enough to fully close the gap.

This finding has implications for the practical deployment of code-mixed speech recognition: when building multilingual ASR systems, attention must be paid both to consistency in transcription conventions and to the domain adaptation capability of acoustic models.

FLEURS Dataset: FLEURS (Few-shot Learning Evaluation of Universal Representations of Speech) is a multilingual speech recognition benchmark released by Google, covering 102 languages with about 10 hours of supervised data and a small amount of few-shot data per language. Its audio mostly comes from public read-speech corpora, with clear pronunciation and relatively uniform domains, differing markedly from everyday conversational code-mixed speech in acoustic conditions, speech rate, and accent. Therefore, when evaluating code-mixed models on FLEURS, performance degradation stems not only from transcription script inconsistency but also from acoustic domain shift between the training domain (Nepali-English spoken dialogue) and the test domain (read-style clean speech). The original paper's point that transcription conventions explain only part of the loss is precisely a reminder to distinguish between two different sources of error: "annotation convention differences" and "acoustic/language model domain shift."

Practical Implications and Limitations

This study demonstrates a feasible path for adapting speech foundation models under limited-data conditions: keep the tokenizer, use zero-shot ASR to assist data cleaning, and fully fine-tune both encoder and decoder. At the same time, the importance of digit segments reminds developers not to casually discard "incompatible" data, but rather to design targeted transcription strategies.

One caveat: the study addresses only the Nepali-English code-mixed scenario, and the generalizability of its conclusions to other language pairs still needs verification. In addition, the 12-epoch fine-tuning result is close to Whisper large-v3-turbo, but the model is smaller (449 million parameters) and may have an advantage in inference efficiency—a point the original paper does not elaborate on.

One-line Summary

Full fine-tuning of SraVaani can reduce WER on Nepali-English code-mixed speech from 48.6% to 16.3%, with digit handling and encoder adaptation being key.

Paragraph Summary

This article focuses on Nepali-English code-mixed speech recognition and introduces an adaptation study based on SraVaani (a 449-million-parameter TDT-CTC hybrid model). To address the tokenizer's inability to handle digits and symbols, the authors generate dual Nepali and English readings for digits and use zero-shot ASR to select transcriptions, thereby retaining digit segments; excluding digit segments raises the WER for digit speech from 14.3% to 21.3%. On the model adaptation side, full fine-tuning reduces the test WER from 48.6% to 17.1%, reaching 16.3% after 12 epochs—comparable to the fine-tuned Whisper large-v3-turbo (16.4%); freezing the encoder and tuning only the decoder raises the WER to 25.5%, showing that encoder adaptation is crucial. Out-of-domain evaluation on FLEURS shows that transcription conventions explain only part of the performance loss, and factors such as acoustic domain shift also affect generalization. The study provides a feasible path for adapting speech foundation models under limited data.