Ruby-OPD: A Breakthrough in Pronunciation Controllability for Flow-Matching TTS

Japanese TTS has long faced a dilemma: either convert text to phoneme sequences through a frontend to ensure pronunciation controllability, at the cost of losing semantic and prosodic cues; or let zero-shot models consume raw text directly to improve naturalness, sacrificing polyphone accuracy and explicit control. Ruby-OPD proposes a two-stage framework that attempts to solve both problems simultaneously on a flow-matching TTS architecture.

The Structural Challenge of Japanese TTS

Japanese is an unsegmented language that mixes kanji (logographic) and kana (phonographic) scripts, and kanji exhibit extensive polyphony. Traditional cascaded systems rely on a frontend to convert text into phoneme sequences, making pronunciation control deterministic—but at the cost of discarding lexical semantics and prosodic cues present in the surface orthography. Recently popular zero-shot TTS models consume raw orthographic text directly, significantly improving naturalness, yet they frequently misread polyphonic characters and lack an explicit pronunciation control interface.

The authors of Ruby-OPD frame this tension as a trade-off between "pronunciation accuracy" and "fine-grained controllability," and propose a two-stage design to bridge it.

Stage One: Ruby Text Representation and Orthography Perturbation

Ruby text is originally a typographic convention in Japanese for annotating readings above kanji. Ruby-OPD uses it as a representational device: through orthography perturbation, the model is exposed to both surface written forms and target pronunciations during training, thereby retaining orthographic information while acquiring reliable pronunciation control signals.

The elegance of this design lies in the fact that it does not fully phonemize text like a traditional frontend. Instead, it teaches the model to treat Ruby annotations as controllable pronunciation anchors while preserving the semantic cues of the original text.

What Is Orthography Perturbation?

Orthography perturbation refers to applying controlled variations to the surface written form of input text during training—for example, randomly removing, replacing, or inserting Ruby reading annotations—so that the model sees multiple representations such as "kanji only," "kanji + reading," and "kana only." The purpose is to prevent the model from over-relying on any single input format: at inference time, even if the user provides only raw text without Ruby annotations, the model can infer pronunciation based on the alignment relationships learned during training. The perturbation strategy must balance coverage against noise—too much perturbation makes pronunciation signals unreliable, while too little prevents generalization to unannotated scenarios.

Stage Two: On-Policy Distillation on Large-Scale Plain-Text Corpora

Ruby-annotated data alone is insufficient to cover the unannotated raw text encountered in real-world scenarios. Ruby-OPD introduces On-Policy Distillation (OPD) to further optimize pronunciation accuracy on large-scale plain-text corpora that require no paired speech.

"On-Policy" means the distillation process uses outputs generated by the model itself under its current policy for learning, rather than relying on fixed teacher-forcing trajectories. This enables the model to self-correct polyphone errors on raw orthographic text without requiring paired speech data—particularly critical for Japanese TTS in low-resource settings.

On-Policy vs. Off-Policy Distillation

In knowledge distillation, the source of the student model's learning data determines the policy type. Off-policy distillation uses fixed teacher model outputs or annotated data as supervision signals; the student can only passively imitate, and the training distribution easily diverges from the real input distribution at inference time. On-policy distillation, by contrast, has the student model first generate its own output sequences for the current input, then corrects these "self-policy-generated" samples using a pronunciation discriminator or reward signal. Because training samples come from the model's current distribution, polyphone error corrections directly target the text forms encountered at inference, leading to more stable convergence. The cost is that training requires online sampling, incurring higher computational overhead than offline distillation and placing greater demands on the reliability of the reward signal.

Benchmark Results

On the Joyo-Kanji-Yomi benchmark, Ruby-OPD achieves the following with raw text input:

• Accuracy: 97.13%
• Target Phoneme Error Rate (Target PER): 2.02%
• Sentence-level Phoneme Error Rate (Sentence PER): 0.70%
• Sentence-level Character Error Rate (Sentence CER): 4.21%

The model also retains strong controllability. These results indicate that the two-stage framework has reached or even matched the pronunciation accuracy of frontend phonemization systems, without giving up the naturalness advantages of raw text input.

The Joyo-Kanji-Yomi Benchmark and PER/CER Metrics

The Joyo-Kanji-Yomi benchmark focuses on the reading accuracy of Joyo Kanji (commonly used Chinese characters), serving as a representative test set for evaluating polyphone handling in Japanese TTS. PER (Phoneme Error Rate) measures the proportion of substitution, deletion, and insertion errors at the phoneme level; Target PER counts only phoneme errors at positions containing the target polyphone, while Sentence PER and Sentence CER measure phoneme and character error rates at the full-sentence level, respectively. A low Target PER indicates accurate judgment of key polyphones, while low sentence-level metrics reflect overall stability of the pronunciation sequence—both are needed for a comprehensive evaluation of pronunciation accuracy.

Implications for TTS Engineering Practice

The value of Ruby-OPD extends beyond Japanese. Any language where orthography and pronunciation mappings are inconsistent—such as Chinese polyphones or heteronyms in English—could benefit from the combination of "Ruby-style annotation + On-Policy distillation." The flow-matching TTS architecture itself excels at generation quality; with the pronunciation controllability gap addressed, its applicability in scenarios requiring precise pronunciation control (e.g., broadcasting, education, accessible reading) will improve noticeably.

Of course, the cost of obtaining Ruby annotations, the design details of perturbation strategies, and the stability of OPD on larger-scale corpora remain engineering questions to watch. But methodologically, shifting pronunciation control from "frontend hard constraints" to "representational soft guidance + in-policy distillation" is a direction worth following.

One-line Summary

Ruby-OPD combines Ruby representation with On-Policy distillation to achieve both pronunciation accuracy and controllability in Japanese TTS.

Paragraph Summary

Ruby-OPD addresses the dilemma in Japanese TTS—where "frontend phonemization ensures pronunciation controllability but loses semantic and prosodic cues" versus "zero-shot models consuming raw text improve naturalness but misread polyphones"—by proposing a two-stage framework. Stage one uses Ruby text representation combined with orthography perturbation, enabling the model to retain surface written forms while acquiring pronunciation anchors; stage two performs On-Policy distillation on large-scale plain-text corpora without paired speech, allowing the model to self-correct polyphone errors on its own policy outputs. On the Joyo-Kanji-Yomi benchmark, it achieves 97.13% accuracy, 2.02% Target PER, 0.70% Sentence PER, and 4.21% Sentence CER with raw text input, while retaining strong controllability. The method has reference value for languages where orthography and pronunciation are inconsistent, such as Chinese polyphones.