Low-Resource Voice Synthesis Through Residual Style Encoding and Uncertainty-Based Domain Adaptation

Authors

  • Hitesh C. Sabharwal Department of Computer Science, Colorado State University, Fort Collins, CO, USA. Author
  • Advika C. Muni Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO, USA. Author

Keywords:

low-resource speech synthesis; residual style encoding; uncertainty estimation; domain adaptation; acoustic modeling; prosody transfer

Abstract

Low-resource voice synthesis must generate intelligible and expressive speech for speakers and styles for which only limited transcribed or untranscribed audio is available. Conventional acoustic models trained on high-resource corpora often transfer poorly because target recordings differ in speaker identity, recording condition, prosody, and affect, while style encoders tend to entangle these factors with phonetic content. This paper develops a methodological framework that combines residual style encoding with uncertainty-based domain adaptation for low-resource voice synthesis. The framework represents a reference style as a residual vector between a reference-derived style code and a speaker-conditioned base style anchor; the residual is injected into an acoustic decoder through adaptive normalization and regularized by a conditional prior. Domain adaptation aligns source and target feature distributions using an uncertainty-weighted adversarial objective, in which per-sample heteroscedastic variance and ensemble disagreement determine the contribution of target frames and pseudo-labeled utterances. A reliability gate combines predictive variance, heteroscedastic uncertainty, and content consistency to suppress unstable target supervision. The framework yields a coherent training objective and explicit design trade-offs involving style disentanglement, adversarial alignment, pseudo-label noise, and low-resource overfitting. Its practical significance lies in providing a principled route for adapting expressive voice synthesis to under-resourced speakers without treating all target observations as equally trustworthy.

References

1. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech 2017 (pp. 4006–4010). https://doi.org/10.21437/Interspeech.2017-1452

2. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R. J., Saurous, R. A., Agiomyrgiannakis, Y., & Wu, Y. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4779–4783). https://doi.org/10.1109/ICASSP.2018.8461368

3. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2019). FastSpeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems (Vol. 32, pp. 3165–3174).

4. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2021). FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations.

5. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems (Vol. 33, pp. 17022–17033).

6. Kim, J., Kim, S., Kong, J., & Yoon, S. (2021). Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. In Advances in Neural Information Processing Systems (Vol. 33, pp. 8067–8077).

7. Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., & Kudinov, M. (2021). Grad-TTS: A diffusion probabilistic model for text-to-speech. In Proceedings of the 38th International Conference on Machine Learning (pp. 8599–8608).

8. Casanova, E., Weber, J., Shulby, C. D., Candido, A., Jr., Oliveira, E. C., & Ponti, M. A. (2022). YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Proceedings of the 39th International Conference on Machine Learning (pp. 2709–2720).

9. Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., & Wei, F. (2023). Neural codec language models are zero-shot text to speech synthesizers. arXiv. https://arxiv.org/abs/2301.02111

10. Shen, K., Ju, Z., Yang, D., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Zhao, S., & Bian, J. (2023). NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv. https://arxiv.org/abs/2304.09116

11. Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59), 1–35.

12. Long, M., Cao, Z., Wang, J., & Jordan, M. I. (2018). Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems (Vol. 31, pp. 1640–1650).

13. Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018). Maximum classifier discrepancy for unsupervised domain adaptation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3723–3732). https://doi.org/10.1109/CVPR.2018.00392

14. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1050–1059).

15. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (Vol. 30, pp. 5574–5584).

16. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In International Conference on Learning Representations.

17. Zhang, Y., Huang, R., Li, R., He, J., Xia, Y., Chen, F., ... & Zhao, Z. (2024, March). Stylesinger: Style transfer for out-of-domain singing voice synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 17, pp. 19597-19605).

18. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. In Advances in Neural Information Processing Systems (Vol. 30, pp. 6306–6315).

19. Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In 2017 IEEE International Conference on Computer Vision (ICCV) (pp. 1501–1510). https://doi.org/10.1109/ICCV.2017.167

20. French, G., Mackiewicz, M., & Fisher, M. (2018). Self-ensembling for visual domain adaptation. In International Conference on Learning Representations.

Downloads

Published

2026-10-22