Emotion-Controllable Speech Synthesis Through Hierarchical Prosody Modeling and Multi-Level Acoustic Control

Authors

  • Triya Mishra Department of Computer Science, University of Houston, Houston, TX, USA. Author
  • Giacinto Tiepolo Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO, USA. Author
  • Alessandra Almagia School of Information Technology, University of Cincinnati, Cincinnati, OH, USA. Author
  • Vadim Hanel Department of Computer Science and Engineering, University of Nevada, Reno, Reno, NV, USA. Author

Keywords:

emotional speech synthesis; hierarchical prosody; multi-level acoustic control; disentangled representation; expressive text-to-speech; prosodic modeling

Abstract

Emotion-controllable text-to-speech requires more than selecting a categorical speaking style; it requires coordinated control of prosodic phenomena that operate at different temporal scales. Utterance-level affect, phrase-level intonation, and syllable- or phone-level microprosody interact, and collapsing them into a single global embedding limits interpretability, controllability, and disentanglement from speaker identity. This manuscript develops a methodological framework for emotion-controllable speech synthesis that combines hierarchical prosody modeling with multi-level acoustic control. The framework factorizes prosody into utterance, phrase, and segment representations, estimates each representation under an explicit emotion condition, and fuses them at the acoustic decoder through level-specific gating. A conditional variational formulation captures utterance-level affective intent, a basis-function model represents phrase-level fundamental-frequency contours and energy, and a microprosody predictor controls local duration and segmental timing. Multi-level acoustic control is realized through latent variables that can be inspected, interpolated, and selectively strengthened during inference. Training objectives combine acoustic reconstruction, prosodic supervision, emotion prediction, disentanglement regularization, and stability constraints. The paper specifies inference procedures, control-strength interpolation, evaluation variables, and boundary conditions concerning data labeling, prosodic parsing, speaker leakage, and naturalness. The contribution is a coherent architecture and analytical account of how hierarchical prosody factorization can make emotional speech synthesis more controllable without treating emotion as an undifferentiated global attribute.

References

1. Taylor, P. (2009). Text-to-speech synthesis. Cambridge University Press.

2. Zen, H., Tokuda, K., & Black, A. W. (2009). Statistical parametric speech synthesis. Speech Communication, 51(11), 1039–1064.

3. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech 2017 (pp. 4006–4010).

4. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R. J., Saurous, R. A., Agiomyrgiannakis, Y., & Wu, Y. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4779–4783).

5. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2021). FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations.

6. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems (Vol. 33, pp. 17022–17033).

7. Skerry-Ryan, R. J., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R. J., Clark, R., & Saurous, R. A. (2018). Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron. In Proceedings of the 35th International Conference on Machine Learning (pp. 4693–4702).

8. Kim, J., Kong, J., & Son, J. (2021). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of the 38th International Conference on Machine Learning (pp. 5530–5540).

9. Schröder, M. (2001). Emotional speech synthesis: A review. In Proceedings of Eurospeech 2001 (pp. 561–564).

10. Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., André, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., & Truong, K. P. (2016). The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2), 190–202.

11. Zhou, K., Sisman, B., Liu, R., & Li, H. (2022). Emotional voice conversion: Theory, databases and ESD. Speech Communication, 137, 1–18.

12. Triantafyllopoulos, A., Schuller, B. W., Iymen, G., Sezgin, M., He, X., Yang, Z., Tzirakis, P., & Tao, J. (2023). An overview of affective speech synthesis and conversion in the deep learning era. Proceedings of the IEEE, 111(10), 1355–1381.

13. Henter, G. E., Lorenzo-Trueba, J., Wang, X., & Yamagishi, J. (2018). Deep encoder-decoder models for unsupervised learning of controllable speech synthesis. arXiv preprint arXiv:1807.11470.

14. Zhang, Y., Jiang, Z., Li, R., Pan, C., He, J., Huang, R., ... & Zhao, Z. (2024, November). Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 1960-1975).

15. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.

16. Morise, M., Yokomori, F., & Ozawa, K. (2016). WORLD: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Transactions on Information and Systems, 99(7), 1877–1884.

17. Busso, C., Bulut, M., Lee, C. C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4), 335–359.

18. Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13(5), e0196391.

19. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005). A database of German emotional speech. In Proceedings of Interspeech 2005 (pp. 1517–1520).

20. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1), 32–80.

Downloads

Published

2026-11-10