Emotion-Aware Singing Voice Generation Through Multimodal Music and Vocal Feature Analysis

Authors

  • Brady Gibson School of Computing, Clemson University, Clemson, SC, USA. Author

Keywords:

singing voice synthesis, emotion-aware generation, multimodal feature fusion, vocal expression, cross-attention, generative modeling

Abstract

Emotion-aware singing voice generation requires a model to coordinate musical score, lyric content, vocal timbre, pitch behavior, and expressive intention without collapsing these factors into a single undifferentiated style signal. Many neural singing and speech synthesis systems treat emotion as a categorical label or a global embedding, which can obscure the separate roles of prosody, spectral envelope, vibrato, breathiness, and timing. This paper develops a methodological framework for emotion-aware singing voice generation through multimodal music and vocal feature analysis. The central contribution is a factorized generative formulation in which score and lyric information define structural constraints, singer identity defines timbre constraints, and continuous valence-arousal affect modulates vocal expression through an emotion-gated fusion pathway. The framework combines frame-level vocal descriptors, including log fundamental frequency, pitch derivatives, energy, harmonic-to-noise balance, aperiodicity, and vibrato strength, with score and lyric encoders and a cross-attention mechanism for alignment-aware conditioning. A conditional generative objective is specified, together with reconstruction, adversarial, emotion-classification, and regularization terms. The analysis identifies principal failure modes, including score-audio misalignment, emotion leakage into singer identity, over-smoothed vibrato, and biased emotion supervision. It also clarifies why emotion conditioning should act separately on prosodic and timbral pathways and why evaluation must combine signal-level, emotion-level, and listener-level criteria. The resulting framework provides a technically grounded basis for expressive singing generation and for disciplined assessment of multimodal affective control.

References

1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.

2. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations.

3. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840–6851.

4. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.

5. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech (pp. 4006–4010).

6. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2019). FastSpeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32, 3165–3174.

7. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33, 17022–17033.

8. Morise, M., Yokomori, F., & Ozawa, K. (2016). WORLD: A high-quality speech analysis, modification and synthesis system. Acoustical Science and Technology, 37(1), 33–39.

9. Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014). COVAREP: A collaborative voice analysis repository for speech technologies. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 960–964).

10. Kim, J. W., Salamon, J., Li, P., & Bello, J. P. (2018). CREPE: A convolutional representation for pitch estimation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 161–165).

11. Koelstra, S., Mühl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012). DEAP: A database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18–31.

12. Yang, Y.-H., & Chen, H. H. (2012). Machine recognition of music emotion: A review. ACM Transactions on Intelligent Systems and Technology, 3(3), Article 40.

13. Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13(5), e0196391.

14. El Ayadi, M., Kamel, M. S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3), 572–587.

15. Eyben, F., Wöllmer, M., & Schuller, B. (2010). OpenSMILE: The Munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia (pp. 1459–1462).

16. Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1–2), 227–256.

17. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.

18. Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R. J., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., & Saurous, R. A. (2018). Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (pp. 5180–5189).

19. Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875.

20. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved training of Wasserstein GANs. Advances in Neural Information Processing Systems, 30, 5767–5777.

Downloads

Published

2026-10-15