Robust Speech Reconstruction Using Probabilistic Acoustic Modeling and Reference-Driven Feature Adaptation

Authors

  • Nora Madigan School of Computing, Clemson University, Clemson, SC, USA. Author
  • Leif Schinke Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL, USA. Author
  • Cornelis Scardino Department of Electrical Engineering and Computer Science, University of Kansas, Lawrence, KS, USA. Author

Keywords:

probabilistic acoustic modeling; speech reconstruction; reference-driven adaptation; variational inference; uncertainty calibration; robust feature fusion

Abstract

Speech reconstruction from degraded observations remains difficult because acoustic evidence is sparse, channel distortions are nonstationary, and deterministic regressors provide point estimates without reliable uncertainty. This manuscript develops a probabilistic acoustic modeling framework with reference-driven feature adaptation for robust speech reconstruction. The observed signal is modeled as a noisy realization of a latent acoustic state under a conditional Gaussian likelihood, while a temporal prior captures continuity across frames. Amortized variational inference estimates the posterior over latent states, separating aleatoric observation uncertainty from epistemic uncertainty in the acoustic state. A reference encoder converts an enrolled utterance or domain prototype into a low-rank adaptation of latent features, and a covariance-weighted fusion rule combines reference-conditioned and evidence-driven estimates. The resulting reconstruction operator is uncertainty aware: it trusts the reference when acoustic evidence is weak and trusts the observed signal when the reference is mismatched. We derive the variational objective, adaptation operator, fusion rule, and training loss, and analyze failure modes under model mismatch, reference contamination, and nonstationary noise. The framework provides a principled route to robust speech reconstruction without assuming that clean targets are available at inference. Its significance lies in linking probabilistic uncertainty, temporal acoustic structure, and reference-conditioned adaptation within a single inferential model, thereby giving reconstruction decisions an explicit statistical interpretation rather than relying on an uncalibrated regression output.

References

1. Loizou, P. C. (2013). Speech enhancement: Theory and practice (2nd ed.). CRC Press.

2. Wang, D., & Chen, J. (2018). Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10), 1702–1726.

3. Ephraim, Y., & Malah, D. (1984). Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator. IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(6), 1109–1121.

4. Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations.

5. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.

6. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (Vol. 33, pp. 6840–6851).

7. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (Vol. 33, pp. 12449–12460).

8. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460.

9. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (pp. 28492–28518).

10. Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.

11. Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45.

12. Roweis, S., & Ghahramani, Z. (1999). A unifying review of linear Gaussian models. Neural Computation, 11(2), 305–345.

13. Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1501–1510).

14. Perez, E., Strub, F., de Vries, H., Dumoulin, V., & Courville, A. (2018). FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).

15. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.

16. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321–1330).

17. Zhang, Y., Huang, R., Li, R., He, J., Xia, Y., Chen, F., ... & Zhao, Z. (2024, March). Stylesinger: Style transfer for out-of-domain singing voice synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 17, pp. 19597-19605).

18. Chou, J.-C., & Lee, H.-Y. (2019). One-shot voice conversion by separating speaker and content representations with instance normalization. In Proceedings of Interspeech 2019 (pp. 664–668).

19. Qian, K., Zhang, Y., Chang, S., Yang, M., & Hasegawa-Johnson, M. (2020). AutoVC: Zero-shot voice style transfer with only autoencoder loss. In Proceedings of the 37th International Conference on Machine Learning (pp. 7630–7639).

20. Zhao, Y., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z., & Toda, T. (2020). Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. arXiv preprint arXiv:2008.12527.

Downloads

Published

2026-11-10