Multilingual Singing Recognition and Music Semantic Understanding for Cross-Cultural Music Analysis
Keywords:
multilingual singing recognition, music semantic understanding, cross-cultural music analysis, audio-language models, corpus governance, fairness, cultural data sovereigntyAbstract
The computational analysis of singing has traditionally developed along separate trajectories from multilingual speech processing and music semantic understanding. Speech-oriented foundation models have produced increasingly reliable lyric transcription for several high-resource languages, while music information retrieval systems have focused heavily on Western instrumental repertoires. However, cross-cultural music analysis requires more than the concatenation of these capabilities. It demands an integrated system architecture that can represent phonetic content, melodic structure, performance style, cultural context, and semantic intent in mutually interpretable ways. This paper provides a system-level examination of multilingual singing recognition and music semantic understanding for cross-cultural music analysis. It discusses architectural choices, including modular pipelines and end-to-end audio-language models, and analyzes trade-offs between scalability, controllability, interpretability, and cultural specificity. The paper further examines data infrastructure, corpus governance, robustness, fairness, deployment sustainability, and policy implications. It argues that cross-cultural singing analysis should be understood not primarily as a statistical pattern recognition problem, but as a socio-technical infrastructure challenge in which data sovereignty, annotation quality, evaluation validity, and deployment context shape what computational systems can responsibly claim to understand. The discussion draws on speech processing, music information retrieval, generative audio systems, and critical dataset scholarship to outline a forward-looking research agenda that integrates technical design with cultural and institutional considerations.
References
1. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (PMLR Vol. 202, pp. 28492–28518). PMLR.
2. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., ... & Roberts, A. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
3. Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. W. (2022). MuLan: A joint embedding of music audio and natural language. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (pp. 559–566). ISMIR.
4. Bertin-Mahieux, T., Ellis, D. P. W., Whitman, B., & Lamere, P. (2011). The Million Song Dataset. In Proceedings of the 12th International Society for Music Information Retrieval Conference (pp. 591–596). ISMIR.
5. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.
6. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., & Collobert, R. (2020). MLS: A large-scale multilingual corpus for speech research. In Proceedings of Interspeech 2020 (pp. 1043–1047). ISCA.
7. Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017). Convolutional recurrent neural networks for music classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 2392–2396). IEEE.
8. Benetos, E., Dixon, S., Duan, Z., & Ewert, S. (2019). Automatic music transcription: Breaking the glass ceiling. In Proceedings of the 20th International Society for Music Information Retrieval Conference (pp. 100–107). ISMIR.
9. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
10. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM.
11. Sturm, B. L., Iglesias, M., Ben-Tal, O., Miron, M., & Gómez, E. (2019). Artificial intelligence and music: Open questions of copyright law and engineering process. Arts, 8(3), 115.
12. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
13. Serra, X. (2011). A multicultural approach in music information research. In Proceedings of the 12th International Society for Music Information Retrieval Conference (pp. 151–156). ISMIR.
14. Defferrard, M., Benzi, K., Vandergheynst, P., & Bresson, X. (2017). FMA: A dataset for music analysis. In Proceedings of the 18th International Society for Music Information Retrieval Conference (pp. 316–323). ISMIR.
15. Donahue, C., Caillon, A., Roberts, A., Manilow, E., Esling, P., Agostinelli, A., ... & Engel, J. (2023). SingSong: Generating musical accompaniments from singing. arXiv preprint arXiv:2301.12662.
16. Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., ... & Auli, M. (2024). Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(19), 1–52.
17. Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2021). AudioCLIP: Extending CLIP to image, text and audio. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 976–980). IEEE.
18. Schlüter, J., & Grill, T. (2015). Exploring data augmentation for improved singing voice detection with neural networks. In Proceedings of the 16th International Society for Music Information Retrieval Conference (pp. 121–126). ISMIR.
19. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J. J., Appleton, G., Axton, M., Baak, A., ... & Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, 160018.
20. Holzapfel, A., Sturm, B. L., & Coeckelbergh, M. (2018). Ethical dimensions of music information retrieval technology. Transactions of the International Society for Music Information Retrieval, 1(1), 44–55.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Engineering Systems and Digital Innovation
This work is licensed under a Creative Commons Attribution 4.0 International License.
