Self-Supervised Singing Representation Learning for Low-Resource Music Information Retrieval
Keywords:
self-supervised learning; singing voice; low-resource music information retrieval; representation learning; audio infrastructure; fairness; sustainabilityAbstract
The representation of singing voice in machine-readable form has become a central problem in music information retrieval, especially in settings where annotated data are scarce or unevenly distributed across languages, genres, and vocal techniques. This paper presents a system-level examination of self-supervised singing representation learning as a framework for low-resource music information retrieval. It argues that singing voice presents distinctive challenges that generic speech and audio models do not fully address, including wide pitch ranges, expressive timing, ornamentation, phoneme prolongation, and strong coupling between linguistic and musical structures. The paper discusses contrastive, masked, and multimodal pre-training strategies, and analyzes their architectural and infrastructural implications. Rather than presenting a single technical model, the study focuses on structural trade-offs among data curation, model capacity, downstream adaptation, evaluation validity, robustness, fairness, and deployment sustainability. It explores how low-resource conditions reshape corpus design, transfer learning, and inference infrastructure. The paper further addresses governance and policy concerns, including informed consent, cultural representation, energy consumption, and the risk of homogenizing vocal expression. By integrating technical architecture with social and institutional considerations, the paper offers a broad research perspective on building singing representations that are not only accurate but also maintainable, equitable, and responsive to diverse musical practices.
References
1. van den Oord, A., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
2. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.
3. Hsu, W. N., Bolte, B., Tsai, Y. H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460.
4. Saeed, A., Grangier, D., & Zeghidour, N. (2021). Contrastive learning of general-purpose audio representations. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3875–3879). IEEE.
5. Spijkervet, J., & Burgoyne, J. A. (2021). Contrastive learning of musical representations. In Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR).
6. Castellon, R., Donahue, C., & Liang, P. (2021). Codified audio language modeling learns useful representations for music information retrieval. In Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR).
7. Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. W. (2022). MuLan: A joint embedding of music audio and natural language. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR).
8. Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2022). Contrastive audio-language learning for music. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR).
9. Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017). Convolutional recurrent neural networks for music classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2392–2396). IEEE.
10. McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015). librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference (pp. 18–25).
11. Bittner, R. M., Salamon, J., Tierney, M., Mauch, M., Cannam, C., & Bello, J. P. (2014). MedleyDB: A multitrack dataset for annotation-intensive MIR research. In Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR).
12. Stoller, D., Ewert, S., & Dixon, S. (2018). Wave-U-Net: A multi-scale neural network for end-to-end audio source separation. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR).
13. Défossez, A., Usunier, N., Bottou, L., & Bach, F. (2019). Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254.
14. Pons, J., & Serra, X. (2019). musicnn: Pre-trained convolutional neural networks for music audio tagging. In Proceedings of the 20th International Society for Music Information Retrieval Conference Late-Breaking Demos.
15. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778).
16. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998–6008).
18. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186).
19. Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & Ellis, D. P. W. (2014). mir_eval: A transparent implementation of common MIR metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR).
20. Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017). Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 776–780). IEEE.
21. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623).
22. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
23. Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 1–21.
24. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645–3650).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Engineering Systems and Digital Innovation
This work is licensed under a Creative Commons Attribution 4.0 International License.
