Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

This paper introduces the Raga Based Music Classification as a part of Music Information Retrieval (MIR) systems for retrieving musical information. The music has been more than just a sound that entertains us. MIR users like musicology researchers, musicians, music learners, music therapists, etc., require not just the sound but the actual strength of the music that controls our body, motivates us, cures mental illness, etc. The backbone of any form of music is its raga. A raga is a particular combination of notes with certain laws which, when carefully observed, preserves and safeguards their integrity and create wonders in music. The existing MIR systems and audio search engines concentrate on content based audio mining and not ragas. This paper discusses the classification of ragas and how it differs in Carnatic, Hindustani and Western musical forms. It also describes the various features that can be extracted from an audio data to represent a raga. Finally, we propose a system architecture to perform raga based classification of music.

Music Information Retrieval (MIR) is an interdisciplinary research area that covers automated extraction of information from audio signals, music databases and services enabling the indexed information searching. In the early stages the primary focus of MIR was on music information through Query-by-Humming (QBH) applications, i.e. on identifying a piece of music by singing (singing/whistling), while more advanced implementations supporting Query-by-Example (QBE) searching resulted in names of audio tracks, song identification, etc. Both QBH and QBE required several steps, among others an optimized signal parametrization and the soft computing approach. Nowadays, MIR is associated with research based on the content analysis that is related to the retrieval of a musical style, genre or music referring to mood or emotions. Even though, this type of music retrieval called Query-by-Category still needs feature extraction and parametrization optimizing, but in this case search of global online music systems and services applications with their millions of users is based on statistical measures. The paper presents details concerning MIR background and answers a question concerning usage of soft computing versus statistics, namely: why and when each of them should be employed.

The article describes the project on music information retrieval that has been carried out at the University of Padova, Italy. The research work has been characterized by the synergy of the modular integration of sound techniques of melody processing and of statistical information retrieval. After illustrating the background from which the project has originated, we describe the complete process, from methodology design through evaluation and system implementation. Conclusions, impacts on research in music information retrieval, and future directions are also described.

The field of Music Information Retrieval is still very immature although it has been addressed by a substantial number of research projects over past three decades. Many problems are faced in music information retrieval due to the nature of music itself. In this paper, the basic representation of audio and video are discussed along with the challenges of music retrieval. The need for music information is discussed using the statistical survey. The paper challenges the common assumption that searching on the pitch alone is satisfactory and discusses the other features which can aid in searching. This assumption may indeed be right for monophonic music but it is insufficient for polyphonic music. Even in monophonic music the possibility of faulty results can’t be ignored.

Music information retrieval (MIR) is a fast-growing research area. One of its aims is to extract musical characteristics from audio. In this study, we assumed the roles of researchers without further technical MIR experience and set out to test in an exploratory way its opportunities and challenges in the specific context of musical emotion perception. Twenty sound engineers rated 60 musical excerpts from a broad range of styles with respect to 22 spectral, musical, and cross-modal features (perceptual features) and perceived emotional expression. In addition, we extracted 86 features (acoustic features) of the excerpts with the MIRtoolbox (Lartillot & Toiviainen, 2007). First, we evaluated the perceptual and extracted acoustic features. Both perceptual and acoustic features posed statistical challenges (e.g., perceptual features were often bimodally distributed, and acoustic features highly correlated). Second, we tested the suitability of the acoustic features for modeling perceived emotional content. Four nearly disjunctive feature sets provided similar results, implying a certain arbitrariness of feature selection. We compared the predictive power of perceptual and acoustic features using linear mixed effects models, but the results were inconclusive. We discuss critical points and make suggestions to further evaluate MIR tools for modeling music perception and processing.

Over the years, Music Information Retrieval (MIR) has proposed various models pretrained on large amounts of music data. Transfer learning showcases the proven effectiveness of pretrained backend models with a broad spectrum of downstream tasks, including auto-tagging and genre classification. However, MIR papers generally do not explore the efficiency of pretrained models for Music Recommender Systems (MRS). In addition, the Recommender Systems community tends to favour traditional end-to-end neural network learning over these models. Our research addresses this gap and evaluates the applicability of six pretrained backend models (MusicFM, Music2Vec, MERT, EncodecMAE, Jukebox, and MusiCNN) in the context of MRS. We assess their performance using three recommendation models: K-nearest neighbours (KNN), shallow neural network, and BERT4Rec. Our findings suggest that pretrained audio representations exhibit significant performance variability between traditional MIR tasks and MRS, indicating that valuable aspects of musical information captured by backend models may differ depending on the task. This study establishes a foundation for further exploration of pretrained audio representations to enhance music recommendation systems.

Representing a significant shift from methods that employ bibliographic metadata for music retrieval, modern Music Information Retrieval (MIR) research seeks to utilize characteristics of music to search databases of musical content via musically expressed queries. These investigations hold significant promise for the future development of commercial and research applications such as searchable databases of popular tunes on the web and databases of historical music. However the challenges are diverse, encompassing music cognition, storage and processing, and traditional information retrieval (IR) issues. This paper focuses on IR issues in MIR, discussing various approaches to music representation and matching techniques. Prototype MIR systems Meldex, Semex, Themefinder and Arthur are briefly described.

In this presentation, we will cover various technologies needed for the semi-automatic encoding of mensural sources into symbolic scores. Mensural sources refer to the polyphonic music sources of the Late Middle Ages and the Renaissance, written on an earlier music notation system called “mensural notation.” Mensural sources are usually written in some separate-parts layout, with the voices kept on different areas of the page; this makes it harder to visualize the vertical sonorities that become apparent only when the voices are lined up in a score layout. However, scoring up the voices requires an expert because, for triple meter, the duration of the notes in mensural notation is not absolute but context-dependent. This presentation focuses on the Music Information Retrieval (MIR) technologies involved in the semi-automatic encoding of mensural sources into machine-readable scores. These technologies include a few symbolic formats (MEI and Humdrum **mens), optical music recognition software (MuRET), automatic scoring-up software for mensural notation (included within an online mensural notation editor called the Measuring Polyphony Editor), and computational music analysis tools for Renaissance music (humlib’s dissonance filter). A real-life example of the digitization and encoding of a seventeenth-century music manuscript from the Americas will illustrate the use of these technologies.

Music Information Retrieval (MIR) is a crucial topic in the domain of information retrieval. According to major characteristics of music, Query-by-Humming system retrieves interesting music by finding melody that contains similar or equal melody to the humming query. Basing on the designed fuzzy inference model a novel Query-by-Humming/Singing system is proposed to extract pitch contour information from WAV and MIDI files in this paper. To verify the effectiveness of the presented work, the MIREX QBSH Database is employed as our experimental database and a large amount of human vocal data is used as query to test the robustness of MIR. Then, the Longest Common Subsequence (LCS) is served as an approximate matching algorithm to identify the most related top 5 music as an evaluation standard for the system. Experimental results show that the proposed system achieves 85% accuracy in the top 5 retrievals.

Jazz improvisation provides a case context for examining information in music; entropy provides a means for representing music for retrieval. Entropy measures are shown to distinguish between different improvisations on the same theme, thus demonstrating their potential for representing jazz information for analysis and retrieval. The calculated entropy measures are calibrated against human representation by means of a case study of an advanced jazz improvisation course, in which synonyms for "entropy" are frequently used by the instructor. The data sets are examined for insights in music information retrieval, music information behavior, and music representation.

Deep Learning has emerged as a powerful set of computational methods achieving great results in a variety of different tasks. Music signal processing, a field with rich commercial applications, seems to benefit too from this data-driven approach. In this paper a review of the state of the art Deep Learning methods applied on music signal processing is provided. A special focus is given in music information retrieval and music generation. In addition, possible future research directions are discussed.

We present the open-source implementation of the first fully automatic and comprehensive DJ system, able to generate seamless music mixes using songs from a given library much like a human DJ does.The proposed system is built on top of several enhanced music information retrieval (MIR) techniques, such as for beat tracking, downbeat tracking, and structural segmentation, to obtain an understanding of the musical structure. Leveraging the understanding of the music tracks offered by these state-of-the-art MIR techniques, the proposed system surpasses existing automatic DJ systems both in accuracy and completeness. To the best of our knowledge, it is the first fully integrated solution that takes all basic DJing best practices into account, from beat and downbeat matching to identification of suitable cue points, determining a suitable cross-fade profile and compiling an interesting playlist that trades off innovation with continuity.To make this possible, we focused on one specific sub-genre of electronic dance music, namely Drum and Bass. This allowed us to exploit genre-specific properties, resulting in a more robust performance and tailored mixing behavior.Evaluation on a corpus of 160 Drum and Bass songs and an additional hold-out set of 220 songs shows that the used MIR algorithms can annotate 91% of the songs with fully correct annotations (tempo, beats, downbeats, and structure for cue points). On these songs, the proposed song selection process and the implemented DJing techniques enable the system to generate mixes of high quality, as confirmed by a subjective user test in which 18 Drum and Bass fans participated.

Musical themes are essential elements in Western classical music. In this paper, we present the Musical Theme Dataset (MTD), a multimodal dataset inspired by “A Dictionary of Musical Themes” by Barlow and Morgenstern from 1948. For a subset of 2067 themes of the printed book, we created several digital representations of the musical themes. Beyond graphical sheet music, we provide symbolic music encodings, audio snippets of music recordings, alignments between the symbolic and audio representations, as well as detailed metadata on the composer, work, recording, and musical characteristics of the themes. In addition to the data, we also make several parsers and web-based interfaces available to access and explore the different modalities and their relations through visualizations and sonifications. These interfaces also include computational tools, bridging the gap between the original dictionary and music information retrieval (MIR) research. The dataset is of relevance for various subfields and tasks in MIR, such as cross-modal music retrieval, music alignment, optical music recognition, music transcription, and computational musicology.

Music information research (MIR) has been a fast growing field of research during the past decade. In traditional MIR, music-related information were extracted from the audio signal using signal processing techniques. These methods, however, cannot capture semantic information that is not encoded in the audio signal, but nonetheless essential to many consumers, e.g., the meaning of the lyrics of a song or the political motivation or background of a singer. The recent launches of Google Music, Amazon.com's Cloud Player and Apple's iCloud with iTunes' Scan and Match show the huge commercial interest behind music distribution and consumption. Given the fact that music is an omnipresent topic on the Web, techniques to mine the Web of Music are vital for music information research.In recent years, the emergence of various Web 2.0 platforms and services dedicated or related to the music and audio domain, like last.fm, YouTube, MusicBrainz, Pandora, or Echo Nest, has been providing novel and powerful, albeit noisy, sources for high level, semantic information on music artists, albums, songs, and others. The abundance of such information provided by the power of the crowd can therefore contribute to music information research and development considerably. On the other hand, the wealth of newly available, semantically meaningful information offered on Web 2.0 platforms also poses new challenges, e.g., dealing with the huge amount and the noisiness of this kind of data, various user biases, hacking, or the cold start problem.Another recent trend are innovative user interfaces to access the large amounts of music available on smart mobile devices that are always connected to the Web. Dealing with the vast amounts of music requires new interaction paradigms and intelligent services that provide, for example, personalized and context-aware music recommendations. The current emergence and confluence of these challenges make this an interesting field for researchers and industry practitioners alike.Addressing these challenges, the International Workshop on Advances in Music Information Research (AdMIRe) workshop served as a forum for theoretical and practical discussions of cutting edge research in the fields of music information extraction, retrieval, and recommendation in the Web of Music. The workshop brought together researchers and developers from the music and audio community, the multimedia community, and the Web mining community and further initiated interesting and fruitful discussions on the future of MIR.This year was AdMIRe's 4th edition, and we received a considerable number of high-quality submissions. A total of 14 submissions from 7 different countries have been reviewed by at least 4 reviewers each, and 9 papers have finally been accepted for publication and presentation at the workshop. They span a wide field, including topics such as Web-based data and microblogging mining, content-based music processing, music similarity and recommendation from hybrid sources, and new evaluation methods.

The exponential growth of computer processing power, cloud data storage, and crowdsourcing model of gathering data bring new possibilities to Music Information Retrieval (MIR) field. MIR is no longer music content retrieval only; the area also comprises the discovery of expressing feelings and emotions contained in music, incorporating other than hearing modalities for helping this issue, users’ profiling, merging music with social media and qualitative recommendations in music services. Moreover, 5G telecommunications networks, characterized by “near-instant and everything in the vicinity talks with one another,” with exponentially faster download and upload speeds, may change the existing models and create a new age of interconnectedness. This paper aims at showing some of the already highly exploited technologies and crowdsourcing models applied to music processing. Several studies are discussed in details, such as, e.g., deep learning applied to music, a way to generate an expanded training sets using 2-D data such spectrograms, mel-cepstrograms, chromagrams, and waveform-based representations of the signal instead of feature vectors in machine learning, allowing to retain all nuances related musical articulation in the signal. Also, a discussion is to be outlined, expanding the issue of the impact of these new technologies on the artistic and aesthetic values of music.