Using listener-based perceptual features as intermediate representations in music information retrieval.

Music information retrieval (MIR) is a fast-growing research area. One of its aims is to extract musical characteristics from audio. In this study, we assumed the roles of researchers without further technical MIR experience and set out to test in an exploratory way its opportunities and challenges in the specific context of musical emotion perception. Twenty sound engineers rated 60 musical excerpts from a broad range of styles with respect to 22 spectral, musical, and cross-modal features (perceptual features) and perceived emotional expression. In addition, we extracted 86 features (acoustic features) of the excerpts with the MIRtoolbox (Lartillot & Toiviainen, 2007). First, we evaluated the perceptual and extracted acoustic features. Both perceptual and acoustic features posed statistical challenges (e.g., perceptual features were often bimodally distributed, and acoustic features highly correlated). Second, we tested the suitability of the acoustic features for modeling perceived emotional content. Four nearly disjunctive feature sets provided similar results, implying a certain arbitrariness of feature selection. We compared the predictive power of perceptual and acoustic features using linear mixed effects models, but the results were inconclusive. We discuss critical points and make suggestions to further evaluate MIR tools for modeling music perception and processing.

"Music information retrieval is currently an active domain of research. An interesting aspect of music information retrieval involves mood classification. While the Western music captured much attention, research on Indian music was limited and mostly based on audio data. In this work, the authors propose a mood taxonomy and describe the framework for developing a multimodal dataset (audio and lyrics) for Hindi songs. We observed differences in mood for several instances of Hindi songs while annotating the audio of such songs in contrast to their corresponding lyrics. Finally, the mood classification frameworks were developed for Hindi songs and they consist of three different systems based on the features of audio, lyrics and both. The mood classification systems based on audio and lyrics achieved F-measures of 58.2% and 55.1%, respectively whereas the multimodal system (combination of both audio and lyrics) achieved the maximum F-measure of 68.6%."

In this paper, we propose new image features called perceptual colour features. The features are based on twenty basic colours called emotional colours, which are used to describe the relationship between colours and emotions in psychological studies. We analyzed a colour image in L*a*b* and L*C*h colour spaces to link to emotions. Then, the perceptual colour features are derived from the histogram of the twenty emotional colours. We have demonstrated effectiveness of the perceptual colour features with an emotion based image retrieval application. Experiments were performed on a set of impressive natural scene images. The retrieval results were analyzed by both objective evaluation and subjective evaluation. The results revealed that the proposed perceptual colour features outperformed the conventional colour histogram features both in terms of precision and consistency with users' perception.

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years.

Multimedia content can be described in versatile ways as its essence is not limited to one view. For music data these multiple views could be a song's audio features as well as its lyrics. Both of these modalities have their advantages as text may be easier to search in and could cover more of the 'content semantics' of a song, while omitting other types of semantic categorisation. (Psycho) acoustic feature sets, on the other hand, provide the means to identify tracks that 'sound similar' while less supporting other kinds of semantic categorisation. Those discerning characteristics of different feature sets meet users' differing information needs. We will explain the nature of text and audio feature sets which describe the same audio tracks. Moreover, we will propose the use of textual data on top of low level audio features for music genre classification. Further, we will show the impact of different combinations of audio features and textual features based on content words.

During a music performance, the musician adds expressiveness to the musical message by changing timing, dynamics, and timbre of the musical events to communicate an expressive intention. Traditionally, the analysis of music expression is based on measurements of the deviations of the acoustic parameters with respect to the written score. In this paper, we employ machine learning techniques to understand the expressive communication and to derive audio features at an intermediate level, between music intended as a structured language and notes intended as sound at a more physical level. We start by extracting audio features from expressive performances that were recorded by asking the musicians to perform in order to convey different expressive intentions. We use a sequential forward selection procedure to rank and select a set of features for a general description of the expressions, and a second one specific for each instrument. We show that higher recognition ratings are achieved by using a set of four features which can be specifically related to qualitative descriptions of the sound by physical metaphors. These audio features can be used to retrieve expressive content on audio data, and to design the next generation of search engines for music information retrieval.

Choral singing is a central part of musical cultures across the world, yet many facets of this widespread form of polyphonic singing are still to be explored. Music information retrieval (MIR) research on choral singing benefits from multitrack recordings of the individual singing voices. However, there exist only few publicly available multitrack datasets on polyphonic singing. In this paper, we present Dagstuhl ChoirSet (DCS), a multitrack dataset of a cappella choral music designed to support MIR research on choral singing. The dataset includes recordings of an amateur vocal ensemble performing two choir pieces in full choir and quartet settings. The audio data was recorded during an MIR seminar at Schloss Dagstuhl using different close-up microphones to capture the individual singers’ voices. In this article, we give detailed insights into all stages of creating DCS: recording process, data preparation, generation of annotations as well as development of suitable interfaces for publicly accessing and reusing the data. Furthermore, we demonstrate the potential of the dataset for MIR research by discussing case studies on choral intonation assessment and multiple fundamental frequency (F0) estimation.

Depression screening with textual and audio features based on large language models and machine learning.

The main objective of the project is to build a system that allows users to predict genre as well as language out of some music . The Automated determination of musical genre and linguistic categorization is a machine learning based prediction model which is used to predict the genres and languages of an audio . Musical genres are categorical labels created by humans to characterize piece of music. A musical genre is characterized by common characteristics typically related to the instrumentation, rhythmic structure and harmonic content of the music. The performance and relative importance of the proposed features are investigated by training the statistical pattern recognition classifiers using real world audio collections and using machine learning algorithm.In this project, we built systems called music genre and language classification and the effects of different feature extraction methods and their various combinations were also investigated. This task is of great interest in music information retrieval and recommendation systems. In recent years, several machine learning algorithms have been developed to tackle this task, using features such spectral, temporal, and tonal characteristics of audio signals. Music genre prediction and language classification is a task that involves automatically predicting a piece of music into a specific genre category as well as language based on its audio features. Keywords-machine-learning,music,music genre and language classification

In this paper, the relationship between the features of the input audio and its corresponding transparent bitrate is studied. This relationship can be used to pre-estimate the transparent bitrate required for an input audio file, so that this bitrate can be used for encoding and the transparent quality can be achieved with the minimum compressed file size. Two features are selected and tested. Firstly, the perceptual feature is extracted from the test sequences using MPEG-4 advanced audio coding. The results of the signal-to-mask ratio and the transparent bitrate are compared. Secondly, the relationship between the music genre of the audio and its transparent bitrate is explored.

SoundSignature is a music application that integrates a custom OpenAI Assistant to analyze users' favorite songs. The system incorporates state-of-the-art Music Information Retrieval (MIR) Python packages to combine extracted acoustic/musical features with the assistant's extensive knowledge of the artists and bands. Capitalizing on this combined knowledge, SoundSignature leverages semantic audio and principles from the emerging Internet of Sounds (IoS) ecosystem, integrating MIR with AI to provide users with personalized insights into the acoustic properties of their music, akin to a musical preference personality report. Users can then interact with the chatbot to explore deeper inquiries about the acoustic analyses performed and how they relate to their musical taste. This interactivity transforms the application, acting not only as an informative resource about familiar and/or favorite songs, but also as an educational platform that enables users to deepen their understanding of musical features, music theory, acoustic properties commonly used in signal processing, and the artists behind the music. Beyond general usability, the application also incorporates several well-established open-source musician-specific tools, such as a chord recognition algorithm (CREMA), a source separation algorithm (DEMUCS), and an audio-to-MIDI converter (basic-pitch). These features allow users without coding skills to access advanced, open-source music processing algorithms simply by interacting with the chatbot (e.g., can you give me the stems of this song?). In this paper, we highlight the application's innovative features and educational potential, and present findings from a pilot user study that evaluates its efficacy and usability.

Jazz improvisation provides a case context for examining information in music; entropy provides a means for representing music for retrieval. Entropy measures are shown to distinguish between different improvisations on the same theme, thus demonstrating their potential for representing jazz information for analysis and retrieval. The calculated entropy measures are calibrated against human representation by means of a case study of an advanced jazz improvisation course, in which synonyms for "entropy" are frequently used by the instructor. The data sets are examined for insights in music information retrieval, music information behavior, and music representation.

This paper introduces the Raga Based Music Classification as a part of Music Information Retrieval (MIR) systems for retrieving musical information. The music has been more than just a sound that entertains us. MIR users like musicology researchers, musicians, music learners, music therapists, etc., require not just the sound but the actual strength of the music that controls our body, motivates us, cures mental illness, etc. The backbone of any form of music is its raga. A raga is a particular combination of notes with certain laws which, when carefully observed, preserves and safeguards their integrity and create wonders in music. The existing MIR systems and audio search engines concentrate on content based audio mining and not ragas. This paper discusses the classification of ragas and how it differs in Carnatic, Hindustani and Western musical forms. It also describes the various features that can be extracted from an audio data to represent a raga. Finally, we propose a system architecture to perform raga based classification of music.

In the music information retrieval (MIR) field, it is highly desirable to know what instruments are used in an audio sample. Musical instrument classification is one of the sub domains of music information retrieval. Many researchers have presented different approaches for identifying western instruments and those approaches proved to be good for instrument identification. In this article, we have presented work done by the various authors to identify musical instrument using various approaches such sparse based representation, bio-inspired hierarchical model, joint modelling, Bayesian networks, neural networks, convolution neural networks, individual partials, clustering, and segmentation.

This paper proposes methods for information processing of audio streams using methods of information geometry. We lay the theoretical groundwork for a framework allowing the treatment of signal information as information entities, suitable for similarity and symbolic computing on audio signals. The theoretical basis of this paper is based on the information geometry of statistical structures representing audio spectrum features, and specifically through the bijection between the generic families of Bregman divergences and that of exponential distributions. The proposed framework, called Music Information Geometry, allows online segmentation of audio streams to metric balls where each ball represents a quasi-stationary continuous chunk of audio, and discusses methods to qualify and quantify information between entities for similarity computing. We define an information geometry that approximates a similarity metric space, redefine general notions in music information retrieval such as similarity between entities, and address methods for dealing with nonstationarity of audio signals. We demonstrate the framework on two sample applications for online audio structure discovery and audio matching.