Multimodal Mood Classification Framework for Hindi Songs

The notion of perceptual features is introduced for describing general music properties based on human perception. This is an attempt at rethinking the concept of features, aiming to approach the underlying human perception mechanisms. Instead of using concepts from music theory such as tones, pitches, and chords, a set of nine features describing overall properties of the music was selected. They were chosen from qualitative measures used in psychology studies and motivated from an ecological approach. The perceptual features were rated in two listening experiments using two different data sets. They were modeled both from symbolic and audio data using different sets of computational features. Ratings of emotional expression were predicted using the perceptual features. The results indicate that (1) at least some of the perceptual features are reliable estimates; (2) emotion ratings could be predicted by a small combination of perceptual features with an explained variance from 75% to 93% for the emotional dimensions activity and valence; (3) the perceptual features could only to a limited extent be modeled using existing audio features. Results clearly indicated that a small number of dedicated features were superior to a "brute force" model using a large number of general audio features.

Multimedia content can be described in versatile ways as its essence is not limited to one view. For music data these multiple views could be a song's audio features as well as its lyrics. Both of these modalities have their advantages as text may be easier to search in and could cover more of the 'content semantics' of a song, while omitting other types of semantic categorisation. (Psycho) acoustic feature sets, on the other hand, provide the means to identify tracks that 'sound similar' while less supporting other kinds of semantic categorisation. Those discerning characteristics of different feature sets meet users' differing information needs. We will explain the nature of text and audio feature sets which describe the same audio tracks. Moreover, we will propose the use of textual data on top of low level audio features for music genre classification. Further, we will show the impact of different combinations of audio features and textual features based on content words.

During a music performance, the musician adds expressiveness to the musical message by changing timing, dynamics, and timbre of the musical events to communicate an expressive intention. Traditionally, the analysis of music expression is based on measurements of the deviations of the acoustic parameters with respect to the written score. In this paper, we employ machine learning techniques to understand the expressive communication and to derive audio features at an intermediate level, between music intended as a structured language and notes intended as sound at a more physical level. We start by extracting audio features from expressive performances that were recorded by asking the musicians to perform in order to convey different expressive intentions. We use a sequential forward selection procedure to rank and select a set of features for a general description of the expressions, and a second one specific for each instrument. We show that higher recognition ratings are achieved by using a set of four features which can be specifically related to qualitative descriptions of the sound by physical metaphors. These audio features can be used to retrieve expressive content on audio data, and to design the next generation of search engines for music information retrieval.

Depression screening with textual and audio features based on large language models and machine learning.

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years.

A multimodal hierarchical approach to speech emotion recognition from audio and text

The main objective of the project is to build a system that allows users to predict genre as well as language out of some music . The Automated determination of musical genre and linguistic categorization is a machine learning based prediction model which is used to predict the genres and languages of an audio . Musical genres are categorical labels created by humans to characterize piece of music. A musical genre is characterized by common characteristics typically related to the instrumentation, rhythmic structure and harmonic content of the music. The performance and relative importance of the proposed features are investigated by training the statistical pattern recognition classifiers using real world audio collections and using machine learning algorithm.In this project, we built systems called music genre and language classification and the effects of different feature extraction methods and their various combinations were also investigated. This task is of great interest in music information retrieval and recommendation systems. In recent years, several machine learning algorithms have been developed to tackle this task, using features such spectral, temporal, and tonal characteristics of audio signals. Music genre prediction and language classification is a task that involves automatically predicting a piece of music into a specific genre category as well as language based on its audio features. Keywords-machine-learning,music,music genre and language classification

Choral singing is a central part of musical cultures across the world, yet many facets of this widespread form of polyphonic singing are still to be explored. Music information retrieval (MIR) research on choral singing benefits from multitrack recordings of the individual singing voices. However, there exist only few publicly available multitrack datasets on polyphonic singing. In this paper, we present Dagstuhl ChoirSet (DCS), a multitrack dataset of a cappella choral music designed to support MIR research on choral singing. The dataset includes recordings of an amateur vocal ensemble performing two choir pieces in full choir and quartet settings. The audio data was recorded during an MIR seminar at Schloss Dagstuhl using different close-up microphones to capture the individual singers’ voices. In this article, we give detailed insights into all stages of creating DCS: recording process, data preparation, generation of annotations as well as development of suitable interfaces for publicly accessing and reusing the data. Furthermore, we demonstrate the potential of the dataset for MIR research by discussing case studies on choral intonation assessment and multiple fundamental frequency (F0) estimation.

This paper introduces the Raga Based Music Classification as a part of Music Information Retrieval (MIR) systems for retrieving musical information. The music has been more than just a sound that entertains us. MIR users like musicology researchers, musicians, music learners, music therapists, etc., require not just the sound but the actual strength of the music that controls our body, motivates us, cures mental illness, etc. The backbone of any form of music is its raga. A raga is a particular combination of notes with certain laws which, when carefully observed, preserves and safeguards their integrity and create wonders in music. The existing MIR systems and audio search engines concentrate on content based audio mining and not ragas. This paper discusses the classification of ragas and how it differs in Carnatic, Hindustani and Western musical forms. It also describes the various features that can be extracted from an audio data to represent a raga. Finally, we propose a system architecture to perform raga based classification of music.

This paper proposes methods for information processing of audio streams using methods of information geometry. We lay the theoretical groundwork for a framework allowing the treatment of signal information as information entities, suitable for similarity and symbolic computing on audio signals. The theoretical basis of this paper is based on the information geometry of statistical structures representing audio spectrum features, and specifically through the bijection between the generic families of Bregman divergences and that of exponential distributions. The proposed framework, called Music Information Geometry, allows online segmentation of audio streams to metric balls where each ball represents a quasi-stationary continuous chunk of audio, and discusses methods to qualify and quantify information between entities for similarity computing. We define an information geometry that approximates a similarity metric space, redefine general notions in music information retrieval such as similarity between entities, and address methods for dealing with nonstationarity of audio signals. We demonstrate the framework on two sample applications for online audio structure discovery and audio matching.

Putting emotional labels on music, or "music mood classification," is important for use in recommendation systems and music therapy. Using fine-tuned machine learning methods, this study aims to improve the accuracy and performance of classification. We used a large dataset with names for different types of music and moods to make sure that the model training was strong. Advanced feature extraction methods picked up both the traits of the audio stream and the lyrics. For audio features, color features, spectral contrast, and mel-frequency cepstral coefficients (MFCCs) were recovered. For poetry analysis, TF-IDF and word embeddings were used, along with natural language processing (NLP) methods. Logistic Regression, SGD Classifier, Gaussian Naive Bayes, Decision Tree, Random Forest, XGB Classifier, SVM Linear, and K-Nearest Neighbors (KNN) were some of the machine learning classification methods we used. Random Forest, XGB Classifier, and SVM Linear all did better than the others. We used grid search and random search to fine-tune the hyperparameters of these top-performing models in order to make them even better. Cross-validation made sure that the models were stable and could be used in other situations. Our results show that the highly tuned Random Forest, XGB, and SVM models greatly improved the accuracy of classification, with the XGB Classifier performing the best. This study adds to music information retrieval by creating a useful method for mood classification that can be used in real-life situations to improve user experiences and create more personalized music services.

Emotion detection, hence, is an important step toward making human-computer interaction a more enhanced process, where systems are made capable of identifying and responding to the emotional state of users. In fact, multimodal emotion detection systems in which both auditory and visual information are fused are emerging, and these approaches toward expressive emotional states are complementary and robust. Multimodal systems enhance the quality of interacting and, through many applications, can diagnose emotional disorders, monitor automotive safety, and improve human-robot interactions. In nature, the high-dimensional space and dynamic threats have resulted in obtaining low accuracy and high computational cost in applying the traditional models based on single-modality data. On the other hand, multimodal systems explore the synergy between audio and visual data, giving better performance and higher accuracy in inferring subtle emotional expressions. The latest improvement was done on these systems using recent advancements in transfer learning and deep learning techniques.That being said, this research Proposal devises a multimodal emotion recognition system integrating speech and face information through transfer learning for improved accuracy and robustness. Serving this purpose, the objectives of this research entail the effective comparison among different transfer-learning strategies, including the impact of pre-trained models in speech-based emotion recognition, and to introduce the role of voice activity detection in the process. Advanced neural network architectures like Spatial Transformer Networks and bidirectional LSTM in facial emotion recognition will also be tested. Early and late fusion strategies will also be used to find the best strategy for combining speech and facial data.This research will target several challenges that involve the complexity of data, balancing of the model performance-robustness balance, computational limitations, and standardization of evaluations in developing a working and robust emotion recognition system to enhance digital interaction and apply in practical areas. The goal is to create a system that oversteps the limitation of single-modality models through state-of-the-art advances in deep learning, as well as front-line improvements in transfer learning, in the manner of emotion detection performance.

Musical themes are essential elements in Western classical music. In this paper, we present the Musical Theme Dataset (MTD), a multimodal dataset inspired by “A Dictionary of Musical Themes” by Barlow and Morgenstern from 1948. For a subset of 2067 themes of the printed book, we created several digital representations of the musical themes. Beyond graphical sheet music, we provide symbolic music encodings, audio snippets of music recordings, alignments between the symbolic and audio representations, as well as detailed metadata on the composer, work, recording, and musical characteristics of the themes. In addition to the data, we also make several parsers and web-based interfaces available to access and explore the different modalities and their relations through visualizations and sonifications. These interfaces also include computational tools, bridging the gap between the original dictionary and music information retrieval (MIR) research. The dataset is of relevance for various subfields and tasks in MIR, such as cross-modal music retrieval, music alignment, optical music recognition, music transcription, and computational musicology.

In the music information retrieval (MIR) field, it is highly desirable to know what instruments are used in an audio sample. Musical instrument classification is one of the sub domains of music information retrieval. Many researchers have presented different approaches for identifying western instruments and those approaches proved to be good for instrument identification. In this article, we have presented work done by the various authors to identify musical instrument using various approaches such sparse based representation, bio-inspired hierarchical model, joint modelling, Bayesian networks, neural networks, convolution neural networks, individual partials, clustering, and segmentation.

Music Information Retrieval (MIR) is an interdisciplinary research area that covers automated extraction of information from audio signals, music databases and services enabling the indexed information searching. In the early stages the primary focus of MIR was on music information through Query-by-Humming (QBH) applications, i.e. on identifying a piece of music by singing (singing/whistling), while more advanced implementations supporting Query-by-Example (QBE) searching resulted in names of audio tracks, song identification, etc. Both QBH and QBE required several steps, among others an optimized signal parametrization and the soft computing approach. Nowadays, MIR is associated with research based on the content analysis that is related to the retrieval of a musical style, genre or music referring to mood or emotions. Even though, this type of music retrieval called Query-by-Category still needs feature extraction and parametrization optimizing, but in this case search of global online music systems and services applications with their millions of users is based on statistical measures. The paper presents details concerning MIR background and answers a question concerning usage of soft computing versus statistics, namely: why and when each of them should be employed.