Machine learning music emotion recognition based on audio features

Depression screening with textual and audio features based on large language models and machine learning.

Early diagnosis of medical conditions in infants is crucial for ensuring timely and effective treatment. However, infants are unable to verbalize their symptoms, making it difficult for healthcare professionals to accurately diagnose their conditions. Crying is often the only way for infants to communicate their needs and discomfort. In this paper, we propose a medical diagnostic system for interpreting infants’ cry audio signals (CAS) using a combination of different audio domain features and deep learning (DL) algorithms. The proposed system utilizes a dataset of labeled audio signals from infants with specific pathologies. The dataset includes two infant pathologies with high mortality rates, neonatal respiratory distress syndrome (RDS), sepsis, and crying. The system employed the harmonic ratio (HR) as a prosodic feature, the Gammatone frequency cepstral coefficients (GFCCs) as a cepstral feature, and image-based features through the spectrogram which are extracted using a convolution neural network (CNN) pretrained model and fused with the other features to benefit multiple domains in improving the classification rate and the accuracy of the model. The different combination of the fused features is then fed into multiple machine learning algorithms including random forest (RF), support vector machine (SVM), and deep neural network (DNN) models. The evaluation of the system using the accuracy, precision, recall, F1-score, confusion matrix, and receiver operating characteristic (ROC) curve, showed promising results for the early diagnosis of medical conditions in infants based on the crying signals only, where the system achieved the highest accuracy of 97.50% using the combination of the spectrogram, HR, and GFCC through the deep learning process. The finding demonstrated the importance of fusing different audio features, especially the spectrogram, through the learning process rather than a simple concatenation and the use of deep learning algorithms in extracting sparsely represented features that can be used later on in the classification problem, which improves the separation between different infants’ pathologies. The results outperformed the published benchmark paper by improving the classification problem to be multiclassification (RDS, sepsis, and healthy), investigating a new type of feature, which is the spectrogram, using a new feature fusion technique, which is fusion, through the learning process using the deep learning model.

Music emotion recognition and generation play a vital role in creating emotionally resonant experiences in human-computer interaction. Deep reinforcement learning (DRL) offers promising capabilities for dynamically modeling and generating music aligned with human emotional states. However, existing methods often struggle with rigid emotional mapping, a lack of real-time adaptability, and limited generalization across diverse musical contexts. To address these limitations, this study proposes an Emotion-Conditioned Deep Reinforcement Learning (EC-DRL) framework that integrates emotion-aware representations into the reward mechanism of a DRL agent. The system employs deep neural networks to extract high-level audio features and map them onto valence-arousal emotion space, guiding the music generation process through a reinforcement learning policy optimized for emotional congruence. This framework is applied in adaptive soundtrack generation for interactive applications such as video games, where music responds dynamically to emotional cues derived from gameplay and user interactions. Experimental results demonstrate that the EC-DRL framework significantly improves emotional accuracy, coherence, and user satisfaction compared to traditional sequence-based generation models. The proposed system shows the potential for creating emotionally intelligent music systems that can adapt in real-time and produce expressive musical output. EC-DRL achieves the mapping accuracy by 98%, emotional congruence score by 0.9%, real-time responsiveness by 280 ms, reward function optimization by 9.5%, audio feature extraction quality by 86%, policy convergence rate by 0.8%, user satisfaction score by 8.9%, and cross-domain generalization by 88%.

The high feature dimensionality is a challenge in music emotion recognition. There is no common consensus on a relation between audio features and emotion. The MER system uses all available features to recognize emotion; however, this is not an optimal solution since it contains irrelevant data acting as noise. In this paper, we introduce a feature selection approach to eliminate redundant features for MER. We created a Selected Feature Set (SFS) based on the feature selection algorithm (FSA) and benchmarked it by training with two models, Support Vector Regression (SVR) and Random Forest (RF) and comparing them against with using the Complete Feature Set (CFS). The result indicates that the performance of MER has improved for both Random Forest (RF) and Support Vector Regression (SVR) models by using SFS. We found using FSA can improve performance in all scenarios, and it has potential benefits for model efficiency and stability for MER task.

During a music performance, the musician adds expressiveness to the musical message by changing timing, dynamics, and timbre of the musical events to communicate an expressive intention. Traditionally, the analysis of music expression is based on measurements of the deviations of the acoustic parameters with respect to the written score. In this paper, we employ machine learning techniques to understand the expressive communication and to derive audio features at an intermediate level, between music intended as a structured language and notes intended as sound at a more physical level. We start by extracting audio features from expressive performances that were recorded by asking the musicians to perform in order to convey different expressive intentions. We use a sequential forward selection procedure to rank and select a set of features for a general description of the expressions, and a second one specific for each instrument. We show that higher recognition ratings are achieved by using a set of four features which can be specifically related to qualitative descriptions of the sound by physical metaphors. These audio features can be used to retrieve expressive content on audio data, and to design the next generation of search engines for music information retrieval.

Music is a fundamental element in every culture, serving as a universal means of expressing our emotions, feelings, and beliefs. This work investigates the link between our moral values and musical choices through lyrics and audio analyses. We align the psychometric scores of 1,480 participants to acoustics and lyrics features obtained from the top 5 songs of their preferred music artists from Facebook Page Likes. We employ a variety of lyric text processing techniques, including lexicon-based approaches and BERT-based embeddings, to identify each song's narrative, moral valence, attitude, and emotions. In addition, we extract both low- and high-level audio features to comprehend the encoded information in participants' musical choices and improve the moral inferences. We propose a Machine Learning approach and assess the predictive power of lyrical and acoustic features separately and in a multimodal framework for predicting moral values. Results indicate that lyrics and audio features from the artists people like inform us about their morality. Though the most predictive features vary per moral value, the models that utilised a combination of lyrics and audio characteristics were the most successful in predicting moral values, outperforming the models that only used basic features such as user demographics, the popularity of the artists, and the number of likes per user. Audio features boosted the accuracy in the prediction of empathy and equality compared to textual features, while the opposite happened for hierarchy and tradition, where higher prediction scores were driven by lyrical features. This demonstrates the importance of both lyrics and audio features in capturing moral values. The insights gained from our study have a broad range of potential uses, including customising the music experience to meet individual needs, music rehabilitation, or even effective communication campaign crafting.

AFExplorer: Visual analysis and interactive selection of audio features

BackgroundPatients with Alzheimer disease (AD) and amnesic mild cognitive impairment (aMCI) have deficits in emotion recognition. However, it has not yet been determined whether patients with AD and aMCI also experience difficulty in recognizing the emotions conveyed by music. This study was conducted to investigate whether musical emotion recognition is impaired or retained in patients with AD and aMCI.MethodsAll patients were recruited from the First Affiliated Hospital of Anhui Medical University between March 1, 2015 and January 31, 2017. Using the musical emotion recognition test, patients with AD (n = 16), patients with aMCI (n = 19), and healthy controls (HCs, n = 16) were required to choose one of four emotional labels (happy, sad, peaceful, and fearful) that matched each musical excerpt. Emotion recognition scores in three groups were compared using one-way analysis of variance (ANOVA) test. We also investigated the relationship between the emotion recognition scores and Mini-Mental State Examination (MMSE) using Pearson's correlation analysis test in patients with AD and aMCI.ResultsCompared to the HC group, both of the patient groups showed deficits in the recognition of fearful musical emotions (HC: 7.88 ± 1.36; aMCI: 5.05 ± 2.34; AD: 3.69 ± 2.02), with results of a one-way ANOVA confirming a significant main effect of group (F(2,50) = 18.70, P < 0.001). No significant differences were present among the three groups for the happy (F(2,50)=2.57, P = 0.09), peaceful (F(2,50) = 0.38, P = 0.09), or sad (F(2,50) = 2.50, P = 0.09) musical emotions. The recognition of fearful musical emotion was positively associated with general cognition, which was evaluated by MMSE in patients with AD and aMCI (r = 0.578, P < 0.001). The correlations between the MMSE scores and recognition of the remaining emotions were not significant (happy, r = 0.228, P = 0.11; peaceful, r = 0.047, P = 0.74; sad, r = 0.207, P = 0.15).ConclusionThis study showed that both patients with AD and aMCI had decreased ability to distinguish fearful emotions, which might be correlated with diminished cognitive function.

<title>Abstract</title> In the field of speech emotion recognition, the choice of audio features can dramatically influence the accuracy and effectiveness of classification systems. This study presents a comprehensive comparative analysis of feature significance, shedding light on how different audio characteristics contribute to the success of emotion recognition methodologies. This paper analyzes speech-based emotion recognition techniques and works with audio analyses using the Ryerson Audio-Visual Database (RAVD) of Emotional Speech and Song, a database consisting of audio analysis on raw audio files. Analysis involved features like Log-Mel Spectrograms (LMS), Mel-Frequency Cepstral Coefficients (MFCCs), pitch, and energy, after raw audio files are pre-processed. We measured the relevance of these features to the classification of emotion through a series of approaches that include Long Short-Term Memory networks, Convolutional Neural Networks, Hidden Markov Models, and Deep Neural Networks. On a 14-class classification problem that covers two genders and seven emotions, we obtained 56% accuracy by using a 4-layer 2-dimensional CNN with Log-Mel Spectrogram features. Our results show the importance of selection of good audio features while complexity is not that important for performance for emotion recognition.

This study introduces a novel approach to emotion recognition by amalgamating information from heterogeneous modalities, specifically audio and video. We employed techniques such as energy, zero crossing rate, and Mel-Frequency Cepstral Coefficients (MFCC) for audio feature extraction, which showed promising results. For video feature extraction, spatialtemporal Gaussian kernels were used to organize video frames within a linear scale space, followed by the application of a Gaussian-weighted function to the second momentum matrix for further feature extraction. The Multimodal Feature Aggregation (MFA) fusion method was employed to unify audio and video features, resulting in a comprehensive dataset. Evaluation through the Fusion of Emotion Recognition Convolutional Neural Network (FERCNN) model, supported by the "TPU VM v3-8" accelerator TPU is a Tensor Processing Unit, showcased notable performance improvements. Using the RAVDESS and CREMAD datasets, accuracies of 94.66%, 95.82%, and 94.36% in the RAVDESS dataset and 79.45%, 96.62%, and 70.14% in the CREMAD dataset for audio, video, and multimodal modalities, respectively, were achieved. These outcomes surpass the capabilities of existing multimodal systems, underscoring the efficacy of our proposed approach. Emotion recognition, particularly through multimodal means, plays a critical role in various domains, including human-computer interfaces, healthcare, legal proceedings, and entertainment. Fusing Audio and Video Modalities to Elevate Human-Computer Interaction and Intelligent System Performance is essential for enhancing communication within these domains. The proposed model is termed "DualVision EmotionNet: DV EmotionNet".

Emotions are a natural component of life, and the human brain's central nervous system regulates their generations and expressions. The capacity of computers to identify and replicate human emotions can be improved by studying the brain mechanisms underlying human emotion display. Audio, which is currently one of the most popular types of multimedia stimulation, may communicate a wide range of emotional meanings due to its aural characteristics. In addition to audio features, EEG features can provide useful novelty for emotion recognition, as these features are the most realistic feedback on human emotion perception. This paper constructed a fused dataset of EEG and audio features based on the SEED EEG dataset. A deep learning model based on Long-Short Term Memory Network (LSTM) is utilized to find the best model for audio emotion identification using the bimodal dataset. We discovered that combining the full-band EEG power spectral density and audio fusion characteristics resulted in the best recognition. Simultaneously, We extracted features from each original stimulus audio and added audio characteristics from each period to examine the influence of emotion recognition and investigate the inherent link between the creation of EEG signals and the original stimulus audio in-depth. The study shows that the method can be used as a potential video emotion indexing method for video information retrieval.

The development of computer technology, multimedia technology and communication technology has made the application of computers more and more widespread in people's lives. The next direction of computer development is to expect to be able to use computers to analyze, simulate and express human subjective thinking activities. As a part of people's lives, music has a strong emotional appeal. The stronger the musical emotion, the more audiences representing the song. Most of the computer technology related to music is still in the storage, tuning, compression, etc. of music, but with the development of computer analysis and automatic recognition technology, today's computer technology can already realize the cognition of music emotions through models and programs and creation. This article first briefly introduces computer analysis and automatic recognition technology, then analyzes and discusses the current representation model of music emotion recognition, and finally tries to design a music emotion recognition model design and proposes to introduce BP neural network into music emotion recognition technology to realize the automatic emotion analysis and recognition of the main music track.

Music emotion recognition (MER) is a challenging field of studies that has been addressed in multiple disciplines such as cognitive science, physiology, psychology, musicology, and arts. In this paper, music emotions are modeled as a set of continuous variables composed of valence and arousal (VA) values based on the Valence-Arousal model. MER is formulated as a regression problem where 548 dimensions of music features were extracted and selected. A wide range of methods including multivariate adaptive regression spline, support vector regression (SVR), radial basis function, random forest regression (RFR), and regression neural networks are adopted to recognize music emotions. Experimental results show that these regression algorithms have led to good regression effect for MER. The optimal R2 statistics and VA values are 29.3% and 62.5%, respectively, which are obtained by the RFR and SVR algorithms in the relief feature space.

Automatic emotion recognition from audio-visual data is a topic that has been broadly explored using data captured in the laboratory. However, these data are not necessarily representative of how emotion is manifested in the real-world. In this paper, we describe our system for the 2016 Emotion Recognition in the Wild challenge. We use the Acted Facial Expressions in the Wild database 6.0 (AFEW 6.0), which contains short clips of popular TV shows and movies and has more variability in the data compared to laboratory recordings. We explore a set of features that incorporate information from facial expressions and speech, in addition to cues from the background music and overall scene. In particular, we propose the use of a feature set composed of dimensional emotion estimates trained from outside acoustic corpora. We design sets of multiclass and pairwise (one-versus-one) classifiers and fuse the resulting systems. Our fusion increases the performance from a baseline of 38.81% to 43.86% and from 40.47% to 46.88%, for validation and test sets, respectively. While the video features perform better than audio features alone, a combination of the two modalities achieves the greatest performance, with gains of 4.4% and 1.4%, with and without information gain, respectively. Because of the flexible design of the fusion, it is easily adaptable to other multimodal learning problems.

Emotion recognition is one of the most important assets that humans possess, which helps them to communicate with each other in the most efficient manner. Humans recognize each other’s emotions through facial features, audio features and body gestures. Limited work is reported in literature, wherein multiple modalities are considered for emotion recognition. In this work, models are trained on facial and speech samples using 1-Dimensional (1D) and 2-Dimensional (2D) Convolutional Neural Networks (CNN) and also trained using pretrained networks such as Visual Geometry Group-16 (VGG-16) and Inception Version 3 (V3). Suitable features are extracted from face and speech to classify the emotions. Analysis of speech features is performed to highlight the difference between the expressiveness based on genders. For the final combined model both facial and speech models are trained separately and combined based on a decision tree, which depends on the individual model’s test case accuracy.