Transfer learning from speech to music: towards language-sensitive emotion recognition models

Music Emotion Recognition (MER) is a computational field of affective computing and audio signal processing. Although previous attempts used traditional machine learning algorithms, e.g., Support Vector Machines and k-Nearest Neighbors, for emotion classification in music, these methods are often challenged by the intricate temporal and spectral nature of sound signals, thereby constraining classification performance. To address this, the work proposes a new model, the Lion-Optimized CNN-BiGRU for Emotion Recognition (LOCBER), leveraging the strengths of Convolutional Neural Networks and Bidirectional Gated Recurrent Units for effective feature extraction and temporal sequence modeling. For additional performance optimization, LOCBER is optimized using the Lion Swarm Optimization (LSO) algorithm, which adjusts model parameters to minimize loss and achieve better accuracy and convergence. This study aims to improve music emotion recognition by integrating a CNN–BiGRU model with Lion Swarm Optimization, achieving 97.7% classification accuracy and outperforming conventional methods by 15.3%. LSO-based optimization achieved improved convergence and stability compared to traditional training processes. The LOCBER model benefits emotion-sensitive music recommendation systems, intelligent user interfaces, and music therapy applications. Experiments demonstrate that by fusion of state-of-the-art neural architectures with bio-inspired optimization, one can substantially improve the performance and validity of MER systems for real-world use.

Emotion recognition of music objects is a promising and important research issues in the field of music information retrieval. Usually, music emotion recognition could be considered as a training/classification problem. However, even given a benchmark (a training data with ground truth) and using effective classification algorithms, music emotion recognition remains a challenging problem. Most previous relevant work focuses only on acoustic music content without considering individual difference (i.e., personalization issues). In addition, assessment of emotions is usually self-reported (e.g., emotion tags) which might introduce inaccuracy and inconsistency. Electroencephalography (EEG) is a non-invasive brain-machine interface which allows external machines to sense neurophysiological signals from the brain without surgery. Such unintrusive EEG signals, captured from the central nervous system, have been utilized for exploring emotions. This paper proposes an evidence-based and personalized model for music emotion recognition. In the training phase for model construction and personalized adaption, based on the IADS (the International Affective Digitized Sound system, a set of acoustic emotional stimuli for experimental investigations of emotion and attention), we construct two predictive and generic models \(AN\!N_1\) (“EEG recordings of standardized group vs. emotions”) and \(AN\!N_2\) (“music audio content vs. emotion”). Both models are trained by an artificial neural network. We then collect a subject’s EEG recordings when listening the selected IADS samples, and apply the \(AN\!N_1\) to determine the subject’s emotion vector. With the generic model and the corresponding individual differences, we construct the personalized model H by the projective transformation. In the testing phase, given a music object, the processing steps are: (1) to extract features from the music audio content, (2) to apply \(AN\!N_2\) to calculate the vector in the arousal-valence emotion space, and (3) to apply the transformation matrix H to determine the personalized emotion vector. Moreover, with respect to a moderate music object, we apply a sliding window on the music object to obtain a sequence of personalized emotion vectors, in which those predicted vectors will be fitted and organized as an emotion trail for revealing dynamics in the affective content of music object. Experimental results suggest the proposed approach is effective.

Electroencephalography (EEG) signals are physiological signals that directly reflect the brain’s electrical activity, holding significant research value in the field of emotion recognition. However, existing methods still face certain limitations in jointly modeling temporal, spatial, and frequency-domain features. To address this, this article proposes an EEG-based emotion recognition model that integrates a three-dimensional convolutional neural network (3DCNN) with a bidirectional long short-term memory network (BiLSTM), referred to as 3DC-BiL. This approach utilizes 3DCNN to extract spatio-temporal features across different frequency bands, and then leverages BiLSTM to enhance temporal modeling capabilities, achieving collaborative fusion of multidimensional features. On the DEAP dataset for binary emotion classification, the proposed model achieves average accuracies of 98.60% and 98.66% on the arousal and valence dimensions, respectively. On the SEED dataset for three-class emotion classification, it achieves an average accuracy of 99.01%, outperforming various mainstream emotion recognition models in recent years. The experimental results demonstrate that the proposed model excels in EEG feature extraction and high-precision emotion classification, providing a new perspective and practical foundation for the research of EEG-based emotion recognition systems.

The development of computer technology, multimedia technology and communication technology has made the application of computers more and more widespread in people's lives. The next direction of computer development is to expect to be able to use computers to analyze, simulate and express human subjective thinking activities. As a part of people's lives, music has a strong emotional appeal. The stronger the musical emotion, the more audiences representing the song. Most of the computer technology related to music is still in the storage, tuning, compression, etc. of music, but with the development of computer analysis and automatic recognition technology, today's computer technology can already realize the cognition of music emotions through models and programs and creation. This article first briefly introduces computer analysis and automatic recognition technology, then analyzes and discusses the current representation model of music emotion recognition, and finally tries to design a music emotion recognition model design and proposes to introduce BP neural network into music emotion recognition technology to realize the automatic emotion analysis and recognition of the main music track.

Web music emotion recognition based on higher effective gene expression programming

ObjectiveIn the field of digital health, emotion recognition is vital for human–computer interaction and clinical neuropsychology. However, existing deep-learning models often lack interpretability, limiting their ability to provide insights into brain activity.MethodsThis study proposes an innovative brain-inspired neuronal competition model for emotion recognition from electroencephalography (EEG). First, a discrete wavelet transform is employed to extract five brain rhythms (delta, theta, alpha, beta, and gamma) as feature inputs, maximizing the preservation of signal information. Then, the proposed model simulates excitatory and inhibitory competitive dynamics between neurons, where a rhythm attention encoder dynamically focuses on key brain rhythms and uses a leaky integrate-and-fire model to simulate the neuronal decision-making process. Finally, emotion recognition is implemented based on a winner-take-all principle.ResultsTen-fold cross-validation experiments on three widely used datasets, DEAP, SEED, and DREAMER, achieve average accuracies of 86.64–92.70% on DEAP, 81.89% on SEED, and 80.38–82.70% on DREAMER, respectively. These results indicate that different brain rhythms exhibit distinct characteristics in various emotion recognition tasks.ConclusionsThis study presents a biologically plausible paradigm for EEG emotion recognition, laying the foundation for understanding the neurophysiological basis of brain activity and developing reliable tools for healthcare applications, such as mental health monitoring and depression detection.

Recognition of emotional content in music is an issue that arises recently. Music received by live applications are often exposed to noise, thus prone to reducing the recognition rate of the application. The solution proposed in this study is a robust music emotion recognition system for live applications. The proposed system consists of two major parts, i.e. subspace-based noise suppression and a hierarchical sparse representation classifier, which is based on sparse coding and a sparse representation classifier (SRC). The music is firstly enhanced by fast subspace based noise suppression. Nine classes of emotion are then used to construct a dictionary, and the vector of coefficients is obtained by sparse coding. The vector can be divided into nine parts, and each of which models a specific emotional class of a signal. Since the proposed descriptor can provide emotional content analysis of different resolutions for emotional music recognition, this work regards vectors of coefficients as feature representations. Finally, a sparse representation based classification method is employed for classification of music into four emotional classes. The experimental results confirm the highly robust performance of the proposed system in emotion recognition in live music.

Atypical and subtle facial expression patterns in individuals with autism spectrum disorder (ASD) pose a significant challenge for automated emotion recognition. This study evaluates and compares the performance of convolutional neural networks (CNNs) and transformer-based deep learning models for facial emotion recognition in this population. Using a labeled dataset of emotional facial images, we assessed eight models across four emotion categories: natural, anger, fear, and joy. Our results demonstrate that transformer models consistently outperformed CNNs in both overall and emotion-specific metrics. Notably, the Swin Transformer achieved the highest performance, with an accuracy of 0.8000 and an F1-score of 0.7889, significantly surpassing all CNN counterparts. While CNNs failed to detect the fear class, transformer models showed a measurable capability in identifying complex emotions such as anger and fear, suggesting an enhanced ability to capture subtle facial cues. Analysis of the confusion matrix further confirmed the transformers’ superior classification balance and generalization. Despite these promising results, the study has limitations, including class imbalance and its reliance solely on facial imagery. Future work should explore multimodal emotion recognition, model interpretability, and personalization for real-world applications. Research also demonstrates the potential of transformer architectures in advancing inclusive, emotion-aware AI systems tailored for autistic individuals.

A novel facial emotion recognition method for stress inference of facial nerve paralysis patients

Music is a powerful force that evokes human emotions. Several investigations of music emotion recognition (MER) have been conducted in recent years. This paper proposes a system for detecting emotion in music that is based on a deep Gaussian process (GP). The system consists of two parts-feature extraction and classification. In the feature extraction part, five types of features that are associated with emotions are selected for representing the music signal; these are rhythm, dynamics, timbre, pitch and tonality. A music clip is decomposed into frames and these features are extracted from each frame. Next, statistical values, such as mean and standard deviation, of frame-based features are calculated to generate a 38-dimensional feature vector. In the classification part, a deep GP is utilized for emotion recognition. We treat classification problem from the perspective of regression. Finally, 9 classes of emotion are categorized by 9 one-versus-all classifiers. The experimental results demonstrate that the proposed system performs well in emotion recognition.

BackgroundPatients with Alzheimer disease (AD) and amnesic mild cognitive impairment (aMCI) have deficits in emotion recognition. However, it has not yet been determined whether patients with AD and aMCI also experience difficulty in recognizing the emotions conveyed by music. This study was conducted to investigate whether musical emotion recognition is impaired or retained in patients with AD and aMCI.MethodsAll patients were recruited from the First Affiliated Hospital of Anhui Medical University between March 1, 2015 and January 31, 2017. Using the musical emotion recognition test, patients with AD (n = 16), patients with aMCI (n = 19), and healthy controls (HCs, n = 16) were required to choose one of four emotional labels (happy, sad, peaceful, and fearful) that matched each musical excerpt. Emotion recognition scores in three groups were compared using one-way analysis of variance (ANOVA) test. We also investigated the relationship between the emotion recognition scores and Mini-Mental State Examination (MMSE) using Pearson's correlation analysis test in patients with AD and aMCI.ResultsCompared to the HC group, both of the patient groups showed deficits in the recognition of fearful musical emotions (HC: 7.88 ± 1.36; aMCI: 5.05 ± 2.34; AD: 3.69 ± 2.02), with results of a one-way ANOVA confirming a significant main effect of group (F(2,50) = 18.70, P < 0.001). No significant differences were present among the three groups for the happy (F(2,50)=2.57, P = 0.09), peaceful (F(2,50) = 0.38, P = 0.09), or sad (F(2,50) = 2.50, P = 0.09) musical emotions. The recognition of fearful musical emotion was positively associated with general cognition, which was evaluated by MMSE in patients with AD and aMCI (r = 0.578, P < 0.001). The correlations between the MMSE scores and recognition of the remaining emotions were not significant (happy, r = 0.228, P = 0.11; peaceful, r = 0.047, P = 0.74; sad, r = 0.207, P = 0.15).ConclusionThis study showed that both patients with AD and aMCI had decreased ability to distinguish fearful emotions, which might be correlated with diminished cognitive function.

Continuous emotion recognition is a challenging task and a key part of human-computer interaction, especially multimodal emotion recognition can effectively improve the accuracy and robustness of recognition. But there are limited emotion data sets, and it is difficult to extract emotion features. We present a multi-level segmented decision level fusion emotion recognition model to improve the performance of emotion recognition. In this paper, we predict multi-modal dimensional emotional state on AVEC2017 dataset. Our model uses Bidirectional Long Short-Term Memory (BLSTM) as multi-level segmented emotional feature learning model, and uses the SVR model as fusion model of the decision layer. The BLSTM can model different forms of emotional information in time, and can also consider the impact of previous and later emotional features on current results. The SVR model can compensate for the redundant information of emotion recognition. At the same time, we also consider annotation delay and temporal pooling in our multi-modal dimensional emotion recognition model. Our multi-modal emotion recognition model achieves significant recognition improvements and provide the robustness. Finally, we compare the baseline methods which used the same dataset, and find that the CCC performance of our method is the best on arousal, which is 0.685. Our research shows that the proposed multi-layer segmentation decision level fusion emotion recognition model is conducive to improving performance.

Emotion identification plays a vital role in human interactions. For this purpose, Computer-vision methods for automatic emotion recognition is nowadays a widely studied topic. One of the most studied approaches for automatic emotion recognition is processing multi-channel Electroencephalogram signals (EEG). This paper presents a new model for emotion recognition using brain maps as input and providing emotion states in terms of arousal and valence as output. Brain maps are a spatial representation of features extracted from EEG signals. The proposed model, called Multi-Task Convolutional Neural Network (MT-CNN), is fed with stacked brain maps of four different waves of different frequency bands: alpha, beta, gamma and theta, using differential entropy and power spectra density and considering observation windows of 0.5s. This model is trained and tested on the DEAP dataset, a well-known dataset for comparison purposes. This work shows that the MT-CNN nerforms better than other methods.

Facial Emotion Recognition (FER) is crucial for applications such as human-computer interaction and mental health diagnostics. This study presents the first empirical comparison of open-source Vision-Language Models (VLMs), including Phi-3.5 Vision and CLIP, against traditional deep learning models—VGG19, ResNet-50, and EfficientNet-B0—on the challenging FER-2013 dataset, which contains 35,887 low-resolution, grayscale images across seven emotion classes. To address the mismatch between VLM training assumptions and the noisy nature of FER data, we introduce a novel pipeline that integrates GFPGAN-based image restoration with FER evaluation. Results show that traditional models, particularly EfficientNet-B0 (86.44%) and ResNet-50 (85.72%), significantly outperform VLMs like CLIP (64.07%) and Phi-3.5 Vision (51.66%), highlighting the limitations of VLMs in low-quality visual tasks. In addition to performance evaluation using precision, recall, F1-score, and accuracy, we provide a detailed computational cost analysis covering preprocessing, training, inference, and evaluation phases, offering practical insights for deployment. This work underscores the need for adapting VLMs to noisy environments and provides a reproducible benchmark for future research in emotion recognition.

The electroencephalogram (EEG) has great attraction in emotion recognition studies due to its resistance to deceptive actions of humans. This is one of the most significant advantages of brain signals in comparison to visual or speech signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that EEG recordings exhibit varying distributions for different people as well as for the same person at different time instances. This nonstationary nature of EEG limits the accuracy of it when subject independency is the priority. The aim of this study is to increase the subject-independent recognition accuracy by exploiting pretrained state-of-the-art Convolutional Neural Network (CNN) architectures. Unlike similar studies that extract spectral band power features from the EEG readings, raw EEG data is used in our study after applying windowing, pre-adjustments and normalization. Removing manual feature extraction from the training system overcomes the risk of eliminating hidden features in the raw data and helps leverage the deep neural network’s power in uncovering unknown features. To improve the classification accuracy further, a median filter is used to eliminate the false detections along a prediction interval of emotions. This method yields a mean cross-subject accuracy of 86.56% and 78.34% on the Shanghai Jiao Tong University Emotion EEG Dataset (SEED) for two and three emotion classes, respectively. It also yields a mean cross-subject accuracy of 72.81% on the Database for Emotion Analysis using Physiological Signals (DEAP) and 81.8% on the Loughborough University Multimodal Emotion Dataset (LUMED) for two emotion classes. Furthermore, the recognition model that has been trained using the SEED dataset was tested with the DEAP dataset, which yields a mean prediction accuracy of 58.1% across all subjects and emotion classes. Results show that in terms of classification accuracy, the proposed approach is superior to, or on par with, the reference subject-independent EEG emotion recognition studies identified in literature and has limited complexity due to the elimination of the need for feature extraction.