Zero-Shot Learning for Multi-Class Emotion Recognition from EEG Signals Using Transformer-Based Models

Accurate recognition and understating of human emotions is an essential skill that can improve the collaboration between humans and machines. In this vein, electroencephalogram (EEG)-based emotion recognition is considered an active research field with challenging issues regarding the analyses of the nonstationary EEG signals and the extraction of salient features that can be used to achieve accurate emotion recognition. In this paper, an EEG-based emotion recognition approach with a novel time-frequency feature extraction technique is presented. In particular, a quadratic time-frequency distribution (QTFD) is employed to construct a high resolution time-frequency representation of the EEG signals and capture the spectral variations of the EEG signals over time. To reduce the dimensionality of the constructed QTFD-based representation, a set of 13 time- and frequency-domain features is extended to the joint time-frequency-domain and employed to quantify the QTFD-based time-frequency representation of the EEG signals. Moreover, to describe different emotion classes, we have utilized the 2D arousal-valence plane to develop four emotion labeling schemes of the EEG signals, such that each emotion labeling scheme defines a set of emotion classes. The extracted time-frequency features are used to construct a set of subject-specific support vector machine classifiers to classify the EEG signals of each subject into the different emotion classes that are defined using each of the four emotion labeling schemes. The performance of the proposed approach is evaluated using a publicly available EEG dataset, namely the DEAPdataset. Moreover, we design three performance evaluation analyses, namely the channel-based analysis, feature-based analysis and neutral class exclusion analysis, to quantify the effects of utilizing different groups of EEG channels that cover various regions in the brain, reducing the dimensionality of the extracted time-frequency features and excluding the EEG signals that correspond to the neutral class, on the capability of the proposed approach to discriminate between different emotion classes. The results reported in the current study demonstrate the efficacy of the proposed QTFD-based approach in recognizing different emotion classes. In particular, the average classification accuracies obtained in differentiating between the various emotion classes defined using each of the four emotion labeling schemes are within the range of –. Moreover, the emotion classification accuracies achieved by our proposed approach are higher than the results reported in several existing state-of-the-art EEG-based emotion recognition studies.

The speech signal consists of linguistic information and also paralinguistic one such as emotion. The modern automatic speech recognition systems have achieved high performance in neutral style speech recognition, but they cannot maintain their high recognition rate for spontaneous speech. So, emotion recognition is an important step toward emotional speech recognition. The accuracy of an emotion recognition system is dependent on different factors such as the type and number of emotional states and selected features, and also the type of classifier. In this paper, a modular neural-support vector machine (SVM) classifier is proposed, and its performance in emotion recognition is compared to Gaussian mixture model, multi-layer perceptron neural network, and C5.0-based classifiers. The most efficient features are also selected by using the analysis of variations method. It is noted that the proposed modular scheme is achieved through a comparative study of different features and characteristics of an individual emotional state with the aim of improving the recognition performance. Empirical results show that even by discarding 22% of features, the average emotion recognition accuracy can be improved by 2.2%. Also, the proposed modular neural-SVM classifier improves the recognition accuracy at least by 8% as compared to the simulated monolithic classifiers.

Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method

EEG-based emotion recognition: A deep learning approach to brain region analysis

Purpose: Listening to music has a great impact on people's emotions and would change brain activity. In other words, music-induced emotions are trackable in electrical brain activities. Therefore, Electroencephalography can be a suitable tool to detect these induced emotions. The present study attempted to use electroencephalography in to recognize four types of emotions (happy, relaxing, stressful, and sad) induced in response to listening to music excerpts, using three classifiers.
 Materials and Methods: In this empirical study, electroencephalography signals were collected from 20 participants, as they were listening to pieces of selected music. The collected data were then pre-processed, and 28 linear and nonlinear features for recognizing the aforementioned emotions were extracted. Feature-space components were then reduced through a principal components analysis. Finally, the first ten components of feature-space were used as input for three classifiers based on Neural Network (NN), K-Nearest Neighbors (KNN), and Support Vector Machine (SVM) algorithms to identify the induced emotions.
 Results: The outputs showed that the suggested method was well capable of emotion recognition. Evaluating the music excerpts, on the self-assessment manikin scale, demonstrated that the labeling of the music tracks was accurate. The highest accuracy found among NN, KNN, and SVM algorithms were %84, %84, and %89 for happy emotions, respectively.
 Conclusion: The findings of this study provide useful insights into emotion classification and brain behavior related to induced emotion extraction. Happiness was the most recognizable emotion and the support vector machine had the highest performance among the classifiers. In the end, the outcomes of the proposed method demonstrate that this system is better than the previous research in EEG-based emotion recognition.

Emotion recognition based on EEG signals is a critical area within affective computing, with applications in mental health monitoring, human-computer interaction, and neuroadaptive systems. However, accurately classifying emotional states from inherently non-stationary and noisy EEG data remains a major challenge. This study explores the classification of three discrete emotions, Humorous, Sad, and Fearful, elicited through video stimuli, using EEG recordings from six participants acquired via a 19-channel Mitsar amplifier at a 500 Hz sampling rate. Preprocessing steps included bandpass filtering (1–40 Hz), epoch segmentation, and multi-domain feature extraction encompassing statistical measures, spectral features, differential entropy, Hjorth parameters, and hemispheric asymmetry indicators. Data augmentation was applied to balance class distributions, particularly for the underrepresented fear category. The resulting features were normalized and structured to support temporal deep learning and classical machine learning models. The classification performance of Long Short-Term Memory (LSTM) networks was evaluated alongside Support Vector Machine (SVM), k-Nearest Neighbors (k-NN), and Random Forest (RF) classifiers. While LSTM demonstrated competency in capturing temporal dependencies, especially in fear recognition, SVM achieved the highest overall accuracy, 94.12%, outperforming LSTM at 85.16%, RF at 90.00%, and k-NN at 78.01%. These results suggest that when robust and discriminative features are employed, traditional models like SVM can surpass deep learning methods, particularly in small-scale EEG datasets with limited temporal complexity. This study underscores the importance of aligning model architecture with feature representation and contributes a comparative evaluation framework for EEG-based emotion recognition systems.

An approach to EEG-based emotion recognition using combined feature extraction method

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

It is well known that there may be significant individual differences in physiological signal patterns for emotional responses. Emotion recognition based on electroencephalogram (EEG) signals is still a challenging task in the context of developing an individual-independent recognition method. In our paper, from the perspective of spatial topology and temporal information of brain emotional patterns in an EEG, we exploit complex networks to characterize EEG signals to effectively extract EEG information for emotion recognition. First, we exploit visibility graphs to construct complex networks from EEG signals. Then, two kinds of network entropy measures (nodal degree entropy and clustering coefficient entropy) are calculated. By applying the AUC method, the effective features are input into the SVM classifier to perform emotion recognition across subjects. The experiment results showed that, for the EEG signals of 62 channels, the features of 18 channels selected by AUC were significant (p < 0.005). For the classification of positive and negative emotions, the average recognition rate was 87.26%; for the classification of positive, negative, and neutral emotions, the average recognition rate was 68.44%. Our method improves mean accuracy by an average of 2.28% compared with other existing methods. Our results fully demonstrate that a more accurate recognition of emotional EEG signals can be achieved relative to the available relevant studies, indicating that our method can provide more generalizability in practical use.

Affective Computing and emotion recognition hold significant importance in healthcare, identity verification, human-computer interaction, and related fields. Accurate identification of emotion is crucial for applications in medicine, education, psychology, and military domains. Electroencephalographic (EEG) signals have gained widespread application in emotion recognition due to their inherent characteristics of being non-concealable and directly reflecting brain activity. In recent years, with the establishment of open datasets and advancements in deep learning, an increasing number of researchers have integrated EEG with deep learning methods for emotion recognition studies. This review summarizes commonly used deep learning models in EEG-based emotion recognition along with their applications in this field, including the design of different network architectures, optimization strategies, and model designs based on EEG signal features. We also discuss limitations from the perspectives of commonality-individuality (C-I) and suggest improvements. The review outlines future research directions and provided a minimal C-I framework to assess models. Through this review, we aim to provide researchers in this field with a comprehensive reference and approach to balance universality and personalization to promote the development of deep learning-based EEG emotion recognition methods.

The electroencephalogram (EEG) has great attraction in emotion recognition studies due to its resistance to deceptive actions of humans. This is one of the most significant advantages of brain signals in comparison to visual or speech signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that EEG recordings exhibit varying distributions for different people as well as for the same person at different time instances. This nonstationary nature of EEG limits the accuracy of it when subject independency is the priority. The aim of this study is to increase the subject-independent recognition accuracy by exploiting pretrained state-of-the-art Convolutional Neural Network (CNN) architectures. Unlike similar studies that extract spectral band power features from the EEG readings, raw EEG data is used in our study after applying windowing, pre-adjustments and normalization. Removing manual feature extraction from the training system overcomes the risk of eliminating hidden features in the raw data and helps leverage the deep neural network’s power in uncovering unknown features. To improve the classification accuracy further, a median filter is used to eliminate the false detections along a prediction interval of emotions. This method yields a mean cross-subject accuracy of 86.56% and 78.34% on the Shanghai Jiao Tong University Emotion EEG Dataset (SEED) for two and three emotion classes, respectively. It also yields a mean cross-subject accuracy of 72.81% on the Database for Emotion Analysis using Physiological Signals (DEAP) and 81.8% on the Loughborough University Multimodal Emotion Dataset (LUMED) for two emotion classes. Furthermore, the recognition model that has been trained using the SEED dataset was tested with the DEAP dataset, which yields a mean prediction accuracy of 58.1% across all subjects and emotion classes. Results show that in terms of classification accuracy, the proposed approach is superior to, or on par with, the reference subject-independent EEG emotion recognition studies identified in literature and has limited complexity due to the elimination of the need for feature extraction.

Individuals with cochlear implants (CIs) show reduced word and auditory emotion recognition abilities relative to their peers with normal hearing. Modern CI processing strategies are designed to preserve acoustic cues requisite for word recognition rather than those cues required for accessing other signal information (e.g., talker gender or emotional state). While word recognition is undoubtedly important for communication, the inaccessibility of this additional signal information in speech may lead to negative social experiences and outcomes for individuals with hearing loss. This study aimed to evaluate whether the emphasis on word recognition preservation in CI processing has unintended consequences on the perception of other talker information, such as emotional state. Twenty-four young adult listeners with normal hearing listened to sentences and either reported a target word in each sentence (word recognition task) or selected the emotion of the talker (emotion recognition task) from a list of options (Angry, Calm, Happy, and Sad). Sentences were blocked by task type (emotion recognition versus word recognition) and processing condition (unprocessed versus 8-channel noise vocoder) and presented randomly within the block at three signal-to-noise ratios (SNRs) in a background of speech-shaped noise. Confusion matrices showed the number of errors in emotion recognition by listeners. Listeners demonstrated better emotion recognition performance than word recognition performance at the same SNR. Unprocessed speech resulted in higher recognition rates than vocoded stimuli. Recognition performance (for both words and emotions) decreased with worsening SNR. Vocoding speech resulted in a greater negative impact on emotion recognition than it did for word recognition. These data confirm prior work that suggests that in background noise, emotional prosodic information in speech is easier to recognize than word information, even after simulated CI processing. However, emotion recognition may be more negatively impacted by background noise and CI processing than word recognition. Future work could explore CI processing strategies that better encode prosodic information and investigate this effect in individuals with CIs as opposed to vocoded simulation. This study emphasized the need for clinicians to consider not only word recognition but also other aspects of speech that are critical to successful social communication.

Electroencephalogram (EEG) signal-based emotion classification is vital in the ever-growing human-computer interface (HCI) applications. However, the chaotic, non-stationary, and person-dependent nature of EEG signals often limits such practical applications. These challenges reduce the ability of the state-of-the-art approaches to effectively distinguish between different emotional states from EEG data, resulting in sub-optimal emotion recognition performance. This work presented a time-frequency (T-F) analysis of EEG signals to localize different EEG frequency rhythms responsible for emotion-related information in the EEG signals. In particular, this work investigated two T-F analysis domains for multichannel EEG signals based on complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) and local mean decomposition (LMD) to extract entropy features that capture specific emotion-relevant traits from the specific EEG channels that are highly responsive to emotions. The CEEMDAN and LMD decompose the EEG signals into different EEG frequency rhythms called mode functions, intrinsic mode functions (IMFs), and product functions (PFs), respectively. Further, various types of entropy feature of these two categories of mode functions, such as approximate entropy (ApEn), sample entropy (SaEn), permutation entropy (PeEn), and bubble entropy (BuEn), are computed for extracting emotion-relevant distinguishing features. Entropy features help quantify EEG’s non-linear behavior and eventually help classify EEG-based emotions precisely. Emotion classification has been achieved using a grid search cross-validation (GSCV) optimized XGboost classifier. Thorough experimentations are conducted to validate the efficacy of the proposed approach on publically accessible datasets named Database for Emotion Analysis of Physiological Signals (DEAP), SJTU emotion EEG dataset (SEED), and SEED-IV. The efficacy of the proposed emotion recognition approach is measured in terms of widely used performance metrics such as accuracy, confusion metrics, receiver operating characteristics (RoC), and area under the curve (AuC). The highest average accuracy is attained using the proposed LMD-domain BuEn features, i.e., 97.8%, 98.6%, and 95.7% using SEED, SEED-IV, and DEAP databases, respectively, outperforming the recent state-of-the-art emotion recognition algorithms.

Speech emotion recognition is a trending research topic these days, with its main motive to improve the human-machine interaction. At present, most of the work in this area utilizes extraction of discriminatory features for the purpose of classification of emotions into various categories. Most of the present work involves use of only Mel Frequency Cepstral Coefficients (MFCCs) as an integral feature for emotion recognition. In some other works, the utterance of words is used for lexical analysis for emotion recognition, which is language dependent. In this paper, two different techniques are utilized for classifying emotions into Angry, Happy or Neutral categories. In the first technique, the maximum cross correlation between audio files is computed for labeling the speech data into one of the three emotion categories. Accordingly, a function is developed in MATLAB for Identification of an emotion for any audio file passed as an argument. The second technique makes use of six discriminatory features, namely, Energy, Volume, MFCC, Zero Crossing Rate, Formants and Spectral Centroid. These features are used as predictors for the purpose of classification of emotions. A variety of classifiers are used through the MATLAB classification learner toolbox, and an accuracy of 91.3% is achieved using the Cubic SVM Classifier. The proposed techniques pave way for a real-time prototype for speech emotion recognition in the near future.

Emotion recognition by artificial intelligence (AI) is a challenging task. A wide variety of research has been done, which demonstrated the utility of audio, imagery, and electroencephalography (EEG) data for automatic emotion recognition. This paper presents a new automated emotion recognition framework, which utilizes electroencephalography (EEG) signals. The proposed method is lightweight, and it consists of four major phases, which include: a reprocessing phase, a feature extraction phase, a feature dimension reduction phase, and a classification phase. A discrete wavelet transforms (DWT) based noise reduction method, which is hereby named multi scale principal component analysis (MSPCA), is utilized during the pre-processing phase, where a Symlets-4 filter is utilized for noise reduction. A tunable Q wavelet transform (TQWT) is utilized as feature extractor. Six different statistical methods are used for dimension reduction. In the classification step, rotation forest ensemble (RFE) classifier is utilized with different classification algorithms such as k-Nearest Neighbor (k-NN), support vector machine (SVM), artificial neural network (ANN), random forest (RF), and four different types of the decision tree (DT) algorithms. The proposed framework achieves over 93 % classification accuracy with RFE + SVM. The results clearly show that the proposed TQWT and RFE based emotion recognition framework is an effective approach for emotion recognition using EEG signals.