Emotion recognition from single-trial EEG based on kernel Fisher's emotion pattern and imbalanced quasiconformal kernel support vector machine.

The electroencephalogram (EEG) has great attraction in emotion recognition studies due to its resistance to deceptive actions of humans. This is one of the most significant advantages of brain signals in comparison to visual or speech signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that EEG recordings exhibit varying distributions for different people as well as for the same person at different time instances. This nonstationary nature of EEG limits the accuracy of it when subject independency is the priority. The aim of this study is to increase the subject-independent recognition accuracy by exploiting pretrained state-of-the-art Convolutional Neural Network (CNN) architectures. Unlike similar studies that extract spectral band power features from the EEG readings, raw EEG data is used in our study after applying windowing, pre-adjustments and normalization. Removing manual feature extraction from the training system overcomes the risk of eliminating hidden features in the raw data and helps leverage the deep neural network’s power in uncovering unknown features. To improve the classification accuracy further, a median filter is used to eliminate the false detections along a prediction interval of emotions. This method yields a mean cross-subject accuracy of 86.56% and 78.34% on the Shanghai Jiao Tong University Emotion EEG Dataset (SEED) for two and three emotion classes, respectively. It also yields a mean cross-subject accuracy of 72.81% on the Database for Emotion Analysis using Physiological Signals (DEAP) and 81.8% on the Loughborough University Multimodal Emotion Dataset (LUMED) for two emotion classes. Furthermore, the recognition model that has been trained using the SEED dataset was tested with the DEAP dataset, which yields a mean prediction accuracy of 58.1% across all subjects and emotion classes. Results show that in terms of classification accuracy, the proposed approach is superior to, or on par with, the reference subject-independent EEG emotion recognition studies identified in literature and has limited complexity due to the elimination of the need for feature extraction.

Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals

Emotion recognition plays a pivotal role in advancing Human-Computer Interaction (HCI) by enabling systems to understand and respond to human emotional states. Among various modalities, Speech Emotion Recognition (SER) stands out as a non-invasive, cost-effective, and temporally efficient approach for detecting emotions. This study leverages the RAVDESS dataset, which provides high-quality audio recordings for emotion classification. The proposed methodology involves preprocessing audio signals to remove noise, extracting temporal and spectral features in both time and frequency domains, and implementing machine learning models for multi-class emotion classification. The study evaluates and compares the performance of several classification models, including Random Forest (RF), Multilayer Perceptron (MLP), Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and Decision Trees (DT). Experimental results demonstrate promising accuracy levels, highlighting the potential of these machine learning techniques in SER. This research contributes to the integration of SER within Brain-Computer Interfaces (BCI) and other emotion-aware applications, paving the way for enhanced interactive systems. Keywords – Speech Emotion Recognition -Machine Learning - Random Forest -Multilayer Perceptron - Support Vector Machine - Convolutional Neural Networks - Decision Tree - RAVDESS Dataset

-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications

This paper investigates attention fatigue detection and multi-grain emotional AI classification for student mental health through an educational brain-computer interface (BCI), with a focus on integrating biomechanical principles to enhance understanding and application. Recognizing the growing importance of students’ mental health and well-being, this study introduces a domain generalization approach in transfer learning to improve cross-subject BCI model accuracy, addressing the challenges of individual variability. The proposed model utilizes only seven electrodes and achieves a 90% accuracy rate in differentiating between two cognitive-behavioral tasks. A truncated weighting algorithm is employed to optimize electrode combinations, enabling effective generalization across subjects. To tackle the practical challenges of emotion recognition in educational settings, the study reduces data sampling points by identifying key brain regions and frequency bands associated with emotions. Machine learning algorithms, including support vector machines (SVM), Bayesian networks, and K-nearest neighbor (KNN), further enhance recognition accuracy. By integrating eye movement and electroencephalography (EEG) signals using deep canonical correlation analysis, the model achieves cumulative accuracy improvements of 15% and 12% compared to unimodal EEG and eye movement data, respectively, across 12 subjects. Incorporating biomechanical principles, the study also examines the mechanical properties of neural tissues and their influence on signal propagation. By analyzing the viscoelastic behavior of brain tissue and its impact on EEG signal transmission, the research provides insights into how mechanical stress and strain affect neural activity. This biomechanical perspective enhances the understanding of individual variability in EEG signals and contributes to the development of more robust and personalized BCI models. The integration of biomechanics with AI-driven emotion classification and attention fatigue detection offers a comprehensive approach to improving student mental health and educational outcomes. This fusion approach demonstrates superior performance in both emotion classification and attention fatigue detection, offering substantial potential for real-time interventions in student mental health and the enhancement of educational outcomes.

Speech emotion recognition (SER) has been actively studied in the recent decade and has achieved promising results. Most state-of-the-art SER methods are based on a classification approach that ultimately outputs the softmax probability over different emotion classes. On the other hand, we have recently introduced an anomalous sound detection approach to improve the SER performance of the neutral class. It uses a neutral speech detector consisting of an autoencoder that reconstructs acoustic and text features in latent space and is trained using only neutral speech data. The experimental result confirmed that the reconstruction error could be successfully used as a cue to decide whether or not the class is neutral and suggested that it could be applied to other emotion classes. In this paper, we propose an SER method based on the reconstruction of acoustic and text features in latent space, in which the reconstructor for different emotion classes, including the neutral class, is used. The proposed method selects the emotion class with the lowest normalized reconstruction error as the SER result. Unlike the classifier approach, one reconstructor is dedicated to each emotion class and trained using only the data of the target emotion class. Therefore, the reconstructor can be trained without being affected by imbalanced training data and also facilitates the application of data augmentation to only a specific emotion class. Our experimental result obtained using the IEMOCAP dataset showed that the proposed method improved the class-average weighted accuracy by 1.7% to 77.8% compared with the state-of - the-art SER methods.

This paper presents the various emotion classification and recognition systems which implement methods aiming at improving Human Machine Interaction. The modalities and approaches used for affect detection vary and contribute to accuracy and efficacy in detecting emotions of human beings. This paper discovers them in a comparison and descriptive manner. Various applications that use the methodologies in different contexts to address the challenges in real time are discussed. This survey also describes the databases that can be used as standard data sets in the process of emotion identification. Thus an integrated discussion of methods, databases used and applications pertaining to the emerging field of Affective Computing (AC) is done and surveyed.This paper presents the various emotion classification and recognition systems which implement methods aiming at improving Human Machine Interaction. The modalities and approaches used for affect detection vary and contribute to accuracy and efficacy in detecting emotions of human beings. This paper discovers them in a comparison and descriptive manner. Various applications that use the methodologies in different contexts to address the challenges in real time are discussed. This survey also describes the databases that can be used as standard data sets in the process of emotion identification. Thus an integrated discussion of methods, databases used and applications pertaining to the emerging field of Affective Computing (AC) is done and surveyed.

Speech emotion recognition is a critical component for achieving natural human–robot interaction. The modulation-filtered cochleagram is a feature based on auditory modulation perception, which contains multi-dimensional spectral–temporal modulation representation. In this study, we propose an emotion recognition framework that utilizes a multi-level attention network to extract high-level emotional feature representations from the modulation-filtered cochleagram. Our approach utilizes channel-level attention and spatial-level attention modules to generate emotional saliency maps of channel and spatial feature representations, capturing significant emotional channel and feature space from the 3D convolution feature maps, respectively. Furthermore, we employ a temporal-level attention module to capture significant emotional regions from the concatenated feature sequence of the emotional saliency maps. Our experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset demonstrate that the modulation-filtered cochleagram significantly improves the prediction performance of categorical emotion compared to other evaluated features. Moreover, our emotion recognition framework achieves comparable unweighted accuracy of 71% in categorical emotion recognition by comparing with several existing approaches. In summary, our study demonstrates the effectiveness of the modulation-filtered cochleagram in speech emotion recognition, and our proposed multi-level attention framework provides a promising direction for future research in this field.

When we are speaking with someone, the first thing we will notice about them is their face. We can easily guess their emotional state by looking at their face, such as which feeling they are experiencing. When we're talking indirectly, however, we can't make the same prediction. As a result, a human may predict the feelings of another human being to some extent. However, a machine such as a computer is incapable of doing so. When we handed up The emotional state of a human being cannot be predicted using an audio clip as input to a computer. As a result, we devised a technique dubbed "Speech Emotion Recognition"[SER] to address this issue. Different strategies can be used to address this voice emotion recognition challenge, but the accuracy is what matters. So we solved this difficulty by comparing the accuracies of two technologies. Here, The most efficient technique for voice emotion detection is the convolution neural network algorithm, which uses several modules for emotion recognition and classifiers to discriminate emotions like happy, surprise, anger, neutral mood, sadness, and so on. [IN-DEE LEARNING] Emotions play a vital role in everyone's lives. Everyone wants to know what other people are up to. As if we can say what the other person thinks when we're talking directly. I'm referring about the facial expressions as well as the tone of voice. However, when it comes to phone calls, we don't know and can't forecast what the person is thinking as an emotion, thus to fix this, we've turned to a new trend known as "Speech Emotion Recognition"[SER]. So, in order to tackle this problem, we used the Machine Learning Language, which includes an efficient algorithm known as SVM, as well as the database Ravdess[Machine Learning].

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

The acoustic cues that convey emotion in speech are similar to those that convey emotion in music, and recognition of emotion in both of these types of cue recruits overlapping networks in the brain. Given the similarities between music and speech prosody, developmental research is uniquely positioned to determine whether recognition of these cues develops in parallel. In the present study, we asked 60 children aged 6 to 11 years, and 51 university students, to judge the emotions of 10 musical excerpts, 10 inflected speech clips, and 10 affect burst clips. We presented stimuli intended to convey happiness, sadness, anger, fear, and pride. Each emotion was presented twice per type of stimulus. We found that recognition of emotions in music and speech developed in parallel, and adult-levels of recognition develop later for these stimuli than for affect bursts. We also found that sad stimuli were most easily recognised, followed by happiness, fear, and then anger. In addition, we found that recognition of emotion in speech and affect bursts can predict emotion recognition in music stimuli independently of age and musical training. Finally, although proud speech and affect bursts were not well recognised, children aged eight years and older showed adult-like responses in recognition of proud music.

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

Brain-Computer Interface (BCI) based emotion recognition is mostly motivated by the supervised technique and may get biased by the emotion category representation. An alternative approach to emotion recognition could be employing an unsupervised technique. We compared a supervised and an unsupervised technique for two-class emotion recognition using time-frequency-based EEG features. Results obtained from the unsupervised technique reveal incongruency between the actual physiological response and the established ground truth (emotion category), whereas an average accuracy of 62.76% is observed for different emotion categories while employing the supervised technique.

Speech emotion recognition is a trending research topic these days, with its main motive to improve the human-machine interaction. At present, most of the work in this area utilizes extraction of discriminatory features for the purpose of classification of emotions into various categories. Most of the present work involves use of only Mel Frequency Cepstral Coefficients (MFCCs) as an integral feature for emotion recognition. In some other works, the utterance of words is used for lexical analysis for emotion recognition, which is language dependent. In this paper, two different techniques are utilized for classifying emotions into Angry, Happy or Neutral categories. In the first technique, the maximum cross correlation between audio files is computed for labeling the speech data into one of the three emotion categories. Accordingly, a function is developed in MATLAB for Identification of an emotion for any audio file passed as an argument. The second technique makes use of six discriminatory features, namely, Energy, Volume, MFCC, Zero Crossing Rate, Formants and Spectral Centroid. These features are used as predictors for the purpose of classification of emotions. A variety of classifiers are used through the MATLAB classification learner toolbox, and an accuracy of 91.3% is achieved using the Cubic SVM Classifier. The proposed techniques pave way for a real-time prototype for speech emotion recognition in the near future.

Speech represents a fundamental medium for conveying human emotions and, as a result, speech-based emotion recognition (SER) systems have become pivotal in advancing human-computer interaction (HCI) across a range of applications. While significant progress has been made in speech emotion recognition over recent years, existing solutions still face several key challenges, in that they: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(i)$</tex-math></inline-formula> rely excessively on subjectively annotated (discrete) labels during training, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(ii)$</tex-math></inline-formula> often overlook the label ambiguity of speech samples that express more than one class of emotions, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(iii)$</tex-math></inline-formula> underutilize unlabeled or ambiguous speech, for which typically a label distribution (or so-called soft labels) is available. To address these issues, we propose in this paper a novel SER model that explicitly handles ambiguous speech samples and overcomes the shortcomings outlined above. Central to our approach is a novel real-time soft-label correction strategy designed to refine the annotations assigned to ambiguous speech. The proposed model leverages both, (explicitly) labeled as well as ambiguous samples and applies the dynamic soft-label correction strategy alongside an enhanced inter-class difference loss function to iteratively optimize the label distributions during training. We theoretically demonstrate that our method is capable of approximating the true emotional distribution of speech even in the presence of label noise, suggesting that utilizing ambiguous speech samples without explicit emotion labels still contributes toward more effective emotion recognition. Furthermore, we integrate the representational power of convolutional neural networks (CNNs) with the contextual modeling capabilities of Wav2Vec 2.0 to enable a comprehensive extraction of spatio-temporal speech features. Experimental results on the IEMOCAP multi-label dataset confirm the effectiveness of our approach, achieving state-of-the-art performance with significant improvements in weighted accuracy (WA) and unweighted accuracy (UA) over competing methods.