Hybrid-Supervised Hypergraph-Enhanced Transformer for Micro-Gesture Based Emotion Recognition

Individuals with cochlear implants (CIs) show reduced word and auditory emotion recognition abilities relative to their peers with normal hearing. Modern CI processing strategies are designed to preserve acoustic cues requisite for word recognition rather than those cues required for accessing other signal information (e.g., talker gender or emotional state). While word recognition is undoubtedly important for communication, the inaccessibility of this additional signal information in speech may lead to negative social experiences and outcomes for individuals with hearing loss. This study aimed to evaluate whether the emphasis on word recognition preservation in CI processing has unintended consequences on the perception of other talker information, such as emotional state. Twenty-four young adult listeners with normal hearing listened to sentences and either reported a target word in each sentence (word recognition task) or selected the emotion of the talker (emotion recognition task) from a list of options (Angry, Calm, Happy, and Sad). Sentences were blocked by task type (emotion recognition versus word recognition) and processing condition (unprocessed versus 8-channel noise vocoder) and presented randomly within the block at three signal-to-noise ratios (SNRs) in a background of speech-shaped noise. Confusion matrices showed the number of errors in emotion recognition by listeners. Listeners demonstrated better emotion recognition performance than word recognition performance at the same SNR. Unprocessed speech resulted in higher recognition rates than vocoded stimuli. Recognition performance (for both words and emotions) decreased with worsening SNR. Vocoding speech resulted in a greater negative impact on emotion recognition than it did for word recognition. These data confirm prior work that suggests that in background noise, emotional prosodic information in speech is easier to recognize than word information, even after simulated CI processing. However, emotion recognition may be more negatively impacted by background noise and CI processing than word recognition. Future work could explore CI processing strategies that better encode prosodic information and investigate this effect in individuals with CIs as opposed to vocoded simulation. This study emphasized the need for clinicians to consider not only word recognition but also other aspects of speech that are critical to successful social communication.

Emotion recognition is considered as one of the most vital and challenging studies in the domain of computer vision. Due to the ongoing pandemic the people’s emotional and mental state is getting affected due to less physical interaction and increased virtual interaction. The motivation of the work lies in the recognition and monitoring of the emotional state of the human beings in live environment with higher accuracy rate and fast recognition time so as to keep them aware about their emotional state in the pandemic situation. Emotion recognition is achieved using the proposed deep convolution neural network (DCNN) using custom Gabor filter with 85.8% accuracy. Emotion recognition is also being investigated and compared with other deep learning models such as AlexNet, VGG-Net for testing the efficacy. When the emotion of the person will be found sad or angry then an alert sound is played and an email alert is automatically sent.KeywordsEmotion recognitionDeep learningGabor filterPandemicReal-time alertDCNNPreprocessing

The speech signal consists of linguistic information and also paralinguistic one such as emotion. The modern automatic speech recognition systems have achieved high performance in neutral style speech recognition, but they cannot maintain their high recognition rate for spontaneous speech. So, emotion recognition is an important step toward emotional speech recognition. The accuracy of an emotion recognition system is dependent on different factors such as the type and number of emotional states and selected features, and also the type of classifier. In this paper, a modular neural-support vector machine (SVM) classifier is proposed, and its performance in emotion recognition is compared to Gaussian mixture model, multi-layer perceptron neural network, and C5.0-based classifiers. The most efficient features are also selected by using the analysis of variations method. It is noted that the proposed modular scheme is achieved through a comparative study of different features and characteristics of an individual emotional state with the aim of improving the recognition performance. Empirical results show that even by discarding 22% of features, the average emotion recognition accuracy can be improved by 2.2%. Also, the proposed modular neural-SVM classifier improves the recognition accuracy at least by 8% as compared to the simulated monolithic classifiers.

Emotional expressions serve a crucial role in interpersonal communication between people while improving social life. In particular, information security systems for visual surveillance that aim to recognize human emotional facial states are highly relevant today. Facial expressions are among the most effective and straightforward means of nonverbal interaction in systems with a human-machine interface. Despite significant scientific and engineering advances in emotion recognition, there are still several challenges in improving the performance of real-time human-machine systems that might work more effectively. In this work, a method of facial geometric feature representation is proposed to improve the operation of security systems. The method is designed to automatically reflect the facial expressions of human emotions in the form of quantitative characteristics of geometric shapes. It uses software-generated landmarks for constructing specific geometric characteristics of the face, which serve as input for the method. Our method consists in forming seven geometric shapes based on predefined landmarks, with the subsequent quantitative expression of these shapes. It was established within the method that the movement of the landmarks when changing facial expressions directly changed the value of each geometric shape. The method outputs the quantitative features of seven shapes, later used to classify emotional facial states. Finally, our method was validated using hyperplane classification. The results of computational experiments confirmed the effectiveness of the proposed method for identifying changes in a person’s emotional state by facial expressions. In addition, the use of simple mathematical calculations in our method has significantly reduced the computational complexity against analogs.

Emotional design is an important development trend of interaction design. Emotional design in products plays a key role in enhancing user experience and inducing user emotional resonance. In recent years, based on the user's emotional experience, the design concept of strengthening product emotional design has become a new direction for most designers to improve their design thinking. In the emotional interaction design, the machine needs to capture the user's key information in real time, recognize the user's emotional state, and use a variety of clues to finally determine the appropriate user model. Based on this background, this research uses a deep learning mechanism for more accurate and effective emotion recognition, thereby optimizing the design of the interactive system and improving the user experience. First of all, this research discusses how to use user characteristics such as speech, facial expression, video, heartbeat, etc., to make machines more accurately recognize human emotions. Through the analysis of various characteristics, the speech is selected as the experimental material. Second, a speech-based emotion recognition method is proposed. The mel-Frequency cepstral coefficient (MFCC) of the speech signal is used as the input of the improved long and short-term memory network (ILSTM). To ensure the integrity of the information and the accuracy of the output at the next moment, ILSTM makes peephole connections in the forget gate and input gate of LSTM, and adds the unit state as input data to the threshold layer. The emotional features obtained by ILSTM are input into the attention layer, and the self-attention mechanism is used to calculate the weight of each frame of speech signal. The speech features with higher weights are used to distinguish different emotions and complete the emotion recognition of the speech signal. Experiments on the EMO-DB and CASIA datasets verify the effectiveness of the model for emotion recognition. Finally, the feasibility of emotional interaction system design is discussed.

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

To date, most of the human emotion recognition systems are intended to sense the emotions and their dominance individually. This paper discusses a fuzzy model for multilevel affective computing based on the dominance dimensional model of emotions. This model can detect any other possible emotions simultaneously at the time of recognition. One hundred and thirty volunteers from various countries with different cultural backgrounds were selected to record their emotional states. These volunteers have been selected from various races and different geographical locations. Twenty-seven different emotions with their strengths in a scale of 5 were questioned through a survey. Recorded emotions were analyzed with the other possible emotions and their levels of dominance to build the fuzzy model. Then this model was integrated into a fuzzy emotion recognition system using three input devices of mouse, keyboard and the touch screen display. Support vector machine classifier detected the other possible emotions of the users along with the directly sensed emotion. The binary system (non-fuzzy) sensed emotions with an incredible accuracy of 93 %. However, it only could sense limited emotions. By integrating this model, the system was able to detect more possible emotions at a time with slightly lower recognition accuracy of 86 %. The recorded false positive rates of this model for four emotions were measured at 16.7 %. The resulted accuracy and its false positive rate are among the top three accurate human emotion recognition (affective computing) systems.

Objective: This study compared emotional recognition in participants with mild cognitive impairment (MCI) and mild to moderate Alzheimer 's disease (AD) against caregivers' perceptions of these participants' emotional states, while exploring the influence of clinical variables. Methods: We included 141 participants (32 with MCI, 50 with mild AD, and 59 with moderate AD) and their primary caregivers. We employed tasks assessing emotional decoding, identification, and correspondence, along with objective evaluations. Results: Participants across all groups showed significant differences in cognition and functionality. However, emotional recognition abilities did not significantly differ between MCI and mild or moderate AD groups. Most cognitive and neuropsychiatric variables had no significant impact on emotion recognition or social functioning. No differences emerged in patients' self-evaluations of social and emotional functioning. Caregiver assessments revealed significant differences only between the MCI and moderate AD groups. Conclusion: Participants with MCI and AD displayed expected clinical progression while retaining some emotional recognition and social functioning capabilities.

Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

Poincaré's section analysis for PPG-based automatic emotion recognition

Faces are fundamental stimuli for social interactions since they provide significant information about people’s identity and emotional states. With the outburst of the COVID-19 pandemic, global use of preventive measures, such as disposable surgical face masks (DSFMs), has been imposed. The massive use of DSFMs covering a large part of the face could interfere with identity and emotion recognition. Thus, the main aim of the current study was (i) to assess how DSFMs affect identity recognition (Experiment 1), (ii) how DSFMs affect emotion recognition (Experiment 2), and (iii) whether individual empathy levels correlate with emotion recognition with DSFMs. The potential relation between identity and emotion recognition with and without DSFMs was also investigated. Two tasks were administered to 101 healthy participants: (i) the Old-new face memory task aimed to assess whether the learning context (i.e., DSFMs on/off) affects recognition performance, whereas (ii) the Facial affect task explored DSFMs’ effect on emotion recognition. Results from the former showed that the stimuli’s features in the learning stage affect recognition performances; that is, faces wearing DSFMs were better recognized if wearing DSFMs at first exposure and vice versa. Results from the Facial affect task showed that DSFMs lead to reduced disgust, happiness, and sadness recognition. No significant correlation emerged between identity and emotion recognition. The Interpersonal Reactivity Index (IRI) was administered to assess affective and cognitive empathy; however, IRI scores did not correlate with either face memory recognition or facial affect recognition. Overall, our results demonstrate (a) a “context effect” for face memory with and without DSFMs; (b) a disruptive effect of DSFMs depending on the expressed emotion; and (c) no correlation between empathy and emotion recognition with DSFMs.

This work presents an approach to emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information (AP) and semantic labels (SLs). For AP-based recognition, acoustic and prosodic features including spectrum, formant, and pitch-related features are extracted from the detected emotional salient segments of the input speech. Three types of models, GMMs, SVMs, and MLPs, are adopted as the base-level classifiers. A Meta Decision Tree (MDT) is then employed for classifier fusion to obtain the AP-based emotion recognition confidence. For SL-based recognition, semantic labels derived from an existing Chinese knowledge base called HowNet are used to automatically extract Emotion Association Rules (EARs) from the recognized word sequence of the affective speech. The maximum entropy model (MaxEnt) is thereafter utilized to characterize the relationship between emotional states and EARs for emotion recognition. Finally, a weighted product fusion method is used to integrate the AP-based and SL-based recognition results for the final emotion decision. For evaluation, 2,033 utterances for four emotional states (Neutral, Happy, Angry, and Sad) are collected. The speaker-independent experimental results reveal that the emotion recognition performance based on MDT can achieve 80.00 percent, which is better than each individual classifier. On the other hand, an average recognition accuracy of 80.92 percent can be obtained for SL-based recognition. Finally, combining acoustic-prosodic information and semantic labels can achieve 83.55 percent, which is superior to either AP-based or SL-Based approaches. Moreover, considering the individual personality trait for personalized application, the recognition accuracy of the proposed approach can be further improved to 85.79 percent.

The aetiology of somatoform disorders is assumed to consist of several factors, as alexithymia, dysfunctional cognitions and emotion regulation. As a consequence of somatoform disorders social withdrawal is discussed. Several studies prove connections between mental problems, emotion recognition, and functioning social relationships. A sample of 35 patients suffering from a somatoform disorder and a sample of 73 persons without any somatoform symptoms were investigated according to their ability to identify emotional facial expressions (computer-assisted presentation of photos). Our results demonstrate significant differences in the ability to decode others' emotional states between people with or without somatoform disorders. In all six tests the patients achieved significantly worse results. Our results suggest that psychotherapeutic treatment of patients with somatoform disorders should also focus on the recognition of emotions and deal with them.

Automatic emotion recognition has become an important trend in the fields of human–computer natural interaction and artificial intelligence. Although gesture is one of the most important components of nonverbal communication, which has a considerable impact on emotion recognition, it is rarely considered in the study of emotion recognition. An important reason is the lack of large open-source emotional databases containing skeletal movement data. In this paper, we extract three-dimensional skeleton information from videos and apply the method to IEMOCAP database to add a new modality. We propose an attention-based convolutional neural network which takes the extracted data as input to predict the speakers’ emotional state. We also propose a graph attention-based fusion method that combines our model with the models using other modalities, to provide complementary information in the emotion classification task and effectively fuse multimodal cues. The combined model utilizes audio signals, text information, and skeletal data. The performance of the model significantly outperforms the bimodal model and other fusion strategies, proving the effectiveness of the method.

AI-based emotion recognition has emerged as a critical component of affect-aware educational technologies, particularly in online, large-scale, and technology-mediated learning environments where direct observation of learners’ emotional states is limited. By leveraging facial expressions, speech, physiological signals, and interaction data, emotion-aware systems aim to support adaptive instruction, learner engagement, and instructional decision-making. Despite rapid technical progress, the integration of emotion recognition into authentic educational practice remains constrained by dataset limitations, cultural bias, weak pedagogical alignment, and unresolved ethical concerns. This article presents a critical and theory-informed review of recent advances in AI-based emotion recognition in education, with particular emphasis on deep learning architectures and multimodal fusion approaches, while highlighting persistent challenges and future research directions.Purpose:This study aims to critically examine the current state of AI-based emotion recognition in educational contexts by analyzing dominant modeling paradigms, sensing modalities, application domains, and ethical implications, in order to identify key gaps between technical development and pedagogical practice.Methods/Study design/approach:The study adopts a narrative literature review approach grounded in educational and affective computing theories. Representative and influential studies were analyzed across facial, speech, physiological, and multimodal emotion recognition systems, with emphasis on deep learning, attention-based, and transformer architectures. The review synthesizes findings related to system performance, educational applications, dataset characteristics, and ethical considerations.Result/Findings:The findings indicate that multimodal emotion recognition systems generally outperform unimodal approaches in detecting learning-relevant affective states such as engagement, frustration, and confusion. Advances in deep learning, particularly attention-based and transformer models, improve temporal modeling and robustness. However, most systems rely on non-educational datasets, demonstrate limited cultural generalizability, and prioritize recognition accuracy over pedagogical interpretability and long-term educational impact. Ethical challenges related to privacy, consent, bias, and emotional surveillance remain insufficiently addressed.Novelty/Originality/Value:This study provides an integrative and pedagogically grounded synthesis that bridges affective computing research with educational theory and governance perspectives. By reframing AI-based emotion recognition as a decision-support mechanism rather than an automated control system, the article offers a conceptual foundation for responsible, culturally inclusive, and ethically aligned deployment of emotion-aware AI in education, serving as a reference for future interdisciplinary research and policy development.