Context-Aware Multimodal Emotion Recognition

In this work, we have proposed STLEV (Siamese Neural Network with Triplet Loss and Long Short-Term Memory for Emotion recognition in Videos) a method for emotion recognition in videos. This method is beneficial for cognitive Human-Computer interaction. Emotion recognition from video is a challenging task as only few frames of a video will contain relevant information for recognizing emotions. Also identifying the apex frame is challenging. Emotion recognition from videos involves extracting the features from the frames of a video and classifying the sequence of features. For feature extraction we have used Siamese Neural Network (SNN) which is metric based meta-learning model. SNN is trained with triplet loss. For classification, Long Shot-Term Memory (LSTM) model is used. In this work we have used spatial and temporal aspects of video. Spatial features are extracted using SNN and temporal features using LSTM. Deep learning models need large amount of data for good accuracy. We have addressed this problem by leveraging few shot learning aspects of metric meta-learning so that fewer samples are needed for training the model used for feature extraction. Experiments done using BU-4DFE dataset achieved an accuracy of 87.5% which demonstrates the efficiency of the method STLEV.

Due to the complexity of its system and the numerous advantages of its implementation, video emotion identification is a popular area of study today. There have been multiple ways implemented. In this project, emotion recognition in videos will be performed using deep learning. The model will include initialization, feature extraction, emotion categorization, and prediction. LeNet and AlexNet, two distinct neural networks, are used to extract features and classify emotions. There will be some parameter tuning to determine if it enhances the employed architecture’s performance, including optimizers and batch size. Each architecture for deep learning will generate a final forecast of four fundamental emotions: disgust, happiness, sadness, and surprise. AlexNet’s performance is enhanced by the SGD optimizer, whereas RMSprop improves LeNet’s performance. Results showed that AlexNet with SGD optimizer provides 93.00% recognition accuracy.

Emotion is a complex psychophysiological phenomenon elicited by external stimuli, exerting a profound influence on cognitive processes, decision-making, and social behavior. Emotion recognition holds broad application potential in healthcare, education, and entertainment. With virtual reality (VR) emerging as a powerful tool, it offers an immersive and controllable experimental environment. Prior studies have confirmed the feasibility and advantages of VR for emotion elicitation and recognition, and multimodal fusion has become a key strategy for enhancing recognition accuracy. However, publicly available VR multimodal emotion datasets remain limited in both scale and diversity due to the scarcity of VR content and the complexity of data collection. The shortage hampers further progress. Moreover, existing multimodal approaches still face challenges such as noise interference, large inter-individual variability, and insufficient model generalization. Achieving robust and accurate physiological signal processing and emotion modeling in VR environments thus remains an open challenge. To address the issues, we constructed a VR experimental environment and selected 10 emotion-eliciting video clips guided by the PAD(Pleasure-Arousal-Dominance) model. Thirty-eight participants (N=38) were recruited, from whom electrodermal activity, eye-tracking, and questionnaire data were collected, yielding 366 valid trials. The newly collected dataset substantially extends the publicly available VREED dataset, enriching VR-based multimodal emotion resources. Furthermore, we propose the MMTED model (Multi-Modal Temporal Emotion Detector), which incorporates baseline calibration and multimodal fusion of electrodermal and eye-tracking signals for emotion recognition. Experimental results demonstrate the strong performance of the MMTED model, achieving accuracies of 85.52% on the public VREED dataset, 89.27% on our self-collected dataset, and 85.29% on their combination.

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

Patients with amygdala lesions were found to be impaired in recognizing the fear emotion both from face and from music. In patients with Parkinson's disease (PD), impairment in recognition of emotions from facial expressions was reported for disgust, fear, sadness and anger, but no studies had yet investigated this population for the recognition of emotions from both face and music. The ability to recognize basic universal emotions (fear, happiness and sadness) from both face and music was investigated in 24 medicated patients with PD and 24 healthy controls. The patient group was tested for language (verbal fluency tasks), memory (digit and spatial span), executive functions (Similarities and Picture Completion subtests of the WAIS III, Brixton and Stroop tests), visual attention (Bells test), and fulfilled self-assessment tests for anxiety and depression. Results showed that the PD group was significantly impaired for recognition of both fear and sadness emotions from facial expressions, whereas their performance in recognition of emotions from musical excerpts was not different from that of the control group. The scores of fear and sadness recognition from faces were neither correlated to scores in tests for executive and cognitive functions, nor to scores in self-assessment scales. We attributed the observed dissociation to the modality (visual vs. auditory) of presentation and to the ecological value of the musical stimuli that we used. We discuss the relevance of our findings for the care of patients with PD.

Multimodal sentiment analysis and emotion recognition represent a major research direction in natural language processing (NLP). With the rapid development of online media, people often express their emotions on a topic in the form of video, and the signals it transmits are multimodal, including language, visual, and audio. Therefore, the traditional unimodal sentiment analysis method is no longer applicable, which requires the establishment of a fusion model of multimodal information to obtain sentiment understanding. In previous studies, scholars used the feature vector cascade method when fusing multimodal data at each time step in the middle layer. This method puts each modal information in the same position and does not distinguish between strong modal information and weak modal information among multiple modalities. At the same time, this method does not pay attention to the embedding characteristics of multimodal signals across the time dimension. In response to the above problems, this paper proposes a new method and model for processing multimodal signals, which takes into account the delay and hysteresis characteristics of multimodal signals across the time dimension. The purpose is to obtain a multimodal fusion feature emotion analysis representation. We evaluate our method on the multimodal sentiment analysis benchmark dataset CMU Multimodal Opinion Sentiment and Emotion Intensity Corpus (CMU-MOSEI). We compare our proposed method with the state-of-the-art model and show excellent results.

A survey of emotion recognition methods with emphasis on E-Learning environments

Recognizing the emotions of the elderly is key to achieving personalized services in smart home environments. Traditional methods have difficulty capturi ng the correlation and temporal information of multimodal features. To this end, the study proposes a multimodal emotion recognition model that integrates the EfficientFace and Transforme r structures to construct an improved attention mechanism. A modal in teraction compensation term is introduced into the similarity calculation to improve the ability to model dynamic dependencies between modalities. Meanwhile, the dynamic importance factor is used to adaptively adjust feature weights. The model was tested o n IEMOCAP and self constructed EMED datasets. The emotion recognition precision reached up to 94.37%, the recall rate reached 93.64%, the F1 value was 94.21, and the specificity reached 9 4.85%. Additionally, the model achieved 96.24% classification accurac y and 94.13% emotional intensity on easily confused categories, such as "disgust" and "contempt," with a minimum detection latency of 0.55 seconds. The results show that the model exhibit s excellent performance in multimodal fusion and emotion recognition for the elderly, and is suitable for the task of smart home emotion monitoring.

Video emotion recognition aims to infer human emotional states from the audio, visual, and text modalities. Previous approaches are centered around designing sophisticated fusion mechanisms, but usually ignore the fact that text contains global semantic information, while speech and face video show more fine-grained temporal dynamics of emotion. From the perspective of cognitive sciences, the process of emotion expression, either through facial expression or speech, is implicitly regulated by high-level semantics. Inspired by this fact, we propose a multimodal interaction enhanced representation learning framework for emotion recognition from face video, where a semantic enhancement module is first designed to guide the audio/visual encoder using the semantic information from text, then the multimodal bottleneck Transformer is adopted to further reinforce the audio and visual representations by modeling the cross-modal dynamic interactions between the two feature sequences. Experimental results on two benchmark emotion databases indicate the superiority of our proposed method. With the semantic enhanced audio and visual features, it outperforms the state-of-the-art models which fuse the features or decisions from the audio, visual and text modalities.

Recent advances in Semantic IoT data integration have highlighted the importance of multimodal fusion in emotion recognition systems. Human emotions, formed through innate learning and communication, are often revealed through speech and facial expressions. In response, this study proposes a hidden Markov model‐based multimodal fusion emotion detection system, combining speech recognition with facial expressions to enhance emotion recognition rates. The integration of such emotion recognition systems with Semantic IoT data can offer unprecedented insights into human behavior and sentiment analysis, contributing to the advancement of data integration techniques in the context of the Internet of Things. Experimental findings indicate that in single‐modal emotion detection, speech recognition achieves a 76% accuracy rate, while facial expression recognition achieves 78%. However, when state information fusion is applied, the recognition rate increases to 95%, surpassing the national average by 19% and 17% for speech and facial expressions, respectively. This demonstrates the effectiveness of multimodal fusion in emotion recognition, leading to higher recognition rates and reduced workload compared to single‐modal approaches.

IntroductionRecent developments in the field of social cognition have led to a renewed interest in basic and social emotion recognition in early stages of Alzheimer's Disease (AD) and FrontoTemporal Dementia (FTD). Despite the growing attention to this issue, only few studies have attempted to investigate emotion recognition using both visual and vocal stimuli. In addition, recent studies have presented conflicting findings regarding the extent of impairment in patients in the early stages of these diseases. The present study aims to investigate emotion understanding (both basic and social emotions), using different tasks with visual and auditory stimuli, to identify supramodal deficits in AD and FTD to provide a reliable tool to better outline their behavioral and emotional profile and useful instruments for their management.MethodsEighteen patients with AD and 15 patients with FTD were included in the study. Healthy control (HCs) subjects were recruited to obtain normative data for basic emotion recognition tests and social emotion recognition tasks. To evaluate basic emotion recognition, the Facial Emotion Recognition Battery (FERB) and the Emotional Prosody Recognition Battery (EPRB) were administered. To evaluate social emotion recognition, the Faux Pas (FP), Reading the Mind in the Eyes (RME), and Reading the Mind in the Voice (RMV) tests were employed.ResultsFTD patients performed significantly worse than HCs in most of the subtests of the basic emotion recognition batteries, where, instead, AD patients were significantly impaired only when required to match emotional facial expression in different individuals (subtask of the FERB). Moreover, FTD patients scored significantly lower in RME and RMV tests compared both to AD patients and to HCs. In addition, ADs were selectively impaired in RMV as respect to HCs.DiscussionFTD patients showed deficits in emotion recognition, affecting both basic and social emotions, whether conveyed through facial expressions or prosody. This result may explain the well-known social behavioral difficulties observed in FTD patients from the early stages of the disease. The fewer and specific deficits in AD patients with comparable MMSE scores may be attributed to the mild degree of impairment, as these deficits may appear later in the progression of AD.

Human emotion recognition has emerged as a critical component in the development of intelligent systems capable of understanding and responding to human emotional states. This comprehensive review examines the latest novel deep learning methods for emotion recognition, encompassing advances in neural architectures, multimodal fusion techniques, and innovative approaches to feature extraction and classification. The field has witnessed remarkable progress through the integration of sophisticated deep learning models including convolutional neural networks (CNNs), long short-term memory networks (LSTMs), transformer architectures, and generative adversarial networks (GANs). Recent breakthroughs have demonstrated exceptional performance improvements, with some methods achieving over 99% accuracy on benchmark datasets such as DEAP, SEED, and MAHNOB-HCI. This review analyzes the evolution from traditional machine learning approaches to state-of-the-art deep learning methodologies, examining their effectiveness across various modalities including electroencephalography (EEG), facial expressions, speech, and physiological signals.The paper provides a systematic analysis of fusion strategies, architectural innovations, and performance metrics while identifying key challenges and future research directions in the rapidly evolving landscape of emotion recognition technology.

In this paper a study on multimodal automatic emotion recognition during a speech-based interaction is presented. A database was constructed consisting of people pronouncing a sentence in a scenario where they interacted with an agent using speech. Ten people pronounced a sentence corresponding to a command while making 8 different emotional expressions. Gender was equally represented, with speakers of several different native languages including French, German, Greek and Italian. Facial expression, gesture and acoustic analysis of speech were used to extract features relevant to emotion. For the automatic classification of unimodal data, bimodal data and multimodal data, a system based on a Bayesian classifier was used. After performing an automatic classification of each modality, the different modalities were combined using a multimodal approach. Fusion of the modalities at the feature level (before running the classifier) and at the results level (combining results from classifier from each modality) were compared. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison to the unimodal systems: the multimodal approach increased the recognition rate by more than 10% when compared to the most successful unimodal system. Bimodal emotion recognition based on all combinations of the modalities (i.e., ‘face-gesture’, ‘face-speech’ and ‘gesture-speech’) was also investigated. The results show that the best pairing is ‘gesture-speech’. Using all three modalities resulted in a 3.3% classification improvement over the best bimodal results.

This paper proposes a database for group level emotion recognition in videos. The motivation is coming from the large number of information which the users are sharing online. This gives us the opportunity to use this perceived affect for various tasks. Most of the work in this area has been restricted to controlled environments. In this paper, we explore the group level emotion and cohesion in a real-world environment. There are several challenges involved in moving from a controlled environment to real-world scenarios such as face tracking limitations, illumination variations, occlusion and type of gatherings. As an attempt to address these challenges, we propose a ‘Video level Group AFfect (VGAF), database containing 1,004 videos downloaded from the web. The collected videos have a large variations in terms of gender, ethnicity, the type of social event, number of people, pose, etc. We have labelled our database for group level emotion and cohesion tasks and proposed a baseline based on the Inception V3 network on the database.

Automatic highlights extraction for drama video using music emotion and human face features