A Multi-Kernel Embedding Fusion Framework for Physiological Signal Based Emotion Recognition

As an essential challenge within the realm of affective computing, emotion recognition assumes a vital role in bestowing computers with a higher level and comprehensive intelligence. Furthermore, it has emerged as a crucial research topic in both human–computer interaction (HCI) and medical rehabilitation related to mental illnesses. However, the related fusion studies for modeling physiological signals in emotion recognition are less based on multimodal coordinated representation and lack the exploration of multimodal physiological signal correlation. In this article, we propose a tensor correlation fusion framework for emotion recognition based on multimodal physiological signals. After extracting effective features from various physiological signals, the coordinated representation module of the framework first simultaneously learns the linear correlation of all input physiological signals based on the covariance tensor. An optimized solution strategy is constructed to obtain the coordinated representation corresponding to each physiological signal. Finally, an emotion recognition module fuses the correlation information of the coordinated representation of different physiological signals as input to the emotion recognition classifier. This framework constructs coordinated representation by introducing a strategy to simultaneously capture the correlation among multiple physiological signals, providing a fresh perspective with a well-defined mathematical foundation for the fusion of multimodal physiological signals in the realm of emotion recognition. The experiments conducted on the DEAP dataset demonstrate that compared with related methods, the framework achieves relatively higher emotion recognition performance while obtaining a coordinated representation of multimodal physiological signal correlations of emotions, all while achieving superior processing speed.

Uncertainty-Aware Graph Contrastive Fusion Network for multimodal physiological signal emotion recognition.

Incongruity-aware multimodal physiology signals fusion for emotion recognition

Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Continuous emotion recognition is a challenging task and a key part of human-computer interaction, especially multimodal emotion recognition can effectively improve the accuracy and robustness of recognition. But there are limited emotion data sets, and it is difficult to extract emotion features. We present a multi-level segmented decision level fusion emotion recognition model to improve the performance of emotion recognition. In this paper, we predict multi-modal dimensional emotional state on AVEC2017 dataset. Our model uses Bidirectional Long Short-Term Memory (BLSTM) as multi-level segmented emotional feature learning model, and uses the SVR model as fusion model of the decision layer. The BLSTM can model different forms of emotional information in time, and can also consider the impact of previous and later emotional features on current results. The SVR model can compensate for the redundant information of emotion recognition. At the same time, we also consider annotation delay and temporal pooling in our multi-modal dimensional emotion recognition model. Our multi-modal emotion recognition model achieves significant recognition improvements and provide the robustness. Finally, we compare the baseline methods which used the same dataset, and find that the CCC performance of our method is the best on arousal, which is 0.685. Our research shows that the proposed multi-layer segmentation decision level fusion emotion recognition model is conducive to improving performance.

Emotion is a complex psychophysiological phenomenon elicited by external stimuli, exerting a profound influence on cognitive processes, decision-making, and social behavior. Emotion recognition holds broad application potential in healthcare, education, and entertainment. With virtual reality (VR) emerging as a powerful tool, it offers an immersive and controllable experimental environment. Prior studies have confirmed the feasibility and advantages of VR for emotion elicitation and recognition, and multimodal fusion has become a key strategy for enhancing recognition accuracy. However, publicly available VR multimodal emotion datasets remain limited in both scale and diversity due to the scarcity of VR content and the complexity of data collection. The shortage hampers further progress. Moreover, existing multimodal approaches still face challenges such as noise interference, large inter-individual variability, and insufficient model generalization. Achieving robust and accurate physiological signal processing and emotion modeling in VR environments thus remains an open challenge. To address the issues, we constructed a VR experimental environment and selected 10 emotion-eliciting video clips guided by the PAD(Pleasure-Arousal-Dominance) model. Thirty-eight participants (N=38) were recruited, from whom electrodermal activity, eye-tracking, and questionnaire data were collected, yielding 366 valid trials. The newly collected dataset substantially extends the publicly available VREED dataset, enriching VR-based multimodal emotion resources. Furthermore, we propose the MMTED model (Multi-Modal Temporal Emotion Detector), which incorporates baseline calibration and multimodal fusion of electrodermal and eye-tracking signals for emotion recognition. Experimental results demonstrate the strong performance of the MMTED model, achieving accuracies of 85.52% on the public VREED dataset, 89.27% on our self-collected dataset, and 85.29% on their combination.

Over the past decade, increasing research efforts have been concentrated on emotion recognition from physiological signals due to their capability on emotion information representation. Existing works mainly focus on exploring the relationship between stimulus and subjects, while ignoring the effects of latent correlations among different subjects, which are important for personalized emotion recognition. To tackle this issue, we aim to conduct emotion recognition using multi-modal physiological signals through an edge-weighted hyper-graph neural network, in which complex relationship among subjects is formulated using hypergraph for each modality respectively. In our model, the differences in significance of influence which various samples leave on the classification can be better represented. The major contribution of this network lies in its concern that the associate strengths between various samples are different, which have different impact on the training result. The hyperedge between the vertices with closer correlation should be assigned a larger weight. Reversely, the looser relation, the minor weight. To evaluate the proposed method, experiments have been conducted on the DEAP dataset and ASCERTAIN dataset. Experimental results and comparison with state-of-the-art methods show that the proposed method can achieve better performance.

Emotions play an important role at our day-to-day activities such as cognitive process, communication and decision making. It is also very essential for interaction between human and machine. Emotion recognition has been receiving significant attention from various research communities and capturing user’s emotional state such as facial expressions, voice and body language, all of which are emerging way to find the human emotions. In recent years, physiological signals based emotion recognition has drawn increasing attention. Most of the physiological signals based methods use well-designed classifiers with hand-crafted features to recognize human emotions. In this paper, we present an approach to perform emotional states classification by end-to-end learning of deep convolutional neural network (CNN), which is inspired by the breakthroughs in the image domain using deep convolutional neural network. The approach is tested using the database “DEAP” including electroencephalogram (EEG) and peripheral physiological signals. We transform EEG into images combine extract hand-crafted features of other peripheral physiological signals, and classify emotions into valence and arousal. The results show this approach is possible to improve classification accuracy.

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

Survey on multimodal approaches to emotion recognition

Selection of Optimal EEG Electrodes for Human Emotion Recognition

Emotion recognition using multimodal physiological signals has gained widespread attention for providing objective insight into emotional states. However, existing models are often computationally complex and parameter heavy, hindering deployment on resource constrained wearable devices. To overcome this, we propose ULER (Ultra Lightweight Emotion Recognition), an efficient framework that integrates lightweight multi-scale convolution (LMSC) with a novel joint attention mechanism (LJA) based on a dynamic feature fusion network (DFFN) for effective multimodal fusion. Using subject dependent and total dataset training strategies, ULER is evaluated on three benchmarks (DEAP, DREAMER, WESAD) and outperforms recent state-of-the-art (SOTA) methods. It achieves accuracies of 99.34%, 99.46%, and 99.23% on DEAP binary valence, binary arousal, and four class tasks, respectively, with only 0.60M parameters, 9.33M FLOPs, and 31.10ms inference latency. In a wearable oriented reduced channel setup (11 EEG channels), ULER also surpasses most SOTA models using standard 32 channels. Principal contributions emphasize: (1) a systematic comparison against recent SOTA methods on multiple datasets; (2) a wearable design with reduced-channel configuration; and (3) the DFFN as a core component for efficient multimodal fusion. This work demonstrates the potential for practical, real-time emotion monitoring in personal healthcare and other scenarios.

Revealing the correspondences and relationships between physiological signals is attractive for bioinformatics and human-computer interaction. Time alignment is a straightforward way to figure out correspondences between time sequential data. However, alignment between multimodal physiological signals is hard to achieve because the similarity metrics are difficult to define if the two physiological signals being investigated are non-linearly correlated, misaligned or quite different in morphology. In this paper, we propose a generalized time alignment method for multimodal physiological signals which (i) learns the feature extractions on physiological signals in a generalized way, and (ii) enables learned features to be in a coordinated space where the similarity between sub-components from two signals can be defined. Furthermore, we applied our alignment based multimodal feature fusion on an evaluation model to perform emotion recognition tasks on the DEAP multimodal physiological signal dataset. The experimental results show that the alignment based feature fusion outperforms the non-aligned feature fusion in most cases.

In the fast pace of life, emotion recognition systems are essential to help monitor mental health and well-being. The continuous development of the Internet of Things (IoT) and Human-Computer Interaction (HCI) improve the availability and accessibility to devices that can capture the facial expressions of a user, while wearable devices can also capture physiological signals and use them for emotion recognition. Meanwhile, machine learning and deep learning methods can provide emotion prediction models. However, the training of the models relies heavily on massive amounts of labeled data. The accuracy of data labels affects the success of the overall system. Research targeting emotion recognition uses the participants' self-reports as labels. However, participants often fail to give accurate self-reports, thus affecting the accuracy of the analysis. In this study, we examine the performance of the self-reports and external annotations for emotion recognition based on visual and physiological signals. Specifically, we use video data, as well as the Electrodermal Activity (EDA), Electroencephalogram (EEG), and Electrocardiogram (ECG) signals collected from wearable devices. We use two machine learning and three deep learning methods to process the signals and train the classifiers. The results show that the classifiers trained with external annotations offer better emotion recognition accuracy than self-reports. Also, the classifiers trained on facial expression offer better emotion prediction accuracy than the physiological signals, and the Deep Convolutional Network model shows the best results.

In this paper a study on multimodal automatic emotion recognition during a speech-based interaction is presented. A database was constructed consisting of people pronouncing a sentence in a scenario where they interacted with an agent using speech. Ten people pronounced a sentence corresponding to a command while making 8 different emotional expressions. Gender was equally represented, with speakers of several different native languages including French, German, Greek and Italian. Facial expression, gesture and acoustic analysis of speech were used to extract features relevant to emotion. For the automatic classification of unimodal data, bimodal data and multimodal data, a system based on a Bayesian classifier was used. After performing an automatic classification of each modality, the different modalities were combined using a multimodal approach. Fusion of the modalities at the feature level (before running the classifier) and at the results level (combining results from classifier from each modality) were compared. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison to the unimodal systems: the multimodal approach increased the recognition rate by more than 10% when compared to the most successful unimodal system. Bimodal emotion recognition based on all combinations of the modalities (i.e., ‘face-gesture’, ‘face-speech’ and ‘gesture-speech’) was also investigated. The results show that the best pairing is ‘gesture-speech’. Using all three modalities resulted in a 3.3% classification improvement over the best bimodal results.