Deep Convolutional Neural Network for Emotion Recognition Using EEG and Peripheral Physiological Signal

Selection of Optimal EEG Electrodes for Human Emotion Recognition

Current multi-modal emotion recognition from physiological signals requires electroencephalogram(EEG) signals and peripheral physiological signals during both training and test. Compared with the peripheral physiological signals, it is more difficult to obtain EEG signals in our daily life. Therefore, we propose a novel approach to recognize emotions from peripheral signals by using EEG features as privileged information, which is only available during training. During training, first, peripheral physiological features and EEG features are extracted. Then, we construct a new peripheral physiological feature space using canonical correlation analysis with the help of EEG features. Finally we train a support vector machine(SVM) to map the new peripheral physiological features to the emotion labels. During test, only peripheral physiological features are used to recognize emotions from the constructed peripheral physiological feature space with the trained SVM model. The experimental results on two benchmark databases show that our proposed approach using EEG features as privileged information outperforms the method which recognizes emotions merely from the peripheral physiological signals.

Recognising emotions is paramount in assessing students’ engagement and improving their educational performance. This study investigates a deep learning and convolutional neural network (CNN)-based facial emotion recognition system to analyse students’ facial expressions in real time. The model proposed improves classical learning methods by offering the analysis of students' emotional states, which, in turn, facilitates adaptive learning and teaching approaches. The system combines feature extraction by CNNs and classification using deep learning, achieving a high accuracy in recognising several different emotions, including happiness, frustration, and confusion. Findings of this study demonstrate the role of AI-powered emotion recognition in transforming the conventional teaching approach to an interactive and learner-centred paradigm. This research approaches the development of intelligent technologies for education through the integration of AI, psychology, and pedagogy to increase students’ academic engagement and performance.

This paper considers a hybrid multimodal model for improvement of the human emotion recognition based on facial expression and body gesture recognition. The paper extends the author’s investigations related to the usage of pre-trained models of deep learning neural networks (DNN) for facial emotion recognition (FER) with addition of the emotions extracted from body language. In order to extract emotions from upper body gestures a second model of DNN was developed and trained with specific datasets. The information regarding recognized emotions obtained by both models is more accurate and can be used in education, medicine, psychology, product advertisement, marketing, human-machine interfaces, etc. In our case, it aims to personalize the lecture material of the students during their online training, taking into account their emotional state.

Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

To address the limitations of the single-modal electroencephalogram (EEG), such as its single physiological dimension, weak anti-interference ability, and inability to fully reflect emotional states, this paper proposes a gated multi-head cross-attention module (GMHCA) for multimodal fusion of EEG, electrooculography (EOG),and electrodermal activity (EDA). This attention module employs three independent and parallel attention computation units to assign independent attention weights to different feature subsets across modalities. Combined with a modality complementarity metric, the gating mechanism suppresses redundant heads and enhances the information transmission of key heads. Through multi-head concatenation, cross-modal interaction results from different perspectives are fused. For the backbone network, a multi-scale convolution and bidirectional long short-term memory network (MC-BiLSTM) is designed for feature extraction, tailored to the characteristics of each modality. Experiments show that this method, which primarily fuses eight-channel EEG with peripheral physiological signals, achieves an emotion recognition accuracy of 89.45%, a 7.68% improvement over single-modal EEG. In addition, in cross-subject experiments conducted on the SEED-IV dataset, the EEG+EOG modality achieved a classification accuracy of 92.73%. All were significantly better than the baseline method. This fully demonstrates the effectiveness of the innovative GMHCA module architecture and MC-BiLSTM feature extraction network proposed in this paper for multimodal fusion methods. Through the novel attention gating mechanism, higher recognition accuracy is achieved while significantly reducing the number of EEG channels, providing new ideas and approaches based on attention mechanisms and gated fusion for multimodal emotion recognition in resource-constrained environments.

Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Affective gaming is a hot field of research that exploits human emotion for the enhancement of player's experience during gameplay. Physiological signal is an effective modality that can provide a better understanding of the emotional states and is very promising to be applied to affective gaming. Most physiological-based affective gaming applications evaluate player's emotion on an overall game fragment. These approaches fail to capture the emotion change in the dynamic game context. In order to achieve a better understanding of psychophysiological response with a better time sensitivity, we present a study that evaluates the psychophysiological responses related to the game events. More specifically, we present a multi-modal database DAG that contains peripheral physiological signals (ECG, EDA, respiration, EMG, temperature), accelerometer signals, facial and screening recordings as well as player's self-reported eventrelated emotion assessment through game playing. We then investigate physiological-based emotion detection and recognition by using machine learning techniques. Common challenges for physiological-based affective model such as signal segmentation, feature normalization, relevant features are addressed. We also discuss factors that influence the performance of the affective models.

Physiological signal-based emotion recognition requires effective fusion of multi-modal physiological signals to improve recognition accuracy. In this paper, a multi-kernel embedding fusion framework (MKEFF) is proposed for multi-modal physiological signal emotion recognition. Specifically, multi-kernel learning and kernel approximation techniques are used to compute the multi-kernel embeddings of the original feature vectors of each modality independently. The embeddings are then fed in parallel to their respective representation learning layer, where the proposed sparse relation learning method is applied to all the modalities to explore the correlation and diversity among them. Finally, a distribution alignment based fusion method is proposed to align each modality in the subspace, and a weighted summation fusion is performed to obtain the fused representations. Extensive cross-subject emotion recognition experiments are conducted on three public datasets, DEAP, DECAF, and SEED-IV, to evaluate the proposed method. The experimental results demonstrate that the proposed method achieves better classification performance and interpretability than the state-of-the-art methods.

With the advancement of machine learning methods, audio sentiment analysis has become an active research area in recent years. For example, business organizations are interested in persuasion tactics from vocal cues and acoustic measures in speech. A typical approach is to find a set of acoustic features from audio data that can indicate or predict a customer's attitude, opinion, or emotion state. For audio signals, acoustic features have been widely used in many machine learning applications, such as music classification, language recognition, emotion recognition, and so on. For emotion recognition, previous work shows that pitch and speech rate features are important features. This thesis work focuses on determining sentiment from call center audio records, each containing a conversation between a sales representative and a customer. The sentiment of an audio record is considered positive if the conversation ended with an appointment being made, and is negative otherwise. In this project, a data processing and machine learning pipeline for this problem has been developed. It consists of three major steps: 1) an audio record is split into segments by speaker turns; 2) acoustic features are extracted from each segment; and 3) classification models are trained on the acoustic features to predict sentiment. Different set of features have been used and different machine learning methods, including classical machine learning algorithms and deep neural networks, have been implemented in the pipeline. In our deep neural network method, the feature vectors of audio segments are stacked in temporal order into a feature matrix, which is fed into deep convolution neural networks as input. Experimental results based on real data shows that acoustic features, such as Mel frequency cepstral coefficients, timbre and Chroma features, are good indicators for sentiment. Temporal information in an audio record can be captured by deep convolutional neural networks for improved prediction accuracy.

As life continues to change in the digital era, it is crucial to perceive a person's emotional state. Affective computing is receiving more attention with the increase in the human-computer interface (HCI). Human emotion recognition employing electroen-cephalogram (EEG) signals has been studied to obtain a person's emotional status for different stimuli. However, it is difficult to identify clear patterns in EEG signals because they have low electrical impulses and are highly sensitive to noise. A deep convolutional neural network (DCNN) was employed in the present study to recognize emotions in EEG signals. For this purpose, a publicly available dataset, DREAMER, was utilized in this study to assess the applicability of the model for emotion classification. The dataset consisted of three-dimensional emotions, that is, valence, arousal, and dominance (VAD). 2D emotions arousal and valence were the most-recognized emotions in existing research. The present study identified the 3D emotions present in the above-mentioned dataset. In this study, raw EEG signals from the DREAMER dataset were pre-processed. Subsequently, three EEG rhythms, theta, alpha, and beta, were extracted using a bandpass filter. The power spectral density (PSD) was computed using fast Fourier transform (FFT) in the feature extraction. Finally, a 1D CNN model is applied to the classification of emotions. In addition, the performance of the proposed model was compared with two machine learning (ML) classifiers: random forest (RF) and extreme Gradient Boosting (XGBoost) classifiers. The highest accuracy (ACC) of 97.6% was obtained using the proposed model in the dominance dimension. The working principles were compared and discussed to determine the suitability of the model for emotion recognition applications.

The connection between emotional states and physical health has attracted widespread attention. The emotional stress assessment can help healthcare professionals figure out the patient's engagement toward the diagnostic plan and optimize the rehabilitation program as feedback. It is of great significance to study the changes of physiological features in the process of emotional change and find out subset of one or several physiological features that can best represent the changes of psychological state in a statistical sense. Previous studies had used the differences in physiological features between discrete emotional states to select feature subsets. However, the emotional state of the human body is continuously changing. The conventional feature selection methods ignored the dynamic process of an individual's emotional stress in real life. Therefore, a dedicated experimental was conducted while three peripheral physiological signals, i.e., ElectroCardioGram (ECG), Galvanic Skin Resistance (GSR), and Blood Volume Pulse (BVP), were continuously acquired. This paper reported a novel feature selection method based on emotional state transition, the experimental results show that the number of physiological features selected by the proposed method in this paper is 13, including 5 features of ECG, 4 features of PPG and 4 features of GSR, respectively, which are superior to PCA method and conventional feature selection method based on discrete emotional states in terms of dimension reduction. The classification results show that the accuracy of the proposed method in emotion recognition based on ECG and PPG is higher than the other two methods. These results suggest that the proposed method can serve as a viable alternative to conventional feature selection methods, and emotional state transition deserves more attention to promote the development of stress assessment.

Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals

Emotion recognition is considered as one of the most vital and challenging studies in the domain of computer vision. Due to the ongoing pandemic the people’s emotional and mental state is getting affected due to less physical interaction and increased virtual interaction. The motivation of the work lies in the recognition and monitoring of the emotional state of the human beings in live environment with higher accuracy rate and fast recognition time so as to keep them aware about their emotional state in the pandemic situation. Emotion recognition is achieved using the proposed deep convolution neural network (DCNN) using custom Gabor filter with 85.8% accuracy. Emotion recognition is also being investigated and compared with other deep learning models such as AlexNet, VGG-Net for testing the efficacy. When the emotion of the person will be found sad or angry then an alert sound is played and an email alert is automatically sent.KeywordsEmotion recognitionDeep learningGabor filterPandemicReal-time alertDCNNPreprocessing

Insulator fault diagnosis is a daily but key task for the power transmission system. Long-term exposure to complex natural environment will cause different insulator defects. As a common defects, missing-cap defects of insulators will not only affect the structural strength of power insulators, but also bring a certain effect to the stable power transmission. With the rapid development of machine learning, some machine learning-based defect recognition methods have been proposed for fast and high-precision power inspection. However, the handcrafted features could not effectively express the aerial images against complex inspection environment to affect detection performance of the shallow learning algorithms. And the detection precision of deep learning algorithms will be affected by the unbalanced small-scale defects. Therefore, the fast and high-precision power inspection still faces a certain challenge in the smart grid. To address the above issues, fusion with the deep convolutional neural network (DCNN) and transfer learning, a novel fault diagnosis algorithm of power insulators is proposed to provide a fast and accurate power inspection scheme. To remove complex backgrounds, a fast insulator location algorithm based on the lightweight YOLOV4 model is proposed which is served for the following defect recognition. On the basis, to imitate human vision, a defect recognition algorithm is proposed based on multi-feature fusion. Meanwhile, to ensure the feature expression ability of transfer learning on power insulators, a novel optimization strategy of transfer learning is proposed to improve the recognition precision. Experiments show that the proposed method could acquire a good recognition performance than other recognition models.