Multi-View Cognitive Prior Graph Convolutional Network for Multimodal Emotion Recognition

Graph Convolutional Network (GCN) excels at EEG recognition by capturing brain connections, but previous studies neglect the important EEG feature itself. In this study, we propose MSFR-GCN, a multi-scale feature reconstruction GCN for recognizing emotion and cognition tasks. Specifically, MSFR-GCN includes the MSFR and feature-pool characteristically, with the MSFR consisting of two sub-modules, multi-scale Squeeze-and-Excitation (MSSE) and multi-scale sample re-weighting (MSSR). MSSE assigns weights to channels and frequency bands based on their separate statistical information, while MSSR assigns sample weights based on combined channel and frequency information. The feature-pool, which pools across the feature dimension, is applied after GCN to retain EEG channel information. The MSFR-GCN achieves excellent results in emotion recognition when first tested on two public datasets, SEED and SEED-IV. Than the MSFR-GCN is tested on our self-collected Emotion and Cognition EEG dataset (ECED) for both emotion and cognition classification tasks. The results show MSFR-GCN's good performance in emotion and cognition classification tasks and reveal the implicit relationship between the two, which may provide aid in the rehabilitation of people with cognitive impairments from an emotional perspective.

A parallel neural networks for emotion recognition based on EEG signals

Due to the changeable, high-dimensional, non-stationary, and other characteristics of electroencephalography (EEG) signals, the recognition of EEG signals is mostly limited to independent individuals. To deal with these issues, we propose a multi-dimensional graph convolution network (MD-GCN), which integrates EEG signals’ temporal and spatial characteristics and can classify emotions more accurately. First, we use that the asymmetry of neuron activity in the left and right hemispheres is very important for emotion prediction to initialize the adjacency matrix and perform preliminary edge prediction without considering node features. Then, we perform the feature fusion on the Inception network and then input it into the graph convolution network to learn the interrelationship between channels. Finally, we visually analyze the adjacency matrix. To evaluate the performance of the model, we conduct experiments on the SEED dataset and the SEED-IV dataset. The results show that the pre-defined adjacency matrix method can improve the accuracy of emotion recognition and the graph convolution has better performance than the same type of convolution. It also theoretically shows that the emotional state is mainly by the interaction of important brain regions.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Various studies have shown that the temporal information captured by conventional long-short-term memory (LSTM) networks is very useful for enhancing multimodal emotion recognition using encephalography (EEG) and other physiological signals. However, the dependency among multiple modalities and high-level temporal-feature learning using deeper LSTM networks is yet to be investigated. Thus, we propose a multimodal residual LSTM (MMResLSTM) network for emotion recognition. The MMResLSTM network shares the weights across the modalities in each LSTM layer to learn the correlation between the EEG and other physiological signals. It contains both the spatial shortcut paths provided by the residual network and temporal shortcut paths provided by LSTM for efficiently learning emotion-related high-level features. The proposed network was evaluated using a publicly available dataset for EEG-based emotion recognition, DEAP. The experimental results indicate that the proposed MMResLSTM network yielded a promising result, with a classification accuracy of 92.87% for arousal and 92.30% for valence.

Graph convolutional neural network (GCNN)-based methods have been widely used in electroencephalogram (EEG)-related works due to their advantages of considering the symmetrical connections of brain regions. However, the current GCNN-based methods do not fully explore other correlations between EEG channels. Many studies have proved that definite causal connections exist between brain regions. Therefore, this article proposes a causal GCNN (CGCNN) using the Granger causality (GC) test to calculate interchannel interactions. First, we consider causal relations between EEG channels and construct an asymmetric causal graph with direction. Then, we adopt depthwise separable convolution to extract emotional features from multichannel EEG signals. Experiments carried out on SEED and SEED-IV show that CGCNN has the ability to represent the causal information flow in different emotional states, and improve the classification accuracy to 93.36% on SEED and 75.48% on SEED-IV, respectively. The results outperform other existing methods, indicating that GC is more effective in revealing the correlations between EEG channels in emotion recognition.

Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

Electroencephalogram (EEG) emotion recognition plays an important role in human–computer interaction. An increasing number of algorithms for emotion recognition have been proposed recently. However, it is still challenging to make efficient use of emotional activity knowledge. In this paper, based on prior knowledge that emotion varies slowly across time, we propose a temporal-difference minimizing neural network (TDMNN) for EEG emotion recognition. We use maximum mean discrepancy (MMD) technology to evaluate the difference in EEG features across time and minimize the difference by a multibranch convolutional recurrent network. State-of-the-art performances are achieved using the proposed method on the SEED, SEED-IV, DEAP and DREAMER datasets, demonstrating the effectiveness of including prior knowledge in EEG emotion recognition.

Electroencephalography (EEG) signals are physiological signals that directly reflect the brain’s electrical activity, holding significant research value in the field of emotion recognition. However, existing methods still face certain limitations in jointly modeling temporal, spatial, and frequency-domain features. To address this, this article proposes an EEG-based emotion recognition model that integrates a three-dimensional convolutional neural network (3DCNN) with a bidirectional long short-term memory network (BiLSTM), referred to as 3DC-BiL. This approach utilizes 3DCNN to extract spatio-temporal features across different frequency bands, and then leverages BiLSTM to enhance temporal modeling capabilities, achieving collaborative fusion of multidimensional features. On the DEAP dataset for binary emotion classification, the proposed model achieves average accuracies of 98.60% and 98.66% on the arousal and valence dimensions, respectively. On the SEED dataset for three-class emotion classification, it achieves an average accuracy of 99.01%, outperforming various mainstream emotion recognition models in recent years. The experimental results demonstrate that the proposed model excels in EEG feature extraction and high-precision emotion classification, providing a new perspective and practical foundation for the research of EEG-based emotion recognition systems.

Facial emotion recognition technology is one of the important topics in current human-computer interaction research. Emotion recognition methods based on deep learning have made great progress. However, with the complexity of the network model, the emotion recognition network has the problems of large amount of computation and long computation cycle, which makes it difficult to deploy and apply in real-time embedded systems with limited hardware resources. Therefore, this paper designs a binary neural network (BNN) emotion recognition algorithm optimized for embedded systems and deploys it on the PYNQ-ZI hardware board for verification. The main network of the algorithm uses Resnet as the benchmark model to perform binary operations and uses a combination of XNOR and popcount to implement convolution operations instead of floating-point multiplication and addition operations, thereby significantly reducing hardware memory usage and computation. By adding a reorg layer and a route layer to cascade multi-dimensional feature maps, the recognition accuracy is improved. In addition, in the preprocessing part, the Viola-Jones (VJ) and local binary pattern rotation invariant (LBPROT) algorithms are used to achieve accurate face detection, and finally emotion recognition is realized through network reasoning. The experimental results show that the proposed real-time facial emotion algorithm shows good performance in resource consumption, detection accuracy and running speed.

Due to the objectivity of emotional expression in the central nervous system, EEG-based emotion recognition can effectively reflect humans' internal emotional states. In recent years, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made significant strides in extracting local features and temporal dependencies from EEG signals. However, CNNs ignore spatial distribution information from EEG electrodes; moreover, RNNs may encounter issues such as exploding/vanishing gradients and high time consumption. To address these limitations, we propose an attention-based temporal graph representation network (ATGRNet) for EEG-based emotion recognition. Firstly, a hierarchical attention mechanism is introduced to integrate feature representations from both frequency bands and channels ordered by priority in EEG signals. Second, a graph convolutional neural network with top-k operation is utilized to capture internal relationships between EEG electrodes under different emotion patterns. Next, a residual-based graph readout mechanism is applied to accumulate the EEG feature node-level representations into graph-level representations. Finally, the obtained graph-level representations are fed into a temporal convolutional network (TCN) to extract the temporal dependencies between EEG frames. We evaluated our proposed ATGRNet on the SEED, DEAP and FACED datasets. The experimental findings show that the proposed ATGRNet surpasses the state-of-the-art graph-based mehtods for EEG-based emotion recognition.

The main core purpose of artificial emotional intelligence is to recognize human emotions. Technologies such as facial, semantic, or brainwave recognition applications have been widely proposed. However, the abovementioned recognition techniques for emotional features require a large number of training samples to obtain high accuracy. Human behaviour pattern can be trained and recognized by the continuous movement of the Spatial Temporal Graph Convolution Network (ST-GCN). However, this technology does not distinguish between the speed of delicate emotions, and the speed of human behaviour and delicate changes of emotions cannot be effectively distinguished. This research paper proposes Spatial Temporal Variation Convolutional Network training for human emotion recognition, using skeleton detection technology to calculate the degree of skeleton point change between consecutive actions and using the nearest neighbour algorithm to classify speed levels and train the ST-GCN recognition model to obtain the emotional state. Application of the speed change recognition ability of the Spatial Temporal Variation Graph Convolution Network (STV-GCN) to artificial emotional intelligence calculation makes it possible to efficiently recognize the delicate actions of happy, sad, fear, and angry in human behaviour. The STV-GCN technology proposed in this paper is compared with ST-GCN and can effectively improve the recognition accuracy by more than 50%.

Uncertainty-Aware Graph Contrastive Fusion Network for multimodal physiological signal emotion recognition.