Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism

Emotion recognition using cross-modal attention from EEG and facial expression

Emotion identification plays a vital role in human interactions. For this purpose, Computer-vision methods for automatic emotion recognition is nowadays a widely studied topic. One of the most studied approaches for automatic emotion recognition is processing multi-channel Electroencephalogram signals (EEG). This paper presents a new model for emotion recognition using brain maps as input and providing emotion states in terms of arousal and valence as output. Brain maps are a spatial representation of features extracted from EEG signals. The proposed model, called Multi-Task Convolutional Neural Network (MT-CNN), is fed with stacked brain maps of four different waves of different frequency bands: alpha, beta, gamma and theta, using differential entropy and power spectra density and considering observation windows of 0.5s. This model is trained and tested on the DEAP dataset, a well-known dataset for comparison purposes. This work shows that the MT-CNN nerforms better than other methods.

Objective: To extract statistical features from EEG signal for human emotion recognition. Methods: Emotions, the “inner” state of a person play a vital role in analysing the state of mind. In this paper, a new method for human emotion recognition using Multi Wavelet Transform (MWT) and random forest (ensemble technique) for classification of human emotions from EEG signals has been proposed. The experiment is conducted on EEG database for emotions i.e., DEAP data set. In present work, the data set used contains EEG recording from 4 participants (S01, S02, S03, S04) from 15 channel (FP1, FP2, F3, P3, F4, T7, T8, P4, O1, PZ, PO3, O2, P7, CP2, and C4) having 40 trails each. The paper explore the capability of proposed features namely Mean, Standard deviation, Variance, Shannon entropy, Hjorth parameters and Band power. Findings: The feature set obtained using Multi-Wavelet are then used as input for MLP, KNN, MC-SVM with Puk kernel function and Random Forest (ensemble) classifier for the classification of emotions. The classification accuracy for different emotion state happy 99.8%, sad 98.3%, exciting 95%, hate 96.4% are obtained by the proposed method. Overall 98.1% for classification of emotions (happy, sad, exciting, hate) from EEG signals. Application: The findings are helpful in monitoring alertness, cognitive engagement of patients with impaired motor functions; recognition of emotions; investigating sleep disorders and physiology. In future, efficient feature extraction algorithm using different multiwavelet functions and with a different set of features which include the temporal dynamics of brain signals in the human cognitive system can be considered.

ADMET property prediction via multi-task graph learning under adaptive auxiliary task selection

In this work, we have proposed STLEV (Siamese Neural Network with Triplet Loss and Long Short-Term Memory for Emotion recognition in Videos) a method for emotion recognition in videos. This method is beneficial for cognitive Human-Computer interaction. Emotion recognition from video is a challenging task as only few frames of a video will contain relevant information for recognizing emotions. Also identifying the apex frame is challenging. Emotion recognition from videos involves extracting the features from the frames of a video and classifying the sequence of features. For feature extraction we have used Siamese Neural Network (SNN) which is metric based meta-learning model. SNN is trained with triplet loss. For classification, Long Shot-Term Memory (LSTM) model is used. In this work we have used spatial and temporal aspects of video. Spatial features are extracted using SNN and temporal features using LSTM. Deep learning models need large amount of data for good accuracy. We have addressed this problem by leveraging few shot learning aspects of metric meta-learning so that fewer samples are needed for training the model used for feature extraction. Experiments done using BU-4DFE dataset achieved an accuracy of 87.5% which demonstrates the efficiency of the method STLEV.

In order to overcome the problems of high recognition error rate and low recognition efficiency existing in the traditional method of emotional recognition of weightlifters, a method of emotional recognition of weightlifters based on a non-negative matrix decomposition algorithm is proposed. The database of emotion recognition matching criteria is established, and the mapping relationship among emotion, facial expression of far mobilisation and weight lifting posture is formed. The real-time motion images of weightlifters are collected, and the image classifier is designed by using the non-negative matrix decomposition algorithm. The real-time images are divided into facial expression images and whole body motion images, and the features of the two images are extracted respectively. The extracted results are matched with the results of the standard database, and the real-time emotion recognition results of weightlifters are obtained. The experimental results show that compared with the traditional emotion recognition method, the proposed method improves the accuracy of emotion recognition by 50%.

In recent years, affective computing has become a research hotspot in the area of natural language processing and Chinese emotion recognition is an important constituent. This paper proposes a method of Chinese emotion recognition based on three-way decisions. Given the emotion dictionary constructed firstly, the grammatical information of sentences, topic features of texts and three-way decisions are integrated and applied into Chinese emotion recognition, thus realizing the multi-label emotion recognition of sentences in Chinese texts. The results of experiments show that the method of Chinese emotion recognition, based on three-way decisions, has achieved excellent results in the emotion recognition of Chinese sentences.

Multimodal fusion: A study on speech-text emotion recognition with the integration of deep learning

Computer vision has been transformed by deep learning, which has made it possible for robots to comprehend and interpret visual data at previously unheard-of levels. This study explores deep learning for computer vision systems: future trends and upcoming technology. Looking ahead, several significant advancements are reshaping this profession. First off, a viable approach is to combine deep learning architectures with generative models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). This synergy makes activities like picture generation, style transfer, and data augmentation easier and enables a more nuanced interpretation of visual data. Second, a key component of deep learning for computer vision is now recognized to be attention mechanisms. These techniques, which are based on human visual attention, allow models to concentrate on particular areas of a picture, improving efficiency and interpretability. Tasks such as object detection, image segmentation, and image captioning demonstrate the critical role of attention mechanisms. Moreover, computer vision is reaching new heights thanks to the combination of deep learning and other cutting-edge technologies. Real-time processing on devices is made possible by the combination of deep learning and edge computing, which lowers latency and improves privacy. Furthermore, new opportunities for applications in augmented reality, robotics, and immersive experiences are being created by the combination of deep learning and 3D vision technologies. In the deep learning space for computer vision, transfer learning and meta-learning are likewise becoming more and more popular [1]. These methods increase performance on a different but related job by utilizing pre-trained models or knowledge from one task. This method minimizes the need for large datasets for training in situations where labeled data is rare. Deep learning in computer vision raises ethical questions that are becoming more and more significant. Concerns about decision interpretability, algorithmic bias, and privacy are becoming more prevalent. Fairness, accountability, and transparency must be given top priority in this field's future developments in order to guarantee the appropriate use of these technologies. In conclusion, there are a lot of promising things that lie ahead for deep learning in computer vision. Advances in transfer learning, attention mechanisms, and the integration of generative models are all leading to more complex and effective vision systems. As these technologies advance, it is critical to address ethical issues in order to build trust and guarantee that deep learning has a good social impact. This study offers a thorough analysis of these recent developments, providing insight into the future direction of deep learning for computer vision.

In summary, the field of computer science has long been intrigued by emotion recognition, which seeks to decode the emotional content hidden within data. Initial approaches to sentiment analysis were predominantly based on single-mode data sources like textual sentiment analysis, speech-based emotion detection, or the study of facial expressions. In recent years, with the increasingly abundant data representations, Many people has gradually pay attention on the multimodal emotion recognition. Multimodal emotion recognition involves not only text, but also audio, image, and video, which is of great significance for enhancing human-computer interaction, improving user experience, and improving emotion-aware applications. This paper thoroughly discusses the research advancements and primary techniques of multimodal emotion recognition tasks, with an emphasis on the aforementioned tasks. Specifically, this paper first introduces representative methods for single-modal emotion recognition based on graphic data, including its basic process, advantages and disadvantages, etc. Secondly, this article introduces pertinent studies on multimodal emotion recognition and offers a quantitative comparison of how different approaches perform on standard multimodal data sets. Lastly, it addresses the complexities inherent in multimodal emotion recognition research and suggests potential areas for future study.

An approach to EEG-based emotion recognition using combined feature extraction method

In order to overcome the problem of high recognition error rate in traditional emotional tendency recognition methods, a Mandarin emotional tendency recognition method based on expression spatiotemporal feature recognition is proposed. This method extracts the spatiotemporal features of the expression of the research object, and uses the data fusion technology to fuse the extracted feature vector. This paper constructs the standard database of Mandarin emotional tendency recognition, and takes the database as the standard of emotional tendency recognition. The fusion feature vector is matched with the standard feature in the database to get the result of Mandarin emotional tendency recognition. The experimental results show that compared with the traditional method for Mandarin emotional tendency recognition, the recognition method based on the spatiotemporal feature of expression can reduce the recognition error by about 75%.

Emotions are integral to human cognition, exerting a profound influence on physiological responses, cognitive processes, and decision-making capabilities. Electroencephalography (EEG)-based emotion classification provides a significant methodological approach for the exploration of emotional states. Despite its potential, most current methodologies face challenges in delineating the representational patterns across different brain regions and in effectively classifying emotions from EEG signals. In response, a novel model for emotion recognition is proposed in this paper, which utilizes a multi-channel attention mechanism, designated as MCAHNN. This model incorporates Householder Reflection to enhance the attention mechanism, facilitating the extraction of inter-channel EEG features and simulating inter-regional brain dynamics. Furthermore, 1D convolution is employed to analyze intra-channel relationships. The proposed model has been evaluated on the publicly available DEAP dataset and further tested on the SEED dataset. Experimental results confirm that the MCAHNN model achieves state-of-the-art performance, demonstrating its effectiveness in classifying emotions within multi-center datasets.Code is publicly available at https://github.com/Oreoreoreor/MCAHNN.

When modeling related tasks in computer vision, Multi-Task Learning (MTL) can outperform Single-Task Learning (STL) due to its ability to capture intrinsic relatedness among tasks. However, MTL may encounter the insufficient training problem, i.e., some tasks in MTL may encounter non-optimal situation compared with STL. A series of studies point out that too much gradient noise would lead to performance degradation in STL, however, in the MTL scenario, Inter-Task Gradient Noise (ITGN) is an additional source of gradient noise for each task, which can also affect the optimization process. In this paper, we point out ITGN as a key factor leading to the insufficient training problem. We define the Gradient-to-Noise Ratio (GNR) to measure the relative magnitude of gradient noise and design the MaxGNR algorithm to alleviate the ITGN interference of each task by maximizing the GNR of each task. We carefully evaluate our MaxGNR algorithm on two standard image MTL datasets: NYUv2 and Cityscapes. The results show that our algorithm outperforms the baselines under identical experimental conditions.KeywordsMulti-task learningGradient noiseWeight strategy

Emotion feature extraction is the key to speech emotional recognition. And ensemble empirical mode decomposition(EEMD) is a newly developed method aimed at eliminating emotion mode mixing present in the original empirical mode decomposition(EMD). To evaluate the performance of this new method, this paper investigates the effect of a parameters pertinent to EEMD: speech emotional envelope. Firstly, a speaker emotional envelope features extraction based on EEMD is proposed in the paper. Using the piecewise power function in speech emotional envelope has a better effect in emotional identification. At the same time, the proposed technique has been utilized for classification of fo ur kinds of emotional(angry, happy, sad and neutral) speech signals. Emotional intrinsic mode functions(IMFe) are obtained by empirical mode decomposition on emotional speech signals, the fast fourier transform(FFT) of each intrinsic mode function is extracte d as the emotional feature coefficient which is used in speaker emotional identification applying by vector quantization. MATLAB is used to ca lculate the characteristic of emotional speech signals using empirical mode decomposition (EEMD). We obtain an emotional envelope by transforming the IMFe of emotional speech signals, and obtain a new method of emotion recognition according to different emotional envelop feature vectors. The results indicate the proposed method works well in speaker emotional identification.