Multimodal Emotion Recognition Using Deep Learning Techniques

A novel facial emotion recognition method for stress inference of facial nerve paralysis patients

In order to overcome the problems of high recognition error rate and low recognition efficiency existing in the traditional method of emotional recognition of weightlifters, a method of emotional recognition of weightlifters based on a non-negative matrix decomposition algorithm is proposed. The database of emotion recognition matching criteria is established, and the mapping relationship among emotion, facial expression of far mobilisation and weight lifting posture is formed. The real-time motion images of weightlifters are collected, and the image classifier is designed by using the non-negative matrix decomposition algorithm. The real-time images are divided into facial expression images and whole body motion images, and the features of the two images are extracted respectively. The extracted results are matched with the results of the standard database, and the real-time emotion recognition results of weightlifters are obtained. The experimental results show that compared with the traditional emotion recognition method, the proposed method improves the accuracy of emotion recognition by 50%.

Humans have traditionally found it simple to identify emotions from facial expressions, but it is far more difficult for a computer system to do the same. The social signal processing subfield of emotion recognition from facial expression is used in a wide range of contexts, particularly for human-computer interaction. Automatic emotion recognition has been the subject of numerous studies, most of which use a machine learning methodology. The recognition of simple emotions like anger, happiness, contempt, fear, sadness, and surprise, however, continues to be a difficult topic in computer vision. Deep learning has recently drawn increased attention as a solution to a variety of practical issues, including emotion recognition. In this study, we improved the convolutional neural network technique to identify 7 fundamental emotions and evaluated several preprocessing techniques to demonstrate how they affected the CNN performance. This research focuses on improving facial features and expressions based on emotional recognition. By identifying or recognising facial expressions that elicit human responses, it is possible for computers to make more accurate predictions about a person's mental state and to provide more tailored responses. As a result, we examine how a deep learning technique that employs a convolutional neural network might improve the detection of emotions based on facial features (CNN). Multiple facial expressions are included in our dataset, which consists of about 32,298 photos for testing and training. The preprocessing system aids in removing noise from the input image, and the pretraining phase aids in revealing face detection after noise removal, including feature extraction. As a result, the existing paper generates the classification of multiple facial reactions like the seven emotions of the facial acting coding system (FACS) without using the optimization technique, but our proposed paper reveals the same seven emotions of the facial acting coding system.

The study of emotions in human-computer interaction is a growing research area. Focusing on automatic emotion recognition, work is being performed in order to achieve good results particularly in speech and facial gesture recognition. In this paper we present a study performed to analyze different machine learning techniques validity in automatic speech emotion recognition area. Using a bilingual affective database, different speech parameters have been calculated for each audio recording. Then, several machine learning techniques have been applied to evaluate their usefulness in speech emotion recognition, including techniques based on evolutive algorithms (EDA) to select speech feature subsets that optimize automatic emotion recognition success rate. Achieved experimental results show a representative increase in the success rate.KeywordsAffective computingMachine Learningspeech features extractionemotion recognition in speech

The paper concerns technology of automatic emotion recognition applied in e-learning environment. During a study of e-learning process the authors applied facial expressions observation via multiple video cameras. Preliminary analysis of the facial expressions using automatic emotion recognition tools revealed several unexpected results, including unavailability of recognition due to face coverage and significant inconsistency between the results obtained from two cameras. The paper presents the experiment on e-learning process and summarizes the observations that constitute limitations of emotion recognition from facial expressions applied in e-learning context. The paper might be of interest to researchers and practitioners who consider automatic emotion recognition as an option in monitoring e-learning processes.

This paper presents a novel approach to perform sentiment analysis of news videos, based on the fusion of audio, textual and visual clues extracted from their contents. The proposed approach aims at contributing to the semiodiscoursive study regarding the construction of the ethos (identity) of this media universe, which has become a central part of the modern-day lives of millions of people. To achieve this goal, we apply state-of-the-art computational methods for (1) automatic emotion recognition from facial expressions, (2) extraction of modulations in the participants' speeches and (3) sentiment analysis from the closed caption associated to the videos of interest. More specifically, we compute features, such as, visual intensities of recognized emotions, field sizes of participants, voicing probability, sound loudness, speech fundamental frequencies and the sentiment scores (polarities) from text sentences in the closed caption. Experimental results with a dataset containing 520 annotated news videos from three Brazilian and one American popular TV newscasts show that our approach achieves an accuracy of up to 84% in the sentiments (tension levels) classification task, thus demonstrating its high potential to be used by media analysts in several applications, especially, in the journalistic domain.

Automated emotion recognition (AEE) plays a crucial role in numerous industries that depend on understanding human emotional responses, such as advertising, technology, and human-robot interaction, particularly within the Information Technology (IT) field. However, current systems can often come up short in comprehensively understanding an individual's emotions, as prior research has mainly focused on assessing facial expressions and categorizing them into seven primary emotions, including neutrality. In this study, we present several Deep Convolutional Neural Network (CNN) models designed specifically for the task of facial emotion recognition, utilizing the FER2013 and RAF datasets. The baseline CNN model is established through a trial-and-error method, and its results are compared with more complex deep learning techniques, including ResNet18, VGGNet16, VGGNet19, and Efficient Net-B0 models. Among these models, the VGGNet19 model achieved the best results with a test accuracy of 71.02% on the FER2013 dataset. In comparison, the ResNet18 model outperformed all other models with an 86.02% test accuracy on the RAF-DB dataset. These results underscore the potential for advancing automated emotion recognition through complex deep-learning techniques.

T Human-Computer Interaction recently received an increased attention. This is not just a matter of making the operation of technical systems as simple as pissuble, but also to enable a possibly natural interaction. In this context, especially the speech-based operation gained an increased attention. For example, modern smart phones and televisions offer a robust voice control, which is attributed to various technical improvements in recent years. Nevertheless, voice control still seems artificial. Only self-contained dialogues with short statements can be managed. Furthermore, just the content of speech is evaluated. The way in which something is said remains unconsidered, although it is known from human communication, that the transmitted emotion is important in order to communicate successfully. A relatively new branch of research, the “Affective Computing”, has, amongst other objectives, the aim to develop technical systems that recognise and interpret emotions and respond to them appropriately. In this case, speech-based automatic emotion recognition has a major role. For emotion recognition, it is important to know how emotions can be presented and how they are expressed. For this purpose, it is helpful to rely on empirical evidences of the psychology of emotions. Unfortunately, there is no uniform representation of emotions. Also the definition of appropriate emotion-distinctive acoustic features is rather descriptive in psychology. Therefore, the automatic detection of emotions is based on proven methods of automatic speech recognition, which have also been shown as appropriate for emotion recognition. Automatic emotion recognition is, as speech recognition, a branch of pattern recognition. Contrary to emotion psychology, it is data-driven, that means insights are gathered from sampled data. For emotion recognition the phases “annotation”, “modelling” and “recognition” are distinguishable. The annotation categorises speech data according to predefined emotion-terms. Modelling generates recognisers to categorise data automatically. Recognition performs a previously unknown allocation of data to emotional classes. In the beginning, automatic emotion recognition was usually based – due to the lack of suitable data sets – on acted and very expressive emotional expressions. In this case, based on features and detection methods known from speech recognition, very good recognition results of over 80% in distinguishing of up to seven emotions could be achieved. However, for human-machine interaction these recognisers were unsuitable because in this case emotions are not that expressive. Therefore, in collaboration with

Emotion recognition using cross-modal attention from EEG and facial expression

Human emotions analysis has been the focus of many studies, especially in the field of Affective Computing, and is important for many applications, e.g. human-computer intelligent interaction, stress analysis, interactive games, animations, etc. Solutions for automatic emotion analysis have also benefited from the development of deep learning approaches and the availability of vast amount of visual facial data on the internet. This paper proposes a novel method for human emotion recognition from a single RGB image. We construct a largescale dataset of facial videos (FaceVid), rich in facial dynamics, identities, expressions, appearance and 3D pose variations. We use this dataset to train a deep Convolutional Neural Network for estimating expression parameters of a 3D Morphable Model and combine it with an effective back-end emotion classifier. Our proposed framework runs at 50 frames per second and is capable of robustly estimating parameters of 3D expression variation and accurately recognizing facial expressions from in the-wild images. We present extensive experimental evaluation that shows that the proposed method outperforms the compared techniques in estimating the 3D expression parameters and achieves state-of-the-art performance in recognising the basic emotions from facial images, as well as recognising stress from facial videos.

A survey of emotion recognition methods with emphasis on E-Learning environments

Emotion feature extraction is the key to speech emotional recognition. And ensemble empirical mode decomposition(EEMD) is a newly developed method aimed at eliminating emotion mode mixing present in the original empirical mode decomposition(EMD). To evaluate the performance of this new method, this paper investigates the effect of a parameters pertinent to EEMD: speech emotional envelope. Firstly, a speaker emotional envelope features extraction based on EEMD is proposed in the paper. Using the piecewise power function in speech emotional envelope has a better effect in emotional identification. At the same time, the proposed technique has been utilized for classification of fo ur kinds of emotional(angry, happy, sad and neutral) speech signals. Emotional intrinsic mode functions(IMFe) are obtained by empirical mode decomposition on emotional speech signals, the fast fourier transform(FFT) of each intrinsic mode function is extracte d as the emotional feature coefficient which is used in speaker emotional identification applying by vector quantization. MATLAB is used to ca lculate the characteristic of emotional speech signals using empirical mode decomposition (EEMD). We obtain an emotional envelope by transforming the IMFe of emotional speech signals, and obtain a new method of emotion recognition according to different emotional envelop feature vectors. The results indicate the proposed method works well in speaker emotional identification.

In this paper, we present a new method for emotion recognition from facial expressions. The proposed algorithm concentrates on only two specific areas (eyes and mouth), reducing features and descriptors and focusing only on these areas. The algorithm extracts characteristics from these two regions of the face and, in a subsequent process, eliminates the less significant characteristics or those that introduce noise into the classifier. The system allows obtaining a reduced set of features to improve the performance of the classification. In the experiments carried out, we obtain precisions of 99.56%. We evaluated the proposed algorithm on two benchmark datasets; we find that SVM consistently outperforms traditional machine learning techniques.

Automatic emotion recognition is a critical part of human-machine interactions. Reflection of emotions and to develop its understanding is crucial to provide dealings across human beings and machine frameworks. This work determines an automatic system that distinguishes different emotions connoted on the face. The framework is deliberated to apply the hybridization of feature extraction and optimization using PCA and PSO, respectively, to accomplish a high precision rate. PCA is used to get high-quality feature vectors for each category of emotion. Swarm intelligence, optimization is applied to get an optimized feature vector which is essential for classifying the features in the testing phase. For exploratory work, the authors have considered the Japanese Female Facial Expression (JAFFE) dataset. A maximum classification rate of 94.97% is achieved with the proposed technique. The proposed framework execution is assessed in terms of the false rejection rate, false acceptance rate, and accuracy.

Emotion identification plays a vital role in human interactions. For this purpose, Computer-vision methods for automatic emotion recognition is nowadays a widely studied topic. One of the most studied approaches for automatic emotion recognition is processing multi-channel Electroencephalogram signals (EEG). This paper presents a new model for emotion recognition using brain maps as input and providing emotion states in terms of arousal and valence as output. Brain maps are a spatial representation of features extracted from EEG signals. The proposed model, called Multi-Task Convolutional Neural Network (MT-CNN), is fed with stacked brain maps of four different waves of different frequency bands: alpha, beta, gamma and theta, using differential entropy and power spectra density and considering observation windows of 0.5s. This model is trained and tested on the DEAP dataset, a well-known dataset for comparison purposes. This work shows that the MT-CNN nerforms better than other methods.