Spatio-temporal Analysis for Infrared Facial Expression Recognition from Videos

Facial expression recognition is a practical application in the field of artificial intelligence. This article focuses on the introduction of the principles of each facial expression recognition method based on deep learning in chronological order, as well as the advantages and disadvantages of each method. The methods of facial expression recognition based on deep learning introduced in this article are as follows: Li Dunyu [1] proposed a method of LBP and BP neural network to realize facial expression recognition. And then the convolutional neural network is widely used, Zhou Benjun [2] merged LBP and convolutional neural network to realize facial expression recognition; Liu [3] establish a model that combines a recurrent neural network with a 3D convolutional neural network, and added a timeline so that the machine can extract and recognize facial expressions in a video; Jung [4] proposed to combine the convolutional neural network and the deep neural network to form a new neural network DTAGN, by using this new network he completes facial expression recognition; Shi Xugan [5] proposed a model of combining deep belief network with multi-layer perceptron (MLP), and successfully applied it to the research of facial expression recognition. By utilizing joint training of feature points and multiple networks Xia Tian [6] finish the research; Zhang Feifei [7] and other scholars proposed multi-pose facial expression recognition based on generating an adversarial network.

Researchers pay much attention to facial expressions recognition due to the rapid development of Artificial Intelligence. Facial expression recognition is used to help human-computer interaction. In addition, facial expression recognition is also used in psychological recognition, Human-computer interaction, assisted driving, and security station in everyday life. But most of the research focused on the machine learning approach rather than deep learning and the emotion classifications are also smaller. This facial expression recognition can be implemented using a deep learning approach. The architecture that is often used and considered to be the best in image classification is Convolutional Neural Network. Therefore, this study builds a Convolutional Neural Network Model with Xception and DenseNet architecture. The accuracy of the two models is compared, with Xception received an accuracy of 70% and DenseNet got 79%.

Facial expression recognition (FER) techniques can be widely used in human-computer interaction, intelligent robots, intelligent monitoring, and other domains. Currently, FER methods based on deep learning have become the mainstream schemes. However, these methods have some problems, such as a large number of parameters, difficulty in being applied to embedded processors, and the fact that recognition accuracy is affected by facial deflection. To solve the problem of a large number of parameters, we propose a DSC-DenseNet model, which improves the standard convolution in DenseNet to depthwise separable convolution (DSC). To solve the problem wherein face deflection affects the recognition effect, we propose a posture normalization model based on GAN: a GAN with two local discriminators (LD-GAN) that strengthen the discriminatory abilities of the expression-related local parts, such as the parts related to the eyes, eyebrows, mouth, and nose. These discriminators improve the model’s ability to retain facial expressions and evidently benefits FER. Quantitative and qualitative experimental results on the Fer2013 and KDEF datasets have consistently shown the superiority of our FER method when working with multi-pose face images.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition

Deep reinforcement learning for robust emotional classification in facial expression recognition

Facial expression recognition has been applied in many fields such as human computer interaction, patient monitoring, neurology, social robot… Although facial expression recognition has gained some encourage results, there are still many challenges such as the change of illumination, blur, etc. Especially, recognizing between sadness and anger expression raises more barriers. In this paper, we proposed a framework using only salient facial regions, but it would be able to improve the accuracy of facial expression recognition. In this framework, one of the most state-of-the-art descriptor, called Pyramid of Local Phase Quantization descriptor (PLPQ) was used to robust with respect to image blur. The experiment achieved 97.7% accuracy recognition rate on the extend Cohn-Canade (CK+) database, and outperformed than other state-of-the-art facial expression recognition methods.

Facial expression recognition is a rapidly growing area of research in computer vision due to its immense applications such as human-computer interaction, psychology and intelligent robotic systems etc. Various methods have been proposed in past two decades aiming to solve different issues in this aspect. Most of the researches used distinct databases for the evaluation of their techniques, so it is very challenging task to infer which one is superior. In this paper, we have implemented three methods of facial expression recognition named: Principal Component Analysis (PCA), Artificial Neural Network (ANN) and Local Binary Pattern (LBP) and evaluated their performances using JAFFE database. Keywords: Facial Expression Recognition, Emotion Recognition, JAFFE Dataset, PCA, LBP, ANN.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Background/Objectives: There is only limited research work is going on in the field of facial expression recognition on low resolution images. Mostly, all the images in the real world will be in low resolution and might also contain noise, so this study is to design a novel convolutional neural network model (FERConvNet), which can perform better on low resolution images. Methods: We proposed a model and then compared with state-of-art models on FER2013 dataset. There is no publicly available dataset, which contains low resolution images for facial expression recognition (Anger, Sad, Disgust, Happy, Surprise, Neutral, Fear), so we created a Low Resolution Facial Expression (LRFE) dataset, which contains more than 6000 images of seven types of facial expressions. The existing FER2013 dataset and LRFE dataset were used. These datasets were divided in the ratio 80:20 for training and testing and validation purpose. A HDM is proposed, which is a combination of Gaussian Filter, Bilateral Filter and Non local means denoising Filter. This hybrid denoising method helps us to increase the performance of the convolutional neural network. The proposed model was then compared with VGG16 and VGG19 models. Findings: The experimental results show that the proposed FERConvNet_HDM approach is effective than VGG16 and VGG19 in facial expression recognition on both FER2013 and LRFE dataset. The proposed FERConvNet_HDM approach achieved 85% accuracy on Fer2013 dataset, outperforming the VGG16 and VGG19 models, whose accuracies are 60% and 53% on Fer2013 dataset respectively. The same FERConvNet_HDM approach when applied on LRFE dataset achieved 95% accuracy. After analyzing the results, our FERConvNet_HDM approach performs better than VGG16 and VGG19 on both Fer2013 and LRFE dataset. Novelty/Applications: HDM with convolutional neural networks, helps in increasing the performance of convolutional neural networks in Facial expression recognition. Keywords: Facial expression recognition; facial emotion; convolutional neural network; deep learning; computer vision

Facial expression recognition (FER) plays a vital role in the field of human-computer interaction. To achieve automatic FER, various approaches based on deep learning (DL) have been presented. However, most of them lack for the extraction of discriminative expression semantic information and suffer from the problem of annotation ambiguity. In this paper, we propose an elaborately designed end-to-end recognition network with contrastive learning and uncertainty-guided relabeling, to recognize facial expressions efficiently and accurately, as well as to alleviate the impact of annotation ambiguity. Specifically, a supervised contrastive loss (SCL) is introduced to promote inter-class separability and intra-class compactness, thus helping the network extract fine-grained discriminative expression features. As for the annotation ambiguity problem, we present an uncertainty estimation-based relabeling module (UERM) to estimate the uncertainty of each sample and relabel the unreliable ones. In addition, to deal with the padding erosion problem, we embed an amending representation module (ARM) into the recognition network. Experimental results on three public benchmarks demonstrate that our proposed method facilitates the recognition performance remarkably with 90.91% on RAF-DB, 88.59% on FERPlus and 61.00% on AffectNet, outperforming current state-of-the-art (SOTA) FER methods. Code will be available at http//github.com/xiaohu-run/fer_supCon .

Facial expression recognition based on Local Binary Patterns: A comprehensive study

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

In this paper, we propose a novel method for facial expression recognition. The facial expression is extracted from human faces by an expression classifier that is learned from boosting Haar feature based look-up-table type weak classifiers. The expression recognition system consists of three modules, face detection, facial feature landmark extraction and facial expression recognition. The implemented system can automatically recognize seven expressions in real time that include anger, disgust, fear, happiness, neutral, sadness and surprise. Experimental results are reported to show its potential applications in human computer interaction.

Face expression recognition plays a crucial role in emotion recognition, human-computer interaction and other fields. The aim of this paper is to implement a deep learning-based monitoring system for real-time facial expression recognition. In this paper, a public expression dataset ExpW is used, which is a widely used public face expression dataset. The dataset contains face images from various scenarios in the real world, covering a rich variety of expressions, encompassing seven major expression categories: anger, disgust, fear, happiness, sadness, surprise, and neutrality. The YOLOv5 target detection algorithm is selected on the deep learning framework for the training and testing of the training set, which is a high-performance algorithm for fast and accurate implementation of the target detection task through a simplified network structure and optimized algorithm design. The experimental evaluation results can be concluded that the system in this paper achieves accurate expression recognition in real-time scenarios. The results of this paper show that real-time face expression recognition monitoring based on deep learning has potential applications in emotion recognition and human-computer interaction.