Real time facial expression recognition with AdaBoost

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Automatic analysis of human facial expressions is one of the challenging problems in machine vision systems. It has many applications in human-computer interactions such as, social signal processing, social robots, deceit detection, interactive video and behavior monitoring. In this paper, we develop a new method for automatic facial expression recognition based on facial muscle anatomy and human face structure. The algorithm finds approximate location of effective facial muscles and extracts features by measuring skin texture in 11 local patches. Seven facial expressions, including neutral are being classified in this study using AdaBoost classifier and other classifiers on MMI databases. Experimental results show that analyzing skin texture from selected local patches gives accurate and efficient information in order to identify different facial expressions.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

There has been much research on computer vision in last three decades. Computer vision methods have been developed for different situations. One example is a detection of human face. For computers face detection is hard. Faces look different from different viewing directions. Facial expressions affect to the look of the face. Each individual person has a unique face. The lightning conditions can vary and so on.However, face detection is currently possible in limited conditions. In addition, there are some methods that can be used for gender recognition [3], face recognition [5] and facial expression recognition [2].Nonetheless, there has been very little research on how to combine these methods. There has also been quite little research on how to apply these methods in human-computer interaction (HCI).Finding sets of techniques that complement each other in a useful way is one research challenge. There are some applications that take advantage of one or two computer vision techniques. For example, Christian and Avery [1] have developed an information kiosk that uses computer vision to detect potential users from a distance. A similar kiosk has been developed by us in the University of Tampere [4]. There are also some games that use simple computer vision techniques for the interaction. However, there are very few applications that use several computer vision techniques together such as face detection, facial expression recognition and gender recognition. Overall, there has been very little effort in combining different techniques.In my research I develop computer vision methods and combine them, so that the combined method can detect face, recognize gender and facial expressions. After successfully combining the methods, it is easier to develop HCI applications that take advantage of computer vision. Applications that can be used by small group of people are my specific interest. These applications allow me to build adaptive user interfaces and analyze the use of computer vision techniques in improving human-computer interaction.

This paper proposes a method of facial expression recognition based on Zernike moments and the minimum classification error (MCE) based hidden Markov model (HMM). In the feature extraction of face, the method of Zernike moments feature extraction based on local feature regions is adopted. First, eyes and mouths are segmented from the facial expression image and Zernike moments feature vectors of eyes and mouth are extracted. In the classification of expression, using MCE criteria is adopted to improve the performance of HMM. The results show that the performance of expression recognition system can be improved to some extent by using Zernike moments and MCE criterion.

Introduction To date, most ways to perform facial expression recognition rely on two-dimensional images, advanced approaches with three-dimensional data exist. These however demand stationary apparatuses and thus lack portability and possibilities to scale deployment. As human emotions, intent and even diseases may condense in distinct facial expressions or changes therein, the need for a portable yet capable solution is signified. Due to the superior informative value of three-dimensional data on facial morphology and because certain syndromes find expression in specific facial dysmorphisms, a solution should allow portable acquisition of true three-dimensional facial scans in real time. In this study we present a novel solution for the three-dimensional acquisition of facial geometry data and the recognition of facial expressions from it. The new technology presented here only requires the use of a smartphone or tablet with an integrated TrueDepth camera and enables real-time acquisition of the geometry and its categorization into distinct facial expressions. Material and Methods Our approach consisted of two parts: First, training data was acquired by asking a collective of 226 medical students to adopt defined facial expressions while their current facial morphology was captured by our specially developed app running on iPads, placed in front of the students. In total, the list of the facial expressions to be shown by the participants consisted of “disappointed”, “stressed”, “happy”, “sad” and “surprised”. Second, the data were used to train a self-normalizing neural network. A set of all factors describing the current facial expression at a time is referred to as “snapshot”. Results In total, over half a million snapshots were recorded in the study. Ultimately, the network achieved an overall accuracy of 80.54% after 400 epochs of training. In test, an overall accuracy of 81.15% was determined. Recall values differed by the category of a snapshot and ranged from 74.79% for “stressed” to 87.61% for “happy”. Precision showed similar results, whereas “sad” achieved the lowest value at 77.48% and “surprised” the highest at 86.87%. Conclusions With the present work it can be demonstrated that respectable results can be achieved even when using data sets with some challenges. Through various measures, already incorporated into an optimized version of our app, it is to be expected that the training results can be significantly improved and made more precise in the future. Currently a follow-up study with the new version of our app that encompasses the suggested alterations and adaptions, is being conducted. We aim to build a large and open database of facial scans not only for facial expression recognition but to perform disease recognition and to monitor diseases’ treatment progresses.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

Facial expression recognition is a practical application in the field of artificial intelligence. This article focuses on the introduction of the principles of each facial expression recognition method based on deep learning in chronological order, as well as the advantages and disadvantages of each method. The methods of facial expression recognition based on deep learning introduced in this article are as follows: Li Dunyu [1] proposed a method of LBP and BP neural network to realize facial expression recognition. And then the convolutional neural network is widely used, Zhou Benjun [2] merged LBP and convolutional neural network to realize facial expression recognition; Liu [3] establish a model that combines a recurrent neural network with a 3D convolutional neural network, and added a timeline so that the machine can extract and recognize facial expressions in a video; Jung [4] proposed to combine the convolutional neural network and the deep neural network to form a new neural network DTAGN, by using this new network he completes facial expression recognition; Shi Xugan [5] proposed a model of combining deep belief network with multi-layer perceptron (MLP), and successfully applied it to the research of facial expression recognition. By utilizing joint training of feature points and multiple networks Xia Tian [6] finish the research; Zhang Feifei [7] and other scholars proposed multi-pose facial expression recognition based on generating an adversarial network.

Facial expression recognition is a rapidly growing area of research in computer vision due to its immense applications such as human-computer interaction, psychology and intelligent robotic systems etc. Various methods have been proposed in past two decades aiming to solve different issues in this aspect. Most of the researches used distinct databases for the evaluation of their techniques, so it is very challenging task to infer which one is superior. In this paper, we have implemented three methods of facial expression recognition named: Principal Component Analysis (PCA), Artificial Neural Network (ANN) and Local Binary Pattern (LBP) and evaluated their performances using JAFFE database. Keywords: Facial Expression Recognition, Emotion Recognition, JAFFE Dataset, PCA, LBP, ANN.

A Facial Expression Recognition (FER) method based on Conv3D-ConvLSTM-SEnet in an online education environment is proposed to address the issue of low accuracy in current classroom expression recognition (FER) methods. Firstly, ConvLSTM is used to integrate the local feature extraction ability of CNN and the temporal modeling ability of LSTM, and based on this, the rich spatial features of the image are characterized. Then, by introducing Depth Separated Convolution (DSC) to change the number of output channels and separate each channel, the Feature Maps (F-M) are sequentially concatenated to obtain a multi-channel output F-M. Finally, based on ConvLSTM and SEnet modules, a Conv3D ConvLSTM-based FER model was proposed by redistributing the extracted abstract features and basic texture features, achieving high accuracy classroom FER. The proposed classroom FER method and the other five methods were compared and analyzed through simulation experiments using the CK[Formula: see text], FER2013 and JAFFE datasets, respectively. The results indicate that the proposed method achieves the highest accuracy, precision, recall, and F1 score, with improvements of at least 1.85%, 2.41%, 1.18% and 2.05%, respectively, compared to the other five methods on the FER2013 dataset. The proposed method can perform FER on students in online classroom teaching, detect their emotions in real-time, and help teachers make better course teaching adjustments based on this.

Face expression recognition plays a crucial role in emotion recognition, human-computer interaction and other fields. The aim of this paper is to implement a deep learning-based monitoring system for real-time facial expression recognition. In this paper, a public expression dataset ExpW is used, which is a widely used public face expression dataset. The dataset contains face images from various scenarios in the real world, covering a rich variety of expressions, encompassing seven major expression categories: anger, disgust, fear, happiness, sadness, surprise, and neutrality. The YOLOv5 target detection algorithm is selected on the deep learning framework for the training and testing of the training set, which is a high-performance algorithm for fast and accurate implementation of the target detection task through a simplified network structure and optimized algorithm design. The experimental evaluation results can be concluded that the system in this paper achieves accurate expression recognition in real-time scenarios. The results of this paper show that real-time face expression recognition monitoring based on deep learning has potential applications in emotion recognition and human-computer interaction.

Facial expressions recognition plays important role in human communication. It has become one of the most challenging tasks in the pattern recognition field. It has many applications such as: human computer interaction, video surveillance, forensic applications, criminal investigations, and in many other fields. In this paper we propose a method for facial expression recognition (FER). This method provides new insights into two issues in FER: feature extraction and robustness. For feature extraction we are using sparse representation approach after applying multiple Gabor filter and then using support vector machine (SVM) as classifier. We conduct extensive experiments on standard facial expressions database to verify the performance of proposed method. And we compare the result with other approach.

This paper presents an analysis of hand-crafted and learned feature extraction methods for real-time facial expression recognition (FER). Our analysis focuses on methods capable of running on mobile devices, including traditional algorithms such as Gabor transform, HOG, LBP, as well as two compact CNN models, named Mobilenet V1 and V2. Additionally, we test the performance of MOTIF, a highly efficient texture feature extractor algorithm. Furthermore, we analyze the contribution of the mouth and front-eyes regions for recognizing the seven basic facial expressions. Experimental results are evaluated on two publicly available datasets. KDEF database which was captured under controlled conditions and RAF database which represents more naturalistic expressions captured in-the-wild. Under the same experimental conditions, MOTIF presents the fastest performance by sacrificing accuracy, while Mobilenet V2 presents the highest results with considerable speed and model size.

ABSTRACTHuman beings show multiple responses to various emotional states, like anger, disgust, surprise, happiness, sadness, and fear. Among various emotions, facial expressions are widely informative as they exhibit a person's intentions and character. Facial expression recognition is used in many applications, such as marketing, research, customer service, neuroscience, and psychology. Traditional unimodal methods for facial expression recognition are ineffective due to the scarcity of data. In this paper, the Residual Fused Neuro‐Fuzzy Network (RF‐NFN) is used for the recognition of facial expressions and detection of emotion type using video and Electroencephalogram (EEG) signals. Here, the video frame is allowed for pre‐processing done by Non‐Local Means (NLM) filtering. Then, pre‐processed video frames and input EEG signals are fed toward feature extraction, which is then followed by feature selection. Finally, facial expressions are recognized and the type of emotion is detected by RF‐NFN, which is designed by incorporation of Hybrid Cascade Neuro‐Fuzzy Network (Hybrid Cascade NFN) and Deep Residual Network (DRN). Moreover, the performance of the RF‐NFN model is validated by three performance measures that exhibited a maximum accuracy of 90.88%, precision of 91.77%, and recall of 94.57%.