Edge texture feature extraction and expression recognition based on curvelet

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

This paper proposes a method of facial expression recognition based on Zernike moments and the minimum classification error (MCE) based hidden Markov model (HMM). In the feature extraction of face, the method of Zernike moments feature extraction based on local feature regions is adopted. First, eyes and mouths are segmented from the facial expression image and Zernike moments feature vectors of eyes and mouth are extracted. In the classification of expression, using MCE criteria is adopted to improve the performance of HMM. The results show that the performance of expression recognition system can be improved to some extent by using Zernike moments and MCE criterion.

Texture characteristics among the landmark points reflected in human faces are important features in the recognition of facial expression. In this paper, we propose a salient feature based facial expression on texture characteristics. Detection of effective landmarks are performed by appearance based models and corresponding texture regions are extracted from face images. Local Binary Pattern (LBP) is used to compute the texture feature. Texture feature is used for normalizing texture signatures. In addition, we introduce stability indices corresponding to texture features which are exploited as features in the present scope. Moreover, statistical features such as moment, skewness, kurtosis and entropy are also supplemented to the feature set. These salient features together are used as an input to Nonlinear AutoRegressive with eXogenous (NARX) for recognition of the human facial expression. The Cohn-Kanade (CK+), Japanese Female Facial Expression (JAFFE), MMI and MUG benchmark databases are used to conduct the experiments and the results justify the effectiveness of the proposed procedure.

A Facial Expression Recognition (FER) method based on Conv3D-ConvLSTM-SEnet in an online education environment is proposed to address the issue of low accuracy in current classroom expression recognition (FER) methods. Firstly, ConvLSTM is used to integrate the local feature extraction ability of CNN and the temporal modeling ability of LSTM, and based on this, the rich spatial features of the image are characterized. Then, by introducing Depth Separated Convolution (DSC) to change the number of output channels and separate each channel, the Feature Maps (F-M) are sequentially concatenated to obtain a multi-channel output F-M. Finally, based on ConvLSTM and SEnet modules, a Conv3D ConvLSTM-based FER model was proposed by redistributing the extracted abstract features and basic texture features, achieving high accuracy classroom FER. The proposed classroom FER method and the other five methods were compared and analyzed through simulation experiments using the CK[Formula: see text], FER2013 and JAFFE datasets, respectively. The results indicate that the proposed method achieves the highest accuracy, precision, recall, and F1 score, with improvements of at least 1.85%, 2.41%, 1.18% and 2.05%, respectively, compared to the other five methods on the FER2013 dataset. The proposed method can perform FER on students in online classroom teaching, detect their emotions in real-time, and help teachers make better course teaching adjustments based on this.

Many Facial expression recognition methods have achieved great success, but they only considered front facial images or facial images close to the front. Besides, unlike in-the-laboratory datasets, the facial images in the real world (or in the wild) are without lighting and pose control, so that it is a big challenge to recognize these facial expressions. In this paper, the PIDViT (i.e., Pose-Invariant Distilled Vision Transformer) using the teacher-student model for the probability distributions of facial expressions of frontal and multi-pose faces was proposed and solved the pose variance and occlusion issues on expression recognition. First, the multi-pose face dataset FairFace-3D was built from the original FairFace and then used to train pose-invariance on the PIDViT. The PIDViT was trained in two stages; stage 1 is to train the PIDViT to achieve the consistency of facial expressions between frontal faces and multi-pose faces, and stage 2 is to use the student model pre-trained in stage 1 and train facial expressions further on target datasets. Finally, comprehensive experiments were conducted on three in the wild facial expression datasets, and the results validates the generalization of the PIDViT and its superiority over most state-of-the-art models.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

Introduction To date, most ways to perform facial expression recognition rely on two-dimensional images, advanced approaches with three-dimensional data exist. These however demand stationary apparatuses and thus lack portability and possibilities to scale deployment. As human emotions, intent and even diseases may condense in distinct facial expressions or changes therein, the need for a portable yet capable solution is signified. Due to the superior informative value of three-dimensional data on facial morphology and because certain syndromes find expression in specific facial dysmorphisms, a solution should allow portable acquisition of true three-dimensional facial scans in real time. In this study we present a novel solution for the three-dimensional acquisition of facial geometry data and the recognition of facial expressions from it. The new technology presented here only requires the use of a smartphone or tablet with an integrated TrueDepth camera and enables real-time acquisition of the geometry and its categorization into distinct facial expressions. Material and Methods Our approach consisted of two parts: First, training data was acquired by asking a collective of 226 medical students to adopt defined facial expressions while their current facial morphology was captured by our specially developed app running on iPads, placed in front of the students. In total, the list of the facial expressions to be shown by the participants consisted of “disappointed”, “stressed”, “happy”, “sad” and “surprised”. Second, the data were used to train a self-normalizing neural network. A set of all factors describing the current facial expression at a time is referred to as “snapshot”. Results In total, over half a million snapshots were recorded in the study. Ultimately, the network achieved an overall accuracy of 80.54% after 400 epochs of training. In test, an overall accuracy of 81.15% was determined. Recall values differed by the category of a snapshot and ranged from 74.79% for “stressed” to 87.61% for “happy”. Precision showed similar results, whereas “sad” achieved the lowest value at 77.48% and “surprised” the highest at 86.87%. Conclusions With the present work it can be demonstrated that respectable results can be achieved even when using data sets with some challenges. Through various measures, already incorporated into an optimized version of our app, it is to be expected that the training results can be significantly improved and made more precise in the future. Currently a follow-up study with the new version of our app that encompasses the suggested alterations and adaptions, is being conducted. We aim to build a large and open database of facial scans not only for facial expression recognition but to perform disease recognition and to monitor diseases’ treatment progresses.

In this paper, we propose a novel method for facial expression recognition. The facial expression is extracted from human faces by an expression classifier that is learned from boosting Haar feature based look-up-table type weak classifiers. The expression recognition system consists of three modules, face detection, facial feature landmark extraction and facial expression recognition. The implemented system can automatically recognize seven expressions in real time that include anger, disgust, fear, happiness, neutral, sadness and surprise. Experimental results are reported to show its potential applications in human computer interaction.

ABSTRACTHuman beings show multiple responses to various emotional states, like anger, disgust, surprise, happiness, sadness, and fear. Among various emotions, facial expressions are widely informative as they exhibit a person's intentions and character. Facial expression recognition is used in many applications, such as marketing, research, customer service, neuroscience, and psychology. Traditional unimodal methods for facial expression recognition are ineffective due to the scarcity of data. In this paper, the Residual Fused Neuro‐Fuzzy Network (RF‐NFN) is used for the recognition of facial expressions and detection of emotion type using video and Electroencephalogram (EEG) signals. Here, the video frame is allowed for pre‐processing done by Non‐Local Means (NLM) filtering. Then, pre‐processed video frames and input EEG signals are fed toward feature extraction, which is then followed by feature selection. Finally, facial expressions are recognized and the type of emotion is detected by RF‐NFN, which is designed by incorporation of Hybrid Cascade Neuro‐Fuzzy Network (Hybrid Cascade NFN) and Deep Residual Network (DRN). Moreover, the performance of the RF‐NFN model is validated by three performance measures that exhibited a maximum accuracy of 90.88%, precision of 91.77%, and recall of 94.57%.

In order to achieve high recognition rate, most facial expression recognition (FER) methods generate sufficient labeled facial images based on generative adversarial networks (GAN) to train model. However, these methods do not estimate the facial pose before passing the images to the generator, which affects the quality of generated images. And mode collapse is prone to occur during the training process, leading to generate a single-style facial images. To solve these problems, a FER model is proposed based on pose conditioned dendritic convolution neural network (PCD-CNN) with pose and expression. Before passing the facial images to the generator, PCD-CNN was used to process facial images, effectively estimating the facial landmarks to detect face and disentangle the pose. In order to accelerate the training speed of the model, PCD-CNN was based on the ShuffleNet-v2 framework. Every landmark of facial image was modeled by a separate ShuffleNet-DeconvNet, maintaining better performance with fewer parameters. To solve the mode collapse during image generation, we theoretically analyzed the causes, and implemented mini-batch processing on the discriminator in the model and directly calculated the statistical characteristics of the mini-batch samples. Experiments were carried out on the Multi-PIE and BU-3DFE facial expression datasets. Compared with current advanced methods, our method achieves higher accuracy 93.08%, and the training process is more stable.

Facial expression recognition is a practical application in the field of artificial intelligence. This article focuses on the introduction of the principles of each facial expression recognition method based on deep learning in chronological order, as well as the advantages and disadvantages of each method. The methods of facial expression recognition based on deep learning introduced in this article are as follows: Li Dunyu [1] proposed a method of LBP and BP neural network to realize facial expression recognition. And then the convolutional neural network is widely used, Zhou Benjun [2] merged LBP and convolutional neural network to realize facial expression recognition; Liu [3] establish a model that combines a recurrent neural network with a 3D convolutional neural network, and added a timeline so that the machine can extract and recognize facial expressions in a video; Jung [4] proposed to combine the convolutional neural network and the deep neural network to form a new neural network DTAGN, by using this new network he completes facial expression recognition; Shi Xugan [5] proposed a model of combining deep belief network with multi-layer perceptron (MLP), and successfully applied it to the research of facial expression recognition. By utilizing joint training of feature points and multiple networks Xia Tian [6] finish the research; Zhang Feifei [7] and other scholars proposed multi-pose facial expression recognition based on generating an adversarial network.

Objectives: Attention-deficit/hyperactivity disorder (ADHD) is the most frequent neurobehavioural disorder in childhood. ADHD is associated with impaired academic performance, cognitive, and emotional deficits. Moreover, comorbid oppositional defiant disorder (ODD) is leading to more severe impairment in social performance. Social cognition involves recognition, encoding, and interpretation of emotions from faces. Basic facial expressions that include sadness, happiness, anger, disgust, fear, and surprise are the easiest emotions to recognize. We aimed to demonstrate facial expression recognition impairments that might occur more frequently in children with co-occurring ADHD/ODD than patients with ADHD only. Thus, children with the co-occurrence of ODD may suffer more severely from social and behavioural difficulties.Methods: Forty patients diagnosed with ADHD and/or co-occurring ADHD/ODD according to DSM-IV-TR criteria were compared with a parallel (by gender, age, and educational state) 11 healthy children as a control group in this study. Clear facial images of each emotion were used as well as two additional sets of photos include 50% blurred images and cropped eye images were added as distractors then all images represented with black and white tone for emotion recognition task via facial expression. Angry expressions presented as target expressions. DSM-IV-Based Screening and Rating Scale for Children and Adolescents with attention deficit and disruptive behaviour disorders, the Conners’ Teachers Rating Scale/Revised Long Form and the Conners’ Parent Rating Scale/Revised Long Form were used to provide diagnostic objectivity.Results: Control group statistically performed better than ADHD group on recognition of emotional facial expressions. Results showed no statistically significant differences between the ADHD and ADHD/ODD group on recognition of emotional facial expressions. However, according to results of emotion recognition task via facial expressions, there were statistically significant differences between pure ADHD and comorbid ADHD+ODD groups in happy and neutral expressions. ADHD/ODD group tend to attribute more meaning to neutral facial expressions. Additionally there was statistically significant difference between control group and ADHD group according to recognition of angry expressions. There were statistically significant differences between the groups according to recognition of sad expressions in all clear, blurred, and eye photographs.Conclusions: Difficulties in recognizing emotional facial expressions were observed in children with ADHD. A statistically significant association was established between presence of ADHD and impaired recognition of facial emotion expressions independent from the scores of the disruptive behaviour rating scale. Comorbid ODD was not associated with recognition of emotional facial expressions including angry expressions. Recognition of angry expressions was not found as a predictor for disruptive behaviour disorders.

Facial expression recognition is a challenging and interesting problem. It has many potential and important applications in data-driven animation, human computer interaction (HCI), social robots, deceit detection and behavior monitoring. In this paper, we present the novel descriptors Pyramid local phase quantization (PLPQ). The effective of our proposed descriptor is evaluated by facial expressions recognition very efficiently and with high accuracy. On the other hand, the proposed framework extracts texture features in a pyramidal fashion only from the perceptual salient region of the face thereby our proposed framework achieved reduction in computation time of feature extraction and improved accuracy. There with the proposed framework achieved accuracy of 96.7% on extended Cohn-Kanade (CK+) posed facial expression database for six basic emotions and exceed the state-of-theart methods for expression recognition using texture features.