Texture signature based facial expression recognition using NARX

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

The recognition rate of person-independent facial expression is generally not high, which limits the practical application of facial expression recognition. Aiming at this problem, this paper analyzes the reasons for the low recognition rate of person-independent facial expression, and proposes a recognition algorithm of person-independent facial expression based on improved LBP (Local Binary Pattern) and HOSVD (Higher-Order Singular Value Decomposition). The algorithm has the following contributions of facial expression recognition framework. In the stage of facial expression feature extraction, the transient features extracted by LDP(Local Directional Pattern) and the persistent features extracted by CBP(Centralized Binary Pattern) are integrated to improve the discrimination of facial expression features. Moreover, in the stage of facial expression classification and recognition, the traditional nearest neighbor classification is changed into k-nearest neighbor pre-classification, and the regional energy calculated by HOSVD is used to determine the similarity of two images for secondary classification. Finally, in the extended Cohn-Kanade dataset and Oulu-CASIA NIR&VIS facial expression database, the theoretical analysis and experimental results show that the method has better recognition effect for solving the problem of person-independent facial expression recognition.

There have been quite a few studies on facial expression recognition over the years, and it is still a challenging subject due to the significant inter-class variability. Facial expression research in this field focuses on the development of techniques to identify, code, and extract facial expressions to improve prediction by computer. With great success of machine learning, the various texture descriptors are exploited to obtain a better performance. This paper proposes a method based on the aggregation between different descriptors Histogram of oriented Gradient (HOG) and Local Binary Pattern (LBP). First stage the input image has pre-processed to detect dace area which helps to extract most significant features. Then, Diagonal-HOG (D-HOG) also has extracted and aggregated all features. Finally, Support Vector Machine (SVM) has been used a classifier to classify each feature as well as aggregated features. We evaluate our method using Japanese Female Facial Expressions database (JAFFE), experimental results showed that the proposed method is accurate and efficient in recognizing facial expressions.

Shape and texture features provide a novel framework for expression recognition by using salient landmarks based triangle initialisation and identification of texture regions. Detection of effective landmarks is achieved by well‐known appearance‐based model to form a grid. Accordingly, several triangles are identified within the grid with respect to the nose landmark reference point. These salient landmarks are also used to find the texture regions. Normalised shape and texture signatures are derived from triangles and texture regions. Stability indices are determined from shape and texture signatures which are also exploited as significant features for recognition of facial expressions. Statistical measures such as range, moment, skewness, kurtosis, and entropy are used to supplement the feature set. Individual and combined features are fed into multilayer perceptron and deep belief network (DBN) network to classify different expression categories encompassing anger, sadness, fear, disgust, surprise, and happiness. The authors investigated the performance of their proposed system on Cohn–Kanade, JAFFE, MMI, MUG, and Wild face benchmark databases. Thorough experimentation establishes the performance superiority of the proposed methodology over other existing competitors. Combining features is also validated through extensive results and analyses.

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

Objectives: Attention-deficit/hyperactivity disorder (ADHD) is the most frequent neurobehavioural disorder in childhood. ADHD is associated with impaired academic performance, cognitive, and emotional deficits. Moreover, comorbid oppositional defiant disorder (ODD) is leading to more severe impairment in social performance. Social cognition involves recognition, encoding, and interpretation of emotions from faces. Basic facial expressions that include sadness, happiness, anger, disgust, fear, and surprise are the easiest emotions to recognize. We aimed to demonstrate facial expression recognition impairments that might occur more frequently in children with co-occurring ADHD/ODD than patients with ADHD only. Thus, children with the co-occurrence of ODD may suffer more severely from social and behavioural difficulties.Methods: Forty patients diagnosed with ADHD and/or co-occurring ADHD/ODD according to DSM-IV-TR criteria were compared with a parallel (by gender, age, and educational state) 11 healthy children as a control group in this study. Clear facial images of each emotion were used as well as two additional sets of photos include 50% blurred images and cropped eye images were added as distractors then all images represented with black and white tone for emotion recognition task via facial expression. Angry expressions presented as target expressions. DSM-IV-Based Screening and Rating Scale for Children and Adolescents with attention deficit and disruptive behaviour disorders, the Conners’ Teachers Rating Scale/Revised Long Form and the Conners’ Parent Rating Scale/Revised Long Form were used to provide diagnostic objectivity.Results: Control group statistically performed better than ADHD group on recognition of emotional facial expressions. Results showed no statistically significant differences between the ADHD and ADHD/ODD group on recognition of emotional facial expressions. However, according to results of emotion recognition task via facial expressions, there were statistically significant differences between pure ADHD and comorbid ADHD+ODD groups in happy and neutral expressions. ADHD/ODD group tend to attribute more meaning to neutral facial expressions. Additionally there was statistically significant difference between control group and ADHD group according to recognition of angry expressions. There were statistically significant differences between the groups according to recognition of sad expressions in all clear, blurred, and eye photographs.Conclusions: Difficulties in recognizing emotional facial expressions were observed in children with ADHD. A statistically significant association was established between presence of ADHD and impaired recognition of facial emotion expressions independent from the scores of the disruptive behaviour rating scale. Comorbid ODD was not associated with recognition of emotional facial expressions including angry expressions. Recognition of angry expressions was not found as a predictor for disruptive behaviour disorders.

In this paper, we proposed a novel algorithm for Facial Expression Recognition (FER) which was based on fusion of gabor texture features and Local Phase Quantization (LPQ). Firstly, the LPQ feature and gabor texture feature were respectively extracted from every expression image. LPQ features are histograms of LPQ transform. Five scales and eight orientations of gabor wavelet filters are used to extract gabor texture features and adaboost algorithm is used to select gabor features. Then we obtain two expression recognition results on both expression features by Sparse Representation-based Classification (SRC) method. Finally, the final expression recognition was performed by fusion of residuals of two SRC algorithms. The experiment results on Japanese Female Facial Expression (JAFFE) database demonstrated that the new algorithm was better than the original two algorithms, and this algorithm had a much higher recognition rate than the traditional algorithm.

Facial expression recognition is one of the most important tasks in human–computer interaction, affective computing, computer vision, and related work. Feature additive pooling and progressive fine‐tuning of the convolutional neural network (CNN) for facial expression recognition in a static image are introduced. Network is proposed that partially employs the visual geometry group (VGG)‐face model pre‐trained on a VGG‐face dataset. The characteristics and distribution of the facial expression images in each database are biased according to the purpose of the publicly available facial database used. To alleviate this problem, a CNN model is developed that merges progressively fine‐tuned CNNs into a single network. Experiments were carried out to validate the presented method using facial expression images from the Cohn–Kanade +, Karolinska directed emotional face, and Japanese female facial expression databases, and cross‐database evaluation results show that the method is superior to state‐of‐the‐art methods.

Recently, facial expression recognition has attracted much attention in machine vision research because of its various applications. Accordingly, many facial expression recognition systems have been proposed. However, the majority of existing systems suffer from a critical problem: geometric variability. It directly affects the performance of geometric feature-based facial expression recognition approaches. Furthermore, it is a crucial challenge in appearance feature-based techniques. This variability appears in both neutral faces and facial expressions. Appropriate face geometry normalization can improve the accuracy of each facial expression recognition system. Therefore, this paper proposes different geometric models or shapes for normalization. Face geometry normalization removes geometric variability of facial images and consequently, appearance feature extraction methods can be accurately utilized to represent facial images. Thus, some expression-based geometric models are proposed for facial image normalization. Next, local binary patterns and local phase quantization are used for appearance feature extraction. A combination of an effective geometric normalization with accurate appearance representations results in more than a 4% accuracy improvement compared to several state-of-the-arts in facial expression recognition. Moreover, utilizing the model of facial expressions which have larger mouth and eye region sizes gives higher accuracy due to the importance of these regions in facial expression.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

Automated recognition of facial expression has attracted significant attention in recent years due to its potential applicability in security and surveillance, human computer interaction, social robotics, and animation. This paper presents a new facial expression recognition method that utilizes bit plane specific local image description in a weighted score level fusion. The motivation is to utilize bit plane slicing to highlight the contribution of a particular bit plane made to the holistic facial appearance, which is then used in a weighted score level fusion in order to boost the recognition performance. A new local image descriptor is proposed specifically to extract local features from bit plane representations that utilizes Fisher linear discriminant to maximize the inter-class distance, while minimizing the intra-class variance. Two well-known facial expression databases, namely the Cohn-Kanade (CK) and the Japanese female facial expression (JAFFE) database have been used to evaluate the performance of the proposed method against existing facial appearance descriptors, such as local binary pattern (LBP), local ternary pattern (LTP), local directional pattern (LDP), and linear discriminant analysis (LDA). Experiments with a total of seven prototypic facial expressions show promising results for the proposed method, as compared with the other existing methods.

Facial expression recognition (FER) based on a new nonlinear feature extraction method, called kernel discriminant plane (KDP), is proposed in this paper. KDP is a nonlinear extension of the Sammon’s optimal discriminant plane (ODP) via the kernel trick. The recognition procedure is divided into two steps: (1) we select 34 fiducial points manually from each facial image and use the coordinates of these points as the input data of the facial image; (2) we construct a multiple binary classifier for the classification purpose of this task. The better performance of the proposed method is confirmed by the Japanese Female Facial Expression (JAFFE) database.

Facial expressions are important in people’s daily communications. Recognising facial expressions also has many important applications in the areas such as healthcare and e-learning. Existing facial expression recognition systems have problems such as background interference. Furthermore, systems using traditional approaches like SVM (Support Vector Machine) have weakness in dealing with unseen images. Systems using deep neural network have problems such as requirement for GPU, longer training time and requirement for large memory. To overcome the shortcomings of pure deep neural network and traditional facial recognition approaches, this paper presents a new facial expression recognition approach which has image pre-processing techniques to remove unnecessary background information and combines deep neural network ResNet50 and a traditional classifier-- the multiclass model for Support Vector Machine to recognise facial expressions. The proposed approach has better recognition accuracy than traditional approaches like Support Vector Machine and doesn’t need GPU. We have compared 3 proposed frameworks with a traditional SVM approach against the Karolinska Directed Emotional Faces (KDEF) Database, the Japanese Female Facial Expression (JAFFE) Database and the extended Cohn-Kanade dataset (CK+), respectively. The experiment results show that the features extracted from the layer 49Relu have the best performance for these three datasets.