Joint Learning for Mask-Aware Facial Expression Recognition Based on Exposed Feature Analysis and Occlusion Feature Enhancement

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

The recognition rate of person-independent facial expression is generally not high, which limits the practical application of facial expression recognition. Aiming at this problem, this paper analyzes the reasons for the low recognition rate of person-independent facial expression, and proposes a recognition algorithm of person-independent facial expression based on improved LBP (Local Binary Pattern) and HOSVD (Higher-Order Singular Value Decomposition). The algorithm has the following contributions of facial expression recognition framework. In the stage of facial expression feature extraction, the transient features extracted by LDP(Local Directional Pattern) and the persistent features extracted by CBP(Centralized Binary Pattern) are integrated to improve the discrimination of facial expression features. Moreover, in the stage of facial expression classification and recognition, the traditional nearest neighbor classification is changed into k-nearest neighbor pre-classification, and the regional energy calculated by HOSVD is used to determine the similarity of two images for secondary classification. Finally, in the extended Cohn-Kanade dataset and Oulu-CASIA NIR&VIS facial expression database, the theoretical analysis and experimental results show that the method has better recognition effect for solving the problem of person-independent facial expression recognition.

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

In this paper, we present a facial expression recognition method using feature-adaptive motion energy analysis. Our method is simplicity-oriented and avoids complicated face model representations or computationally expensive algorithms to estimate facial motions. Instead, the proposed method uses a simplified action-based face model to reduce the computational complexity of the entire facial expression analysis and recognition process. Feature-adaptive motion energy analysis estimates facial motions in a cost-effective manner by assigning more computational complexity on selected discriminative facial features. Facial motion intensity and orientation evaluation are then performed accordingly. Both facial motion intensity and orientation evaluation are based on simple calculations by exploiting a few motion energy values in the difference image, or optimizing the characteristics of feature-adaptive facial feature regions. For facial expression classification, a computationally inexpensive decision tree is used since the information gain heuristics of ID3 decision tree forces the classification to be done with minimal Boolean comparisons. The feasibility of the proposed method is shown through the experimental results as the proposed method recognized every facial expression in the JAFFE database by up to 75% with very low computational complexity.

Objectives: Attention-deficit/hyperactivity disorder (ADHD) is the most frequent neurobehavioural disorder in childhood. ADHD is associated with impaired academic performance, cognitive, and emotional deficits. Moreover, comorbid oppositional defiant disorder (ODD) is leading to more severe impairment in social performance. Social cognition involves recognition, encoding, and interpretation of emotions from faces. Basic facial expressions that include sadness, happiness, anger, disgust, fear, and surprise are the easiest emotions to recognize. We aimed to demonstrate facial expression recognition impairments that might occur more frequently in children with co-occurring ADHD/ODD than patients with ADHD only. Thus, children with the co-occurrence of ODD may suffer more severely from social and behavioural difficulties.Methods: Forty patients diagnosed with ADHD and/or co-occurring ADHD/ODD according to DSM-IV-TR criteria were compared with a parallel (by gender, age, and educational state) 11 healthy children as a control group in this study. Clear facial images of each emotion were used as well as two additional sets of photos include 50% blurred images and cropped eye images were added as distractors then all images represented with black and white tone for emotion recognition task via facial expression. Angry expressions presented as target expressions. DSM-IV-Based Screening and Rating Scale for Children and Adolescents with attention deficit and disruptive behaviour disorders, the Conners’ Teachers Rating Scale/Revised Long Form and the Conners’ Parent Rating Scale/Revised Long Form were used to provide diagnostic objectivity.Results: Control group statistically performed better than ADHD group on recognition of emotional facial expressions. Results showed no statistically significant differences between the ADHD and ADHD/ODD group on recognition of emotional facial expressions. However, according to results of emotion recognition task via facial expressions, there were statistically significant differences between pure ADHD and comorbid ADHD+ODD groups in happy and neutral expressions. ADHD/ODD group tend to attribute more meaning to neutral facial expressions. Additionally there was statistically significant difference between control group and ADHD group according to recognition of angry expressions. There were statistically significant differences between the groups according to recognition of sad expressions in all clear, blurred, and eye photographs.Conclusions: Difficulties in recognizing emotional facial expressions were observed in children with ADHD. A statistically significant association was established between presence of ADHD and impaired recognition of facial emotion expressions independent from the scores of the disruptive behaviour rating scale. Comorbid ODD was not associated with recognition of emotional facial expressions including angry expressions. Recognition of angry expressions was not found as a predictor for disruptive behaviour disorders.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

Facial expression recognition (FER) has been dramatically developed in recent years, thanks to the advancements in related fields, especially machine learning, image processing and human recognition. Accordingly, the impact and potential usage of automatic FER have been growing in a wide range of applications, including human-computer interaction, robot control and driver state surveillance. However, to date, robust recognition of facial expressions from images and videos is still a challenging task due to the difficulty in accurately extracting the useful emotional features. These features are often represented in different forms, such as static, dynamic, point-based geometric or region-based appearance. Facial movement features, which include feature position and shape changes, are generally caused by the movements of facial elements and muscles during the course of emotional expression. The facial elements, especially key elements, will constantly change their positions when subjects are expressing emotions. As a consequence, the same feature in different images usually has different positions. In some cases, the shape of the feature may also be distorted due to the subtle facial muscle movements. Therefore, for any feature representing a certain emotion, the geometric-based position and appearance-based shape normally changes from one image to another image in image databases, as well as in videos. This kind of movement features represents a rich pool of both static and dynamic characteristics of expressions, which playa critical role for FER. The vast majority of the past work on FER does not take the dynamics of facial expressions into account. Some efforts have been made on capturing and utilizing facial movement features, and almost all of them are static based. These efforts try to adopt either geometric features of the tracked facial points, or appearance difference between holistic facial regions in consequent frames or texture and motion changes in local facial regions. Although achieved promising results, these approaches often require accurate location and tracking of facial points, which remains problematic.

This paper proposes a unified facial motion tracking and expression recognition framework for monocular video. For retrieving facial motion, an online weight adaptive statistical appearance method is embedded into the particle filtering strategy by using a deformable facial mesh model served as an intermediate to bring input images into correspondence by means of registration and deformation. For recognizing facial expression, facial animation and facial expression are estimated sequentially for fast and efficient applications, in which facial expression is recognized by static anatomical facial expression knowledge. In addition, facial animation and facial expression are simultaneously estimated for robust and precise applications, in which facial expression is recognized by fusing static and dynamic facial expression knowledge. Experiments demonstrate the high tracking robustness and accuracy as well as the high facial expression recognition score of the proposed framework.

Recognition of facial expressions becomes significant issue between human and robot interactions. The purpose of this paper is to study the alignment and tracking of facial features with optical flow and component-based active appearance model, and then analyze fitted points to recognize facial expressions. Using this method with accurate analysis and tracking of facial features in robot or computer. Consequently robot or computer can easily recognize user's facial expressions and emotional variation, and then response properly. We apply some realtime techniques and Active Appearance Model (AAM) on the cameras. A high-quality AAM alignment results depend on apposite selections of initial positions. Nevertheless it takes a lot of time when we apply image pyramid to get precise results. In this paper, we introduce a new method to apply AAM fitting and further solve above problems. In our fitting plan, we apply partial AAM fitting separately on mouth and eyes. Therefore we could make more efficient facial features alignment and then it becomes able to implement tracking to real-world video and realtime alignment. To get more stable partial AAM, we use multi-level optical flow to determine initial positions of facial feature models. It is relative easier to analyze user's emotional information and get accurate positions of facial features for further application in real world environments by the algorithm we developed.

Recognition of facial expression has many applications including human-computer interaction, human emotion analysis, personality development, cognitive science, health-care, virtual reality, image retrieval, etc. In this paper we propose a new method for recognition of facial expression using local region specific mean optical flow and local binary pattern feature descriptor with support vector machine classification. In general, facial expression recognition techniques divide the face into regular grid (holistic representation) and the facial features are extracted. However, in this paper we divide the face into domain specific local regions. At first a robust optical flow is utilized to get mean optical flow in different directions for each local region which considers both local statistic motion information and its spatial location. The features are used only from the key frames; which are detected based on maximal mean optical flow magnitude within a sequence w.r.t. neutral frame. Now, the region specific local binary pattern is extracted from key frame and concatenated with mean optical flow features. The performance of the proposed facial expression recognition system has been validated on CK+ facial expression dataset.

Several researchers have attempted automatic recognition and tracking of facial expressions in video frames of neutral and expressive faces. But, it is very hard to detect a neutral face in practical applications because the distinction between facial movement and head movement is difficult to make. To solve this distinction problem, we tried to initialize the head position using the central point of each eye and the tip of the nose to extract facial movements. Based on the detected neutral faces, the changing processes of facial expressions can be recognized by means of detecting the reference face (neutral face) and an algorithm of the recognition for facial expressions. In short, we assume that this method of detecting neutral faces can be applied to the recognition of facial expressions.

Facial Expression Recognition (FER) is a critical method for evaluating the emotional states of patients with mental disorders, playing a significant role in treatment monitoring. However, due to privacy constraints, facial expression data from patients with mental disorders is severely limited. Additionally, the more complex inter-class and intra-class similarities compared to healthy individuals make accurate recognition of facial expressions challenging. Therefore, we propose a Voluntary Facial Expression Mimicry (VFEM) experiment, which collected facial expression data from schizophrenia, depression, and anxiety. This experiment establishes the first dataset designed for facial expression recognition tasks exclusively composed of patients with mental disorders. Simultaneously, based on VFEM, we propose a Vision Transformer FER model tailored for Complex mental disorder patients (CmdVIT). CmdVIT integrates crucial facial expression features through both explicit and implicit mechanisms, including explicit visual center positional encoding and implicit sparse attention center loss function. These two key components enhance positional information and minimize the facial feature space distance between conventional attention and critical attention, effectively suppressing inter-class and intra-class similarities. In various FER tasks for different mental disorders in VFEM, CmdVIT achieves more competitive performance compared to contemporary benchmark models. Our works are available at https://github.com/yjy-97/CmdVIT.

BackgroundPrevious reports have suggested impairment in facial expression recognition in delinquents, but controversy remains with respect to how such recognition is impaired. To address this issue, we investigated facial expression recognition in delinquents in detail.MethodsWe tested 24 male adolescent/young adult delinquents incarcerated in correctional facilities. We compared their performances with those of 24 age- and gender-matched control participants. Using standard photographs of facial expressions illustrating six basic emotions, participants matched each emotional facial expression with an appropriate verbal label.ResultsDelinquents were less accurate in the recognition of facial expressions that conveyed disgust than were control participants. The delinquents misrecognized the facial expressions of disgust as anger more frequently than did controls.ConclusionThese results suggest that one of the underpinnings of delinquency might be impaired recognition of emotional facial expressions, with a specific bias toward interpreting disgusted expressions as hostile angry expressions.