Fusion of static and temporal predictors for unconstrained facial expression recognition

Facial expression is one way humans convey their emotional states. Accurate recognition of facial expressions is essential in perceptual human-computer interface, robotics and mimetic games. This paper presents a novel approach to facial expression recognition from static images that combines fixed and adaptive 2-D filters in a hierarchical structure. The fixed filters are used to extract primitive features. They are followed by the adaptive filters that are trained to extract more complex facial features. Both types of filters are non-linear and are based on the biological mechanism of shunting inhibition. The features are finally classified by a support vector machine. The proposed approach is evaluated on the JAFFE database with seven types of facial expressions: anger, disgust, fear, happiness, neutral, sadness and surprise. It achieves a classification rate of 96.7%, which compares favorably with several existing techniques for facial expression recognition tested on the same database.

Facial expression recognition plays a vital role in realizing a highly intelligent human-machine interface, and has recently attracted much attention. In this paper, we propose a new facial expression recognition method that utilizes the 2D DCT, k-means algorithm and vector matching. This technique is based on two main intuitive ideas: (i) complicated facial expression categories such as "anger" and "sadness", may be divided into several subcategories with different subfeature spaces where the recognition task can be performed with higher accuracy and (ii) the k-means algorithm may be used to cluster these subcategories. A new image database with five facial expressions (neutral, smile, anger, sadness, surprise) of 60 women was constructed using a computationally efficient projection-based technique. Experimental results using the new database and an existing one (60 men) reveal that the new technique outperforms the standard vector matching technique and two recently developed methods using fixed-size and constructive one-hidden-layer neural networks. The mean recognition rate can be as high as 95% for the two databases.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

This paper presents a novel facial expression recognition (FER) technique based on support vector machine (SVM) for the FER. Here it is called the FERS technique. First, the FERS technique develops a face detection method that combines the Haar-like features method with the self-quotient image (SQI) filter. As a result, the FERS technique possesses better detection rate because the face detection method gets more accurate in locating face regions of an image. The main reason is that the SQI filter can overcome the insufficient light and shade light. Subsequently, three schemes, the angular radial transform (ART), the discrete cosine transform (DCT) and the Gabor filter (GF), are simultaneously employed in the design of the feature extraction for facial expression in the FERS technique. More specifically, they are employed in constructing a set of training patterns for the training of an SVM. The FERS technique then exploits the trained SVM to recognize the facial expression for a query face image. Finally, experimental results show that the recognition performance of the FERS technique can be better than that of other existing methods under consideration in the paper.

Human facial expression recognition (FER) has attracted much attention in recent years because of its importance in realizing highly intelligent human-machine interfaces. In this paper, we propose a new FER technique that utilizes the 2-D DCT of full size facial images and a decision tree with feedforward neural network (NN) based nodes. The first NN-based node of the decision tree is designated to separate one group of facial expressions with members "smile" and "surprise" from another group that contains "anger" and "sadness". This node can reduce the confusion between the category members of the two groups. Two NN-based nodes that follow the first node are established for each group to separate their two members. As a result, the original recognition problem with four categories is divided into three subproblems, each having only two members to distinguish. This work is the first trial to use NN, decision tree and 2-D DCT simultaneously within a single recognition task. To demonstrate the capability of the proposed recognition technique, we use two databases, including a recently constructed one, which contain 2-D front face images of 60 men and 60 women, respectively. Experimental results reveal that the new technique outperforms, on the whole, the simple vector matching and K-means based vector matching techniques and two recently developed methods using fixed-size and constructive neural networks. The mean recognition rates of the new technique have been found as high as 97.5% and 93.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

Facial expression recognition (FER) techniques can be widely used in human-computer interaction, intelligent robots, intelligent monitoring, and other domains. Currently, FER methods based on deep learning have become the mainstream schemes. However, these methods have some problems, such as a large number of parameters, difficulty in being applied to embedded processors, and the fact that recognition accuracy is affected by facial deflection. To solve the problem of a large number of parameters, we propose a DSC-DenseNet model, which improves the standard convolution in DenseNet to depthwise separable convolution (DSC). To solve the problem wherein face deflection affects the recognition effect, we propose a posture normalization model based on GAN: a GAN with two local discriminators (LD-GAN) that strengthen the discriminatory abilities of the expression-related local parts, such as the parts related to the eyes, eyebrows, mouth, and nose. These discriminators improve the model’s ability to retain facial expressions and evidently benefits FER. Quantitative and qualitative experimental results on the Fer2013 and KDEF datasets have consistently shown the superiority of our FER method when working with multi-pose face images.

The early diagnosis of chronic diseases plays a vital role in the field of healthcare communities and biomedical, where it is necessary for detecting the disease at an initial phase to reduce the death rate. This paper investigates the use of feature selection, dimensionality reduction and classification techniques to predict and diagnose chronic disease. The appropriate selection of attributes plays a crucial role in improving the classification accuracy of the diagnosis systems. Additionally, dimensionality reduction techniques effectively improve the overall performance of the machine learning algorithms. On chronic disease databases, the classification techniques deliver efficient predictive results by developing intelligent, adaptive and automated system. Parallel and adaptive classification techniques are also analyzed in chronic disease diagnosis which is used to stimulate the classification procedure and to improve the computational cost and time. This survey article represents the overview of feature selection, dimensionality reduction and classification techniques and their inherent benefits and drawbacks.

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

Recognition of facial expression has many applications including human-computer interaction, human emotion analysis, personality development, cognitive science, health-care, virtual reality, image retrieval, etc. In this paper we propose a new method for recognition of facial expression using local region specific mean optical flow and local binary pattern feature descriptor with support vector machine classification. In general, facial expression recognition techniques divide the face into regular grid (holistic representation) and the facial features are extracted. However, in this paper we divide the face into domain specific local regions. At first a robust optical flow is utilized to get mean optical flow in different directions for each local region which considers both local statistic motion information and its spatial location. The features are used only from the key frames; which are detected based on maximal mean optical flow magnitude within a sequence w.r.t. neutral frame. Now, the region specific local binary pattern is extracted from key frame and concatenated with mean optical flow features. The performance of the proposed facial expression recognition system has been validated on CK+ facial expression dataset.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

The last two years have seen a rapid rise in the duration of time that both adults and children spend on screens, driven by the recent COVID-19 health pandemic. A key adverse effect is digital eye strain (DES). Recent trends in human-computer interaction and user experience have proposed voice or gesture-guided designs that present more effective and less intrusive automated solutions. These approaches inspired the design of a solution that uses facial expression recognition (FER) techniques to detect DES and autonomously adapt the application to enhance the user’s experience. This study sourced and adapted popular open FER datasets for DES studies, trained convolutional neural network models for DES expression recognition, and designed a self-adaptive solution as a proof of concept. Initial experimental results yielded a model with an accuracy of 77% and resulted in the adaptation of the user application based on the FER classification results. We also provide the developed application, model source code, and adapted dataset used for further improvements in the area. Future work should focus on detecting posture, ergonomics, or distance from the screen.

The recognition rate of person-independent facial expression is generally not high, which limits the practical application of facial expression recognition. Aiming at this problem, this paper analyzes the reasons for the low recognition rate of person-independent facial expression, and proposes a recognition algorithm of person-independent facial expression based on improved LBP (Local Binary Pattern) and HOSVD (Higher-Order Singular Value Decomposition). The algorithm has the following contributions of facial expression recognition framework. In the stage of facial expression feature extraction, the transient features extracted by LDP(Local Directional Pattern) and the persistent features extracted by CBP(Centralized Binary Pattern) are integrated to improve the discrimination of facial expression features. Moreover, in the stage of facial expression classification and recognition, the traditional nearest neighbor classification is changed into k-nearest neighbor pre-classification, and the regional energy calculated by HOSVD is used to determine the similarity of two images for secondary classification. Finally, in the extended Cohn-Kanade dataset and Oulu-CASIA NIR&VIS facial expression database, the theoretical analysis and experimental results show that the method has better recognition effect for solving the problem of person-independent facial expression recognition.

Facial expression recognition (FER) has been widely studied since it can be used for various applications. However, most of FER techniques focus on discriminating typical facial expressions such as six basic facial expressions. Spontaneous facial expressions are not limited to such typical ones because the intensity of a facial expression varies depending on the intensity of an emotion. In order to utilize FER for real-world applications, therefore, it is necessary to discriminate slight difference of facial expressions. In this paper, we propose an effective FER method to recognize spontaneous facial expressions using ensemble learning which combines a number of naive Bayes classifiers. In addition, a method to estimate the intensity of facial expression is also proposed by using the classification results of the classifiers. The effectiveness of these methods are evaluated through an FER experiment and an experiment to estimate the intensity of facial expressions using a data set including spontaneous facial expressions.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.