Facial expression recognition by fusion of gabor texture features and local phase quantization

Aiming at the limitation of single feature classification, this paper proposes a facial expression recognition algorithm based on feature fusion and sparse representation residual fusion. The proposed method is composed of three components. Firstly, a feature fusion method based on local binary quantization (LBP) and local phase quantization (LPQ) is proposed. The LBP/LPQ features are an expression feature with image space and frequency domain. Second, Curvelet transform is used to extract the global features of facial expression, which is a very effective multi-scale, band-pass and direction analysis tool. Finally, the LBP/LPQ features and the Curvelet feature are classified by the sparse representation based classification (SRC) with stronger discrimination and robustness respectively. According to the sparse residuals of LBP/LPQ+SRC and Curvelet+SRC, a classification method of residual fusion is proposed. Experiments on JAFFE database and Cohn-Kanade database show that the proposed method effectively improves the ability of facial expression recognition and achieves a high recognition rate.

In this paper, we present a facial expression recognition method based on Gabor feature and sparse representation. Sparse Representation based Classification (SRC) has been widely used in computer vision and pattern recognition. And Gabor filter banks can be used to approximately model the signal processing in visual primary cortex. We believe that the nature of the attractive performance of SRC and Gabor feature lies in that they both followed the natures of signal perception of retina and information processing of cortex in human vision. Therefore, we combined the Gabor feature and SRC for facial expression recognition. The comparison experiments of proposed Gabor+SRC algorithm and straightforward SRC application are conducted on JAFFE database. And the experimental results showed the attractive performance of the proposed Gabor+SRC method.

Texture characteristics among the landmark points reflected in human faces are important features in the recognition of facial expression. In this paper, we propose a salient feature based facial expression on texture characteristics. Detection of effective landmarks are performed by appearance based models and corresponding texture regions are extracted from face images. Local Binary Pattern (LBP) is used to compute the texture feature. Texture feature is used for normalizing texture signatures. In addition, we introduce stability indices corresponding to texture features which are exploited as features in the present scope. Moreover, statistical features such as moment, skewness, kurtosis and entropy are also supplemented to the feature set. These salient features together are used as an input to Nonlinear AutoRegressive with eXogenous (NARX) for recognition of the human facial expression. The Cohn-Kanade (CK+), Japanese Female Facial Expression (JAFFE), MMI and MUG benchmark databases are used to conduct the experiments and the results justify the effectiveness of the proposed procedure.

The theory of compressive sensing applies the sparse representation to the extraction of useful information from signals and brings a breakthrough to the theory of signal sampling. Based on compressive sensing, sparse representation-based classification (SRC) is proposed. SRC uses the compressibility of the image data to represent the facial image sparsely and could solve the problems of both massive calculation and information loss in dealing with signals. SRC does not, however, deal with the effects of variable illumination, posture and incomplete face image, which could result in severe performance degradation. This paper studies the differences between SRC recognition and human recognition. We find that there is an obvious disadvantage in the SRC algorithm, and it will significantly affect the face recognition performance in actual environment, especially for the variable illumination, posture and incomplete face image. To overcome the disadvantage of SRC algorithm, we propose an SRC-based twice face recognition algorithm named T_SRC. T_SRC uses bidirectional PCA, linear discriminant analysis and GradientFace to execute multichannel analysis, which could extract more "holistic/configural" face features in actual environment than by using SRC algorithm directly. Based on the multichannel analysis, we identify the test image by SRC firstly. Then, by analyzing the residual, this algorithm could decide whether the twice recognition is needed. If the twice recognition is needed, T_SRC extracts the facial details ("featural" face features) by the improved Harris point and Gabor filter detector. We suppose that the facial details are more stable than the whole face in actual environment, and later experiments verify our assumption. At last, this algorithm identifies the class of the test image by SRC again. The results of the experiments prove that the T_SRC algorithm has better recognition rate than SRC.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Mapping cotton cultivated area combining remote sensing with a fused representation-based classification algorithm

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

Facial Expression Recognition (FER) is a critical component of affective computing, with deep learning models dominating performance metrics. In contrast, geometric approaches based on the Facial Action Coding System (FACS) offer explainability through using triangles aligned to facial landmarks. The notable points of these triangles capture the deformation of muscles. However, restricting the feature extraction to notable points may be suboptimal. This paper introduces a novel method for optimizing the extraction of features by searching for optimal inner points in 22 facial triangles applying three metaheuristics: Differential Evolution (DE), Particle Swarm Optimization (PSO), and Convex Partition (CP). This results in a set of 59 geometric-based descriptors that capture muscle deformation more accurately. The proposed method was evaluated using five machine learning classifiers on two benchmark databases: the Karolinska Directed Emotional Faces (KDEF) and the Japanese Female Facial Expression (JAFFE). Experimental results demonstrate significant performance improvements. The combination of DE with a Multi-Layer Perceptron (MLP) achieved an accuracy of 0.91 on the KDEF database, while Support Vector Machine (SVM) optimized via CP attained an accuracy of 0.81 on the JAFFE database. Statistical analysis confirms that optimized descriptors yield higher accuracy than previous geometric methods.

We propose a generalized Sparse Representation-based Classification (SRC) algorithm for open set recognition where not all classes presented during testing are known during training. The SRC algorithm uses class reconstruction errors for classification. As most of the discriminative information for open set recognition is hidden in the tail part of the matched and sum of non-matched reconstruction error distributions, we model the tail of those two error distributions using the statistical Extreme Value Theory (EVT). Then we simplify the open set recognition problem into a set of hypothesis testing problems. The confidence scores corresponding to the tail distributions of a novel test sample are then fused to determine its identity. The effectiveness of the proposed method is demonstrated using four publicly available image and object classification datasets and it is shown that this method can perform significantly better than many competitive open set recognition algorithms.

Facial expressions can be seen as a form of non-verbal communication as well as a primary means of conveying social information among humans.Automatic facial expression recognition (FER) can be applied to a wide range of scenarios in human-computer interaction, facial animation, entertainment, and psychology studies. For feature representation in a FER system, various texture descriptors have been employed to derive an effective solution for this system. However, these individual texture descriptor-based FER systems have often failed to achieve effective performance in the recognition of facial expressions. In this sense, it is necessary to further improve the general performance of a facial expression recognition system, evaluating different feature representations. In this paper, a novel local descriptor for a facial expression recognition system is proposed, designated the level of difference descriptor (LOD). The main goal is to use this descriptor as a supplement to state-of-the-art local descriptors to further improve the performance of a FER system in terms of classification accuracy. Furthermore, the fusion of various texture features for devising a robust feature representation for multi-view facial expression recognition is presented.

Facial expression is an important channel for human communication and can be applied in many real applications. One critical step for facial expression recognition (FER) is to accurately extract emotional features. Current approaches on FER in static images have not fully considered and utilized the features of facial element and muscle movements, which represent static and dynamic, as well as geometric and appearance characteristics of facial expressions. This paper proposes an approach to solve this limitation using "salient” distance features, which are obtained by extracting patch-based 3D Gabor features, selecting the "salient” patches, and performing patch matching operations. The experimental results demonstrate high correct recognition rate (CRR), significant performance improvements due to the consideration of facial element and muscle movements, promising results under face registration errors, and fast processing time. Comparison with the state-of-the-art performance confirms that the proposed approach achieves the highest CRR on the JAFFE database and is among the top performers on the Cohn-Kanade (CK) database.

In this paper, we propose a 3D information-based facial expression recognition network (3D-FERNet), which effectively combines identity, 3D and 2D expression information for facial expression recognition (FER). The 3DFERNet model consists of three feature encoders to extract identity, 2d expression, and 3d expression features, and one semantic-based feature fusion module to combine these features for further classification. Firstly, the identity encoder is constructed by a face identity recognition network, which is used to perform identity-conditional FER and remove identity bias of FER prediction results. Secondly, we design the 3D expression encoder by a 3D face reconstruction network, which is trained under a semi-supervised method with a variety of unlabeled expression data. The generated 3D features can capture the subtle differences between similar expression cases. Thirdly, for the 2D expression encoder, we can leverage most existing FER networks as the backbone. Finally, after obtaining these facial features, we propose a semantic-based feature fusion module, which is based on the attention mechanism for feature combination. Experiments show that the proposed 3D-FERNet achieves state-of-the-art classification accuracy in multiple benchmark data sets. Moreover, the identity feature encoder and 3D feature encoder can serve as two general modules to plug in most existing FER models.

Facial expression recognition (FER) based on a new nonlinear feature extraction method, called kernel discriminant plane (KDP), is proposed in this paper. KDP is a nonlinear extension of the Sammon’s optimal discriminant plane (ODP) via the kernel trick. The recognition procedure is divided into two steps: (1) we select 34 fiducial points manually from each facial image and use the coordinates of these points as the input data of the facial image; (2) we construct a multiple binary classifier for the classification purpose of this task. The better performance of the proposed method is confirmed by the Japanese Female Facial Expression (JAFFE) database.

Facial expressions are important in people’s daily communications. Recognising facial expressions also has many important applications in the areas such as healthcare and e-learning. Existing facial expression recognition systems have problems such as background interference. Furthermore, systems using traditional approaches like SVM (Support Vector Machine) have weakness in dealing with unseen images. Systems using deep neural network have problems such as requirement for GPU, longer training time and requirement for large memory. To overcome the shortcomings of pure deep neural network and traditional facial recognition approaches, this paper presents a new facial expression recognition approach which has image pre-processing techniques to remove unnecessary background information and combines deep neural network ResNet50 and a traditional classifier-- the multiclass model for Support Vector Machine to recognise facial expressions. The proposed approach has better recognition accuracy than traditional approaches like Support Vector Machine and doesn’t need GPU. We have compared 3 proposed frameworks with a traditional SVM approach against the Karolinska Directed Emotional Faces (KDEF) Database, the Japanese Female Facial Expression (JAFFE) Database and the extended Cohn-Kanade dataset (CK+), respectively. The experiment results show that the features extracted from the layer 49Relu have the best performance for these three datasets.

In order to improve the recognition of facial expression recognition rate, facial expression recognition algorithm is based on a LBP and SVM decision tree. First will facial expression image is converted to LBP characteristic spectrum using LBP algorithm, and then the LBP characteristic spectrum into LBP histogram feature sequence, finally completes the classification and recognition of facial expression by SVM decision tree algorithm, and prove the effectiveness of the proposed method in the recognition of facial expression database in JAFFE.