Facial Expression recognition using Local Binary Patterns and Kullback Leibler divergence

The recognition rate of person-independent facial expression is generally not high, which limits the practical application of facial expression recognition. Aiming at this problem, this paper analyzes the reasons for the low recognition rate of person-independent facial expression, and proposes a recognition algorithm of person-independent facial expression based on improved LBP (Local Binary Pattern) and HOSVD (Higher-Order Singular Value Decomposition). The algorithm has the following contributions of facial expression recognition framework. In the stage of facial expression feature extraction, the transient features extracted by LDP(Local Directional Pattern) and the persistent features extracted by CBP(Centralized Binary Pattern) are integrated to improve the discrimination of facial expression features. Moreover, in the stage of facial expression classification and recognition, the traditional nearest neighbor classification is changed into k-nearest neighbor pre-classification, and the regional energy calculated by HOSVD is used to determine the similarity of two images for secondary classification. Finally, in the extended Cohn-Kanade dataset and Oulu-CASIA NIR&VIS facial expression database, the theoretical analysis and experimental results show that the method has better recognition effect for solving the problem of person-independent facial expression recognition.

The most expressive manner which human can reveal his emotional states is facial expression. Automatic facial expression recognition is an emerging field of study having extensive applications among which the human-computer interaction (HCI) has received lots of attentions in recent years. The features extracted from facial images, in order to recognize facial expressions, play an essential role in effectiveness of the facial image descriptors. Local binary pattern (LBP) texture descriptors have been known as simple, yet efficient descriptors which are noticeably used for extracting facial patterns from images. Recently, a generalized form of local binary pattern has been introduced which can offer a more precise image description than simple LBP descriptors. Consequently, it would be expected that taking the advantage of using these new LBP texture descriptors will produce more promising results in comparison with use of simple local binary pattern descriptors. In this paper, a novel method has been proposed for image feature extraction using these new image texture descriptors (generalized LBP); then, the obtained results have been compared to the results produced when applying simple LBP descriptors. Furthermore, K-NN and SVM have been used as classifiers in the proposed approach. Finally, a comparison between the proposed method and the existing local binary pattern algorithms for facial expression recognition concludes the superiority of the proposed algorithm over its existing counterparts.

The Local Binary Patterns (LBP) feature extraction method is a theoretically and computationally simple and efficient methodology for texture analysis. The LBP operator is used in many applications such as facial expression recognition and face recognition. The original LBP is based on hard thresholding the neighborhood of each pixel, which makes texture representation sensitive to noise. In addition, LBP cannot distinguish between a strong and a weak pattern. In order to enhance the LBP approach, Fuzzy Local Binary Patterns (FLBP) is proposed. In FLBP, any neighborhood does not represented only by one code, but, it is represented by all existing codes with different degrees. In FLBP, any fuzzy Intersection and Union operators may be used. In this study, the following operators are applied and their results are compared together: Dot-Sum, Min-Max and normalized Min-Max. Based on the extensive experiments, the fuzzy Min-Max operators are more useful and can improve the accuracy in the application of Facial Expression Recognition (FER) about 4% (i.e., form 82.98% to 86.88%).

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

Robust automatic facial expression detection method based on sparse representation plus LBP map

Facial Expression Recognition (FER) is an important area in human computer interaction. FER has different applications such as analysis of student behaviour in virtual class room, driver mood detection, security systems, and medicine. The analysis of facial expressions is an interesting and exciting problem. Feature extraction plays important role in any FER system. Local Binary Pattern (LBP) and its variants are popular for feature extraction due to simplicity in computation and monotonic illumination invariant property. However, the performance of LBP is poor in the presence of noise. This work proposes a novel approach for feature extraction to improve the performance of the FER. In this approach, the LBP is calculated considering 4-neighbors and diagonal neighbours separately. Further, for affective feature description, the concept of adaptive window and averaging in radial directions is introduced. This approach reduces the length of the feature vector as well as immune to noise. Support Vector Machine (SVM) is considered for classification. Recognition rate and confusion matrix are used to assess the performance of the proposed algorithm. Extensive experimental results on JAFFE, CK, FERG and FEI face databases show significant improvement in recognition rate compared to the available techniques both in noise free and noisy conditions.

Recently, facial expression recognition has attracted much attention in machine vision research because of its various applications. Accordingly, many facial expression recognition systems have been proposed. However, the majority of existing systems suffer from a critical problem: geometric variability. It directly affects the performance of geometric feature-based facial expression recognition approaches. Furthermore, it is a crucial challenge in appearance feature-based techniques. This variability appears in both neutral faces and facial expressions. Appropriate face geometry normalization can improve the accuracy of each facial expression recognition system. Therefore, this paper proposes different geometric models or shapes for normalization. Face geometry normalization removes geometric variability of facial images and consequently, appearance feature extraction methods can be accurately utilized to represent facial images. Thus, some expression-based geometric models are proposed for facial image normalization. Next, local binary patterns and local phase quantization are used for appearance feature extraction. A combination of an effective geometric normalization with accurate appearance representations results in more than a 4% accuracy improvement compared to several state-of-the-arts in facial expression recognition. Moreover, utilizing the model of facial expressions which have larger mouth and eye region sizes gives higher accuracy due to the importance of these regions in facial expression.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Facial expression which carries rich information of body behavior is the leading carrier of human affective and the symbol of intelligence. The main purpose of this paper is to recognize 3D human facial expression. The research in this paper includes the expression feature extraction algorithm and fusion with different kinds of feature. To contain more local texture feature information, we proposed a new feature of 3D facial expression named Local Threshold Binary Pattern (LTBP) which based on Local Binary Pattern (LBP). We calculate the difference of gray value standard between neighboring pixels and the center pixel as a threshold to binary instead of the traditional LBP operation which only comparison of size between neighboring pixels and the center pixel. After we get the LTBP feature, we fuse the LTBP and HOG (Histogram of Oriented Gradient) features to get multi-feature fusion for 3D facial expression recognition. Our algorithm of 3D facial expression recognition comprises three steps: (1) extracting two sets of feature vectors and establishing the correlation criterion function between the two sets of feature vectors; (2) solving the two sets canonical projective vectors and extracting their canonical correlation features by the framework of canonical correlation analysis algorithm; (3) doing feature fusion for classification by using proposed strategy. We have performed comprehensive experiments on the BU-3DFE database which is presently the largest available 3D face database. We have achieved verification rates of more than 90% for the 3D facial expression recognition.

Texture characteristics among the landmark points reflected in human faces are important features in the recognition of facial expression. In this paper, we propose a salient feature based facial expression on texture characteristics. Detection of effective landmarks are performed by appearance based models and corresponding texture regions are extracted from face images. Local Binary Pattern (LBP) is used to compute the texture feature. Texture feature is used for normalizing texture signatures. In addition, we introduce stability indices corresponding to texture features which are exploited as features in the present scope. Moreover, statistical features such as moment, skewness, kurtosis and entropy are also supplemented to the feature set. These salient features together are used as an input to Nonlinear AutoRegressive with eXogenous (NARX) for recognition of the human facial expression. The Cohn-Kanade (CK+), Japanese Female Facial Expression (JAFFE), MMI and MUG benchmark databases are used to conduct the experiments and the results justify the effectiveness of the proposed procedure.

Humans interact socially with the help of facial expressions. Even health states or pains are reflected through facial expressions and hence can be useful in healthcare. Here, a facial expression recognition system is proposed. The bandlet transform is performed on face image to generate quadtree. Then on the output of bandlet transform centre symmetric - local binary pattern (CS-LBP) is applied. A feature vector of the image is generated by taking the histogram of CS-LBP. The support vector machine (SVM) is used to classify expressions in six categories. The experiments are performed using a publically available CK+ dataset. The initial results with LBP and CS-LBP are reported.

The existing local binary pattern (LBP) operators have three disadvantages: (1) They produce rather long histograms, which slow down the recognition speed especially on large-scale face database; (2) Under some certain circumstance, they miss the local structure as they don't consider the effect of the center pixel; (3) The binary data produced by them are sensitive to noise. Aiming at these problems, we proposed centralized binary pattern (CBP) operator. CBP operator has several advantages: (1) It reduces significantly the histograms' dimensionality by comparing pairs of neighbors in the operator; (2) It considers the center pixel point's effect and gives it the largest weight, thus improving discrimination; (3) By modifying the sign function of existing LBP operator, it decreases the white noise's influence on face images. Moreover, for the purpose of improving the robustness to small perturbation (deformation) of expressional images, we introduced image Euclidean distance (IMED) and embedded it in CBP. Experiments on two well-known facial expression databases demonstrate that the proposed method outperforms other modern approaches and show that IMED can enhance the performance of CBP in facial expression recognition.

Facial expression recognition based on Local Binary Patterns: A comprehensive study

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

This paper proposes a novel face representation approach, local Gabor binary mapping pattern (LGBMP), for multi-view gender classification. In this approach, a face image is first represented as a series of Gabor magnitude pictures (GMP) by applying multi-scale and multi-orientation Gabor filters. Each GMP is then encoded as a LGBP image where a uniform local binary pattern (LBP) operator is used. After that, each LGBP image is divided into non-overlapping rectangular regions, from which spatial histograms are extracted. Although an LGBP feature vector can be obtained by fitting together the regional histograms, it can not be employed in pattern classification due to its high dimension. We propose that each regional LGBP feature be mapped onto a one-dimensional subspace independently before they are concatenated as a whole feature vector. This is attractive since we reduce the feature dimension and also preserve the spatial information of LGBP image. Two ways have been proposed to map the regional LGBP feature in this paper. One is so-called LGBMP-LDA using linear discriminant analysis (LDA) for dimensionality reduction while the other is to project the regional LGBP feature onto the class center connecting line, namely, LGBMP-CCL. As a result, despite several decades of Gabor filters, the final feature dimension is even less than that of the feature extracted by using LBP directly on gray-scale images. The classification tasks in our work are performed by support vector machines (SVM). The experimental results on the CAS-PEAL face database indicate that the proposed approach achieves higher accuracy than the others such as SVMs+Gray-scale pixel, SVMs+Gabor and SVMs+LBP approach, more particularly, it has the lowest dimension of feature vector.