Local saliency-inspired binary patterns for automatic recognition of multi-view facial expression

The most expressive manner which human can reveal his emotional states is facial expression. Automatic facial expression recognition is an emerging field of study having extensive applications among which the human-computer interaction (HCI) has received lots of attentions in recent years. The features extracted from facial images, in order to recognize facial expressions, play an essential role in effectiveness of the facial image descriptors. Local binary pattern (LBP) texture descriptors have been known as simple, yet efficient descriptors which are noticeably used for extracting facial patterns from images. Recently, a generalized form of local binary pattern has been introduced which can offer a more precise image description than simple LBP descriptors. Consequently, it would be expected that taking the advantage of using these new LBP texture descriptors will produce more promising results in comparison with use of simple local binary pattern descriptors. In this paper, a novel method has been proposed for image feature extraction using these new image texture descriptors (generalized LBP); then, the obtained results have been compared to the results produced when applying simple LBP descriptors. Furthermore, K-NN and SVM have been used as classifiers in the proposed approach. Finally, a comparison between the proposed method and the existing local binary pattern algorithms for facial expression recognition concludes the superiority of the proposed algorithm over its existing counterparts.

Several researchers have attempted automatic recognition and tracking of facial expressions in video frames of neutral and expressive faces. But, it is very hard to detect a neutral face in practical applications because the distinction between facial movement and head movement is difficult to make. To solve this distinction problem, we tried to initialize the head position using the central point of each eye and the tip of the nose to extract facial movements. Based on the detected neutral faces, the changing processes of facial expressions can be recognized by means of detecting the reference face (neutral face) and an algorithm of the recognition for facial expressions. In short, we assume that this method of detecting neutral faces can be applied to the recognition of facial expressions.

Every person expresses himself or herself with different facial expressions. Nowadays, recognition of facial expression is a popular topic for researchers due to the practical function in a wide variety of areas. The learning of human facial expressions is a tough and tricky domain in pattern research society. Each individual facial expression is produced by non-rigid object contortions, and these contortions are man-dependent. The main motto of this paper was to invent and apply a structure for the automatic recognition of human facial expression.

Automatic facial expression recognition is a fascinating and challenging problem, and impacts important applications in many areas such as human-computer interaction (HCI) and robotics. In this paper an automatic facial expression recognition method is proposed, we are applying face detection methods to an image from the dataset to get face image and its important parts like eyes, nose and mouth automatically. Local binary patterns are used as feature extractor and for classification a strong machine learning classification tool support vector machine is used. Our experiments illustrate that the LBP provide a compact and discriminative facial representation and by adopting Support Vector Machines we obtained the best recognition performance of 95.83% on Cohn-Kanade database, which is better than contemporary methods. We experimentally illustrate that eyes and mouth play a significant role in facial expression recognition.

Automatic recognition of facial expression remains a challenging task in computer vision, because of large amount of image data and subtle difference in facial expressions. In this paper, we present a customizable system for facial expression recognition using non-negative matrix factorization (NMF).Our system can be described in two major phases: the first phase uses preprocessing and the NMF for decomposing facial expression images into a set of basis images; the second phase incorporates dimensionality reduction and similarity to classify if an input image depicts one of the five facial expressions (i.e., normal, smile, sad, surprise, and angry) given known facial expressions as template images. Our results clearly demonstrate that the system was able to provide an effective basis for characterizing facial feature regions and reasonable classification in various facial expressions.

Automatic facial expression recognition (FER) plays a crucial role in realizing the adaptable and individualized tutoring in affective computer-based learning environment. Although many research efforts have been conducted to enhance a greater understanding of FER, a successful accurate recognition of the spontaneous facial expressions in real e-learning environment is still challenging due to its low change in intensity and short duration. In this paper, we propose a new dual-modality spatiotemporal feature representation learning for recognizing facial expression in e-learning using the hybrid deep neural network. Except facial expression class information, representative expression states (e.g., onset, apex, offset of expressions) are utilized for expression recognition in our study. Spatiotemporal geometrical feature representations and spatial-temporal appearance feature representations are learned with a hybrid deep neural network. The dual-modality feature fusion representations are used to recognize facial expressions. The comprehensive experiments have been conducted on two spontaneous micro-expression datasets ($$\hbox {CAS(ME)}^2$$ and CASME II). The experimental results showed that the proposed method achieved higher recognition accuracy compared to the state-of-the-art methods. Moreover, multiple metrics were adopted to provide more insight into the performance of the proposed method.

Recently, there has been a huge demand for assistive technology for industrial, commercial, automobile, and societal applications. In some of these applications, there is a requirement of an efficient and accurate system for automatic facial expression recognition (FER). Therefore, FER has gained enormous interest among computer vision researchers. Although there has been a plethora of work available in the literature, an automatic FER system has not yet reached the desired level of robustness and performance. In most of these works, there has been the dominance of appearance-based methods primarily consisting of local binary pattern (LBP), local directional pattern (LDP), local ternary pattern (LTP), gradient local ternary pattern (GLTP), and improved local ternary pattern (IGLTP). Keeping in view the popularity of appearance-based methods, in this paper, we have proposed an appearance-based descriptor called Improved Adaptive Local Ternary Pattern (IALTP) for automatic FER. This new descriptor is an improved version of ALTP, which has been proved to be effective in face recognition. We have investigated ALTP in more details and have proposed some improvements like the use of uniform patterns and dimensionality reduction via principal component analysis (PCA). The reduced features are then classified using kernel extreme learning machine (K-ELM) classifier. In order to validate the performance of the proposed method, experiments have been conducted on three different FER datasets using well-known evaluation measures such as accuracy, precision, recall, and F1-Score. The proposed approach has also been compared with some of the state-of-the-art works in literature and found to be more accurate and efficient.

The automatic recognition of facial expressions is challenging for many reasons, and there is still scope for research in the control of sample quality, the construction of effective feature extraction model and multi feature fusion. This study proposes a facial expression recognition method that combines transfer learning, attention mechanism, and feature fusion technology. In this method, various features are extracted from several pre-trained deep learning networks; the recognition weight of each sample is then learned via a quality attention network. This weight can be used to distinguish the quality of samples and suppress the contribution of low-quality samples in the training process. Finally, for a variety of features, this study proposes a fusion method based on feature clustering and correction. The experimental results indicate that the proposed method can significantly improve the accuracy of expression recognition and achieve state-of-the-art recognition accuracy for RAF-DB and FER2013+ datasets.

Automatic recognition of facial expressions and head gestures plays an important role in a wide range of research area including sign language recognition and human-computer interaction. In this work, we adopt the well-performing self-similarity based action recognition method to classify facial expressions and head gestures. Additionally, we propose a novel approach for facial gesture recognition based on the histogram of tracked facial landmarks. We fuse the presented techniques with our previous Hidden Markov Model based approach [1] and get 15% increase in classification results.

Automatic Facial Expression Recognition (FER) is one of the most active topics in the domain of computer vision and pattern recognition. In this paper, we focus on discrete facial expression recognition by using 4D data (i.e. 3D range image sequences), and present a novel method to address such an issue. The Local Phase Quantisation from Three Orthogonal Planes (LPQ-TOP) descriptor is applied to extract both the static and dynamic clues conveyed in facial expressions. On the one hand, it locally captures the shape attributes in each 3D face model (facial range image). On the other hand, it detects the latent temporal information and represents dynamic changes occurred in facial muscle actions. The SVM classifier is finally used to predict the expression type. The experiments are carried out on the BU-4DFE database, and the achieved results demonstrate the effectiveness of the proposed method.

An efficient, global and local image-processing based extraction and tracking of intransient facial features and automatic recognition of facial expressions from both static and dynamic 2D image/video sequences is presented. Expression classification is based on Facial Action Coding System (FACS) a lower and upper face action units (AUs), and discrimination is performed using Probabilistic Neural Networks (PNN) and a Rule-Based system. For the upper face detection and tracking, we use systems based on a novel two-step active contour tracking system while for the upper face, cross-correlation based tracking system is used to detect and track of Facial Feature Points (FFPs). Extracted FFPs are used to extract some geometric features to form a feature vector which is used to classify input image or image sequences into AUs and basic emotions. Experimental results show robust detection and tracking and reasonable classification where an average recognition rate is 96.11% for six basic emotions in facial image sequences and 94% for five basic emotions in static face images.KeywordsActive contoursAction UnitsFacial ExpressionsProbabilistic Neural Networks

Deep convolution network based emotion analysis towards mental health care

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

Facial expressions convey non-verbal information between humans in face-to-face interactions. Automatic facial expression recognition, which plays a vital role in human-machine interfaces, has attracted increasing attention from researchers since the early nineties. Classical machine learning approaches often require a complex feature extraction process and produce poor results. In this paper, we apply recent advances in deep learning to propose effective deep Convolutional Neural Networks (CNNs) that can accurately interpret semantic information available in faces in an automated manner without hand-designing of features descriptors. We also apply different loss functions and training tricks in order to learn CNNs with a strong classification power. The experimental results show that our proposed networks outperform state-of-the-art methods on the well-known FERC-2013 dataset provided on the Kaggle facial expression recognition competition. In comparison to the winning model of this competition, the number of parameters in our proposed networks intensively decreases, that accelerates the overall performance speed and makes the proposed networks well suitable for real-time systems.