Combining Deep Neural Network with Traditional Classifier to Recognize Facial Expressions

Facial Expression Recognition (FER) is a critical component of affective computing, with deep learning models dominating performance metrics. In contrast, geometric approaches based on the Facial Action Coding System (FACS) offer explainability through using triangles aligned to facial landmarks. The notable points of these triangles capture the deformation of muscles. However, restricting the feature extraction to notable points may be suboptimal. This paper introduces a novel method for optimizing the extraction of features by searching for optimal inner points in 22 facial triangles applying three metaheuristics: Differential Evolution (DE), Particle Swarm Optimization (PSO), and Convex Partition (CP). This results in a set of 59 geometric-based descriptors that capture muscle deformation more accurately. The proposed method was evaluated using five machine learning classifiers on two benchmark databases: the Karolinska Directed Emotional Faces (KDEF) and the Japanese Female Facial Expression (JAFFE). Experimental results demonstrate significant performance improvements. The combination of DE with a Multi-Layer Perceptron (MLP) achieved an accuracy of 0.91 on the KDEF database, while Support Vector Machine (SVM) optimized via CP attained an accuracy of 0.81 on the JAFFE database. Statistical analysis confirms that optimized descriptors yield higher accuracy than previous geometric methods.

Facial Expression Recognition (FER) is an important topic that is used in many areas. FER categorizes facial expressions according to human emotions. Most networks are designed for facial emotion recognition but still have some problems, such as performance degradation and the lowest classification accuracy. To achieve greater classification accuracy, this paper proposes a new Leaky Rectified Triangle Linear Unit (LRTLU) activation function based on the Deep Convolutional Neural Network (DCNN). The input images are pre-processed using the new Adaptive Bilateral Filter Contourlet Transform (ABFCT) filtering algorithm. The face is then detected in the filtered image using the Chehra face detector. From the detected face image, facial landmarks are extracted using a cascading regression tree, and important features are extracted based on the detected landmarks. The extracted feature set is then passed as input to the Leaky Rectified Triangle Linear Unit Activation Function Based Deep Convolutional Neural Network (LRTLU-DCNN), which classifies the input image expressions into six emotions, such as happiness, sadness, neutrality, anger, disgust, and surprise. Experimentation of the proposed method is carried out using the Extended Cohn-Kanade (CK+) and Japanese Female Facial Expression (JAFFE) datasets. The proposed work provides a classification accuracy of 99.67347% for the CK+ dataset along with 99.65986% for the JAFFE dataset.

Texture characteristics among the landmark points reflected in human faces are important features in the recognition of facial expression. In this paper, we propose a salient feature based facial expression on texture characteristics. Detection of effective landmarks are performed by appearance based models and corresponding texture regions are extracted from face images. Local Binary Pattern (LBP) is used to compute the texture feature. Texture feature is used for normalizing texture signatures. In addition, we introduce stability indices corresponding to texture features which are exploited as features in the present scope. Moreover, statistical features such as moment, skewness, kurtosis and entropy are also supplemented to the feature set. These salient features together are used as an input to Nonlinear AutoRegressive with eXogenous (NARX) for recognition of the human facial expression. The Cohn-Kanade (CK+), Japanese Female Facial Expression (JAFFE), MMI and MUG benchmark databases are used to conduct the experiments and the results justify the effectiveness of the proposed procedure.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

There have been quite a few studies on facial expression recognition over the years, and it is still a challenging subject due to the significant inter-class variability. Facial expression research in this field focuses on the development of techniques to identify, code, and extract facial expressions to improve prediction by computer. With great success of machine learning, the various texture descriptors are exploited to obtain a better performance. This paper proposes a method based on the aggregation between different descriptors Histogram of oriented Gradient (HOG) and Local Binary Pattern (LBP). First stage the input image has pre-processed to detect dace area which helps to extract most significant features. Then, Diagonal-HOG (D-HOG) also has extracted and aggregated all features. Finally, Support Vector Machine (SVM) has been used a classifier to classify each feature as well as aggregated features. We evaluate our method using Japanese Female Facial Expressions database (JAFFE), experimental results showed that the proposed method is accurate and efficient in recognizing facial expressions.

A novel approach to facial expression recognition (FER) based on the combination of Non-negative Matrix Factorization (NMF) and Support Vector Machine (SVM) was proposed. One key step in FER is to extract expression features from the original face images. NMF is an effective approach to extract expression features because NMF decomposition makes the reconstruction of expression images in a non-subtractive way and is much similar to the process of forming unity from parts. The proposed algorithm first processes facial expression image with histogram equalization operator. Then NMF method is used for feature dimension reduction and SVM for classification. Finally, the algorithm was implemented with Matlab and experimented in Japanese female facial expression database (JAFEE database). A recognition rate of 66.19% was obtained and showed the effectiveness of the proposed algorithm

Quantum-inspired binary gravitational search algorithm to recognize the facial expressions

Dataset classification: An efficient feature extraction approach for grammatical facial expression recognition

Facial Expression Recognition (FER) is an important area in human computer interaction. FER has different applications such as analysis of student behaviour in virtual class room, driver mood detection, security systems, and medicine. The analysis of facial expressions is an interesting and exciting problem. Feature extraction plays important role in any FER system. Local Binary Pattern (LBP) and its variants are popular for feature extraction due to simplicity in computation and monotonic illumination invariant property. However, the performance of LBP is poor in the presence of noise. This work proposes a novel approach for feature extraction to improve the performance of the FER. In this approach, the LBP is calculated considering 4-neighbors and diagonal neighbours separately. Further, for affective feature description, the concept of adaptive window and averaging in radial directions is introduced. This approach reduces the length of the feature vector as well as immune to noise. Support Vector Machine (SVM) is considered for classification. Recognition rate and confusion matrix are used to assess the performance of the proposed algorithm. Extensive experimental results on JAFFE, CK, FERG and FEI face databases show significant improvement in recognition rate compared to the available techniques both in noise free and noisy conditions.

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.

Facial expression is an important form of human nonverbal communication. Recognition of this nonverbal sign may enable developers to understand the feedback of the servers of smart devices or advertising. Existing approaches for facial expression recognition are mainly based on cameras or on-body sensors, which are either sensitive to lighting conditions or require users to wear cumbersome devices on their faces. In this paper, we propose a new facial expression recognition system based on Wi-Fi signals, named WiFace. Our key intuition is that facial muscle movements in different expressions will induce distinctive waveform patterns in the time-series of Channel State Information (CSI) in Wi-Fi signals. We develop a series of algorithms to process the CSI signals and extract the most representative waveform patterns for facial expression classification. We build a fully-functional prototype of WiFace using commercial off-the-shelf devices, which can recognize 6 typical facial expressions. We conduct extensive experiments to evaluate the performance of WiFace, and the experimental results show that the average recognition accuracy is 94.0%.

In this paper, a new method for extracting expression-independent face features based on HOSVD (Higher-Order Singular Value Decomposition) is proposed and used for face recognition. In the new method, it is assumed that a facial expression could be represented by the facial expressions in the training set. In addition, the expression with higher similarity to the expression of test person has higher probability to represent the expression of test person. Expression-similarity weighted expression feature, which is the optimal estimation based on Bayesian estimation theory and the assumption, is used to estimate the face feature of the test person. As a result, the estimated face feature can reduce the influence of expression caused by insufficient training data and becomes less expression-dependent, and can be more robust to new expressions. The proposed method has been applied to Japanese Female Facial Expression (JAFFE) database. Expression-independent experimental results show the superiority of proposed method over the existing methods in terms of recognition rate and accumulative recognition rate.

The ability to recognize facial expressions of emotion is vita l for effective social interaction. Facial expressions convey emotional cues (Adolphs, 2002, 2003; Darwin, 1872/1965; Ekman, 1992, 1993) and accurate recognition of these cues is a necessary step in the evaluation of interpersonal interactions and for the subsequent application of appropriate social skills (Collins & Nowicki, 2001; Hampson, van Anders, & Mullin, 2006; Philippot & Feldman, 1990). Research has shown support for the association between facial expression recognition (FER) abilities and social competency (Custrini & Feldman, 1989; Lancelot & Nowicki, 1997; Nowicki & Duke, 1994); relationship difficulties (Barth & Bastiani, 1997; Blair & Coles, 2000; Lancelot & Nowicki, 1997); and various psychological and psychiatric conditions (Blair et al., 2004; Penn & Combs, 2000; Russell, Chu & Phillips, 2006; Silver, Goodman, Knoll, & Isakov, 2004; van Beek & Dubas, 2008), including anxiety (Easter et al., 2005), bipolar disorder (Brotman et al., 2008), and psychopathology (Blair et al., 2004). Other studies have shown support for associations between FER deficits and childhood maltreatment (Pears & Fisher, 2005; Pollak, Cicchetti, Hornung, & Reed, 2000) and attachment (Cooley, 2005; Magai, Distel & Liker, 1995; Niedenthal, Brauer, Robin & Innes-Ker, 2002). According to emotions theorist Izard (2002), the inability to recognize nonverbal forms of emotion expression can negatively affect intra-and interpersonal behavior, and may serve as a risk factor for poor adjustment and future adverse outcomes. Although the particular mechanisms underlying FER are under debate, the study of the ability to recognize facial expressions of emotion continues to be the subject of research within various fields, including psychology (Hall, 2006; Ekman, Friesen, & Tomkins, 1971; Nowicki & Carton, 1992), psychiatry (Herba & Phillips, 2004; Wang, Lee, Sigman, & Dapretto, 2007), as well as computer science and philosophy (Adolphs, 2002). Facial expression recognition research findings have been provided from varied perspectives, such as neuroscience (Wang et al., 2007); social-cognitive processing and early childhood experiences (Masten et al., 2008; Mullins & Duke, 2004; & Pollack & Sinha, 2000); as well as perception (Pollak, Messner, Kisler, & Cohn, 2009). Other researchers have integrated information from differing fields, such as social psychology and perception (Calder, Young, Keane, & Dean, 2000; Pollak et al., 2009) and behavior studies and neuroscience (Herba & Phillips, 2004). Thus, facial recognition research appears to be bridging some of the natural gaps caused by distinct fields of investigation. The authors of this paper will attempt to present findings from FER studies that are particularly relevant for behavioral psychologists. Following a brief history of the study of FER, we highlight findings from various fields (e.g., social psychology, nonverbal processing), that focused on FER with particular clinical populations. Discussion of suggested clinical implications and applications from a clinical behavioral perspective, and suggestions for future research directions, concludes this article on the importance of FER in clinical work. History of Facial Expression Recognition Study Current research and public interest in facial expression recognition stems from a rich history. Scientific study and understanding of emotion is thought to have begun in the 19th century with Charles Darwin's The Expression of the Emotions in Man and Animals (originally published in 1872) and G.G. Duchenne de Bologne's The Mechanism of Human Facial Expression (originally published in 1862) (Mayne & Bonanno, 2001). These early works focused on the important role of facial displays in emotional life and introduced the theory that emotions may be understood as biologically-based reflex behaviors serving adaptive functions. …

Facial feature extraction and recognition plays a prominent role in human non-verbal interaction and it is one of the crucial factors among pose, speech, facial expression, behaviour and actions which are used in conveying information about the intentions and emotions of a human being. In this article an extended local binary pattern is used for the feature extraction process and a principal component analysis (PCA) is used for dimensionality reduction. The projections of the sample and model images are calculated and compared by Euclidean distance method. The combination of extended local binary pattern and PCA (ELBP+PCA) improves the accuracy of the recognition rate and also diminishes the evaluation complexity. The evaluation of proposed facial expression recognition approach will focus on the performance of the recognition rate. A series of tests are performed for the validation of algorithms and to compare the accuracy of the methods on the JAFFE, Extended Cohn-Kanade images database.