Towards Affective TV with Facial Expression Recognition

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

The ability to recognize facial expressions of emotion is vita l for effective social interaction. Facial expressions convey emotional cues (Adolphs, 2002, 2003; Darwin, 1872/1965; Ekman, 1992, 1993) and accurate recognition of these cues is a necessary step in the evaluation of interpersonal interactions and for the subsequent application of appropriate social skills (Collins & Nowicki, 2001; Hampson, van Anders, & Mullin, 2006; Philippot & Feldman, 1990). Research has shown support for the association between facial expression recognition (FER) abilities and social competency (Custrini & Feldman, 1989; Lancelot & Nowicki, 1997; Nowicki & Duke, 1994); relationship difficulties (Barth & Bastiani, 1997; Blair & Coles, 2000; Lancelot & Nowicki, 1997); and various psychological and psychiatric conditions (Blair et al., 2004; Penn & Combs, 2000; Russell, Chu & Phillips, 2006; Silver, Goodman, Knoll, & Isakov, 2004; van Beek & Dubas, 2008), including anxiety (Easter et al., 2005), bipolar disorder (Brotman et al., 2008), and psychopathology (Blair et al., 2004). Other studies have shown support for associations between FER deficits and childhood maltreatment (Pears & Fisher, 2005; Pollak, Cicchetti, Hornung, & Reed, 2000) and attachment (Cooley, 2005; Magai, Distel & Liker, 1995; Niedenthal, Brauer, Robin & Innes-Ker, 2002). According to emotions theorist Izard (2002), the inability to recognize nonverbal forms of emotion expression can negatively affect intra-and interpersonal behavior, and may serve as a risk factor for poor adjustment and future adverse outcomes. Although the particular mechanisms underlying FER are under debate, the study of the ability to recognize facial expressions of emotion continues to be the subject of research within various fields, including psychology (Hall, 2006; Ekman, Friesen, & Tomkins, 1971; Nowicki & Carton, 1992), psychiatry (Herba & Phillips, 2004; Wang, Lee, Sigman, & Dapretto, 2007), as well as computer science and philosophy (Adolphs, 2002). Facial expression recognition research findings have been provided from varied perspectives, such as neuroscience (Wang et al., 2007); social-cognitive processing and early childhood experiences (Masten et al., 2008; Mullins & Duke, 2004; & Pollack & Sinha, 2000); as well as perception (Pollak, Messner, Kisler, & Cohn, 2009). Other researchers have integrated information from differing fields, such as social psychology and perception (Calder, Young, Keane, & Dean, 2000; Pollak et al., 2009) and behavior studies and neuroscience (Herba & Phillips, 2004). Thus, facial recognition research appears to be bridging some of the natural gaps caused by distinct fields of investigation. The authors of this paper will attempt to present findings from FER studies that are particularly relevant for behavioral psychologists. Following a brief history of the study of FER, we highlight findings from various fields (e.g., social psychology, nonverbal processing), that focused on FER with particular clinical populations. Discussion of suggested clinical implications and applications from a clinical behavioral perspective, and suggestions for future research directions, concludes this article on the importance of FER in clinical work. History of Facial Expression Recognition Study Current research and public interest in facial expression recognition stems from a rich history. Scientific study and understanding of emotion is thought to have begun in the 19th century with Charles Darwin's The Expression of the Emotions in Man and Animals (originally published in 1872) and G.G. Duchenne de Bologne's The Mechanism of Human Facial Expression (originally published in 1862) (Mayne & Bonanno, 2001). These early works focused on the important role of facial displays in emotional life and introduced the theory that emotions may be understood as biologically-based reflex behaviors serving adaptive functions. …

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

These Human facial expressions convey a lot of information visually rather than articulately. Facial expression recognition plays a crucial role in the area of human-machine interaction. Automatic facial expression recognition system has many applications including, but not limited to, human behaviour understanding, detection of mental disorders, and synthetic human expressions. Recognition of facial expression by computer with high recognition rate is still a challenging task. Two popular methods utilized mostly in the literature for the automatic FER systems are based on geometry and appearance. Facial Expression Recognition usually performed in four-stages consisting of pre-processing, face detection, feature extraction, and expression classification. The human face is an important part of an individual’s body and plays an important role in knowing the individual’s mood. The face is where a human expresses all his basic emotions. In the existing system, they examine the mental state manually by assessing them but have many disadvantages like we cannot predict any accurate solutions based on the assessment score because we might be not sure what kind of emotions the human user would be all time. To overcome this problem, a novel system is proposed to suggest an effective solution for predicting the mental state dynamically, we propose a hybrid architecture invoking facial based emotion sequence, PEN test and IQ test. By consistent monitoring of a human’s emotion and subjecting to PEN and IQ tests, the human’s mental state is routed. Combination of the above three techniques provides promising results for mental state and self-control

Cognitive domains related to attention and executive functions (a set of cognitive processes that regulate, control, and manage other cognitive abilities) seem to influence the recognition of facial expressions in people with Alzheimer's disease (AD). We examined the relationship between facial expression recognition, global cognition and executive function in people with AD according to their cognitive level. In a cross-sectional design, we included 130 participants with AD divided into three groups based on their Mini-Mental State Examination (MMSE) scores: MMSE 1 (scores 23-28), MMSE 2 (scores 17-22), and MMSE 3 (scores 11-16). Facial expression recognition ability was analyzed using the Faces Test. Executive function was analyzed using the Trail Making Test (TMT), the Verbal Fluency Test (VFT), the Semantic Fluency Test (SFT), the Digit Span Forward (DSF) and Backward (DSB) tests, and the Clock Drawing Test (CDT). In MMSE 1 group difficulties in divided attention and cognitive flexibility impacted the accuracy of facial expression recognition. In the MMSE 2 group, facial expression recognition was related to impairment in working memory. In the MMSE 3 group, the impact on facial expression recognition was directly related to visuoconstructive abilities. We observed that the executive resources involved in each evaluated group differed in terms of facial recognition task performance efficacy. Interventions at stimulating executive and visuoconstructive abilities in people with AD may contribute to better preservation of facial expression recognition.

Face recognition has many important applications in areas such as public surveillance and security, identity verification in the digital world, and modeling techniques in multimedia data management. Facial expression recognition is also important for targeted marketing, medical analysis, and human–robot interaction. In this paper, we survey a few techniques for facial analysis. We compare the cloud platform AWS Rekognition, convolutional neural networks, transfer learning from pre-trained neural nets, and traditional feature extraction using facial landmarks for this analysis. Although not comprehensive, this survey covers a lot of ground in the state-of-the-art solutions for facial analysis. We show that to get high accuracy, good-quality data and processing power must be provided in large quantities. We present the results of our experiments which have been conducted over six different public as well as proprietary image data sets.

Face and facial expression recognition is a broad research domain in machine learning domain. Non-negative matrix factorization (NMF) is a very recent technique for data decomposition and image analysis. Here we propose face identification system as well as a facial expression recognition, which is a system based on NMF. We get a significant result for face recognition. We test on CK+ and JAFFE dataset and we find the face identification accuracy is nearly 99% and 96.5% respectively. But the facial expression recognition (FER) rate is not as good as it required for the real life implementation. To increase the detection rate for facial expression recognition, our propose fusion based NMF, named as OEPA-NMF, where OEPA means Optimal Expression specific Parts Accumulation. Our experimental result shows OEPA-NMF outperforms the prevalence NMF for facial expression recognition. As face identification using NMF has a good accuracy rate, so we are not interested to apply OEPA-NMF for face identification.

The potential of facial and facial expression recognitions has gained increased interest in social interactions and biometric identification. Earlier facial identification methods suffer from drawbacks due to the lower identification accuracy under difficult lighting conditions. This paper presents two novel new descriptors called Histogram of Directional Gradient (HDG) and Histogram of Directional Gradient Generalized (HDGG) to extracting discriminant facial expression features for better classification accuracy with good efficiency than existing classifiers. The proposed descriptors are based on the directional local gradients combined with SVM (Support Vector Machine) linear classification. To build an efficient face and facial expression recognition, features with reduced dimension are used to boost the performance of the classification. Experiments are conducted on two public-domain datasets: JAFFE for facial expression recognition and YALE for face recognition. The experiment results show the best overall accuracy of 92.12% compared to other existing works. It demonstrates a fast execution time for face recognition ranging from 0.4 to 0.7 s in all evaluated databases.

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

Recognition of facial expression has many applications including human-computer interaction, human emotion analysis, personality development, cognitive science, health-care, virtual reality, image retrieval, etc. In this paper we propose a new method for recognition of facial expression using local region specific mean optical flow and local binary pattern feature descriptor with support vector machine classification. In general, facial expression recognition techniques divide the face into regular grid (holistic representation) and the facial features are extracted. However, in this paper we divide the face into domain specific local regions. At first a robust optical flow is utilized to get mean optical flow in different directions for each local region which considers both local statistic motion information and its spatial location. The features are used only from the key frames; which are detected based on maximal mean optical flow magnitude within a sequence w.r.t. neutral frame. Now, the region specific local binary pattern is extracted from key frame and concatenated with mean optical flow features. The performance of the proposed facial expression recognition system has been validated on CK+ facial expression dataset.

Face alignment and face recognition are two fundamental problems in the facial analysis community. For face alignment, it forms the basis for the accurate face recognition, age estimation, and facial expression recognition. For face recognition, it has been widely applied in various practical scenarios such as access control system, massive surveillance, human computer interaction, etc.. There mainly exist two lines of works in these two fields, namely holistic face alignment and recognition, and partial face alignment and recognition. Numerous holistic face alignment and recognition works

In the current scenario face identification and recognition is an important technique in surveillance. The face is a necessary biometric in humans. Therefore face detection plays a major job in computer vision applications. Several face recognition and emotions classification approaches have been presented throughout the last few decades of research to improve the rate of face recognition for thermal pictures. However, in real-time, lighting conditions might change due to several factors, such as the different times of capture, weather, etc. Due to variations in lighting intensity, the performance of the facial expression recognition system is not good. This paper proposed a model for human thermal face detection and expression classification. Four main steps were involved in this research. Initially, the Difference of the Gaussian (DOG) filter is utilized to crop the input thermal images and then normalize the images using the median filter in pre-processing step. Then, Efficient Net is used for extracting features such as shape, location, and occurrences from thermal face images. After that, detect human faces utilized by the YOLOv4 technique to better emotions classification. Finally, classify the emotions on faces by using the DenseNet technique into seven emotions such as happy, sad, disgust, surprise, anger, fear, and neutral. The proposed method outperforms state-of-art techniques for face recognition on thermal pictures, and classifies the expressions, according to experimentations on the RGB-D-T database. The accuracy, precision, recall, and f1-score metrics will be utilized with the database to assess the efficacy of the proposed methodology. The proposed models achieve a high classification accuracy of 95.97% on the RGB-D-T database. Furthermore, the outcomes show good precision for various face recognition tasks.

The automatic recognition of human faces presents a significant challenge to the pattern recognition research community. Typically, human faces are very similar in structure with minor differences from person to person. They are actually within one class of "human face". Furthermore, lighting conditions change, while facial expressions and pose variations further complicate the face recognition task as one of the difficult problems in pattern analysis. This paper proposes a novel concept: namely, that faces can be recognized using a line edge map (LEM). The LEM, a compact face feature, is generated for face coding and recognition. A thorough investigation of the proposed concept is conducted which covers all aspects of human face recognition, i.e. face recognition under (1) controlled/ideal conditions and size variations, (2) varying lighting conditions, (3) varying facial expressions, and (4) varying pose. The system performance is also compared with the eigenface method, one of the best face recognition techniques, and with reported experimental results of other methods. A face pre-filtering technique is proposed to speed up the search process. It is a very encouraging to find that the proposed face recognition technique has performed better than the eigenface method in most of the comparison experiments. This research demonstrates that the LEM, together with the proposed generic line-segment Hausdorff distance measure, provides a new method for face coding and recognition.

Facial expressions can be seen as a form of non-verbal communication as well as a primary means of conveying social information among humans.Automatic facial expression recognition (FER) can be applied to a wide range of scenarios in human-computer interaction, facial animation, entertainment, and psychology studies. For feature representation in a FER system, various texture descriptors have been employed to derive an effective solution for this system. However, these individual texture descriptor-based FER systems have often failed to achieve effective performance in the recognition of facial expressions. In this sense, it is necessary to further improve the general performance of a facial expression recognition system, evaluating different feature representations. In this paper, a novel local descriptor for a facial expression recognition system is proposed, designated the level of difference descriptor (LOD). The main goal is to use this descriptor as a supplement to state-of-the-art local descriptors to further improve the performance of a FER system in terms of classification accuracy. Furthermore, the fusion of various texture features for devising a robust feature representation for multi-view facial expression recognition is presented.