Facial Expression Recognition Based on Anatomical Structure of Human Face

Enhancing Humans Trust in Robots through Explanations

Intelligent human-computer interaction (HCI) integrates versatile tools such as perceptual recognition, machine learning, affective computing, and emotion cognition to enhance the ways humans interact with computers. Facial expression analysis is one of the essential medium of behavior interpretation and emotion modeling. In this paper, we modify and develop a reconstruction method utilizing Principal Component Analysis (PCA) to perform facial expression recognition. A framework of hierarchical radial basis function network (HRBFN) is further proposed to classify facial expressions based on local features extraction by PCA technique from lips and eyes images. It decomposes the acquired data into a small set of characteristic features. The objective of this research is to develop a more efficient approach to discriminate between seven prototypic facial expressions, such as neutral, smile, anger, surprise, fear, disgust, and sadness. A constructive procedure is detailed and the system performance is evaluated on a public database ”Japanese Females Facial Expression (JAFFE).” We conclude that local images of lips and eyes can be treated as cues for facial expression. As anticipated, the experimental results demonstrate the potential capabilities of the proposed approach.

There has been much research on computer vision in last three decades. Computer vision methods have been developed for different situations. One example is a detection of human face. For computers face detection is hard. Faces look different from different viewing directions. Facial expressions affect to the look of the face. Each individual person has a unique face. The lightning conditions can vary and so on.However, face detection is currently possible in limited conditions. In addition, there are some methods that can be used for gender recognition [3], face recognition [5] and facial expression recognition [2].Nonetheless, there has been very little research on how to combine these methods. There has also been quite little research on how to apply these methods in human-computer interaction (HCI).Finding sets of techniques that complement each other in a useful way is one research challenge. There are some applications that take advantage of one or two computer vision techniques. For example, Christian and Avery [1] have developed an information kiosk that uses computer vision to detect potential users from a distance. A similar kiosk has been developed by us in the University of Tampere [4]. There are also some games that use simple computer vision techniques for the interaction. However, there are very few applications that use several computer vision techniques together such as face detection, facial expression recognition and gender recognition. Overall, there has been very little effort in combining different techniques.In my research I develop computer vision methods and combine them, so that the combined method can detect face, recognize gender and facial expressions. After successfully combining the methods, it is easier to develop HCI applications that take advantage of computer vision. Applications that can be used by small group of people are my specific interest. These applications allow me to build adaptive user interfaces and analyze the use of computer vision techniques in improving human-computer interaction.

Smart cameras are becoming more popular in human computer interaction (HCI). One of HCI research areas, multimodal user interface (MMUI) allows user to interact with a computer by using his or her natural communication modalities, such as speech, pen, touch, gestures, eye gaze, and facial expression. This paper presents the hardware and software co-design and implementation of KLT (Kanade Lucas Tomasi) tracking algorithm in a FPGA-based smart camera prototype for recognize simple hand gestures consisting of a CMOS image sensor capture unit and FPGA main video processor. This tracking system uses face and hand detections as a tool to detect and track gesture (face and hand motion). This gesture tracking system that are based on Harris Keypoint Detection algorithm and KLT tracking algorithm has been successfully implemented in FPGA and shows promising result in real time performance.

One of the most representative manners which human shows his inner feelings is through facial expressions. Automatic recognition of these gestures is of great importance in human computer interaction (HCI) applications. In order to automatically recognize facial expressions, local binary patterns (LBPs), which are powerful and computationally simple descriptors, have been extensively utilized for feature extraction from facial images. To improve image description accuracy, we have employed a new version of LBP, namely high order joint derivative local binary pattern (HJDLBP) in this paper which reveals spatial relations between patterns in different resolutions. Furthermore, a rotation invariant radial partitioning paradigm is utilized to localize LBP histogram calculation regions. Using radial partitioning instead of blocking eliminates head rotation effects in real world applications and improves classification accuracy. A bunch of support vector machines are trained by feature vectors to classify different facial expressions. To evaluate the proposed method we have used known and publicly available JAFFE dataset for experiments and compared the obtained results with results from state of the art approaches which use original LBP and rectangular partitioning. The comparison results show the effectiveness of the new descriptor in combination with radial partitioning for automatic facial expression categorization.

In this paper, we propose a novel method for facial expression recognition. The facial expression is extracted from human faces by an expression classifier that is learned from boosting Haar feature based look-up-table type weak classifiers. The expression recognition system consists of three modules, face detection, facial feature landmark extraction and facial expression recognition. The implemented system can automatically recognize seven expressions in real time that include anger, disgust, fear, happiness, neutral, sadness and surprise. Experimental results are reported to show its potential applications in human computer interaction.

Making a computer generate its own emotion is an important part of the affective computing, and this would have wide applications in human-computer interaction and artificial intelligence. In this paper, we will describe an emotion generation model for a multimodal virtual human. The relationship among the emotion, mood and personality are discussed firstly, and the PAD (pleasure-arousal-dominance) emotion space is used to define the emotion and the mood. Then, we use a random graphical model to generate emotion based on the evaluation of the overall influence of mood, previous emotion and the outside stimulations. Finally, a 3D virtual human head with facial expressions is designed to show the emotion generation outputs. Experimental results demonstrate that the emotion generation model based on random graphs works effectively and meets the basic principle of human emotion generation.

Automatic facial expression recognition (FER) has gained much attention due to its applications in human-computer interaction. Among the approaches to improve FER tasks, this paper focuses on deep architecture with the attention mechanism. We propose a novel Masking Idea to boost the performance of CNN in facial expression task. It uses a segmentation network to refine feature maps, enabling the network to focus on relevant information to make correct decisions. In experiments, we combine the ubiquitous Deep Residual Network and Unetlike architecture to produce a Residual Masking Network. The proposed method holds state-of-the-art (SOTA) accuracy on the well-known FER2013 and private VEMO datasets

There are few reports on the relationship between facial expression formation and mass of the muscle responsible for facial expression. We analyzed the facial expression using facial action coding system (FACS) and examined the muscle mass and characteristics of the facial expression muscles using multi-detector row computed tomography (MDCT) and magnetic resonance imaging (MRI). Moreover, the relation between these was statistically evaluated. Ten healthy women in their 40s (43.4±3.0years, 40-49) were enrolled. The expressive faces were analyzed by facial expression analysis software based on the FACS. The muscle mass and characteristics of the facial expression muscles were investigated using MDCT/MRI. The correlation between an integrated expression intensity value (IEIV) for FACS of the widest possible grin and muscle mass was analyzed. The mean values between the two categorized groups (G-1 and G-2) based on fat infiltration into the muscle were compared. A positive correlation is found between the IEIV and the muscle mass. The IEIV of G-1 is significantly larger than the corresponding value of G-2. Hence, the results indicated that the subjects with high IEIV and expressive face had thicker facial expression muscles and little fat infiltration into the muscles. Our objective imaging diagnostic study using FACS, MDCT, and MRI corroborated the anti-aging medical science about the facial expression muscles related to youthful facial appearance. The results of this research could contribute to the elucidation of the mechanisms involved in the facial aging process and to the development of cosmetology.

This study examined the association of the experience of emotion and somatic changes. The study compared reported somatic changes generally experienced when anxious with the actual association of the experience of emotion and somatic changes as measured during a specific event. Emotions were measured as both general negative emotion as well as specific emotions: anger, disgust, fear, sadness and shame. Participants were volunteers from a Victorian university who agreed to watch a video depicting the dramatisation of child abuse. Throughout the video, participants indicated their experience of emotion. Measures were also taken throughout the procedure of facial expression and Galvanic Skin Response (GSR). In order to examine emotional-congruence, subjects were divided into three groups. These groups were divided according to the congruence of subjects' experienced emotion with autonomic changes and facial expressivity. Groups were divided separately for each of the emotion types. Where there was little difference between the reported experience of emotion and that, which would have been expected from the observed somatic changes, the subject was deemed to be in the Congruent Group. Subjects whose reported experience of emotion was greater or less than would be expected from observed somatic changes were allocated to the Over-reporter and Under-Reporter Groups respectively. This data was then compared to participants' reports of the number of somatic symptoms usually experienced when anxious. It was found that participants who under-report the experience of general negative-emotion compared with their observed somatic changes (both GSR and facial expressivity) had lower trait-somatic-anxiety (reported fewer somatic symptoms usually experienced when anxious). There was no significant difference between the Congruent Group and Over-Reporter Group.;The Under-Reporter Groups had significantly lower trait-somatic-anxiety than the Congruent Group when emotional-congruence was defined by fear and GSR, anger and GSR and sadness and facial expressivity. The actual association of shame and disgust with either somatic change, sadness with autonomic change and anger and fear with facial expressivity was unrelated to the number of somatic symptoms reported to be usually experienced when anxious. The results supported the idea that subjective reports of the number of somatic symptoms reported to be usually experienced when anxious reflect the actual association of somatic change and experience, but with limitations. The actual association of experience of fear with autonomic change seems to reflect the number of somatic symptoms reported to be usually experienced when anxious more than other emotions. Further for those for whom the experience of anger and negative-emotion has a greater association with somatic change, there was a greater number of somatic symptoms reported to be usually experienced when anxious. This would suggest that some people have a greater association of some experiences of emotion and somatic change. Furthermore, while there is an association between reported somatic changes generally experienced when anxious with the actual association of the experience of emotion and somatic changes as measured during a specific event, this was dependant on the association of the emotion types rather than being generalised for all emotions.

Facial expression recognition is a challenging and interesting problem. It has many potential and important applications in data-driven animation, human computer interaction (HCI), social robots, deceit detection and behavior monitoring. In this paper, we present the novel descriptors Pyramid local phase quantization (PLPQ). The effective of our proposed descriptor is evaluated by facial expressions recognition very efficiently and with high accuracy. On the other hand, the proposed framework extracts texture features in a pyramidal fashion only from the perceptual salient region of the face thereby our proposed framework achieved reduction in computation time of feature extraction and improved accuracy. There with the proposed framework achieved accuracy of 96.7% on extended Cohn-Kanade (CK+) posed facial expression database for six basic emotions and exceed the state-of-theart methods for expression recognition using texture features.

There are two major approaches to creating 3DCG facial expressions: The first is based on facial muscle simulation and the second is the blend-shape approach. The blend shape approach is more familiar to creators than the facial muscle approach when they synthesize the facial expressions of 3DCG characters. However, the facial muscle model has the advantage of being physics-based. It can, therefore, produce realistic facial expressions and create facial expressions using fewer parameters than the blend shape approach, thereby reducing processing time and computational requirements. We introduce a method which can be used to synthesize individual facial expressions based on the facial muscle model [Waters 1987].

Face expression recognition plays a crucial role in emotion recognition, human-computer interaction and other fields. The aim of this paper is to implement a deep learning-based monitoring system for real-time facial expression recognition. In this paper, a public expression dataset ExpW is used, which is a widely used public face expression dataset. The dataset contains face images from various scenarios in the real world, covering a rich variety of expressions, encompassing seven major expression categories: anger, disgust, fear, happiness, sadness, surprise, and neutrality. The YOLOv5 target detection algorithm is selected on the deep learning framework for the training and testing of the training set, which is a high-performance algorithm for fast and accurate implementation of the target detection task through a simplified network structure and optimized algorithm design. The experimental evaluation results can be concluded that the system in this paper achieves accurate expression recognition in real-time scenarios. The results of this paper show that real-time face expression recognition monitoring based on deep learning has potential applications in emotion recognition and human-computer interaction.

The lack of visibility in text-based chat can hinder communication, especially when nonverbal cues are instrumental to the production and understanding of messages. However, communicating rich nonverbal cues such as facial expressions may be technologically more costly (e.g., demand of bandwidth for video streaming) and socially less desirable (e.g., disclosing other personal and context information through video). We consider how to balance the tension by supporting people to convey facial expressions without compromising the benefits of invisibility in text communication. We present KinChat, an enhanced text chat tool that integrates motion sensing and 2D graphical visualization as a technique to convey information of key facial features during text conversations. We conducted two studies to examine how KinChat influences the de-identification and awareness of facial cues in comparison to other techniques using raw and blurring-processed videos, as well as its impact on real-time text chat. We show that feature-based visualization of facial expression can preserve both awareness of facial cues and non-identifiability at the same time, leading to better understanding and reduced anxiety.

Expressing emotions through various modalities is a crucial function not only for humans but also for robots. The mapping method from facial expressions to the basic emotions is widely used in research on robot emotional expressions. This method claims that there are specific facial muscle activation patterns for each emotional expression and people can perceive these emotions by reading these patterns. However, recent research on human behavior reveals that some emotional expressions, such as the emotion “intense”, are difficult to judge as positive or negative by just looking at the facial expression alone. Nevertheless, it has not been investigated whether robots can also express ambiguous facial expressions with no clear valence and whether the addition of body expressions can make the facial valence clearer to humans. This paper shows that an ambiguous facial expression of an android can be perceived more clearly by viewers when body postures and movements are added. We conducted three experiments and online surveys among North American residents with 94, 114 and 114 participants, respectively. In Experiment 1, by calculating the entropy, we found that the facial expression “intense” was difficult to judge as positive or negative when they were only shown the facial expression. In Experiments 2 and 3, by analyzing ANOVA, we confirmed that participants were better at judging the facial valence when they were shown the whole body of the android, even though the facial expression was the same as in Experiment 1. These results suggest that facial and body expressions by robots should be designed jointly to achieve better communication with humans. In order to achieve smoother cooperative human-robot interaction, such as education by robots, emotion expressions conveyed through a combination of both the face and the body of the robot is necessary to convey the robot’s intentions or desires to humans.