Facial Emotion Expressions in Human–Robot Interaction: A Survey

Objectives: Attention-deficit/hyperactivity disorder (ADHD) is the most frequent neurobehavioural disorder in childhood. ADHD is associated with impaired academic performance, cognitive, and emotional deficits. Moreover, comorbid oppositional defiant disorder (ODD) is leading to more severe impairment in social performance. Social cognition involves recognition, encoding, and interpretation of emotions from faces. Basic facial expressions that include sadness, happiness, anger, disgust, fear, and surprise are the easiest emotions to recognize. We aimed to demonstrate facial expression recognition impairments that might occur more frequently in children with co-occurring ADHD/ODD than patients with ADHD only. Thus, children with the co-occurrence of ODD may suffer more severely from social and behavioural difficulties.Methods: Forty patients diagnosed with ADHD and/or co-occurring ADHD/ODD according to DSM-IV-TR criteria were compared with a parallel (by gender, age, and educational state) 11 healthy children as a control group in this study. Clear facial images of each emotion were used as well as two additional sets of photos include 50% blurred images and cropped eye images were added as distractors then all images represented with black and white tone for emotion recognition task via facial expression. Angry expressions presented as target expressions. DSM-IV-Based Screening and Rating Scale for Children and Adolescents with attention deficit and disruptive behaviour disorders, the Conners’ Teachers Rating Scale/Revised Long Form and the Conners’ Parent Rating Scale/Revised Long Form were used to provide diagnostic objectivity.Results: Control group statistically performed better than ADHD group on recognition of emotional facial expressions. Results showed no statistically significant differences between the ADHD and ADHD/ODD group on recognition of emotional facial expressions. However, according to results of emotion recognition task via facial expressions, there were statistically significant differences between pure ADHD and comorbid ADHD+ODD groups in happy and neutral expressions. ADHD/ODD group tend to attribute more meaning to neutral facial expressions. Additionally there was statistically significant difference between control group and ADHD group according to recognition of angry expressions. There were statistically significant differences between the groups according to recognition of sad expressions in all clear, blurred, and eye photographs.Conclusions: Difficulties in recognizing emotional facial expressions were observed in children with ADHD. A statistically significant association was established between presence of ADHD and impaired recognition of facial emotion expressions independent from the scores of the disruptive behaviour rating scale. Comorbid ODD was not associated with recognition of emotional facial expressions including angry expressions. Recognition of angry expressions was not found as a predictor for disruptive behaviour disorders.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

Accurately recognizing facial emotional expressions is important in psychiatrist-versus-patient interactions. This might be difficult when the physician and patients are from different cultures. More than two decades of research on facial expressions have documented the universality of the emotions of anger, contempt, disgust, fear, happiness, sadness, and surprise. In contrast, some research data supported the concept that there are significant cultural differences in the judgment of emotion. In this pilot study, the recognition of emotional facial expressions in 123 Japanese subjects was evaluated using the Japanese and Caucasian Facial Expression of Emotion (JACFEE) photos. The results indicated that Japanese subjects experienced difficulties in recognizing some emotional facial expressions and misunderstood others as depicted by the posers, when compared to previous studies using American subjects. Interestingly, the sex and cultural background of the poser did not appear to influence the accuracy of recognition. The data suggest that in this young Japanese sample, judgment of certain emotional facial expressions was significantly different from the Americans. Further exploration in this area is warranted due to its importance in cross-cultural clinician-patient interactions.

BackgroundPrevious reports have suggested impairment in facial expression recognition in delinquents, but controversy remains with respect to how such recognition is impaired. To address this issue, we investigated facial expression recognition in delinquents in detail.MethodsWe tested 24 male adolescent/young adult delinquents incarcerated in correctional facilities. We compared their performances with those of 24 age- and gender-matched control participants. Using standard photographs of facial expressions illustrating six basic emotions, participants matched each emotional facial expression with an appropriate verbal label.ResultsDelinquents were less accurate in the recognition of facial expressions that conveyed disgust than were control participants. The delinquents misrecognized the facial expressions of disgust as anger more frequently than did controls.ConclusionThese results suggest that one of the underpinnings of delinquency might be impaired recognition of emotional facial expressions, with a specific bias toward interpreting disgusted expressions as hostile angry expressions.

Every person expresses himself or herself with different facial expressions. Nowadays, recognition of facial expression is a popular topic for researchers due to the practical function in a wide variety of areas. The learning of human facial expressions is a tough and tricky domain in pattern research society. Each individual facial expression is produced by non-rigid object contortions, and these contortions are man-dependent. The main motto of this paper was to invent and apply a structure for the automatic recognition of human facial expression.

This paper proposes a unified facial motion tracking and expression recognition framework for monocular video. For retrieving facial motion, an online weight adaptive statistical appearance method is embedded into the particle filtering strategy by using a deformable facial mesh model served as an intermediate to bring input images into correspondence by means of registration and deformation. For recognizing facial expression, facial animation and facial expression are estimated sequentially for fast and efficient applications, in which facial expression is recognized by static anatomical facial expression knowledge. In addition, facial animation and facial expression are simultaneously estimated for robust and precise applications, in which facial expression is recognized by fusing static and dynamic facial expression knowledge. Experiments demonstrate the high tracking robustness and accuracy as well as the high facial expression recognition score of the proposed framework.

The recognition of emotional facial expressions is an important means to adjust behavior in social interactions. As facial expressions widely differ in their duration and degree of expressiveness, they often manifest with short and transient expressions below the level of awareness. In this combined behavioral and fMRI study, we aimed at examining whether or not consciously accessible (subliminal) emotional facial expressions influence empathic judgments and which brain activations are related to it. We hypothesized that subliminal facial expressions of emotions masked with neutral expressions of the same faces induce an empathic processing similar to consciously accessible (supraliminal) facial expressions. Our behavioral data in 23 healthy subjects showed that subliminal emotional facial expressions of 40 ms duration affect the judgments of the subsequent neutral facial expressions. In the fMRI study in 12 healthy subjects it was found that both, supra- and subliminal emotional facial expressions shared a widespread network of brain areas including the fusiform gyrus, the temporo-parietal junction, and the inferior, dorsolateral, and medial frontal cortex. Compared with subliminal facial expressions, supraliminal facial expressions led to a greater activation of left occipital and fusiform face areas. We conclude that masked subliminal emotional information is suited to trigger processing in brain areas which have been implicated in empathy and, thereby in social encounters

This study investigated the ability of patients with Relapsing-Remitting Multiple Sclerosis (RRMS) to recognize emotional facial expressions. Cognitive deterioration, depression, alexithymia and facial expression recognition ability were assessed in fiftyfive patients and twenty-one controls. Facial expression recognition ability was measured by a forced-choice labeling procedure of five emotional facial expressions (anger, fear, sadness, happiness, none). RRMS patients exhibited a global impairment in the recognition of facial emotion (p = .00049), specifically for anger (p = .01), sadness (p = .0001), and fear, (p = .011). Deficit in emotion recognition was independent from disability (assessed by EDSS score). This deficit was correlated with depression and partially with cognitive deterioration. These results should be discussed in term of global cortico-cortical disconnections.

Facial expressions recognition and synthesis are important research fields to study how human beings reflect to environments in affective computing. With the rapid development of mathematical theory on multivariate statistics and multi-media technology especially image processing, facial expressions recognition researchers have achieved many useful results. Recently studies show that approaches to facial modeling and expressions recognition and synthesis analysis could be adapted to control security or even real-time health monitoring in the real world. Similarity, how to achieve facial expressions recognition and synthesis for independent-free mechanism is a central design in general. Based on the cognitive analysis of independent user's affective facial recognition researches we proposed an emotions composition model to dope out what are the user's new affective facial expressions. At this point, principal component, cluster and discriminate analysis were applied to show and verify independent user's affective expressions can be composited for synthesis by basic facial expressions such as happy, neutral and unhappy. Experiments were conducted to prove our models to be very significant.

In order to solve the problems of low average gradient and long recognition time in traditional facial expression recognition method, a multi-scale detail enhancement method for facial dynamic expression recognition of sprint athletes is proposed. A principal component analysis method was used to establish the facial expression feature subspace of sprinters, to project and reduce the dimension of the facial dynamic expression feature vector of sprinters, and to obtain the low frequency information and high frequency information of the facial image of sprinters by bilateral filtering. The multi-scale details of expression are enhanced by using side suppression network model and improving image S curve. The feature vector of facial dynamic expression is input into support vector machine to recognise the facial dynamic expression of sprinter. Experimental results show that the average value of annoying gradient is about 98 and the shortest time s. is about 1.9.

While the recognition of emotional expressions has been extensively studied, the behavioural response to these expressions has not. In the interpersonal circumplex, behaviour is defined in terms of communion and agency. In this study, we examined behavioural responses to both facial and postural expressions of emotion. We presented 101 Romanian students with facial and postural stimuli involving individuals ('targets') expressing happiness, sadness, anger, or fear. Using an interpersonal grid, participants simultaneously indicated how communal (i.e., quarrelsome or agreeable) and agentic (i.e., dominant or submissive) they would be towards people displaying these expressions. Participants were agreeable-dominant towards targets showing happy facial expressions and primarily quarrelsome towards targets with angry or fearful facial expressions. Responses to targets showing sad facial expressions were neutral on both dimensions of interpersonal behaviour. Postural versus facial expressions of happiness and anger elicited similar behavioural responses. Participants responded in a quarrelsome-submissive way to fearful postural expressions and in an agreeable way to sad postural expressions. Behavioural responses to the various facial expressions were largely comparable to those previously observed in Dutch students. Observed differences may be explained from participants' cultural background. Responses to the postural expressions largely matched responses to the facial expressions.

Different styles of social interaction are one of the core characteristics of autism spectrum disorder (ASD). Social differences among individuals with ASD often include difficulty in discerning the emotions of neurotypical people based on their facial expressions. This review first covers the rich body of literature studying differences in facial emotion recognition (FER) in those with ASD, including behavioral studies and neurological findings. In particular, we highlight subtle emotion recognition and various factors related to inconsistent findings in behavioral studies of FER in ASD. Then, we discuss the dual problem of FER – namely facial emotion expression (FEE) or the production of facial expressions of emotion. Despite being less studied, social interaction involves both the ability to recognize emotions and to produce appropriate facial expressions. How others perceive facial expressions of emotion in those with ASD has remained an under-researched area. Finally, we propose a method for teaching FER [FER teaching hierarchy (FERTH)] based on recent research investigating FER in ASD, considering the use of posed vs. genuine emotions and static vs. dynamic stimuli. We also propose two possible teaching approaches: (1) a standard method of teaching progressively from simple drawings and cartoon characters to more complex audio-visual video clips of genuine human expressions of emotion with context clues or (2) teaching in a field of images that includes posed and genuine emotions to improve generalizability before progressing to more complex audio-visual stimuli. Lastly, we advocate for autism interventionists to use FER stimuli developed primarily for research purposes to facilitate the incorporation of well-controlled stimuli to teach FER and bridge the gap between intervention and research in this area.

Background/Study Context: Elderly people do not categorize emotional facial expressions as accurately as younger people, particularly negative emotions. Although age-related impairments in decoding emotions in facial expressions are well documented, the causes of this deficit are poorly understood. This study examined the potential mechanisms that account for this age-related categorization deficit by assessing its dependence on presentation time.Methods: Thirty young (19–27 years old) and 31 older (68–78 years old) Chinese adults were asked to categorize the six basic emotions in facial expressions, each presented for 120, 200, 600, or 1000 ms, before and after exposure to a neutral facial expression.Results: Shortened presentation times caused an age-related deficit in the recognition of happy faces, whereas no deficit was observed at longer exposure times. An age-related deficit was observed for all negative emotions but was not exacerbated by shorter presentation times.Conclusion: Age-related deficits in categorization of positive and negative emotions are caused by different mechanisms. Because negative emotions are perceptually similar, they cause high categorization demands. Elderly people may need more evidence in favor of the target emotion than younger people, and they make mistakes if this surplus of evidence is missing. In contrast, perceptually distinct happy faces were easily identified, and elderly people only failed when the presentation time was too short for their slower perceptual processing.

Facial expression recognition impairment has been reported in Parkinson's disease. While some authors have referred to specific emotional disabilities, others view them as secondary to executive deficits frequently described in the disease, such as working memory. The present study aims to analyze the relationship between working memory and facial expression recognition abilities in Parkinson's disease. We observed 50 patients with Parkinson's disease and 49 healthy controls by means of an n-back procedure with four types of stimuli: emotional facial expressions, gender, spatial locations, and non-sense syllables. Other executive and visuospatial neuropsychological tests were also administered. Results showed that Parkinson's disease patients with high levels of disability performed worse than healthy individuals on the emotional facial expression and spatial location tasks. Moreover, spatial location task performance was correlated with executive neuropsychological scores, but emotional facial expression was not. Thus, working memory seems to be altered in Parkinson's disease, particularly in tasks that involve the appreciation of spatial relationships in stimuli. Additionally, non-executive, facial emotional recognition difficulty seems to be present and related to disease progression.