Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot

For supervised training of automatic facial expression recognition systems, adequate ground truth labels that describe relevant facial expression categories are necessary. One possibility is to label facial expressions into emotion categories. Another approach is to label facial expressions independently from any interpretation attempts. This can be achieved via the facial action coding system (FACS). In this paper we present a novel approach that allows to automatically cluster FACScodes into meaningful categories. Our approach exploits the fact that FACScodes can be seen as documents containing terms -the action units (AUs) present in the codes-and so text modeling methods that capture co-occurrence information in low-dimensional spaces can be used. The FACScode derived descriptions are computed by Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Analysis (PLSA). We show that, as a high-level description of facial actions, the newly derived codes constitute a competitive alternative to both basic emotion and FACScodes. We have used them to train different types of artificial neural networks

Different styles of social interaction are one of the core characteristics of autism spectrum disorder (ASD). Social differences among individuals with ASD often include difficulty in discerning the emotions of neurotypical people based on their facial expressions. This review first covers the rich body of literature studying differences in facial emotion recognition (FER) in those with ASD, including behavioral studies and neurological findings. In particular, we highlight subtle emotion recognition and various factors related to inconsistent findings in behavioral studies of FER in ASD. Then, we discuss the dual problem of FER – namely facial emotion expression (FEE) or the production of facial expressions of emotion. Despite being less studied, social interaction involves both the ability to recognize emotions and to produce appropriate facial expressions. How others perceive facial expressions of emotion in those with ASD has remained an under-researched area. Finally, we propose a method for teaching FER [FER teaching hierarchy (FERTH)] based on recent research investigating FER in ASD, considering the use of posed vs. genuine emotions and static vs. dynamic stimuli. We also propose two possible teaching approaches: (1) a standard method of teaching progressively from simple drawings and cartoon characters to more complex audio-visual video clips of genuine human expressions of emotion with context clues or (2) teaching in a field of images that includes posed and genuine emotions to improve generalizability before progressing to more complex audio-visual stimuli. Lastly, we advocate for autism interventionists to use FER stimuli developed primarily for research purposes to facilitate the incorporation of well-controlled stimuli to teach FER and bridge the gap between intervention and research in this area.

The ability to perceive human facial emotions is an essential feature of various multi-modal applications, especially in the intelligent human-computer interaction (HCI) area. In recent decades, considerable efforts have been put into researching automatic facial emotion recognition (FER). However, most of the existing FER methods only focus on either basic emotions such as the seven/eight categories (e.g., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">happiness, anger</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">surprise</i> ) or abstract dimensions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">valence, arousal, etc.</i> ), while neglecting the fruitful nature of emotion statements. In real-world scenarios, there is definitely a larger vocabulary for describing human's inner feelings as well as their reflection on facial expressions. In this work, we propose to address the semantic richness issue in the FER problem, with an emphasis on the granularity of the emotion concepts. Particularly, we take inspiration from former psycho-linguistic research, which conducted a prototypicality rating study and chose 135 emotion names from hundreds of English emotion terms. Based on the 135 emotion categories, we investigate the corresponding facial expressions by collecting a large-scale 135-class FER image dataset and propose a consequent facial emotion recognition framework. To demonstrate the accessibility of prompting FER research to a fine-grained level, we conduct extensive evaluations on the dataset credibility and the accompanying baseline classification model. The qualitative and quantitative results prove that the problem is meaningful and our solution is effective. To the best of our knowledge, this is the first work aimed at exploiting such a large semantic space for emotion representation in the FER problem.

Impaired facial emotion recognition and reduced amygdalar volume in schizophrenia

The ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

Automatic facial expression analysis is an important aspect of human machine interaction as the face is an important communicative medium. We use our face to signal interest, disagreement, intentions or mood through subtle facial motions and expressions. Work on automatic facial expression analysis can roughly be divided into the recognition of prototypic facial expressions such as the six basic emotional states and the recognition of atomic facial muscle actions (action units, AUs). Detection of AUs rather than emotions makes facial expression detection independent of culture-dependent interpretation, reduces the dimensionality of the problem and reduces the amount of training data required. Classic psychological studies suggest that humans consciously map AUs onto the basic emotion categories using a finite number of rules. On the other hand, recent studies suggest that humans recognize emotions unconsciously with a process that is perhaps best modeled by artificial neural networks (ANNs). This paper investigates these two claims. A comparison is made between detection of emotions directly from features vs. a two-step approach where we first detect AUs and use the AUs as input to either a rulebase or an ANN to recognize emotions. The results suggest that the two-step approach is possible with a small loss of accuracy and that biologically inspired classification techniques outperform those that approach the classification problem from a logical perspective, suggesting that biologically inspired classifiers are more suitable for computer-based analysis of facial behavior than logic inspired methods

Persons with epilepsy suffer from problems with social adaptation and face numerous social issues, even when the seizures are well-controlled. Facial emotion recognition (FER), one of the key components of social cognition, has been shown to be impaired in persons with mesial temporal lobe epilepsy (MTLE). To clarify the impairment of neural networks in persons with MTLE, we performed functional magnetic resonance imaging (fMRI) studies of MTLE persons using six dynamic tasks involving FER. We evaluated brain responses to realistic dynamic facial emotional expressions involving six basic emotions (fear, happiness, anger, sadness, disgust and surprise) in nine persons with left MTLE and ten healthy controls (HCs) using fMRI. We noted brain responses to facial emotions in regions related to FER, such as the anterior insular cortex, medial frontal gyrus, superior frontal gyrus and cerebellum by our moving tasks which involved video tasks used to evaluate FER. Persons with left MTLE showed a reduced response in the right calcarine cortex compared to that of HCs. This is the first study to evaluate task-related fMRI on exposure to six basic emotions involving moving tasks in persons with epilepsy. FER deficit in persons with left MTLE may be partially associated with calcarine activity.

Twenty-three acute schizophrenics, 21 acute major depressives (Research Diagnostic Criteria), and 15 normal controls participated in a study on facial expression and emotional face recognition. Under clinical conditions, spontaneous facial expression was assessed according to the affective flattening section of the Scale for the Assessment of Negative Symptoms. Under experimental laboratory conditions involuntary (emotion-eliciting interview) and voluntary facial expression (imitation and simulation of six basic emotions) were recorded on videotape, from which a raterbased analysis of intensity or correctness of facial activity was obtained. Emotional face recognition was also assessed under experimental conditions using the same stimulus material. All subjects were assessed twice (within 4 weeks), controlling for change of the psychopathological status in the patient groups. In schizophrenics, neuroleptic drug influence was controlled by random allocation to treatment with either haloperidol or perazine. The main findings were that schizophrenics and depressives are characterized by different quantitative, qualitative, and temporal patterns of affect-related dysfunctions. In particular, schizophrenics demonstrated a trait-like deficit in affect recognition and in their spontaneous and voluntary facial activity, irrespective of medication, drug type and dosage, or extrapyramidal side-effects. In depressives a stable deficit could be demonstrated only in their involuntary expression under emotion-eliciting interview conditions, whereas in the postacute phase a reduction in their voluntary expression became apparent. Differences in patterns of affect-related behavioral deficits may reflect dysfunctions in different underlying psychobiological systems.

The authors examine facial emotion recognition and unirhinal olfactory performance in 19 schizophrenia patients and 14 comparison subjects. In patients, right nostril odor identification performance was positively related to overall emotion recognition accuracy, specifically, sad facial expressions. Olfactory and emotion recognition abilities appear significantly linked in schizophrenia.

Sixty healthy women were included to the study. They were divided two group labeled women with PMS (n=33) and without PMS (n=27), which is accordance with the Premenstrual Assessment Form. Then, The Facial Emotion Recognition Test (56 mixed photos with happy, surprised, fearful, sad, angry, disgusted and neutral facial expressions from Ekman & Friesen's series) was performed on each group in both the luteal and follicular phases. The women with PMS were significantly worse in recognizing sad (p=0.003) and surprised (p=0.019) faces in the luteal phase compared to the follicular phase, whereas women without PMS were significantly worse in recognizing sad faces (p=0.008) in the luteal phase compared to the follicular phase. There were no significant differences between women with and without PMS in either the luteal phases or in the follicular phases according to facial emotion recognition (for each, p>0.05). Conclusıon: The women with PMS do not differ from women without PMS in recognizing facial emotions accurately. The low accuracy rate in the recognition of sad and surprised facial emotions in the luteal phase may lead PMS women to have more social problems.

People living with Human Immunodeficiency Virus (PLHIV) have been reported to show poor facial emotion recognition. However, these studies presented participants with facial emotion photographs whereas in real life facial emotion recognition hardly involves inferring emotions from static faces. Moreover, emotion recognition from other sensory modalities, such as auditory, has hardly been explored. There’s also a dearth of studies examining emotion regulation difficulties in this group. The present study, thus, explored facial (using facial emotion videos) and auditory emotion recognition as well as difficulties in emotion regulation (using the Hindi version of Difficulties in Emotion Regulation Scale) in 60 PLHIV and 60 people without HIV (PWoHIV). Additionally, the association of HIV duration (duration since diagnosis of HIV), viral load, and Clusters of differentiation 4 (CD4) count with emotion recognition and regulation difficulties in PLHIV was explored. Findings from one-way ANCOVA (with education and socioeconomic status as covariates) revealed significantly impaired auditory emotion recognition (particularly for fear) among PLHIV than PWoHIV. The former also showed significantly poorer facial emotion recognition for surprise. PLHIV also self-reported significantly more emotion regulation difficulties than PWoHIV, specifically Nonacceptance of their response to negative emotions and limited access to emotion regulation Strategies. CD4 count was negatively correlated with emotion regulation difficulties, particularly for accomplishing goal-directed behaviour when experiencing negative emotions (Goals) and Strategies. Besides the novel addition to the literature regarding impaired auditory emotion recognition in PLHIV, these findings can help develop targeted interventions to improve emotion recognition and emotion regulation for PLHIV.

Many children with autism spectrum disorders (ASDs) demonstrate facial emotion recognition and expression impairments. These impairments may contribute to social disability and may put children with ASDs at risk for developing further mental health problems. In this pilot study, we examined the use of a coach- and computer-assisted facial emotion training program for children with ASDs. The intervention components focused on (a) increasing attention to relevant facial emotion cues, (b) increasing facial emotion recognition speed, and (c) using imitation to build skills of facial emotion expression. Three pilot participants demonstrated improved facial emotion recognition (accuracy and speed) of dynamic and static presentations of facial expressions and self-expression. Some improvements persisted 5 weeks after training. Results support the acceptability and feasibility of the training program. These preliminary findings are promising and suggest the need for replication with larger samples and further assessment of acceptability, feasibility, and efficacy.

To examine the impact of music as a cross-modal prime on facial emotion recognition ability in autistic children, this study compares the priming effect of music with that of faces as an intramodal prime and nonverbal sounds as the same cross-modal prime. The response time and accuracy of facial emotion recognition (happy and sad) were compared among 21 neurotypical children and 17 autistic children under various priming stimuli. A data analysis revealed that autistic children exhibited worse recognition of facial emotional expressions and demonstrated longer reaction time than neurotypical children. Unlike the other two stimuli (facial expressions and nonverbal sounds), music as a cross-modal prime resulted in slightly higher response accuracy for emotionally congruent conditions compared with emotionally incongruent conditions among autistic children. Furthermore, reaction time was significantly prolonged under emotionally congruent conditions than under emotionally incongruent conditions. This suggests that autistic children demonstrate a greater propensity to allocate additional time to improve the accuracy of their judgments under congruent conditions compared with incongruent conditions. Therefore, music intervention has significant potential for supporting autistic children to empathize with faces by playing emotionally congruent music and improving their facial emotion recognition abilities.

The facial expressions recognition (FER) is crucial to many applications. As technology advances and our needs evolve, compound emotion recognition is becoming increasingly important, along with basic emotion recognition. In the literature, Although, FER can be conducted primarily using multiple sensors. However, research shows that using facial images/videos to recognize facial expressions is better because visual presentation can convey more efficiently. Among state-of-the-art methods for FER systems, to improve the accuracy of the basic and compound FER systems, detection of facial action units (AUs) must be combined to detect basic and compound facial expressions. State-of-the-art results show that machine learning and deep learning-based approaches are more potent than conventional FER approaches. This paper surveys various learning frameworks for facial emotion recognition systems for detecting basic and compound emotions using the diverse database and summarizing state-of-the-art results to give good understanding of impact of each learning framework used in FER systems.

Objective: Clozapine and risperidone are used for treatment-resistant schizophrenia and known to improve the positive and negative symptoms. However, there are some conflicts about effects on social cognition, which is measured with facial emotion recognition ability. The impairments in facial emotion recognition ability have frequently been in different stages of the illness and might have negative influences on psychosocial functioning. In the present study, we aimed to examine clozapine and risperidone effects recognizing facial emotions in patient with treatment-resistant schizophrenia.Methods: Thirty-four patients were screened for the study, and 19 patients were included. All patients were evaluated with Positive and Negative Syndrome Scale (PANSS), Calgary Depression Scale for Schizophrenia, and Functional Remission of General Schizophrenia Scale at baseline and after 16–20 weeks of clozapine (n = 12) or risperidone (n = 7) treatment. Furthermore, the Facial Emotion Recognition Test was performed before and after treatment. The test included the photos of four male and four female models (totally 56 mixed photos) with happy, surprised, fearful, sad, angry, disgusted, and neutral facial expressions from Ekman and Friesen’s catalog.Results: The mean dose of the index drug in clozapine group was 295.83 ± 103.26 mg/day. The mean positive (p = .002), negative (p = .050) general psychopathology (p = .002), and total score (p = .002) according to the PANSS were significantly improved after treatment. The mean dose of the index drug in risperidone group was 6.86 ± 1.57 mg/day. The mean positive symptom (p = .018) and total score (p = .041) were significantly improved after treatment but negative symptom scale (p = .396) and general psychopathology (p = .149) scores did not change. There were no significant differences between baseline and after treatment in clozapine and risperidone group according to the accuracy rate of facial emotion recognition expressions (p > .05 for each). At baseline phase, the patients were significantly impaired in recognizing disgusted faces in risperidone than those in clozapine group (p = .032) and it was significantly poorer after treatment with risperidone than with clozapine (p = .031). The patients responded significantly faster after the treatment to all facial emotions except for fearful faces (p = .355).Conclusions: Clozapine and risperidone were not found to have extensive effects on the ability to recognize facial emotions because of ineffectiveness to negative symptoms as in our study. We speculated that the higher dopaminergic receptor occupancy rate of risperidone in insular cortex than that of clozapine might be related with hypo-activation of insula that was associated with particular deficit in ability to recognize expressions of disgust in patients with schizophrenia. Impaired facial emotion recognition ability is present even in first-episode psychosis, which might be a trait marker in schizophrenia.