Semantic-Rich Facial Emotional Expression Recognition

Different styles of social interaction are one of the core characteristics of autism spectrum disorder (ASD). Social differences among individuals with ASD often include difficulty in discerning the emotions of neurotypical people based on their facial expressions. This review first covers the rich body of literature studying differences in facial emotion recognition (FER) in those with ASD, including behavioral studies and neurological findings. In particular, we highlight subtle emotion recognition and various factors related to inconsistent findings in behavioral studies of FER in ASD. Then, we discuss the dual problem of FER – namely facial emotion expression (FEE) or the production of facial expressions of emotion. Despite being less studied, social interaction involves both the ability to recognize emotions and to produce appropriate facial expressions. How others perceive facial expressions of emotion in those with ASD has remained an under-researched area. Finally, we propose a method for teaching FER [FER teaching hierarchy (FERTH)] based on recent research investigating FER in ASD, considering the use of posed vs. genuine emotions and static vs. dynamic stimuli. We also propose two possible teaching approaches: (1) a standard method of teaching progressively from simple drawings and cartoon characters to more complex audio-visual video clips of genuine human expressions of emotion with context clues or (2) teaching in a field of images that includes posed and genuine emotions to improve generalizability before progressing to more complex audio-visual stimuli. Lastly, we advocate for autism interventionists to use FER stimuli developed primarily for research purposes to facilitate the incorporation of well-controlled stimuli to teach FER and bridge the gap between intervention and research in this area.

The ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

People living with Human Immunodeficiency Virus (PLHIV) have been reported to show poor facial emotion recognition. However, these studies presented participants with facial emotion photographs whereas in real life facial emotion recognition hardly involves inferring emotions from static faces. Moreover, emotion recognition from other sensory modalities, such as auditory, has hardly been explored. There’s also a dearth of studies examining emotion regulation difficulties in this group. The present study, thus, explored facial (using facial emotion videos) and auditory emotion recognition as well as difficulties in emotion regulation (using the Hindi version of Difficulties in Emotion Regulation Scale) in 60 PLHIV and 60 people without HIV (PWoHIV). Additionally, the association of HIV duration (duration since diagnosis of HIV), viral load, and Clusters of differentiation 4 (CD4) count with emotion recognition and regulation difficulties in PLHIV was explored. Findings from one-way ANCOVA (with education and socioeconomic status as covariates) revealed significantly impaired auditory emotion recognition (particularly for fear) among PLHIV than PWoHIV. The former also showed significantly poorer facial emotion recognition for surprise. PLHIV also self-reported significantly more emotion regulation difficulties than PWoHIV, specifically Nonacceptance of their response to negative emotions and limited access to emotion regulation Strategies. CD4 count was negatively correlated with emotion regulation difficulties, particularly for accomplishing goal-directed behaviour when experiencing negative emotions (Goals) and Strategies. Besides the novel addition to the literature regarding impaired auditory emotion recognition in PLHIV, these findings can help develop targeted interventions to improve emotion recognition and emotion regulation for PLHIV.

The authors examine facial emotion recognition and unirhinal olfactory performance in 19 schizophrenia patients and 14 comparison subjects. In patients, right nostril odor identification performance was positively related to overall emotion recognition accuracy, specifically, sad facial expressions. Olfactory and emotion recognition abilities appear significantly linked in schizophrenia.

Persons with epilepsy suffer from problems with social adaptation and face numerous social issues, even when the seizures are well-controlled. Facial emotion recognition (FER), one of the key components of social cognition, has been shown to be impaired in persons with mesial temporal lobe epilepsy (MTLE). To clarify the impairment of neural networks in persons with MTLE, we performed functional magnetic resonance imaging (fMRI) studies of MTLE persons using six dynamic tasks involving FER. We evaluated brain responses to realistic dynamic facial emotional expressions involving six basic emotions (fear, happiness, anger, sadness, disgust and surprise) in nine persons with left MTLE and ten healthy controls (HCs) using fMRI. We noted brain responses to facial emotions in regions related to FER, such as the anterior insular cortex, medial frontal gyrus, superior frontal gyrus and cerebellum by our moving tasks which involved video tasks used to evaluate FER. Persons with left MTLE showed a reduced response in the right calcarine cortex compared to that of HCs. This is the first study to evaluate task-related fMRI on exposure to six basic emotions involving moving tasks in persons with epilepsy. FER deficit in persons with left MTLE may be partially associated with calcarine activity.

Facial emotion recognition (FER) is the technology or process of identifying and interpreting human emotions based on the analysis of facial expressions. It involves using computer algorithms and machine learning techniques to detect and classify emotional states from images or videos of human faces. Further, FER plays a vital role in recognizing and understanding human emotions to better interpret someone’s feelings, intentions, and attitudes. In the present time, it is widely used in various fields such as healthcare, human–computer interaction, law enforcement, security, and beyond. FER has a wide range of practical applications across various industries including Emotion Monitoring, Adaptive Learning, and Virtual Assistants. This paper presents a comparative analysis of FER algorithms, focusing on deep learning approaches. The performance of different datasets, including FER2013, JAFFE, AffectNet, and Cohn–Kanade, is evaluated using convolutional neural networks (CNNs), deep face, attentional convolutional networks (ACNs), and deep belief networks (DBNs). Among the tested algorithms, DBNs outperformed other algorithms, reaching the highest accuracy of 98.82%. These results emphasize the effectiveness of deep learning techniques, particularly DBNs, in FER. Additionally, outlining the advantages and disadvantages of current research on facial emotion identification might direct future research efforts in the direction of the most profitable directions.

Impaired facial emotion recognition and reduced amygdalar volume in schizophrenia

Facial expression recognition via ResNet-50

Sixty healthy women were included to the study. They were divided two group labeled women with PMS (n=33) and without PMS (n=27), which is accordance with the Premenstrual Assessment Form. Then, The Facial Emotion Recognition Test (56 mixed photos with happy, surprised, fearful, sad, angry, disgusted and neutral facial expressions from Ekman & Friesen's series) was performed on each group in both the luteal and follicular phases. The women with PMS were significantly worse in recognizing sad (p=0.003) and surprised (p=0.019) faces in the luteal phase compared to the follicular phase, whereas women without PMS were significantly worse in recognizing sad faces (p=0.008) in the luteal phase compared to the follicular phase. There were no significant differences between women with and without PMS in either the luteal phases or in the follicular phases according to facial emotion recognition (for each, p>0.05). Conclusıon: The women with PMS do not differ from women without PMS in recognizing facial emotions accurately. The low accuracy rate in the recognition of sad and surprised facial emotions in the luteal phase may lead PMS women to have more social problems.

Parkinson's disease is characterised by the presence of motor symptoms including hypomimia, and by non-motor symptoms including alterations in facial recognition of basic emotions. Few studies have investigated this alteration and its relationship to the severity of hypomimia. The objective is to study the relationship between hypomimia and the facial recognition of basic emotions in subjects with Parkinson's disease. Twenty-three patients and 29 controls were evaluated with the test battery for basic emotion facial recognition. The patients were divided into two subgroups according to the intensity of their hypomimia. The comparison in battery test performance between the minimal/mild hypomimia and moderate/severe hypomimia groups was statistically significant in favour of the former group. This finding shows a close relationship between expression and facial recognition of emotions, which could be explained through the mechanism of motor simulation.

Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot

Many children with autism spectrum disorders (ASDs) demonstrate facial emotion recognition and expression impairments. These impairments may contribute to social disability and may put children with ASDs at risk for developing further mental health problems. In this pilot study, we examined the use of a coach- and computer-assisted facial emotion training program for children with ASDs. The intervention components focused on (a) increasing attention to relevant facial emotion cues, (b) increasing facial emotion recognition speed, and (c) using imitation to build skills of facial emotion expression. Three pilot participants demonstrated improved facial emotion recognition (accuracy and speed) of dynamic and static presentations of facial expressions and self-expression. Some improvements persisted 5 weeks after training. Results support the acceptability and feasibility of the training program. These preliminary findings are promising and suggest the need for replication with larger samples and further assessment of acceptability, feasibility, and efficacy.

Automated facial emotion recognition (FER) is one of the important fields of human-computer interaction (HCI). FER remains challenging due to facial accessories, non-uniform illumination, pose variation, etc. Emotion detection exploiting conventional algorithms has the demerit of mutual optimization of classification and feature extraction. Artificial intelligence (AI) techniques can be employed to identify FER automatically. Deep learning (DL) driven FER models have recently allowed for designing an end-to-end learning process. Therefore, this study designs a Henry Gas Solubility Optimization with Deep Learning Based FER (HGSO-DLFER) technique for HCI. The HGSO-DLFER technique aims to recognize and identify various kinds of facial emotions. To accomplish this, the HGSO-DLFER technique employs adaptive fuzzy filtering (AFF) for noise removal. In addition, the MobileNet model is used for feature vector generation, and the HGSO algorithm optimally chooses its hyperparameter scan. For the recognition of facial emotions, the HGSO-DLFER technique uses an autoencoder (AE) classifier with a Nadam optimizer. A widespread experimental analysis is made to facilitate a better understanding of the FER results by the HGSO-DLFER technique. The comparative analysis showed the effective performance of the HGSO-DLFER technique over other FER techniques with maximum accuracy of 98.65%.

To examine the impact of music as a cross-modal prime on facial emotion recognition ability in autistic children, this study compares the priming effect of music with that of faces as an intramodal prime and nonverbal sounds as the same cross-modal prime. The response time and accuracy of facial emotion recognition (happy and sad) were compared among 21 neurotypical children and 17 autistic children under various priming stimuli. A data analysis revealed that autistic children exhibited worse recognition of facial emotional expressions and demonstrated longer reaction time than neurotypical children. Unlike the other two stimuli (facial expressions and nonverbal sounds), music as a cross-modal prime resulted in slightly higher response accuracy for emotionally congruent conditions compared with emotionally incongruent conditions among autistic children. Furthermore, reaction time was significantly prolonged under emotionally congruent conditions than under emotionally incongruent conditions. This suggests that autistic children demonstrate a greater propensity to allocate additional time to improve the accuracy of their judgments under congruent conditions compared with incongruent conditions. Therefore, music intervention has significant potential for supporting autistic children to empathize with faces by playing emotionally congruent music and improving their facial emotion recognition abilities.

Objective: Clozapine and risperidone are used for treatment-resistant schizophrenia and known to improve the positive and negative symptoms. However, there are some conflicts about effects on social cognition, which is measured with facial emotion recognition ability. The impairments in facial emotion recognition ability have frequently been in different stages of the illness and might have negative influences on psychosocial functioning. In the present study, we aimed to examine clozapine and risperidone effects recognizing facial emotions in patient with treatment-resistant schizophrenia.Methods: Thirty-four patients were screened for the study, and 19 patients were included. All patients were evaluated with Positive and Negative Syndrome Scale (PANSS), Calgary Depression Scale for Schizophrenia, and Functional Remission of General Schizophrenia Scale at baseline and after 16–20 weeks of clozapine (n = 12) or risperidone (n = 7) treatment. Furthermore, the Facial Emotion Recognition Test was performed before and after treatment. The test included the photos of four male and four female models (totally 56 mixed photos) with happy, surprised, fearful, sad, angry, disgusted, and neutral facial expressions from Ekman and Friesen’s catalog.Results: The mean dose of the index drug in clozapine group was 295.83 ± 103.26 mg/day. The mean positive (p = .002), negative (p = .050) general psychopathology (p = .002), and total score (p = .002) according to the PANSS were significantly improved after treatment. The mean dose of the index drug in risperidone group was 6.86 ± 1.57 mg/day. The mean positive symptom (p = .018) and total score (p = .041) were significantly improved after treatment but negative symptom scale (p = .396) and general psychopathology (p = .149) scores did not change. There were no significant differences between baseline and after treatment in clozapine and risperidone group according to the accuracy rate of facial emotion recognition expressions (p > .05 for each). At baseline phase, the patients were significantly impaired in recognizing disgusted faces in risperidone than those in clozapine group (p = .032) and it was significantly poorer after treatment with risperidone than with clozapine (p = .031). The patients responded significantly faster after the treatment to all facial emotions except for fearful faces (p = .355).Conclusions: Clozapine and risperidone were not found to have extensive effects on the ability to recognize facial emotions because of ineffectiveness to negative symptoms as in our study. We speculated that the higher dopaminergic receptor occupancy rate of risperidone in insular cortex than that of clozapine might be related with hypo-activation of insula that was associated with particular deficit in ability to recognize expressions of disgust in patients with schizophrenia. Impaired facial emotion recognition ability is present even in first-episode psychosis, which might be a trait marker in schizophrenia.