Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors’ Satisfaction

In order to overcome the problem that the accuracy and efficiency of Mandarin emotion recognition is low based on singal-modal, a tri-modal Mandarin emotion recognition method based on the feature-level fusion of three important modal of speech, facial expression and body gesture is proposed. Firstly, different emotional features are extracted from speech, facial expressions and body gestures, and then the three modal emotion features are fused by IAGA method. Finally, SVM classifier is used to construct prediction model and emotion recognition is completed on it. The CHEAVD Chinease multi-modal emotion dataset is utilized to evaluate our proposed methods. Compared with the traditional mono-modal and dual-modal emotion recognition, the experimental results show that the dual-modal fusion has higher emotion recognition rate than the mono-modal, and the tri-modal fusion has higher emotion recognition rate than the dual-modal, which verifies the effectiveness of the tri-modal Mandarin emotion recognition

This study empirically analyzed the influence of media on a visitor's satisfaction and re-visit by setting up 5 sorts of exhibition media as text, docent, virtual museum, website and Smartphone App based on practical exchange and communication between a visitor and an exhibition provider. The research results are as follows: First, all these 5 sorts of media suggested by this study were found to be able to have an influence on a visitor's satisfaction and re-visit. Second, it was found that the factors preferentially affecting a visitor's satisfaction and re-visit were 'Docent' and 'Website' factors. Accordingly, it is judged that it's necessary to more preferentially consider 'Docent' and 'Website' factors than the other factors in order to increase a visitor's satisfaction and re-visit. Third, it's possible to interpret that the bigger a visitor's satisfaction, the higher a visitor's intention to re-visit through the verification that a visitor's satisfaction will have a significant influence on a visitor's re-visit. The implication of this study lies in the fact that this study attempted at an integrated approach by analyzing the relative size of the influence through across-the-board suggestion of influential factors on a museum visitor's satisfaction and re-visit.

Patients with amygdala lesions were found to be impaired in recognizing the fear emotion both from face and from music. In patients with Parkinson's disease (PD), impairment in recognition of emotions from facial expressions was reported for disgust, fear, sadness and anger, but no studies had yet investigated this population for the recognition of emotions from both face and music. The ability to recognize basic universal emotions (fear, happiness and sadness) from both face and music was investigated in 24 medicated patients with PD and 24 healthy controls. The patient group was tested for language (verbal fluency tasks), memory (digit and spatial span), executive functions (Similarities and Picture Completion subtests of the WAIS III, Brixton and Stroop tests), visual attention (Bells test), and fulfilled self-assessment tests for anxiety and depression. Results showed that the PD group was significantly impaired for recognition of both fear and sadness emotions from facial expressions, whereas their performance in recognition of emotions from musical excerpts was not different from that of the control group. The scores of fear and sadness recognition from faces were neither correlated to scores in tests for executive and cognitive functions, nor to scores in self-assessment scales. We attributed the observed dissociation to the modality (visual vs. auditory) of presentation and to the ecological value of the musical stimuli that we used. We discuss the relevance of our findings for the care of patients with PD.

In this paper a study on multimodal automatic emotion recognition during a speech-based interaction is presented. A database was constructed consisting of people pronouncing a sentence in a scenario where they interacted with an agent using speech. Ten people pronounced a sentence corresponding to a command while making 8 different emotional expressions. Gender was equally represented, with speakers of several different native languages including French, German, Greek and Italian. Facial expression, gesture and acoustic analysis of speech were used to extract features relevant to emotion. For the automatic classification of unimodal data, bimodal data and multimodal data, a system based on a Bayesian classifier was used. After performing an automatic classification of each modality, the different modalities were combined using a multimodal approach. Fusion of the modalities at the feature level (before running the classifier) and at the results level (combining results from classifier from each modality) were compared. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison to the unimodal systems: the multimodal approach increased the recognition rate by more than 10% when compared to the most successful unimodal system. Bimodal emotion recognition based on all combinations of the modalities (i.e., ‘face-gesture’, ‘face-speech’ and ‘gesture-speech’) was also investigated. The results show that the best pairing is ‘gesture-speech’. Using all three modalities resulted in a 3.3% classification improvement over the best bimodal results.

Humans have traditionally found it simple to identify emotions from facial expressions, but it is far more difficult for a computer system to do the same. The social signal processing subfield of emotion recognition from facial expression is used in a wide range of contexts, particularly for human-computer interaction. Automatic emotion recognition has been the subject of numerous studies, most of which use a machine learning methodology. The recognition of simple emotions like anger, happiness, contempt, fear, sadness, and surprise, however, continues to be a difficult topic in computer vision. Deep learning has recently drawn increased attention as a solution to a variety of practical issues, including emotion recognition. In this study, we improved the convolutional neural network technique to identify 7 fundamental emotions and evaluated several preprocessing techniques to demonstrate how they affected the CNN performance. This research focuses on improving facial features and expressions based on emotional recognition. By identifying or recognising facial expressions that elicit human responses, it is possible for computers to make more accurate predictions about a person's mental state and to provide more tailored responses. As a result, we examine how a deep learning technique that employs a convolutional neural network might improve the detection of emotions based on facial features (CNN). Multiple facial expressions are included in our dataset, which consists of about 32,298 photos for testing and training. The preprocessing system aids in removing noise from the input image, and the pretraining phase aids in revealing face detection after noise removal, including feature extraction. As a result, the existing paper generates the classification of multiple facial reactions like the seven emotions of the facial acting coding system (FACS) without using the optimization technique, but our proposed paper reveals the same seven emotions of the facial acting coding system.

Emotion recognition is widely used to understand human interaction and social behavior. The tendency for subtle differences in facial expressions to influence the emotion completely and the limitations in dataset collection pose a challenge for machines to accurately and automatically detect emotion from one image. The study was conducted using a facial expression dataset from Kaggle’s emotion recognition competition in 2013. We empirically compared different machine learning techniques, explored data augmentation methods, and applied a multiplier technique to further improve accuracy. With both quantitative and qualitative evaluation, we find that Convolutional Neural Network, Random Noise of standard deviation 1, and a multiplier value of 3 have the highest accuracy. We also provide insight on several counter-intuitive results in the study.

Emotion recognition through facial expression analysis is an emerging research in Artificial Intelligence which faces many challenges. The problem is the variation of facial expressions that displays human emotions. Humans can subjectively express the same emotions in various ways. To overcome the problem of ambiguity in emotion expression, a fuzzy approach is developed to analyze the facial components in determining the type of emotion. In this study, we proposed a framework for fuzzy emotion recognition as a representation of the psychologist knowledge. Three stages in the fuzzy emotion recognition were facial feature extraction with Active Appearance Model; Semantic-linguistic facial features extraction; fuzzy emotion recognition with Fuzzy Emotion Classification. System performance testing provided the best results on extended Cohn Kanade (CK+) facial expression dataset, with the accuracy of linguistic facial component recognition 0.98, and accuracy of fuzzy emotion recognition 0.90. Testing was also performed on custom-made Indonesian Mixed Emotion Dataset (IMED) which resulted in accuracy of 0.87. The fuzzy emotion recognition has a potential to be applied in various real problems such as virtual counseling, stress detection, lie detection, and e-commerce.

Annotated emotional image datasets of Chinese university students in real classrooms for deep learning

IntroductionRecent developments in the field of social cognition have led to a renewed interest in basic and social emotion recognition in early stages of Alzheimer's Disease (AD) and FrontoTemporal Dementia (FTD). Despite the growing attention to this issue, only few studies have attempted to investigate emotion recognition using both visual and vocal stimuli. In addition, recent studies have presented conflicting findings regarding the extent of impairment in patients in the early stages of these diseases. The present study aims to investigate emotion understanding (both basic and social emotions), using different tasks with visual and auditory stimuli, to identify supramodal deficits in AD and FTD to provide a reliable tool to better outline their behavioral and emotional profile and useful instruments for their management.MethodsEighteen patients with AD and 15 patients with FTD were included in the study. Healthy control (HCs) subjects were recruited to obtain normative data for basic emotion recognition tests and social emotion recognition tasks. To evaluate basic emotion recognition, the Facial Emotion Recognition Battery (FERB) and the Emotional Prosody Recognition Battery (EPRB) were administered. To evaluate social emotion recognition, the Faux Pas (FP), Reading the Mind in the Eyes (RME), and Reading the Mind in the Voice (RMV) tests were employed.ResultsFTD patients performed significantly worse than HCs in most of the subtests of the basic emotion recognition batteries, where, instead, AD patients were significantly impaired only when required to match emotional facial expression in different individuals (subtask of the FERB). Moreover, FTD patients scored significantly lower in RME and RMV tests compared both to AD patients and to HCs. In addition, ADs were selectively impaired in RMV as respect to HCs.DiscussionFTD patients showed deficits in emotion recognition, affecting both basic and social emotions, whether conveyed through facial expressions or prosody. This result may explain the well-known social behavioral difficulties observed in FTD patients from the early stages of the disease. The fewer and specific deficits in AD patients with comparable MMSE scores may be attributed to the mild degree of impairment, as these deficits may appear later in the progression of AD.

Emotions play a vital role in people’s everyday life. It is a mental state that does not arise through free will and is often accompanied by physiological changes. Therefore monitoring these changes is important as they are perceptions of emotional changes and can help in identifying matters of concern at an early stage before they become serious. Emotion recognition has become an important subject when it comes to human-machine interaction. Various methods have been used in the past to detect and evaluate human emotions. The most commonly used techniques include the use of textual information, facial expressions, speech, body gestures and physiological signals. In this project we have developed an emotion recognition system based on information provided by the physiological signals. These signals are obtained from a skin temperature sensor, a heart rate sensor, and a blood pressure sensor. The amplified and filtered signals from the sensors are input into the microcontroller where all the processing takes place. The microcontroller wirelessly transmits data to a computer where it is stored for data analyses and feature extraction for emotion recognition. The three basic emotions observed in this project are happy (excited), stressed and neutral (relaxed).

A survey of emotion recognition methods with emphasis on E-Learning environments

BackgroundRecognition of others' emotions is an important aspect of interpersonal communication. In major depression, a significant emotion recognition impairment has been reported. It remains unclear whether the ability to recognize emotion from facial expressions is also impaired in anxiety disorders. There is a need to review and integrate the published literature on emotional expression recognition in anxiety disorders and major depression.Methodology/Principal FindingsA detailed literature search was used to identify studies on explicit emotion recognition in patients with anxiety disorders and major depression compared to healthy participants. Eighteen studies provided sufficient information to be included. The differences on emotion recognition impairment between patients and controls (Cohen's d) with corresponding confidence intervals were computed for each study. Over all studies, adults with anxiety disorders had a significant impairment in emotion recognition (d = −0.35). In children with anxiety disorders no significant impairment of emotion recognition was found (d = −0.03). Major depression was associated with an even larger impairment in recognition of facial expressions of emotion (d = −0.58).Conclusions/SignificanceResults from the current analysis support the hypothesis that adults with anxiety disorders or major depression both have a deficit in recognizing facial expression of emotions, and that this deficit is more pronounced in major depression than in anxiety.

3D face generation is a critical task for immersive multimedia applications, where a key challenge is the joint synthesis of expressive geometry and BRDF textures. Existing methods often struggle with geometric-textural coherence and corresponding reflectance modeling. To overcome these limitations, we present ExpDiff, a framework that generates expression meshes and corresponding BRDF textures from a single neutral-expression face. Our method employs an attention-based diffusion model to learn the semantic transition across expressions. To ensure correspondence between geometry and texture, we introduce a unified representation that explicitly models geometric-textural interaction, which is encoded into a shared latent space by models pre-trained on a vast dataset for strong generalization. To achieve semantically coherent and physically consistent generation, we propose to guide the denoising direction with specially designed textual prompts. We further construct two novel facial expression datasets, J-Reflectance, for ultra-high-quality assets, and FFHQ-BRDFExp for diverse identities, both of which are publicly released to advance the community. Extensive experiments demonstrate our method's superior performance in photo-realistic facial expression synthesis. Project page: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://cyh-sj.github.io/expdiff/</uri>.

This paper considers a hybrid multimodal model for improvement of the human emotion recognition based on facial expression and body gesture recognition. The paper extends the author’s investigations related to the usage of pre-trained models of deep learning neural networks (DNN) for facial emotion recognition (FER) with addition of the emotions extracted from body language. In order to extract emotions from upper body gestures a second model of DNN was developed and trained with specific datasets. The information regarding recognized emotions obtained by both models is more accurate and can be used in education, medicine, psychology, product advertisement, marketing, human-machine interfaces, etc. In our case, it aims to personalize the lecture material of the students during their online training, taking into account their emotional state.

Objectives: Attention-deficit/hyperactivity disorder (ADHD) is the most frequent neurobehavioural disorder in childhood. ADHD is associated with impaired academic performance, cognitive, and emotional deficits. Moreover, comorbid oppositional defiant disorder (ODD) is leading to more severe impairment in social performance. Social cognition involves recognition, encoding, and interpretation of emotions from faces. Basic facial expressions that include sadness, happiness, anger, disgust, fear, and surprise are the easiest emotions to recognize. We aimed to demonstrate facial expression recognition impairments that might occur more frequently in children with co-occurring ADHD/ODD than patients with ADHD only. Thus, children with the co-occurrence of ODD may suffer more severely from social and behavioural difficulties.Methods: Forty patients diagnosed with ADHD and/or co-occurring ADHD/ODD according to DSM-IV-TR criteria were compared with a parallel (by gender, age, and educational state) 11 healthy children as a control group in this study. Clear facial images of each emotion were used as well as two additional sets of photos include 50% blurred images and cropped eye images were added as distractors then all images represented with black and white tone for emotion recognition task via facial expression. Angry expressions presented as target expressions. DSM-IV-Based Screening and Rating Scale for Children and Adolescents with attention deficit and disruptive behaviour disorders, the Conners’ Teachers Rating Scale/Revised Long Form and the Conners’ Parent Rating Scale/Revised Long Form were used to provide diagnostic objectivity.Results: Control group statistically performed better than ADHD group on recognition of emotional facial expressions. Results showed no statistically significant differences between the ADHD and ADHD/ODD group on recognition of emotional facial expressions. However, according to results of emotion recognition task via facial expressions, there were statistically significant differences between pure ADHD and comorbid ADHD+ODD groups in happy and neutral expressions. ADHD/ODD group tend to attribute more meaning to neutral facial expressions. Additionally there was statistically significant difference between control group and ADHD group according to recognition of angry expressions. There were statistically significant differences between the groups according to recognition of sad expressions in all clear, blurred, and eye photographs.Conclusions: Difficulties in recognizing emotional facial expressions were observed in children with ADHD. A statistically significant association was established between presence of ADHD and impaired recognition of facial emotion expressions independent from the scores of the disruptive behaviour rating scale. Comorbid ODD was not associated with recognition of emotional facial expressions including angry expressions. Recognition of angry expressions was not found as a predictor for disruptive behaviour disorders.