Single Emotion Recognition from Multiple Faces with Fuzzy System

Context: Measuring user satisfaction is an essential tool to create value in digital transformation. Measurement allows you to identify future purchase intentions, user loyalty, and retention. The traditional way of measuring satisfaction using self-assessment has problems, such as subjectivity. Therefore, a more objective approach is needed that allows automatic measurement of satisfaction. Objective: This research aims to make a systematic literature mapping (SLM) to identify the state-of-the-art use of Artificial Intelligence tools and techniques related to emotion recognition visually and then use the results to propose a model that can lead to an automated user satisfaction measurement. We present the model with a work process, an exploratory experiment, and an application project to measure the user experience from visual emotion recognition, automatically calculate the satisfaction assessment of a user when using a digital product, and present the positive, neutral, and negative points of using this product. Methods: A systematic literature mapping was used to identify the primary studies related to techniques, benefits, and challenges that associate the measurement of user satisfaction through the recognition of emotions with the use of Artificial Intelligence (AI). In the proposed model, an experiment was planned in two phases to validate research hypotheses related to the objective of the work. The experiment, which will apply the projected application, will be carried out by videoconferencing with the approach of exploratory usability testing on a web product. Results: 10 primary studies were identified in different areas of knowledge: restaurants, television systems, retail stores, artistic shows, and usability testing in a call center system. Two systematic reviews of the literature were also identified. The technique most commonly used in primary studies is the convolutional neural network (CNN). The use of cloud services for emotion recognition was also verified. Benefits related to user feedback, such as user profile mapping, were reported, and challenges for emotion recognition were found, such as user privacy and capture environment inadequacies. Besides, a model to automatically measure user satisfaction in a scalable and replicable way was proposed. Conclusions: The possibilities of applying emotion recognition are countless in terms of contexts, techniques, forms, and components. Despite this, it was possible to identify good practices that could guide the creation of a tool to measure the satisfaction of using a digital product through emotion recognition and Artificial Intelligence. The main contribution of this work is the proposal of a model that guides the creation of an automatic satisfaction measurement tool to identify positive, neutral, and negative points in the use of a digital product. This research also defines a measurement experimentation process that any organization or researcher can reproduce.

Companion robots play an important role to accompany humans and provide emotional support, such as reducing human social isolation and loneliness. Based on recognizing human partner's mental states, a companion robot is able to dynamically adjust its behaviors, and make human-robot interaction smoother and natural. Human emotion has been recognized by many modalities like facial expression and voice. Neurophysiological signals have shown promising results in emotion recognition, since it is an innate signal of human brain which cannot be faked. In this paper, emotional state recognition using a neurophysiology method is studied to guide and modulate companion-robot navigation to enhance its social capabilities. Electroencephalogram (EEG), a type of neurophysiological signals, is used to recognize human emotional state, and then feed into a navigation path planning algorithm for controlling a companion robot's routes. Simulation results show that mobile robot presents navigation behaviors modulated by dynamic human emotional states.

Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method

Background: One of the significant problems in human societies that has existed for a long time. Unfortunately, this problem has assumed more serious dimensions in our communities. One of the challenges faced by methamphetamine abusers is a disorder in emotion regulation, social cognition, and recognition of emotional states. Objectives: In this regard, the purpose of this research was to investigate the effect of emotion regulation training on the recognition of emotional states and social cognition in methamphetamine abusers. Patients and Methods: This research is classified as applied in terms of purpose and quasi-experimental in terms of design, using a pre-test and post-test format with a control group. The study sample consisted of 30 methamphetamine abusers who referred to the TC treatment center (TC) of the Baran Institute in Tabriz in 2024. Sampling was done purposefully, and 15 individuals were randomly assigned to the experimental group and 15 to the control group. To collect data, Bakker's Emotional Processing Questionnaire (2007) and Chen and Lin's Social Cognition Questionnaire (2019) were used. A pre-test was administered to both groups, after which the experimental group received an emotion regulation training protocol over 8 sessions. The research hypotheses were tested using the univariate covariance method. Results: The research findings indicated that emotion regulation training had a positive and significant effect on the recognition of emotional states and social cognition in methamphetamine abusers, increasing the levels of these variables (P < 0.05). Conclusions: This study demonstrated that emotion regulation training improved the recognition of emotional states and social cognition in methamphetamine abusers. Therefore, by offering effective strategies in this area, meaningful steps can be taken to reduce the problems faced by drug abusers.

To effectively fuse speech and visual features, this letter proposes a multi-modal emotion recognition method by fusing correlation features of speech-visual. Firstly, speech and visual features are extracted by two-dimensional convolutional neural network (2D-CNN) and three-dimensional convolutional neural network (3D-CNN), respectively. Secondly, the speech and visual features is processed by feature correlation analysis algorithm in multi-modal fusion. In addition, the class information of speech and visual features are also applied to the feature correlation analysis algorithm, which can effectively fuse speech and visual features and improve the performance of multi-modal emotion recognition. Finally, support vector machines (SVM) completes the classification of multi-modal speech and visual emotion recognition. Experimental results on the RML, eNTERFACE05, BAUM-1 s datasets show that the recognition rate of our method is higher than other state-of-the-art methods.

The rich semantic information contained in images is an important clue to explore visual emotions. Therefore, exploring the correlation between visual emotion and the semantic relationship of objects, and extracting more effective semantic features through explicit or implicit modeling is very important for visual emotion analysis. In this paper, a novel Graph-based Object Semantic Refinement (GOSR) model is proposed to extract multi-level semantic features for visual emotion classification, in which graph structures is used to represent the object semantics and their position relationships of an image, and Graph Convolutional Networks (GCN) is used to refine object information by the aggregating neighbor object with their position relationships. The different convolutional layer’s features from GCN are further fused by Gated Recurrent Units (GRU) networks to achieve high-level semantic features. Then a framework with two branches to leverage visual and semantic information for visual sentiment analysis is proposed, which uses convolutional neural networks to extract visual features from images, and collaborates with semantic features from GOSR model to achieve better emotion recognition results. Besides, for alleviating the potentially unreasonable predictions and promote models collaboration, a novel tendency loss function based on the correlations among emotion labels is proposed to adjust the output activation value other than the target label. Extensive experiments on four widely used benchmark datasets show that our proposed method can achieve competitive performance and outperform most of the state-of-the-art methods on visual emotion recognition.

Tracking of regions and object boundaries in an image sequence is a well studied problem in image processing and computer vision. So far, numerous approaches tracking different features of the objects (contours, regions or points of interest) have been presented. Most of these approaches have problems with robustness. Typical reasons are noisy images, objects with identical features or partial occlusions of the tracked features. In this paper we propose a novel level set based tracking approach, that allows robust tracking on noisy images. Our framework is able to track multiple regions in an image sequence, where a level set function is assigned to every region. For already known or learned objects, transformation invariant shape priors can be added to ensure a robust tracking even under partial occlusions. Furthermore, we introduce a simple decision function to maintain the desired topology for multiple regions. Experimental results demonstrate the method for arbitrary numbers of shape priors. The approach can even handle full occlusions and objects which are temporarily hidden in containers.KeywordsActive ContourNoisy ImageRobust TrackingShape PriorGeodesic Active ContourThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Patients with cerebellar lesions present some affective and cognitive disorders, defining a peculiar pattern of cognitive impairment, so-called cerebellar cognitive affective syndrome. This pattern has been confirmed in many genotypes of spinocerebellar ataxias (SCA), a group of genetically defined pathologies characterized by the degeneration of the cerebellum and its connections. Recently, in SCA patients, some authors focused the interest on social cognition evidencing an impairment of theory of mind and basic emotion recognition by verbal material. The recognition of emotions in faces is an essential component of social cognition; therefore, we assessed this ability in SCA patients, expanding the study from the basic verbal emotions to the basic and social visual emotion recognition. We assessed facial emotion recognition using two basic and social emotion tasks in a group of SCA patients together with a complete clinical and neuropsychological evaluation. We compared results with the performance of a control group. We demonstrated a significant difference between patients and controls both in basic and social emotion recognition, although we found a specific impairment only for social emotions. The deficit was not correlated to clinical and demographic features. The cognitive and psychological profile did not explain the impairment in emotion recognition. This result supports the hypothesis that the impairment in social emotion recognition could be specifically related to a defect in the corticocerebellar network.

Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals

The purpose of this paper is the design of a mobile bio-feedback device that collects bio-signals from people and it assigns them to anxiety levels. Student emotional state recognition might be approved a significant support providing self-regulation besides teaching and coaching quality. The physiological reactions of the body affect the conductivity of the skin, skin temperature and heart rate. Based on this information the body produces bio-signals that can be processed to recognize basic human emotional states. Usually stressful situations result in increased skin conductivity. Another physiological reaction to increased human anxiety is decreased skin temperature. Also people under stressful conditions usually have increased heart rate values. Emotional state recognition and regulation may be an important factor in achieving better performance. Furthermore we could assert that there is often a skeptical attitude toward human contact with sensors which sometimes are annoying and may prove inconvenient especially when attached to fingers. In this work we focus on improving interaction with the device by designing a mobile version that gives users freedom of movement and mobility. There is a widespread adoption of mobile smartphones and tablets today in our everyday lives. So we can assume that everyone has access to one and developing an interface that links the bio-feedback device to them will broaden the scenarios where it could be used. This paper will also examine an activity where a mobile version of the device could be beneficial for the user, suggest a prototype design for a mobile adaptation of the user interface and lastly describe some new use cases that highlight the positive effect a mobile version of the device can have.

Most of the electroencephalography (EEG) based emotion recognition systems rely on single stimulus to evoke emotions. EEG data is mostly recorded with higher number of electrodes that can lead to data redundancy and longer experimental setup time. The question “whether the configuration with lesser number of electrodes is common amongst different stimuli presentation paradigms” remains unanswered. There are publicly available datasets for EEG based human emotional states recognition. Since this work is focused towards classifying emotions while subjects are experiencing different stimuli, therefore we need to perform new experiments. Keeping aforementioned issues in consideration, this work presents a novel experimental study that records EEG data for three different human emotional states evoked with four different stimuli presentation paradigms. A methodology based on iterative Genetic Algorithm in combination with majority voting has been used to achieve configuration with reduced number of EEG electrodes keeping in consideration minimum loss of classification accuracy. The results obtained are comparable with recent studies. Stimulus independent configurations with lesser number of electrodes lead towards low computational complexity as well as reduced set up time for future EEG based smart systems for emotions recognition

The development of a Social Intelligence System based on artificial intelligence is one of the cutting edge technologies in Assistive Robotics. Such systems need to create an empathic interaction with the users; therefore, it os required to include an Emotion Recognition (ER) framework which has to run, in near real-time, together with several other intelligent services. Most of the low-cost commercial robots, however, although more accessible by users and healthcare facilities, have to balance costs and effectiveness, resulting in under-performing hardware in terms of memory and processing unit. This aspect makes the design of the systems challenging, requiring a trade-off between the accuracy and the complexity of the adopted models. This paper proposes a compact and robust service for Assistive Robotics, called Lightweight EMotion recognitiON (LEMON), which uses image processing, Computer Vision and Deep Learning (DL) algorithms to recognize facial expressions. Specifically, the proposed DL model is based on Residual Convolutional Neural Networks with the combination of Dilated and Standard Convolution Layers. The first remarkable result is the few numbers (i.e., 1.6 Million) of parameters characterizing our model. In addition, Dilated Convolutions expand receptive fields exponentially with preserving resolution, less computation and memory cost to recognize the distinction among facial expressions by capturing the displacement of the pixels. Finally, to reduce the dying ReLU problem and improve the stability of the model, we apply an Exponential Linear Unit (ELU) activation function in the initial layers of the model. We have performed training and evaluation (via one- and five-fold cross validation) of the model with five datasets available in the community and one mixed dataset created by taking samples from all of them. With respect to the other approaches, our model achieves comparable results with a significant reduction in terms of the number of parameters.

We propose to use a comprehensive path model of vocal emotion communication, encompassing encoding, transmission, and decoding processes, to empirically model data sets on emotion expression and recognition. The utility of the approach is demonstrated for two data sets from two different cultures and languages, based on corpora of vocal emotion enactment by professional actors and emotion inference by naïve listeners. Lens model equations, hierarchical regression, and multivariate path analysis are used to compare the relative contributions of objectively measured acoustic cues in the enacted expressions and subjective voice cues as perceived by listeners to the variance in emotion inference from vocal expressions for four emotion families (fear, anger, happiness, and sadness). While the results confirm the central role of arousal in vocal emotion communication, the utility of applying an extended path modeling framework is demonstrated by the identification of unique combinations of distal cues and proximal percepts carrying information about specific emotion families, independent of arousal. The statistical models generated show that more sophisticated acoustic parameters need to be developed to explain the distal underpinnings of subjective voice quality percepts that account for much of the variance in emotion inference, in particular voice instability and roughness. The general approach advocated here, as well as the specific results, open up new research strategies for work in psychology (specifically emotion and social perception research) and engineering and computer science (specifically research and development in the domain of affective computing, particularly on automatic emotion detection and synthetic emotion expression in avatars).

Information about the emotional state of a person can be inferred from facial expressions. Emotion recognition has become an active research area in recent years in various fields such as Human Robot Interaction (HRI), medicine, intelligent vehicle, etc., The challenges in emotion recognition from images with pose variations, motivates researchers to explore further. In this paper, we have proposed a method based on geometric features, considering images of 7 yaw angles (−45°,−30°,−15°,0°,+15°,+30°,+45°) from BU3DFE database. Most of the work that has been reported considered only positive yaw angles. In this work, we have included both positive and negative yaw angles. In the proposed method, feature extraction is carried out by concatenating distance and angle vectors between the feature points, and classification is performed using neural network. The results obtained for images with pose variations are encouraging and comparable with literature where work has been performed on pitch and yaw angles. Using our proposed method non-frontal views achieve similar accuracy when compared to frontal view thus making it pose invariant. The proposed method may be implemented for pitch and yaw angles in future.

This paper conducts a preliminary exploration of Artificial Intelligence (AI) for emotion recognition, particularly in its business applications. Employing adaptive technologies like machine learning algorithms and computer vision, AI systems analyze human emotions through facial expressions, speech patterns, and physiological signals. Ethical considerations and responsible deployment of these technologies are emphasized through an intense literature review. The study employs a comprehensive bibliometric analysis, utilizing tools such as VOSViewer, to trace the evolution of emotion-aware AI in business. Three key steps involve surveying the literature on emotion analysis, summarizing information on emotion in various contexts, and categorizing methods based on their areas of expertise. Comparative studies on emotion datasets reveal advancements in model fusion methods, exceeding human accuracy and enhancing applications in customer service and market research. The bibliometric analysis sheds light on a shift towards sophisticated, multimodal approaches in emotion recognition research, addressing challenges such as imbalanced datasets and interpretability issues. Visualizations depict keyword distributions in research papers, emphasizing the significance of “emotion recognition” and “deep learning.” The study concludes by offering insights gained from network visualization, showcasing core keywords and their density in research papers. Based on the literature, a SWOT analysis is also conducted to identify the strengths, weaknesses, opportunities, and threats associated with applying emotion recognition to business. Strengths include the technology’s high accuracy and real-time analysis capabilities, enabling diverse applications such as customer service and product quality improvement. However, weaknesses include data bias affecting the AI model’s quality and challenges in processing complex emotional expressions. Opportunities lie in the increasing number of studies, market size, and improving research outcomes, while threats include privacy concerns and growing competition.