A new information fusion method for SVM-based robotic audio-visual emotion recognition

In human-computer interaction, it is an urgent problem to use facial expressions and speech information to identify the user's continuous emotions, and the key factors affecting the recognition accuracy are the data deficiencies during the fusion of speech and facial information, and the abnormal frames in the video. In order to solve these problems, a user emotion recognition system based on the fusion of facial expressions and speech multimodality is designed. In the part of facial expressions, Gabor transform continuous emotion recognition method based on data increments is proposed. In the part of speech information, Mel-scale Frequency Cepstral Coefficients (MFCC) is used to extract speech features, and user emotions are recognize through transfer learning. Finally, in the late fusion, multiple linear regression is used for multi-modality to verify the method in this paper. This paper uses the AVEC2013 dataset with Arousal-Valence label to conduct a valid experiment on the proposed method. The experimental results prove that the method improves the accuracy of user emotion recognition.

A survey of emotion recognition methods with emphasis on E-Learning environments

Individuals with cochlear implants (CIs) show reduced word and auditory emotion recognition abilities relative to their peers with normal hearing. Modern CI processing strategies are designed to preserve acoustic cues requisite for word recognition rather than those cues required for accessing other signal information (e.g., talker gender or emotional state). While word recognition is undoubtedly important for communication, the inaccessibility of this additional signal information in speech may lead to negative social experiences and outcomes for individuals with hearing loss. This study aimed to evaluate whether the emphasis on word recognition preservation in CI processing has unintended consequences on the perception of other talker information, such as emotional state. Twenty-four young adult listeners with normal hearing listened to sentences and either reported a target word in each sentence (word recognition task) or selected the emotion of the talker (emotion recognition task) from a list of options (Angry, Calm, Happy, and Sad). Sentences were blocked by task type (emotion recognition versus word recognition) and processing condition (unprocessed versus 8-channel noise vocoder) and presented randomly within the block at three signal-to-noise ratios (SNRs) in a background of speech-shaped noise. Confusion matrices showed the number of errors in emotion recognition by listeners. Listeners demonstrated better emotion recognition performance than word recognition performance at the same SNR. Unprocessed speech resulted in higher recognition rates than vocoded stimuli. Recognition performance (for both words and emotions) decreased with worsening SNR. Vocoding speech resulted in a greater negative impact on emotion recognition than it did for word recognition. These data confirm prior work that suggests that in background noise, emotional prosodic information in speech is easier to recognize than word information, even after simulated CI processing. However, emotion recognition may be more negatively impacted by background noise and CI processing than word recognition. Future work could explore CI processing strategies that better encode prosodic information and investigate this effect in individuals with CIs as opposed to vocoded simulation. This study emphasized the need for clinicians to consider not only word recognition but also other aspects of speech that are critical to successful social communication.

Patients with amygdala lesions were found to be impaired in recognizing the fear emotion both from face and from music. In patients with Parkinson's disease (PD), impairment in recognition of emotions from facial expressions was reported for disgust, fear, sadness and anger, but no studies had yet investigated this population for the recognition of emotions from both face and music. The ability to recognize basic universal emotions (fear, happiness and sadness) from both face and music was investigated in 24 medicated patients with PD and 24 healthy controls. The patient group was tested for language (verbal fluency tasks), memory (digit and spatial span), executive functions (Similarities and Picture Completion subtests of the WAIS III, Brixton and Stroop tests), visual attention (Bells test), and fulfilled self-assessment tests for anxiety and depression. Results showed that the PD group was significantly impaired for recognition of both fear and sadness emotions from facial expressions, whereas their performance in recognition of emotions from musical excerpts was not different from that of the control group. The scores of fear and sadness recognition from faces were neither correlated to scores in tests for executive and cognitive functions, nor to scores in self-assessment scales. We attributed the observed dissociation to the modality (visual vs. auditory) of presentation and to the ecological value of the musical stimuli that we used. We discuss the relevance of our findings for the care of patients with PD.

Emotional state recognition is an important component for efficient human-computer interaction. Most existing works address this problem using 2D features, but they are sensitive to head pose, clutter, and variations in lighting conditions. The general 3D based methods only consider geometric information for feature extraction. In this paper, we present a real 3D visual features based method for human emotion recognition. 3D geometric information plus colour/density information of the facial expressions are extracted by 3D Gabor library to construct visual feature vectors. The filter's scale, orientation, and shape of the library are specified according to the appearance patterns of the 3D facial expressions. An improved kernel canonical correlation analysis (IKCCA) algorithm is proposed for final decision. From training samples, the semantic ratings that describe the different facial expressions are computed by IKCCA to generate a seven dimensional semantic expression vector. It is applied for learning the correlation with different testing samples. According to this correlation, we estimate the associated expression vector and perform expression classification. From experiment results, our proposed method demonstrates impressive performance.

IntroductionRecent developments in the field of social cognition have led to a renewed interest in basic and social emotion recognition in early stages of Alzheimer's Disease (AD) and FrontoTemporal Dementia (FTD). Despite the growing attention to this issue, only few studies have attempted to investigate emotion recognition using both visual and vocal stimuli. In addition, recent studies have presented conflicting findings regarding the extent of impairment in patients in the early stages of these diseases. The present study aims to investigate emotion understanding (both basic and social emotions), using different tasks with visual and auditory stimuli, to identify supramodal deficits in AD and FTD to provide a reliable tool to better outline their behavioral and emotional profile and useful instruments for their management.MethodsEighteen patients with AD and 15 patients with FTD were included in the study. Healthy control (HCs) subjects were recruited to obtain normative data for basic emotion recognition tests and social emotion recognition tasks. To evaluate basic emotion recognition, the Facial Emotion Recognition Battery (FERB) and the Emotional Prosody Recognition Battery (EPRB) were administered. To evaluate social emotion recognition, the Faux Pas (FP), Reading the Mind in the Eyes (RME), and Reading the Mind in the Voice (RMV) tests were employed.ResultsFTD patients performed significantly worse than HCs in most of the subtests of the basic emotion recognition batteries, where, instead, AD patients were significantly impaired only when required to match emotional facial expression in different individuals (subtask of the FERB). Moreover, FTD patients scored significantly lower in RME and RMV tests compared both to AD patients and to HCs. In addition, ADs were selectively impaired in RMV as respect to HCs.DiscussionFTD patients showed deficits in emotion recognition, affecting both basic and social emotions, whether conveyed through facial expressions or prosody. This result may explain the well-known social behavioral difficulties observed in FTD patients from the early stages of the disease. The fewer and specific deficits in AD patients with comparable MMSE scores may be attributed to the mild degree of impairment, as these deficits may appear later in the progression of AD.

Recent advances in Semantic IoT data integration have highlighted the importance of multimodal fusion in emotion recognition systems. Human emotions, formed through innate learning and communication, are often revealed through speech and facial expressions. In response, this study proposes a hidden Markov model‐based multimodal fusion emotion detection system, combining speech recognition with facial expressions to enhance emotion recognition rates. The integration of such emotion recognition systems with Semantic IoT data can offer unprecedented insights into human behavior and sentiment analysis, contributing to the advancement of data integration techniques in the context of the Internet of Things. Experimental findings indicate that in single‐modal emotion detection, speech recognition achieves a 76% accuracy rate, while facial expression recognition achieves 78%. However, when state information fusion is applied, the recognition rate increases to 95%, surpassing the national average by 19% and 17% for speech and facial expressions, respectively. This demonstrates the effectiveness of multimodal fusion in emotion recognition, leading to higher recognition rates and reduced workload compared to single‐modal approaches.

In order to overcome the problems of high recognition error rate and low recognition efficiency existing in the traditional method of emotional recognition of weightlifters, a method of emotional recognition of weightlifters based on a non-negative matrix decomposition algorithm is proposed. The database of emotion recognition matching criteria is established, and the mapping relationship among emotion, facial expression of far mobilisation and weight lifting posture is formed. The real-time motion images of weightlifters are collected, and the image classifier is designed by using the non-negative matrix decomposition algorithm. The real-time images are divided into facial expression images and whole body motion images, and the features of the two images are extracted respectively. The extracted results are matched with the results of the standard database, and the real-time emotion recognition results of weightlifters are obtained. The experimental results show that compared with the traditional emotion recognition method, the proposed method improves the accuracy of emotion recognition by 50%.

Humans have traditionally found it simple to identify emotions from facial expressions, but it is far more difficult for a computer system to do the same. The social signal processing subfield of emotion recognition from facial expression is used in a wide range of contexts, particularly for human-computer interaction. Automatic emotion recognition has been the subject of numerous studies, most of which use a machine learning methodology. The recognition of simple emotions like anger, happiness, contempt, fear, sadness, and surprise, however, continues to be a difficult topic in computer vision. Deep learning has recently drawn increased attention as a solution to a variety of practical issues, including emotion recognition. In this study, we improved the convolutional neural network technique to identify 7 fundamental emotions and evaluated several preprocessing techniques to demonstrate how they affected the CNN performance. This research focuses on improving facial features and expressions based on emotional recognition. By identifying or recognising facial expressions that elicit human responses, it is possible for computers to make more accurate predictions about a person's mental state and to provide more tailored responses. As a result, we examine how a deep learning technique that employs a convolutional neural network might improve the detection of emotions based on facial features (CNN). Multiple facial expressions are included in our dataset, which consists of about 32,298 photos for testing and training. The preprocessing system aids in removing noise from the input image, and the pretraining phase aids in revealing face detection after noise removal, including feature extraction. As a result, the existing paper generates the classification of multiple facial reactions like the seven emotions of the facial acting coding system (FACS) without using the optimization technique, but our proposed paper reveals the same seven emotions of the facial acting coding system.

BackgroundRecognition of others' emotions is an important aspect of interpersonal communication. In major depression, a significant emotion recognition impairment has been reported. It remains unclear whether the ability to recognize emotion from facial expressions is also impaired in anxiety disorders. There is a need to review and integrate the published literature on emotional expression recognition in anxiety disorders and major depression.Methodology/Principal FindingsA detailed literature search was used to identify studies on explicit emotion recognition in patients with anxiety disorders and major depression compared to healthy participants. Eighteen studies provided sufficient information to be included. The differences on emotion recognition impairment between patients and controls (Cohen's d) with corresponding confidence intervals were computed for each study. Over all studies, adults with anxiety disorders had a significant impairment in emotion recognition (d = −0.35). In children with anxiety disorders no significant impairment of emotion recognition was found (d = −0.03). Major depression was associated with an even larger impairment in recognition of facial expressions of emotion (d = −0.58).Conclusions/SignificanceResults from the current analysis support the hypothesis that adults with anxiety disorders or major depression both have a deficit in recognizing facial expression of emotions, and that this deficit is more pronounced in major depression than in anxiety.

In order to quickly distinguish infertile eggs from fertile eggs, the hyperspectral imaging technology consisting of imaging and spectral information was used for detecting the fertile information of eggs. Before hatching eggs were incubated, a hyperspectral imaging system (wavelength between 400 to 1 000 nm) was used to acquire the images one-by-one manually. The characteristic information of ratios of length to short axis, elongation, roundness and the ratios of the yolk area to the whole area was extracted based on the images. The normalization method was used as the spectral data preprocessing, and then 155 spectral characteristic variables were extracted from 520 nm waveband through the correlation coefficient method. Principal component analysis (PCA) method was adopted to reduce the dimensions of image-spectrum fusion information; the top six principal components were extracted. Support vector machine (SVM) method was used to establish classification of fertile and infertile eggs models, which are based on image, spectrum and image-spectrum fusion information respectively. The accuracy rates of the SVM models were 84.00%, 90.00% and 93.00% respectively. The experimental results show that the model based on image-spectrum fusion information technology is superior to the single information model. Hyperspectral transmission imaging technology is effective and feasible to detect the fertile hatching eggs before incubation. Keywords: hyperspectral image, hatching eggs, information fusion, nondestructive detection DOI: 10.3965/j.ijabe.20150804.1672 Citation: Zhu Z H, Liu T, Xie D J, Wang Q H, Ma M H. Nondestructive detection of infertile hatching eggs based on spectral and imaging information. Int J Agric & Biol Eng, 2015; 8(4): 69－76.

Technology-assisted intervention has the potential to adaptively individualize and improve outcomes of traditional schizophrenia SZ intervention. Virtual reality VR technology, in particular, has the potential to simulate real world social and communication interactions and hence could be useful as a therapeutic platform for SZ. Emotional face recognition is considered among the core building blocks of social communication. Studies have shown that emotional face processing and understanding is impaired in patients with SZ. The current study develops a novel VR-based system that presents avatars that can change their facial emotion dynamically for emotion recognition tasks. Additionally, this system allows real-time measurement of physiological signals and eye gaze during the emotion recognition tasks, which can be used to gain insight about the emotion recognition process in SZ population. This study further compares VR-based facial emotion recognition with that of the more traditional emotion recognition from static faces using a small usability study. Results from the usability study suggest that VR could be a viable platform for SZ intervention and implicit signals such as physiological signals and eye gaze can be utilized to better understand the underlying pattern that is not available from user reports and performance alone.

In this paper, we present a new method for emotion recognition from facial expressions. The proposed algorithm concentrates on only two specific areas (eyes and mouth), reducing features and descriptors and focusing only on these areas. The algorithm extracts characteristics from these two regions of the face and, in a subsequent process, eliminates the less significant characteristics or those that introduce noise into the classifier. The system allows obtaining a reduced set of features to improve the performance of the classification. In the experiments carried out, we obtain precisions of 99.56%. We evaluated the proposed algorithm on two benchmark datasets; we find that SVM consistently outperforms traditional machine learning techniques.

Normal recognition of emotional similarity between facial expressions following bilateral amygdala damage

In order to overcome the problem that the accuracy and efficiency of Mandarin emotion recognition is low based on singal-modal, a tri-modal Mandarin emotion recognition method based on the feature-level fusion of three important modal of speech, facial expression and body gesture is proposed. Firstly, different emotional features are extracted from speech, facial expressions and body gestures, and then the three modal emotion features are fused by IAGA method. Finally, SVM classifier is used to construct prediction model and emotion recognition is completed on it. The CHEAVD Chinease multi-modal emotion dataset is utilized to evaluate our proposed methods. Compared with the traditional mono-modal and dual-modal emotion recognition, the experimental results show that the dual-modal fusion has higher emotion recognition rate than the mono-modal, and the tri-modal fusion has higher emotion recognition rate than the dual-modal, which verifies the effectiveness of the tri-modal Mandarin emotion recognition