Automatic highlights extraction for drama video using music emotion and human face features

Alexithymia, a multifaceted personality construct, is known to be related to difficulties in the decoding of emotional facial expressions, especially in case of suboptimal stimuli. The present study investigated whether and which facets of alexithymia are related to impairments in the recognition of emotions in faces with face masks. Accuracy and speed of emotion recognition were examined in a block of faces with and a block of faces without face masks in a sample of 102 healthy individuals. The order of blocks varied between participants. Emotions were recognized better and faster in unmasked than in masked faces. Recognition performance was worst and slowest for participants starting the task with masked faces. In the whole sample, there were no correlations of alexithymia facets with accuracy and speed of emotion recognition for masked and unmasked faces. In participants starting the task with masked faces, the facet externally oriented thinking was positively correlated with reaction latencies of correct responses for masked faces. Our findings indicate that an externally oriented thinking style could be linked to a less efficient identification of emotions from faces wearing masks when task difficulty is high and support the utility of a facet approach in alexithymia research.

In this work, we have proposed STLEV (Siamese Neural Network with Triplet Loss and Long Short-Term Memory for Emotion recognition in Videos) a method for emotion recognition in videos. This method is beneficial for cognitive Human-Computer interaction. Emotion recognition from video is a challenging task as only few frames of a video will contain relevant information for recognizing emotions. Also identifying the apex frame is challenging. Emotion recognition from videos involves extracting the features from the frames of a video and classifying the sequence of features. For feature extraction we have used Siamese Neural Network (SNN) which is metric based meta-learning model. SNN is trained with triplet loss. For classification, Long Shot-Term Memory (LSTM) model is used. In this work we have used spatial and temporal aspects of video. Spatial features are extracted using SNN and temporal features using LSTM. Deep learning models need large amount of data for good accuracy. We have addressed this problem by leveraging few shot learning aspects of metric meta-learning so that fewer samples are needed for training the model used for feature extraction. Experiments done using BU-4DFE dataset achieved an accuracy of 87.5% which demonstrates the efficiency of the method STLEV.

Patients with amygdala lesions were found to be impaired in recognizing the fear emotion both from face and from music. In patients with Parkinson's disease (PD), impairment in recognition of emotions from facial expressions was reported for disgust, fear, sadness and anger, but no studies had yet investigated this population for the recognition of emotions from both face and music. The ability to recognize basic universal emotions (fear, happiness and sadness) from both face and music was investigated in 24 medicated patients with PD and 24 healthy controls. The patient group was tested for language (verbal fluency tasks), memory (digit and spatial span), executive functions (Similarities and Picture Completion subtests of the WAIS III, Brixton and Stroop tests), visual attention (Bells test), and fulfilled self-assessment tests for anxiety and depression. Results showed that the PD group was significantly impaired for recognition of both fear and sadness emotions from facial expressions, whereas their performance in recognition of emotions from musical excerpts was not different from that of the control group. The scores of fear and sadness recognition from faces were neither correlated to scores in tests for executive and cognitive functions, nor to scores in self-assessment scales. We attributed the observed dissociation to the modality (visual vs. auditory) of presentation and to the ecological value of the musical stimuli that we used. We discuss the relevance of our findings for the care of patients with PD.

Due to the complexity of its system and the numerous advantages of its implementation, video emotion identification is a popular area of study today. There have been multiple ways implemented. In this project, emotion recognition in videos will be performed using deep learning. The model will include initialization, feature extraction, emotion categorization, and prediction. LeNet and AlexNet, two distinct neural networks, are used to extract features and classify emotions. There will be some parameter tuning to determine if it enhances the employed architecture’s performance, including optimizers and batch size. Each architecture for deep learning will generate a final forecast of four fundamental emotions: disgust, happiness, sadness, and surprise. AlexNet’s performance is enhanced by the SGD optimizer, whereas RMSprop improves LeNet’s performance. Results showed that AlexNet with SGD optimizer provides 93.00% recognition accuracy.

In this paper, we mainly study the emotion recognition algorithm based on ECG signals, extract the correlation feature and time-frequency domain statistical feature of ECG signals, and introduce SVW, CART and KNN three classification algorithms commonly used in emotion recognition. By comparing the accuracy of emotion recognition in the application of three classification algorithms between the correlation features of ECG signals and traditional time-frequency domain features, we found that the use of correlation features of ECG signals can get a higher recognition rate, which is 16.7%∼19.7% higher than that of the traditional feature. In addition, among the three classification algorithms, KNN algorithm can get the highest emotion recognition accuracy. In order to further improve the accuracy of emotion recognition, Max-Min Ant System is combined with KNN classification algorithm in this paper to optimize the feature combination. The overall recognition rate reaches 92%, which is 16.9% higher than the accuracy of emotion recognition directly using KNN classification algorithm.

IntroductionUnderstanding how children recognize facial emotions is essential for explaining the development of social communication. During the COVID-19 pandemic, widespread mask-wearing partially occluded facial expressions, potentially influencing children's emotion recognition processes.MethodsThis study investigated age-related differences in emotion recognition accuracy and gaze fixation patterns in young children using eye-tracking technology. Participants included 36 three-year-olds and 45 five-year-olds. Children viewed facial expressions representing four emotions (joy, sadness, anger, and fear) presented in both masked and unmasked conditions while fixation duration and recognition accuracy were recorded.ResultsThe results showed significant effects of age, emotion type, and mask-wearing on emotion recognition accuracy. Overall recognition accuracy decreased under masked conditions. Eye-tracking analysis revealed that children allocated significantly greater visual attention to the eye region when faces were masked, whereas gaze distribution between the eyes and mouth was more balanced in unmasked conditions.DiscussionThese findings suggest that facial occlusion caused by mask-wearing alters visual attention strategies and emotion recognition processes in early childhood. The results provide important insights into how young children adapt their perceptual strategies when key facial cues are partially obscured.

A dual framework for implicit and explicit emotion recognition: An ensemble of language models and computational linguistics

Automatic emotion recognition from speech is a demanding and challenging problem. It is difficult to differentiate between the emotional states of humans. The major problem with this task is to extract the important features from the speech in case of hand-crafted features. The accuracy for emotion recognition can be increased using deep learning approaches which use high level features of speech signals. In this work, an algorithm is proposed using deep learning to extract the high-level features from raw data with high accuracy irrespective of language and speakers (male/females) of speech corpora. For this, the .wav files are converted into the RGB spectrograms (images) and normalized to size (224x224x3) for fine-tuning these for Deep Convolutional Neural Network (DCNN) to recognize emotions. DCNN model is trained in two stages. From stage-1 the optimal learning rate is identified using the Learning Rate (LR) range test and then the model is trained again with optimal learning rate in stage-2. Special stride is used for down-sampling the features with reduced model size. The emotions considered are happiness, sadness, anger, fear, disgust, boredom/surprise and neutral. The proposed algorithm is tested on three popular public speech corpora EMODB (German), EMOVO (Italian), and SAVEE (British English). The accuracy of emotion recognition reported is better as compared to the existing studies for different languages and speakers.

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

Making human–computer interaction more organic and personalized for users essentially demands advancement in human emotion recognition. Emotions are perceived by humans considering multiple factors such as facial expressions, voice tonality, and information context. Although significant research has been conducted in the area of unimodal/multimodal emotion recognition in videos using acoustic/visual features, few papers have explored the potential of textual information obtained from the video utterances. Humans experience emotions through their audio-visual and linguistic senses, making it quintessential to take the latter into account. This paper outlines two different algorithms for recognizing multimodal emotional expressions in online videos. In addition to acoustic (speech), visual (facial), and textual (utterances) feature extraction using BERT, we utilize bidirectional LSTMs to capture the context between utterances. To obtain richer sequential information, we also implement a multi-head self-attention mechanism. Our analysis utilizes the benchmarking CMU multimodal opinion sentiment and emotion intensity (CMU-MOSEI) dataset, which is the largest dataset for sentiment analysis and emotion recognition to date. Our experiments result in improved F1 scores in comparison to the baseline models.KeywordMultimodalEmotionRecognitionCMU-MOSEIMulti-head attentionBERTContext-aware

People living with Human Immunodeficiency Virus (PLHIV) have been reported to show poor facial emotion recognition. However, these studies presented participants with facial emotion photographs whereas in real life facial emotion recognition hardly involves inferring emotions from static faces. Moreover, emotion recognition from other sensory modalities, such as auditory, has hardly been explored. There’s also a dearth of studies examining emotion regulation difficulties in this group. The present study, thus, explored facial (using facial emotion videos) and auditory emotion recognition as well as difficulties in emotion regulation (using the Hindi version of Difficulties in Emotion Regulation Scale) in 60 PLHIV and 60 people without HIV (PWoHIV). Additionally, the association of HIV duration (duration since diagnosis of HIV), viral load, and Clusters of differentiation 4 (CD4) count with emotion recognition and regulation difficulties in PLHIV was explored. Findings from one-way ANCOVA (with education and socioeconomic status as covariates) revealed significantly impaired auditory emotion recognition (particularly for fear) among PLHIV than PWoHIV. The former also showed significantly poorer facial emotion recognition for surprise. PLHIV also self-reported significantly more emotion regulation difficulties than PWoHIV, specifically Nonacceptance of their response to negative emotions and limited access to emotion regulation Strategies. CD4 count was negatively correlated with emotion regulation difficulties, particularly for accomplishing goal-directed behaviour when experiencing negative emotions (Goals) and Strategies. Besides the novel addition to the literature regarding impaired auditory emotion recognition in PLHIV, these findings can help develop targeted interventions to improve emotion recognition and emotion regulation for PLHIV.

In order to overcome the problem that the accuracy and efficiency of Mandarin emotion recognition is low based on singal-modal, a tri-modal Mandarin emotion recognition method based on the feature-level fusion of three important modal of speech, facial expression and body gesture is proposed. Firstly, different emotional features are extracted from speech, facial expressions and body gestures, and then the three modal emotion features are fused by IAGA method. Finally, SVM classifier is used to construct prediction model and emotion recognition is completed on it. The CHEAVD Chinease multi-modal emotion dataset is utilized to evaluate our proposed methods. Compared with the traditional mono-modal and dual-modal emotion recognition, the experimental results show that the dual-modal fusion has higher emotion recognition rate than the mono-modal, and the tri-modal fusion has higher emotion recognition rate than the dual-modal, which verifies the effectiveness of the tri-modal Mandarin emotion recognition

We investigated neural correlates of Emotion Recognition Accuracy (ERA) using the Assessment of Contextualized Emotions (ACE). ACE infuses context by presenting emotion expressions in a naturalistic group setting and distinguishes between accurately perceiving intended emotions (signal), and bias due to perceiving additional, secondary emotions (noise). This social perception process is argued to induce perspective taking in addition to pattern matching in ERA. Thirty participants were presented with an fMRI-compatible adaptation of the ACE consisting of blocks of neutral and emotional faces in single and group-embedded settings. Participants rated the central character’s expressions categorically or using scalar scales in consequent fMRI scans. Distinct brain activations were associated with the perception of emotional vs. neutral faces in the four conditions. Moreover, accuracy and bias scores from the original ACE task performed on another day were associated with brain activation during the scalar (vs. categorical) condition for emotional (vs. neutral) faces embedded in group. These findings suggest distinct cognitive mechanisms linked to each type of emotional rating and highlight the importance of considering cognitive bias in the assessment of social emotion perception.

Current literature suggests that theory of mind (ToM) and recognition of facial emotion are impaired in behavioral variant frontotemporal dementia (bvFTD). In contrast, studies suggest that ToM is spared in Alzheimer disease (AD). However, there is controversy whether recognition of emotion in faces is impaired in AD. This study challenges the concepts that ToM is preserved in AD and that recognition of facial emotion is impaired in bvFTD. ToM, recognition of facial emotion, and identification of emotions associated with video vignettes were studied in bvFTD, AD, and normal controls. ToM was assessed using false-belief and visual perspective-taking tasks. Identification of facial emotion was tested using Ekman and Friesen's pictures of facial affect. After adjusting for relevant covariates, there were significant ToM deficits in bvFTD and AD compared with controls, whereas neither group was impaired in the identification of emotions associated with video vignettes. There was borderline impairment in recognizing angry faces in bvFTD. Patients with AD showed significant deficits on false belief and visual perspective taking, and bvFTD patients were impaired on second-order false belief. We report novel findings challenging the concepts that ToM is spared in AD and that recognition of facial emotion is impaired in bvFTD.

Technology-assisted intervention has the potential to adaptively individualize and improve outcomes of traditional schizophrenia SZ intervention. Virtual reality VR technology, in particular, has the potential to simulate real world social and communication interactions and hence could be useful as a therapeutic platform for SZ. Emotional face recognition is considered among the core building blocks of social communication. Studies have shown that emotional face processing and understanding is impaired in patients with SZ. The current study develops a novel VR-based system that presents avatars that can change their facial emotion dynamically for emotion recognition tasks. Additionally, this system allows real-time measurement of physiological signals and eye gaze during the emotion recognition tasks, which can be used to gain insight about the emotion recognition process in SZ population. This study further compares VR-based facial emotion recognition with that of the more traditional emotion recognition from static faces using a small usability study. Results from the usability study suggest that VR could be a viable platform for SZ intervention and implicit signals such as physiological signals and eye gaze can be utilized to better understand the underlying pattern that is not available from user reports and performance alone.