Emotion Recognition from Audio and Visual Data using F-score based Fusion

Emotions are a crucial aspect of human life and the researchers have tried to build an automatic emotion recognition system that helps to provide important real-world applications. The psychologists have shown that emotions differ across culture, considering this fact, we provide and describe the first audio-visual Arabic emotional dataset which called (AVANEmo). In this work we aim to fill the gap between studies of emotion recognition for Arabic content and other languages by provided an Arabic dataset which is a major and fundamental part of build emotion recognition application. Our dataset contains 3000 clips for video and audio data, and it covers six basic emotional labels (Happy, Sad, Angry, Surprise, Disgust, Neutral). Also, we provide some baseline experiments to measure the primitive performance for automated audio and visual emotion recognition application using the AVANEmo dataset. The best accuracy that we achieved was 54.5% and 57.9% using the audio and visual data respectively. The data will be available for distribution to researchers.

A novel spatio-temporal convolutional neural framework for multimodal emotion recognition

Emotions are quite important in our daily communications and recent years have witnessed a lot of research works to develop reliable emotion recognition systems based on various types data sources such as audio and video. Since there is no apparently visual information of human faces, emotion analysis based on only audio data is a very challenging task. In this work, a novel emotion recognition is proposed based on robust features and machine learning from audio speech. For a person independent emotion recognition system, audio data is used as input to the system from which, Mel Frequency Cepstrum Coefficients (MFCC) are calculated as features. The MFCC features are then followed by discriminant analysis to minimize the inner-class scatterings while maximizing the inter-class scatterings. The robust discriminant features are then applied with an efficient and fast deep learning approach Neural Structured Learning (NSL) for emotion training and recognition. The proposed approach of combining MFCC, discriminant analysis and NSL generated superior recognition rates compared to other traditional approaches such as MFCC-DBN, MFCC-CNN, and MFCC-RNN during the experiments on an emotion dataset of audio speeches. The system can be adopted in smart environments such as homes or clinics to provide affective healthcare. Since NSL is fast and easy to implement, it can be tried on edge devices with limited datasets collected from edge sensors. Hence, we can push the decision-making step towards where data resides rather than conventionally processing of data and making decisions from far away of the data sources. The proposed approach can be applied in different practical applications such as understanding peoples’ emotions in their daily life and stress from the voice of the pilots or air traffic controllers in air traffic management systems.

Humans interact with others in several ways, such as speech, gesture, eye contact etc. Among them, speech is the most effective way of communication through which people can readily exchange information without the need for any other tool. Emotions color the speech, and can make the meaning more complex and tell about how it is said. A Mandarin speech based emotion classification method is presented. Five basic human emotions, anger, boredom, happiness, neutral and sadness, are investigated. The features extracted include 16 LPC (linear predictive cepstrum) coefficients and 20 MFCC (Mel-frequency cepstral coefficients) components, and the presented recognizer is based on two statistical pattern recognition techniques, the minimum-distance method and the nearest class mean method. For minimum-distance emotion recognition, an average accuracy of 79.1% is obtained. For the nearest class mean emotion recognition, higher accuracy of 89.1% is achieved.

Human emotion recognition has been widely used and intensively studied in many areas such as e-Commerce, online education, healthcare, human-computer interaction and recently human-centric software engineering (HCSE). HCSE investigates human factors in the entire software development lifecycle, and human emotion can be used in many scenarios such as requirement gathering and usability testing. However, even though existing studies have already shown the advantages of emotion recognition with multi-source data such as text, audio and video, the current research and practice in HCSE are primarily based on single-source data. In addition, emotion recognition in HCSE faces several challenges such as multiple participants, changing environments and real-time requirement. To tackle these challenges, this paper proposes Edge4Emotion, a novel edge computing-based multisource human emotion recognition platform for HCSE. Edge4Emotion takes the advantage of the edge computing paradigm to efficiently support the collection of multi-source data such as audio, video and physiological data from various IoT devices, and emotion recognition with both single- and multisource models. As an on-going project, this paper focuses on the platform design and the preliminary evaluation of the platform with representative emotion recognition applications. The platform will be further extended to include more multi-source learning models and serve as an open-source platform for the development and evaluation of multi-source emotion recognition models for HCSE.

Emotion detection, hence, is an important step toward making human-computer interaction a more enhanced process, where systems are made capable of identifying and responding to the emotional state of users. In fact, multimodal emotion detection systems in which both auditory and visual information are fused are emerging, and these approaches toward expressive emotional states are complementary and robust. Multimodal systems enhance the quality of interacting and, through many applications, can diagnose emotional disorders, monitor automotive safety, and improve human-robot interactions. In nature, the high-dimensional space and dynamic threats have resulted in obtaining low accuracy and high computational cost in applying the traditional models based on single-modality data. On the other hand, multimodal systems explore the synergy between audio and visual data, giving better performance and higher accuracy in inferring subtle emotional expressions. The latest improvement was done on these systems using recent advancements in transfer learning and deep learning techniques.That being said, this research Proposal devises a multimodal emotion recognition system integrating speech and face information through transfer learning for improved accuracy and robustness. Serving this purpose, the objectives of this research entail the effective comparison among different transfer-learning strategies, including the impact of pre-trained models in speech-based emotion recognition, and to introduce the role of voice activity detection in the process. Advanced neural network architectures like Spatial Transformer Networks and bidirectional LSTM in facial emotion recognition will also be tested. Early and late fusion strategies will also be used to find the best strategy for combining speech and facial data.This research will target several challenges that involve the complexity of data, balancing of the model performance-robustness balance, computational limitations, and standardization of evaluations in developing a working and robust emotion recognition system to enhance digital interaction and apply in practical areas. The goal is to create a system that oversteps the limitation of single-modality models through state-of-the-art advances in deep learning, as well as front-line improvements in transfer learning, in the manner of emotion detection performance.

A vector quantized masked autoencoder for audiovisual speech emotion recognition

Predicting outdoor ultrafine particle number concentrations, particle size, and noise using street-level images and audio data

Many studies have been conducted emotion recognition using audio signals as it is easy to collect. However, the accuracy is lower than other methods such as using facial images or video signals. In this paper, we propose an emotion recognition using speech signals and text simultaneously to achieve better performance. For training, we generate 43 feature vectors like mfcc, spectral features and harmonic features from audio data. Also 256 embedding vectors is extracted from text data using pretrained Tacotron encoder. Feature vectors and text embedding vectors are fed into each LSTM layer and fully connected layer which produces a probability distribution over predicted output classes. By combining the average of both results, the data is assigned to one of four emotion categories : anger, happiness, sadness, neutrality. Our proposed model outperforms previous state-of-the-art methods when they use Korean emotional speech dataset.

This paper provides an overview of my PhD project that focuses on recognizing emotions in dementia by analyzing multi-modal expressions in autobiographical memories of older adults with dementia. The project aims for a better understanding how dementia influences emotional expressions and how dementia differs from the normal aging process. For this reason, spontaneous emotions will be elicited in autobiographical memories in two groups of older adults, one with dementia the other without, for comparison. Audio, video and physiological data will be collected at their home resulting in real-life environments. The emotional expressions can then be analyzed by extracting verbal, non-verbal, facial and gestural features from the audio, video and physiological data collected. In addition, a longitudinal study will be conducted with the older adults with dementia to investigate the longitudinal effect of dementia on emotions. A database of the emotional memories of these vulnerable groups will then be developed to contribute to the advancement of technologies for (automatic) multi-modal emotion recognition. The database will then be made available for the research community. Lastly, we will also develop visualization and statistical models to assess multi-modal patterns of emotion expression in these groups.

Emotion recognition, an important research problem in human-robot interactions, is primarily achieved by extracting human emotions from audio and visual data. State-of-the-art performance is reported by audio-visual sensor fusion algorithms using deep learning models such as CNN, RNN, and LSTM. However, the RNN and LSTM are shown to be limited in handling the long-term dependencies over the entire input sequence. In this work, we propose to improve the performance of audio-visual emotion recognition using a novel transformer-based model, containing three transformer branches, named multimodal transformers. The three transformer branches, in our work, compute the audio self-attention, the video self-attention, and the audio-video cross attention. The self-attention branches identify the most relevant information in the audio and video input, while the cross-attention branch identifies the most relevant audio-video interactive information. The relevant information from these three branches report the best performance in our ablation study. We also propose a novel temporal embedding scheme, termed block embedding, to add the temporal information to the visual feature, derived from the multiple frames in the video. The proposed architecture is validated using the RAVDESS, CREMA-D, and SAVEE audio-visual public datasets. A detailed ablation study and comparative analysis with baseline models is performed. The results show that the proposed multi-modal transformer framework is better than the baseline methods.

<p><span id="docs-internal-guid-c2be589c-7fff-55df-f7fc-a27d9748d2d4"><span>Human emotion recognition</span><span> is an upcoming research field of human computer interaction based on facial gestures and is being used for real-time analysis in classifying cognitive affective states from a facial video data. Since computers have become an integral part of life, many researchers are using emotion recognition and classification of data based on audio and text. But these approaches offer limited accuracy and relevance in emotion classification. Therefore we have introduced and analyzed a hybrid approach which could outperform the existing strategies that uses an innovative approach supported by selection of audio and video data characteristics for classification. The research uses SVM for classifying the data using audio-visual savee database and the results obtained show maximum classification accuracy with respect to audio data about 91.6 could be improved to 99.2% after the application of hybrid strategy. </span></span></p>

Content Based Video Retrieval (CBVR) is so popular these days, because of the increased utilization of video based analytical systems. Video based analytics is quite effective than image analysis, as a series of actions are captured by the video. This ends up with better decision making ability. The CBVR systems play an important role in boosting the human–computer interaction. This paper presents a multimodal CBVR that takes both the visual and audio information into account for retrieving relevant videos to the user. Two modules are employed by this work to deal with video and audio data. The video data is processed to detect the significant frame from shots and is achieved by Lion Optimization Algorithm (LOA). The features are extracted from the visual data and with respect to the audio data, MHEC and LPCC features are extracted. The extracted features are clustered by Kernelized Fuzzy C Mean (KFCM) algorithm. Finally, the feature database is formed and is utilized in the query matching process during the testing phase. The performance of the proposed work is tested in terms of precision, recall, F-measure and time consumption rates. The proposed CBVR system proves better performance than the existing approaches and is evident through attained results.

Obtaining a compact and discriminative representation of facial and body expressions is a difficult problem in emotion recognition. Part of the difficulty is capturing microexpressions, i.e., short, involuntary expressions that last for only a fraction of a second: at a micro-temporal scale, there are so many other subtle face and body movements that do not convey semantically meaningful information. We present a novel approach to this problem by exploiting the sparsity of the frequent micro-temporal motion patterns. Local space-time features are extracted over the face and body region for a very short time period, e.g., few milliseconds. A codebook of microexpressions is learned from the data and used to encode the features in a sparse manner. This allows us to obtain a representation that captures the most salient motion patterns of the face and body at a micro-temporal scale. Experiments performed on the AVEC 2012 dataset show our approach achieving the best published performance on the arousal dimension based solely on visual features. We also report experimental results on audio-visual emotion recognition, comparing early and late data fusion techniques.

The purpose of this study is to propose an emotion recognition algorithm for piano music based on multimodal learning. Multi-modal learning is a learning method that combines various types of data. By combining different modal information, learning tasks can be completed more comprehensively and accurately. In the field of music emotion recognition, multimodal learning can extract the emotional features of music from different angles, and improve the recognition accuracy and robustness. In this study, the multi-modal learning method is mainly used, and the representative features are extracted by combining the audio signal, score information and the performance of the piano music, and the machine learning algorithm is used to classify the emotions. The audio signal is preprocessed, including noise reduction, cutting and other operations, and then the features of the audio signal are obtained by feature extraction methods, such as short-time Fourier transform (STFT) and formant analysis (RPA). Digitally convert the score, extract the structure, melody, harmony and other characteristics of the music, and obtain the emotional information of the music by analyzing the relationship between symbols and notes in the score. By analyzing the performance skills, strength, speed and other parameters of the player, the characteristics related to the emotional expression of the player are extracted. Machine learning algorithm is used to train and classify the fusion features, and the emotion recognition of piano music is realized. In this study, the emotion recognition algorithm of piano music based on multimodal learning can mine the emotion information of music at different levels, improve the accuracy and robustness of emotion recognition, and provide useful reference for the field of music emotion calculation.