Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis

The study of emotions in human-computer interaction is a growing research area. Focusing on automatic emotion recognition, work is being performed in order to achieve good results particularly in speech and facial gesture recognition. In this paper we present a study performed to analyze different machine learning techniques validity in automatic speech emotion recognition area. Using a bilingual affective database, different speech parameters have been calculated for each audio recording. Then, several machine learning techniques have been applied to evaluate their usefulness in speech emotion recognition, including techniques based on evolutive algorithms (EDA) to select speech feature subsets that optimize automatic emotion recognition success rate. Achieved experimental results show a representative increase in the success rate.KeywordsAffective computingMachine Learningspeech features extractionemotion recognition in speech

Automatic emotion recognition is a critical part of human-machine interactions. Reflection of emotions and to develop its understanding is crucial to provide dealings across human beings and machine frameworks. This work determines an automatic system that distinguishes different emotions connoted on the face. The framework is deliberated to apply the hybridization of feature extraction and optimization using PCA and PSO, respectively, to accomplish a high precision rate. PCA is used to get high-quality feature vectors for each category of emotion. Swarm intelligence, optimization is applied to get an optimized feature vector which is essential for classifying the features in the testing phase. For exploratory work, the authors have considered the Japanese Female Facial Expression (JAFFE) dataset. A maximum classification rate of 94.97% is achieved with the proposed technique. The proposed framework execution is assessed in terms of the false rejection rate, false acceptance rate, and accuracy.

Emotion is a central component of verbal communication between humans. Due to advances in machine learning and the development of affective computing, automatic emotion recognition is both increasingly possible and increasingly sought-after. In this project, deep learning architectures, such as long short-term memory networks, are used to classify emotion from the prosody of human speech. Emotional groupings are considered both categorically (i.e., happiness, anger, etc.) and with respect to a three-dimensional emotion space divided into axes of valence, activation, and dominance. The network is trained on publicly available emotional speech corpuses and tested on a new dataset as part of a larger project to understand the dynamics of collaborative groupwork. Specifically, small groups were asked to complete a lunar survival task while recorded in a smart meeting room equipped with multimodal sensors. The results of the emotion classification, as well as other non-verbal speech metrics, are correlated with user-reported rankings including emergent group leader and major group contributor, in the hopes of better understanding the complex dynamics of team meetings. [Work supported by NSF IIP-1631674, a Northeastern University Tier 1 Seed Grant, and the Cognitive and Immersive Systems Laboratory (CISL).]

Automatic emotion recognition is a challenging task since emotion is communicated through different modalities. Deep Convolution Neural Networks (DCNN) and transfer learning have shown success in automatic emotion recognition using different modalities. However significant improvement in accuracy is still required for practical applications. Existing methods are still not effective in modelling the temporal relationships within emotional expressions or in identifying the salient features from different modes and fusing them to improve accuracies. In this paper, we present an automatic emotion recognition system using audio and visual modalities. VGG19 models are used to capture frame level facial features followed by a Long Short Term Memory (LSTM) to capture their temporal distribution at a segment level. A separate VGG19 model captures auditory features from Mel Frequency Cepstral Coefficients (MFCC). The extracted auditory and visual features are fused together and a Deep Neural Network (DNN) with attention is used in classification using majority voting. Voice Activity Detection (VAD) on the audio stream improves performance by reducing the outliers in learning. The system is evaluated using Leave One Subject Out (LOSO) and K-fold cross-validation and our system outperforms state of the art methods on two challenging databases.

A dual framework for implicit and explicit emotion recognition: An ensemble of language models and computational linguistics

T Human-Computer Interaction recently received an increased attention. This is not just a matter of making the operation of technical systems as simple as pissuble, but also to enable a possibly natural interaction. In this context, especially the speech-based operation gained an increased attention. For example, modern smart phones and televisions offer a robust voice control, which is attributed to various technical improvements in recent years. Nevertheless, voice control still seems artificial. Only self-contained dialogues with short statements can be managed. Furthermore, just the content of speech is evaluated. The way in which something is said remains unconsidered, although it is known from human communication, that the transmitted emotion is important in order to communicate successfully. A relatively new branch of research, the “Affective Computing”, has, amongst other objectives, the aim to develop technical systems that recognise and interpret emotions and respond to them appropriately. In this case, speech-based automatic emotion recognition has a major role. For emotion recognition, it is important to know how emotions can be presented and how they are expressed. For this purpose, it is helpful to rely on empirical evidences of the psychology of emotions. Unfortunately, there is no uniform representation of emotions. Also the definition of appropriate emotion-distinctive acoustic features is rather descriptive in psychology. Therefore, the automatic detection of emotions is based on proven methods of automatic speech recognition, which have also been shown as appropriate for emotion recognition. Automatic emotion recognition is, as speech recognition, a branch of pattern recognition. Contrary to emotion psychology, it is data-driven, that means insights are gathered from sampled data. For emotion recognition the phases “annotation”, “modelling” and “recognition” are distinguishable. The annotation categorises speech data according to predefined emotion-terms. Modelling generates recognisers to categorise data automatically. Recognition performs a previously unknown allocation of data to emotional classes. In the beginning, automatic emotion recognition was usually based – due to the lack of suitable data sets – on acted and very expressive emotional expressions. In this case, based on features and detection methods known from speech recognition, very good recognition results of over 80% in distinguishing of up to seven emotions could be achieved. However, for human-machine interaction these recognisers were unsuitable because in this case emotions are not that expressive. Therefore, in collaboration with

PurposeUncertainty is an under-respected issue when it comes to automatic assessment of human emotion by machines. The purpose of this paper is to highlight the existent approaches towards such measurement of uncertainty, and identify further research need.Design/methodology/approachThe discussion is based on a literature review.FindingsTechnical solutions towards measurement of uncertainty in automatic emotion recognition (AER) exist but need to be extended to respect a range of so far underrepresented sources of uncertainty. These then need to be integrated into systems available to general users.Research limitations/implicationsNot all sources of uncertainty in automatic emotion recognition (AER) including emotion representation and annotation can be touched upon in this communication.Practical implicationsAER systems shall be enhanced by more meaningful and complete information provision on the uncertainty underlying their estimates. Limitations of their applicability should be communicated to users.Social implicationsUsers of automatic emotion recognition technology will become aware of their limitations, potentially leading to a fairer usage in crucial application context.Originality/valueThere is no previous discussion including the technical view point on extended uncertainty measurement in automatic emotion recognition.

The paper concerns technology of automatic emotion recognition applied in e-learning environment. During a study of e-learning process the authors applied facial expressions observation via multiple video cameras. Preliminary analysis of the facial expressions using automatic emotion recognition tools revealed several unexpected results, including unavailability of recognition due to face coverage and significant inconsistency between the results obtained from two cameras. The paper presents the experiment on e-learning process and summarizes the observations that constitute limitations of emotion recognition from facial expressions applied in e-learning context. The paper might be of interest to researchers and practitioners who consider automatic emotion recognition as an option in monitoring e-learning processes.

Performance Analysis of Malayalam Language Speech Emotion Recognition System Using ANN/SVM

Purpose of research.Emotions play one of the key roles in the regulation of human behaviour. Solving the problem of automatic recognition of emotions makes it possible to increase the effectiveness of operation of a whole range of digital systems such as security systems, human-machine interfaces, e-commerce systems, etc. At the same time, the low efficiency of modern approaches to recognizing emotions in speech can be noted. This work studies automatic recognition of emotions in speech applying machine learning methods.Methods.The article describes and tests an approach to automatic recognition of emotions in speech based on multitask learning of deep convolution neural networks of AlexNet and VGG architectures using automatic selection of the weight coefficients for each task when calculating the final loss value during learning. All the models were trained on a sample of the IEMOCAP dataset with four emotional categories of ‘anger’, ‘happiness’, ‘neutral emotion’, ‘sadness’. The log-mel spectrograms of statements processed by a specialized algorithm are used as input data.Results.The considered models were tested on the basis of numerical metrics: the share of correctly recognized instances, accuracy, completeness, f-measure. For all of the above metrics, an improvement in the quality of emotion recognition by the proposed model was obtained in comparison with the two basic single-task models as well as with known solutions. This result is achieved through the use of automatic weighting of the values of the loss functions from individual tasks when forming the final value of the error in the learning process.Conclusion.The resulting improvement in the quality of emotion recognition in comparison with the known solutions confirms the feasibility of applying multitask learning to increase the accuracy of emotion recognition models. The developed approach makes it possible to achieve a uniform and simultaneous reduction of errors of individual tasks, and is used in the field of emotions recognition in speech for the first time.

Research in emotion recognition seeks to develop insights into the temporal properties of emotion. However, automatic emotion recognition from spontaneous speech is challenging due to non-ideal recording conditions and highly ambiguous ground truth labels. Further, emotion recognition systems typically work with noisy high-dimensional data, rendering it difficult to find representative features and train an effective classifier. We tackle this problem by using Deep Belief Networks, which can model complex and non-linear high-level relationships between low-level features. We propose and evaluate a suite of hybrid classifiers based on Hidden Markov Models and Deep Belief Networks. We achieve state-of-the-art results on FAU Aibo, a benchmark dataset in emotion recognition [1]. Our work provides insights into important similarities and differences between speech and emotion.

Can edges help convolution neural networks in emotion recognition?

Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals

Since the inception of the Web 2.0, World Wide Web is widely being used as a platform by customers and manufactures to share experiences and opinions regarding products, services, marketing campaigns, social events, etc. As a result, there is enormous growth in user-generated contents (e.g. customer reviews), providing an opportunity for data analysts to computationally evaluate users’ sentiments and emotions for developing real-life applications for business intelligence, product recommendation, enhanced customer services, and target marketing. Since users’ feedbaks (aka reviews) are very useful for products development and marketing, large business houses and corporates are taking interest in opinion mining and sentiment analysis systems. In this chapter, we propose the design of an Ontology Learning and Sentiment Analysis (OntoLSA) system for ontology learning and sentiment analysis using rule-based and machine learning approaches. The rule-based approach aims to identify candidate concepts, which are analyzed using a customized HITS algorithm to compile a list of feasible concepts. Feasible concepts and their relationships (both structural and non-structural) are used to generate a domain ontology, which is later on used for opinion mining and sentiment analysis . The proposed system is also integrated with a visualization module to facilitate users to navigate through review documents at different levels of granularity using a graphical user interface.

Automatic emotion recognition from speech plays a fundamental role towards advanced emotional intelligence in human-machine interaction systems. The discriminative knowledge from speech for effective emotion recognition may come from multiple physical properties such as energy spectrum, frequency, prosody, which could be collected as multi-view representations. However, the current works fail to fully explore the underlying interactive relations among multiple speech representations for emotion recognition. In this paper, we propose a novel Collective Multi-view Relation Network (CMRN) to exploit the intrinsic characteristics of multi-view speech representations for discriminative speech emotion recognition. Generally, the proposed CMRN consists of three sub-networks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> view-specific attention network, multi-view shared attention network and collective relation network. Specifically, the view-specific attention network is designed to excavate the distinguishable view-specific features deduced from the original speech. By contrast, the multi-view shared attention network is conceived to capture the collaborative knowledge from multiple views. Moreover, a well-designed collective relation network is explicitly constructed to characterize the shared-specific correlations, which could reflect the underlying physical interaction capabilities. As such, the decision phase can comprehensively leverage the shared and view-specific information of multiple representations, such that the final privileged deciding principle can aggregate the heterogeneous information of multi-view features to make accurate emotion recognition. Extensive experiments on two benchmark datasets demonstrate the superb performance of the proposed method in comparison with some state-of-the-art methods.