Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Artificial intelligence (AI) and machine learning (ML) are employed to make systems smarter. Today, the speech emotion recognition (SER) system evaluates the emotional state of the speaker by investigating his/her speech signal. Emotion recognition is a challenging task for a machine. In addition, making it smarter so that the emotions are efficiently recognized by AI is equally challenging. The speech signal is quite hard to examine using signal processing methods because it consists of different frequencies and features that vary according to emotions, such as anger, fear, sadness, happiness, boredom, disgust, and surprise. Even though different algorithms are being developed for the SER, the success rates are very low according to the languages, the emotions, and the databases. In this paper, we propose a new lightweight effective SER model that has a low computational complexity and a high recognition accuracy. The suggested method uses the convolutional neural network (CNN) approach to learn the deep frequency features by using a plain rectangular filter with a modified pooling strategy that have more discriminative power for the SER. The proposed CNN model was trained on the extracted frequency features from the speech data and was then tested to predict the emotions. The proposed SER model was evaluated over two benchmarks, which included the interactive emotional dyadic motion capture (IEMOCAP) and the berlin emotional speech database (EMO-DB) speech datasets, and it obtained 77.01% and 92.02% recognition results. The experimental results demonstrated that the proposed CNN-based SER system can achieve a better recognition performance than the state-of-the-art SER systems.

Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition

It has been a challenging task to identify meaningful clues in speech/voice data to recognize emotion. Speech Emotion Recognition (SER) system is a collection of methodologies that process the speech data, extract useful features, use efficient classifier/classifiers to finally detect the embedded emotion from the six basic emotions, i.e., happiness, surprise, anger, sorrow, disgust and fear available. Increasing applications of SER are found in health care system, security, entertainment, education, BPOs, marketing, defense, customer care and Human Computer Interface (HCI) systems. This paper focuses on building a web-based interface for recognizing emotion from an audio file uploaded by user. The web app for SER is built on the flask framework following the supervised learning approach using Python. The system has been tested on publicly available TESS dataset with both Machine Learning (ML) and Deep Learning (DL) based models. Performance comparison of all the trained models is done with state of the art. Results are quite promising pointing towards developing a robust SER system.

Over the past years, speech emotion recognition (SER) studies have been gaining much interest in the fields of affective computing and human-computer interaction (HCI). The idea was to improve the interaction between human beings and machines. In this paper, an SER system that classifies and recognise six basic emotions (anger, sadness, disgust, fear, happiness, and neutral) from speech spoken in Sepedi language (one of South Africa's official languages) is discussed. Speech recordings were collected from the Sepedi language speakers and TV drama broadcast to create emotional speech corpora. 34 speech features were then extracted from the speech corpora, using the pyAudioAnalysis tool, to train and compare different algorithms using 10 folds cross-validation. The experiments were conducted using WEKA data-mining software. The results showed that Auto-WEKA outperforms all the standard algorithms (SVM. KNN and MLP). Recorded speech corpus yielded good recognition accuracy compare to TV broadcast speech corpus.

The Speech Emotion Recognition (SER) system is one of the essential human-computer interface applications. Despite the rapid advancement of technology, there is still a gap in SER research in the Arabic language corpus. The goal of this research is to build an Arabic-based SER based on a feature set that has both high performance and low computational cost. Two novel feature sets were implemented using a mix of spectral and prosodic features. An Arabic semi-natural corpus “EYASE” was adopted for testing the proposed system. Five machine learning classifiers using the different feature sets were implemented. Featureset-2 showed promising results in association with the SVM classifier. To validate the results, a survey analysis was implemented on the Arabic-Egyptian corpus “EYASE” used.

The acoustic cues that convey emotion in speech are similar to those that convey emotion in music, and recognition of emotion in both of these types of cue recruits overlapping networks in the brain. Given the similarities between music and speech prosody, developmental research is uniquely positioned to determine whether recognition of these cues develops in parallel. In the present study, we asked 60 children aged 6 to 11 years, and 51 university students, to judge the emotions of 10 musical excerpts, 10 inflected speech clips, and 10 affect burst clips. We presented stimuli intended to convey happiness, sadness, anger, fear, and pride. Each emotion was presented twice per type of stimulus. We found that recognition of emotions in music and speech developed in parallel, and adult-levels of recognition develop later for these stimuli than for affect bursts. We also found that sad stimuli were most easily recognised, followed by happiness, fear, and then anger. In addition, we found that recognition of emotion in speech and affect bursts can predict emotion recognition in music stimuli independently of age and musical training. Finally, although proud speech and affect bursts were not well recognised, children aged eight years and older showed adult-like responses in recognition of proud music.

Speech Emotion Recognition (SER) plays a vital role in various real-world applications, from mental health diagnostics to human-computer interaction. This research examines the integration of Stacked Long Short-Term Memory (LSTM) networks and Hidden Markov Models (HMM) for SER, addressing the limitations of traditional models like Gaussian Mixture Models (GMM) and Support Vector Machines (SVM). While GMMs and SVMs offer decent performance, their inability to model temporal dynamics limits their accuracy in detecting complex emotions. The proposed hybrid model leverages LSTM’s strength in handling sequential data and HMM’s ability to model emotional transitions, making it highly suitable for real-world noisy environments. Preprocessing techniques such as MFCC and LPCC are applied to enhance feature extraction, and benchmark datasets like IEMOCAP and RAVDESS are used for evaluation. Finally, The Systematic review highlights the superior role of the hybrid model's performance on SER and sets the stage for a significant shift in future research in addressing bias and fairness in SER systems by combining LSTM and HMM as hybrid model. In recent studies, context-aware emotion recognition models are commonly evaluated using existing datasets such as IEMOCAP, MELD, DailyDialog, and SEMAINE, which provide conversational and multimodal emotional data. Features used often include acoustic features as MFCCs, spectrograms, prosodic features, linguistic features as BERT or transformer embeddings, and visual cues such as facial landmarks or video frames. Many models also incorporate contextual embeddings from transformers, graph attention networks, or sequential memory modules to capture speaker history and emotional shifts.

Speech emotion recognition (SER) is a hot topic in academia. One of the key issues in improving the performance of SER systems is the choice of speech emotion features. In order to establish a robust speech emotion recognition system, it is essential to select the features which can be a perfect representation of speech emotion attributes. Researchers has done a lot of work, proposed a variety of emotional features and made great progress. Although each kind of features were proven to be effective, most of methods are based on a single type. In this paper, we proposed a method of feature fusion based on deep learning, combining spectral-based features and pitch-based hyper-prosodic features. The experiments show that this method improves the performance of speech emotion recognition system.

Speech emotion recognition has always been a challenging task due to the difference in emotion expression and perception. Currently, in the supervised speech emotion recognition systems, the soft label overcomes the disadvantage of the hard label losing annotations variability and emotion perception subjectivity, but it only considers the emotion perceptions of a few annotators and thus still brings high statistical error. For this issue, this paper redefines the target and designs a novel loss function (denoted as inter-class difference loss), which enables the network to adaptively learn an emotion distribution in all utterances. This not only restricts the negative class probability less than the positive class probability, but also limits the negative class probability close to zero. To make the speech emotion recognition system more efficient, this paper proposes an end-to-end network, called response residual network (R-ResNet), which incorporates the ResNet for features extraction, together with the emotion response module for data augmentation and variable-length data processing. Finally, the experimental results not only demonstrate the advanced performance of our work, but also confirm that the ambiguous utterances contain emotional characteristics. In addition, another interesting finding is that, on the unbalanced dataset, the batch normalization (BN) after addition performs better than BN before addition.

Identification of emotions along with the semantics from the speech signals is still one of the most challenging tasks because of the arising difficulty in the extraction and selection of the appropriate emotion features. The speech emotion recognition (SER) system has become one of the active research topics in recent years. Many researchers proposed various models to enhance the machine responses during the man–machine interactions by identifying the emotional state of the customer. This research work aims to propose a multi-class classification model to extract and classify the emotional state of the speaker into eight classes, namely happy, sad, surprised, disgust, neutral, calm, angry, and fearful. In this research work, two machine learning classifiers, namely k-nearest neighbors and random forest, are used along with the multi-layer perceptron classifier of deep learning for determining the emotional features from speech signals. The dataset for this research is obtained from the RAVDESS. Along with the audio speeches of 24 actors, the audio song dataset is also utilized for the better training of the proposed model. The performance of all three algorithms is compared and evaluated by using the confusion matrix and accuracy plots. The empirical results show that the KNN classification model of ML has performed better than other proposed algorithms with increased accuracy of 76.25%.

Speech Emotion Recognition (SER) plays a crucial role in applications involving human-machine interaction. However, the scarcity of suitable emotional speech datasets presents a major challenge for accurate SER systems. Deep Neural Network (DNN)-based solutions currently in use require substantial labelled data for successful training. Previous studies have proposed strategies to expand the training set in this framework by leveraging available emotion speech corpora. This paper assesses the impact of a cross-corpus training extension for a SER system using self-supervised (SS) representations, namely HuBERT and WavLM. The feasibility of training systems with just a few minutes of in-domain audio is also analyzed. The experimental results demonstrate that augmenting the training set with EmoDB (German), RAVDESS, and CREMA-D (English) datasets leads to improved SER accuracy on the IEMOCAP dataset. By combining a cross-corpus training extension and SS representations, state-of-the-art performance is achieved. These findings suggest that the cross-corpus strategy effectively addresses the scarcity of labelled data and enhances the performance of SER systems.

Emotional speech recognition: Resources, features, and methods

An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition

Speech is a commonly used signal for interaction between humans, this leads to the usage of speech for human and machine interactions as well. Improvements in this interactive system reach toward speech emotion recognition (SER) system. SER gives sufficient intelligence for efficient natural communication between humans and machines. SER system classifies emotional states such as sadness, angry, neutral, and happiness from the speaker's utterances. This paper describes speech features and machine learning models that can be used for SER. For effective classification and to learn multidimensional complex data, a deep learning algorithm is used in this system. This paper also presents the preliminary results of a system with an MFCC feature and an LSTM algorithm.

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features