Speech Emotion Recognition Model Using Deep Learning

An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition

Emotion recognition from audio data holds immense potential in revolutionizing humancomputer interaction (HMI), affective computing, and psychological health monitoring.This paper delves into a novel deep learning approach that leverages the strengths of multimodal features mined from audio signals.We propose a model that transcends the disadvantages of existing methods by combining Mel-Frequency Cepstral Coefficients (MFCCs) with high-level representations extracted from a pre-trained DenseNet architecture.MFCCs provide a compressed representation of the audio signal's spectral characteristics, capturing crucial emotional cues like pitch and intensity.These learned patterns can translate to the domain of audio emotion recognition, enabling the model to identify subtle emotional nuances that might be difficult to capture with traditional feature engineering techniques.Our deep learning model, comprised of dense layers, fosters robust performance in accurately classifying emotions across diverse categories.We used a Melspectrograms-based LSTM model for speech emotion recognition that effectively identifies various emotions.We rigorously evaluate the proposed approach on the TESS dataset.The experimental results are truly compelling, showcasing a staggering accuracy of 100%.This exceptional performance signifies the effectiveness of the multimodal approach in extracting and interpreting emotional cues from audio data.

Speech is a powerful means of expressing thoughts, emotions, and perspectives. However, accurately determining the emotions conveyed through speech remains a challenging task. Existing manual methods for analyzing speech to recognize emotions are prone to errors, limiting our understanding and response to individuals’ emotional states. To address diverse accents, an automated system capable of real-time emotion prediction from human speech is needed. This paper introduces a speech emotion recognition (SER) system that leverages supervised learning techniques to tackle cross-accent diversity. Distinctively, the system extracts a comprehensive set of nine speech features—Zero Crossing Rate, Mel Spectrum, Pitch, Root Mean Square values, Mel Frequency Cepstral Coefficients, chroma-stft, and three spectral features (Centroid, Contrast, and Roll-off) for refined speech signal processing and recognition. Seven machine learning models are employed, encompassing Random Forest, Logistic Regression, Decision Tree, Support Vector Machines, Gaussian Naive Bayes, K-Nearest Neighbors, ensemble learning, and four individual, hybrid deep learning models including Long short-term memory (LSTM) and 1-Dimensional Convolutional Neural Network (1D-CNN) with stratified cross-validation. Audio samples from diverse English regions are combined to train the models. The performance evaluation results of conventional machine learning and deep learning models indicate that the Random Forest-based feature selection model achieves the highest accuracy of up to 76% among the conventional machine learning models. Simultaneously, the 1D-CNN model with stratified cross-validation reaches up to 99% accuracy. The proposed framework enhances the cross-accent emotion recognition accuracy up to 86.3%, 89.87%, 90.27%, and 84.96% by margins of 14.71%, 10.15%, 9.6%, and 16.52% respectively.

A novel concatenated 1D-CNN model for speech emotion recognition

MFF-SAug: Multi feature fusion with spectrogram augmentation of speech emotion recognition using convolution neural network

Artificial intelligence (AI) and machine learning (ML) are employed to make systems smarter. Today, the speech emotion recognition (SER) system evaluates the emotional state of the speaker by investigating his/her speech signal. Emotion recognition is a challenging task for a machine. In addition, making it smarter so that the emotions are efficiently recognized by AI is equally challenging. The speech signal is quite hard to examine using signal processing methods because it consists of different frequencies and features that vary according to emotions, such as anger, fear, sadness, happiness, boredom, disgust, and surprise. Even though different algorithms are being developed for the SER, the success rates are very low according to the languages, the emotions, and the databases. In this paper, we propose a new lightweight effective SER model that has a low computational complexity and a high recognition accuracy. The suggested method uses the convolutional neural network (CNN) approach to learn the deep frequency features by using a plain rectangular filter with a modified pooling strategy that have more discriminative power for the SER. The proposed CNN model was trained on the extracted frequency features from the speech data and was then tested to predict the emotions. The proposed SER model was evaluated over two benchmarks, which included the interactive emotional dyadic motion capture (IEMOCAP) and the berlin emotional speech database (EMO-DB) speech datasets, and it obtained 77.01% and 92.02% recognition results. The experimental results demonstrated that the proposed CNN-based SER system can achieve a better recognition performance than the state-of-the-art SER systems.

Speech emotion recognition (SER) is an emerging technology that utilizes speech sounds to identify a speaker’s emotional state. Computational intelligence is receiving increasing attention from academics, health, and social media applications. This research was conducted to identify emotional states in verbal communication. We applied a publicly available dataset called RAVDEES. The data augmentation process involved adding noise, applying time stretching, shifting, and pitch, and extracting the features zero cross rate (ZCR), chroma shift, Mel-Frequency Cepstral Coefficients (MFCC), and a spectrogram. In addition, we used many pretrained deep learning models, such as VGG16, ResNet50, Xception, InceptionV3, and DenseNet121. Out of all of the deep learning models, Xception yielded superior outcomes. Furthermore, we improved performance by changing the Xception model to include hyperparameters and additional layers. We used a variety of performance evaluation parameters to test the proposed model. These included F1-score, accuracy, misclassification rate (MCR), precision, sensitivity, specificity, negative predictive value, false negative rate, false positive rate, false discovery rate, false omission rate, and false discovery rate. The model that we suggested demonstrated an overall accuracy of 98%, with an MCR of 2%. Additionally, it attained precision, sensitivity, and specificity values of 91.99%, 91.78%, and 98.68%, respectively. Additional models attained an F1-score of 91.83%. Our suggested model demonstrated superiority compared to other cutting-edge techniques

Children's emotions expression concentrates in the acoustic aspects such as the tones and timbres of the voice instead of the semantics, and there are a lot of lengthy fragments in their speech. This paper proposes an emotion recognition model using the time series deep learning technology, named attention based Bi-directional Long Short-Term Memory (CNN-BiLSTM) to extract the emotional features. After preprocessing the speech signal, the forty-dimensional Mel Frequency Cepstral Coefficients (MFCC) related parameters are extracted, including the dynamic and static features. And these frequency domain features are enhanced by convolutional neural networks (CNNs) as the emotional features of children's speech recognition. BiLSTM is used to solve the problem of poor performance of long-term dependent learning features, and attention mechanism is used for only a few frames contain emotional features in the children speech signal. Compared with the related speech emotion recognition models such as LSTM-CNN and 2D-CNN-LSTM, our proposed speech emotion recognition model improves the accuracy up to 71.6% on the FAU-AIBO children's speech emotion database.

Emotions are ubiquitous to human life. Previous studies suggest that ninety percent of the life experience of humankind is affected by at least one emotion, and that emotional state guides their thoughts and behavior. The neurological study suggests that the design of human brain ensures that the emotional brain almost always influences the rational brain. Which means every rational decision is affected by the underlying emotional state. For decades, the influence of emotional states on human life has been overlooked by the AI researchers until recently. For the last twenty years, the study on different aspects of emotions and its detection for the more significant cause is gaining considerable attention and importance. With the advent of the Fourth Industrial Revolution (4IR), advancements in Artificial Intelligence (AI) is already touching new heights and in this context successful detection of emotional states by AI systems will be another big leap towards more realistic social AI agents. This chapter will discuss in detail the importance and recent technological advancements towards emotion recognition (ER) in the backdrop of the Fourth Industrial Revolution. ER uses different types of human physical and behavioral expressions, such as facial features, speech features, and their variants. Deep Learning (DL) has emerged as one of the prominent pillars of 4IR. ER task extensively uses DL techniques for classifying different emotions. So, this chapter includes a discussion of relevant technical aspects of DL. Finally, it includes a brief description of a newly proposed Speech Emotion Recognition (SER) model as a case study of the ER systems.

Recognizing speech emotions is an important subject in pattern recognition. This work is about studying the effect of extracting the minimum possible number of features on the speech emotion recognition (SER) system. In this paper, three experiments performed to reach the best way that gives good accuracy. The first one extracting only three features: zero crossing rate (ZCR), mean, and standard deviation (SD) from emotional speech samples, the second one extracting only the first 12 Mel frequency cepstral coefficient (MFCC) features, and the last experiment applying feature fusion between the mentioned features. In all experiments, the features are classified using five types of classification techniques, which are the Random Forest (RF), k-Nearest Neighbor (k-NN), Sequential Minimal Optimization (SMO), Naive Bayes (NB), and Decision Tree (DT). The performance of the system validated over Surrey Audio-Visual Expressed Emotion (SAVEE) dataset for seven emotions. The results of the experiments showed given good accuracy compared with the previous studies using a fusion of a few numbers of features with the RF classifier.

Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition

Speech emotion recognition (SER) poses significant challenges within human-computer interaction, primarily because of complex emotional expressions and signal variability. This paper introduces Bi-CALM for RAVDESS, a novel architecture that integrates convolutional neural networks for feature extraction, bidirectional LSTMs for temporal context processing, and an attention mechanism to highlight emotion-salient features. Our approach addresses key limitations in existing SER systems by optimizing the network structure to balance performance and generalization while mitigating overfitting. Experimental validation on the RAVDESS dataset demonstrates the effectiveness of our proposed model, achieving 91.5% test accuracy—significantly outperforming conventional approaches. The Bi-CALM architecture provides a comprehensive framework designed to identify emotional states from Speech, presenting promising use-cases in areas such as psychological health assessment, customer service, and affective computing applications.

Human emotions are playing a significant role in the understanding of human behaviour. There are multiple ways of recognizing human emotions, and one of them is through human speech. This paper aims to present an approach for designing a Speech Emotion Recognition (SER) system for an industrial training station. While assembling a product, the end user emotions can be monitored and used as a parameter for adapting the training station. The proposed method is using a phase vocoder for time-stretching and an Artificial Neural Network (ANN) for classification of five typical different emotions. As input for the ANN classifier, features like Mel Frequency Cepstral Coefficients (MFCCs), short-term energy, zero-crossing rate, pitch and the speech rate were extracted. The proposed method was evaluated on the Ryerson Audio-Visual Database of Emotion Speech and Song (RAVDESS) and shows promising results when compared to other methods such as zero-padding.

A new recognition method based on Gaussian mixture model for speech emotion recognition is proposed in this paper. To improve the effectiveness of feature extraction and accuracy of emotion recognition, extraction of Mel frequency cepstrum coefficient combined with Gaussian mixture model is used to recognize speech emotion. According to feature parameters extraction method by analyzing the principle of vocalization theory, emotion models based on Gaussian mixture model are generated and the similarity of their templates is obtained. A series of experiments is performed with recorded speech based on Gaussian mixture model and indicates the system gains high performance and better robustness.

In this digitally growing era speech emotion recognition plays significant role in several applications such as Human Computer Interface (HCI), lie detection, automotive system to assist steering, intelligent tutoring system, audio mining, security, Telecommunication, Interaction between a human and machine at home, hospitals, shops etc. Speech is a unique human characteristic used as a tool to communicate and express one’s perspective to others. Speech emotion recognition is extracting the emotions of the speaker from his or her speech signal. Feature extraction, Feature selection and classifier are three main stages of the emotion recognition. The main aim of this work is to improve the speech emotion recognition rate of a system using the different feature extraction algorithms. The work emphasizes on the preprocessing of the received audio samples where the noise from speech samples is removed using filters. In next step, the Mel Frequency Cepstral Coefficients (MFCC), Discrete Wavelet Transform (DWT), pitch, energy and Zero crossing rate (ZCR) algorithms are used for extracting the features. In feature selection stage Global feature algorithm is used to remove redundant information from features and to identify the emotions from extracted features machine learning classification algorithms are used. These feature extraction algorithms are validated for universal emotions comprising Anger, Happiness, Sad and Neutral.