Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features.

An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition

Unsupervised feature selection and NMF de-noising for robust Speech Emotion Recognition

There is an imminent need for guidelines and standard test sets to allow direct and fair comparisons of speech emotion recognition (SER). While resources, such as the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database, have emerged as widely-adopted reference corpora for researchers to develop and test models for SER, published work reveals a wide range of assumptions and variety in its use that challenge reproducibility and generalization. Based on a critical review of the latest advances in SER using IEMOCAP as the use case, our work aims at two contributions: First, using an analysis of the recent literature, including assumptions made and metrics used therein, we provide a set of SER evaluation guidelines. Second, using recent publications with open-sourced implementations, we focus on reproducibility assessment in SER.

Speech emotion recognition (SER) is a rapidly evolving field in affective computing and human-computer interaction. In general, a SER system extracts and classifies prominent elements called features from a pre-processed speech signal to target the presence of speaker's certain emotion. This paper explores the utilization of deep learning classifiers in SER and surveys available datasets in both Indic and international languages. The paper highlights the significance of SER in enhancing human-computer interaction and presents deep learning as an effective approach to handle the complexity of speech signals. Various deep learning architectures, including Convolution Neural Networks (CNNs), Recurrent Neural Network (RNNs), and hybrid models, are analysed in terms of training methodology, and performance on benchmark datasets. Additionally, the paper conducts a comprehensive survey of publicly available datasets for speech emotion recognition, considering emotional categories, language diversity, recording conditions, and sample sizes. Challenges in adapting deep learning models to these datasets, such as data augmentation and cross-lingual transfer learning, are discussed. Moreover, the CNN based model is analysed on accuracy, precision, recall and F-1 score on Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) dataset with the value 84%, 85%, 84% and 84% resp. The review concludes with key findings, emphasizing the strengths and limitations of deep learning classifiers for SER. It identifies the need for standardized evaluation protocols, exploration of transfer learning across languages, and development of robust and culturally diverse datasets as future research directions.

Emotion recognition from speech plays a crucial role in enhancing human–computer interaction by enabling systems to interpret and respond to users’ emotional states. This study develops and evaluates a Speech Emotion Recognition (SER) system using three machine learning techniques; Support Vector Machines (SVM), Multilayer Perceptron (MLP), and Convolutional Neural Networks (CNNs). The system is trained and tested on the RAVDESS dataset, which contains 1,440 professionally recorded audio samples representing a wide range of emotions. Our approach involves careful preprocessing of the audio signals, extraction of key acoustic features, and comparative performance evaluation of the three models using standard metrics. Results show that each model exhibits unique strengths and limitations, with CNNs achieving the most robust feature learning and generalization. The study underscores the importance of diverse feature representation for accurate emotion classification and provides insight into how different model architectures handle emotional nuances in speech. Identified challenges such as dataset diversity, feature selection, and computational complexity are discussed, along with recommendations for future research to improve SER systems’ real-world adaptability. This work contributes to ongoing efforts toward developing emotionally aware technologies that can enhance natural human–machine communication.

It has been a challenging task to identify meaningful clues in speech/voice data to recognize emotion. Speech Emotion Recognition (SER) system is a collection of methodologies that process the speech data, extract useful features, use efficient classifier/classifiers to finally detect the embedded emotion from the six basic emotions, i.e., happiness, surprise, anger, sorrow, disgust and fear available. Increasing applications of SER are found in health care system, security, entertainment, education, BPOs, marketing, defense, customer care and Human Computer Interface (HCI) systems. This paper focuses on building a web-based interface for recognizing emotion from an audio file uploaded by user. The web app for SER is built on the flask framework following the supervised learning approach using Python. The system has been tested on publicly available TESS dataset with both Machine Learning (ML) and Deep Learning (DL) based models. Performance comparison of all the trained models is done with state of the art. Results are quite promising pointing towards developing a robust SER system.

Identification of emotions along with the semantics from the speech signals is still one of the most challenging tasks because of the arising difficulty in the extraction and selection of the appropriate emotion features. The speech emotion recognition (SER) system has become one of the active research topics in recent years. Many researchers proposed various models to enhance the machine responses during the man–machine interactions by identifying the emotional state of the customer. This research work aims to propose a multi-class classification model to extract and classify the emotional state of the speaker into eight classes, namely happy, sad, surprised, disgust, neutral, calm, angry, and fearful. In this research work, two machine learning classifiers, namely k-nearest neighbors and random forest, are used along with the multi-layer perceptron classifier of deep learning for determining the emotional features from speech signals. The dataset for this research is obtained from the RAVDESS. Along with the audio speeches of 24 actors, the audio song dataset is also utilized for the better training of the proposed model. The performance of all three algorithms is compared and evaluated by using the confusion matrix and accuracy plots. The empirical results show that the KNN classification model of ML has performed better than other proposed algorithms with increased accuracy of 76.25%.

Speech Emotion Recognition (SER) allows systems to interpret emotions in human speech, creating more natural and responsive interactions between people and machines. Due to the complex nature of emotion detection, several deep learning techniques have been utilized, yet limited research have focused on optimizing key hyperparameters of Convolutional Neural Network (CNN) for a more efficient system. Hence, this research optimized CNN with Mantis Search Algorithm (MSA) due to its ease of implementation, ability to preserve population diversity during the optimization process, ability to escape from the local optima and balance between exploration and exploitation operators. Audio data for four emotions: anger, fear, happiness and neutrality were acquired from Toronto Emotional Speech Set (TESS) available on Kaggle.com. The audio data were then converted into text using speech-to-text code and preprocessed using Natural Language Processing (NLP) techniques: tokenization, removal of stop words, lemmatization, removal of punctuations and lowercase conversion. Mantis Search Algorithm was then applied to optimize CNN for optimal selection of filter size and learning rate. The optimized CNN (MSA-CNN) was implemented using MATLAB R2023a software. The performance of the system was evaluated and compared with CNN classifier using False Positive Rate (FPR), Specificity (Spec), Sensitivity (Sen), Precision (Prec), Accuracy (Acc), and Recognition Time (RT). The optimized speech emotion recognition system showed improved values over CNN on all the metrics considered.

The Speech Emotion Recognition (SER) system is one of the essential human-computer interface applications. Despite the rapid advancement of technology, there is still a gap in SER research in the Arabic language corpus. The goal of this research is to build an Arabic-based SER based on a feature set that has both high performance and low computational cost. Two novel feature sets were implemented using a mix of spectral and prosodic features. An Arabic semi-natural corpus “EYASE” was adopted for testing the proposed system. Five machine learning classifiers using the different feature sets were implemented. Featureset-2 showed promising results in association with the SVM classifier. To validate the results, a survey analysis was implemented on the Arabic-Egyptian corpus “EYASE” used.

Emotions are an integral part of social interactions, serving vital functions in the management of our interpersonal relationships. Cues to emotional meaning can be conveyed through a variety of modalities, from facial expressions and body gestures to the linguistic and non-linguistic, or prosodic aspects of speech. As automatic speech systems continue to grow in use, the ability to recognize emotional cues from these modalities is no longer solely expected in human communication, but also in human-computer interactions with the goal of developing more user-friendly and effective applications. This thesis explores how speech emotion recognition (SER) systems can be more robust to factors of language, culture, and context through the lens of emotional speech and emotional prosody from bilingual English-speaking Singaporeans of Chinese, Malay, or Indian ethnicity. Specifically, the thesis aims to examine how Singaporeans from the three ethnic groups may vary in their perception of emotions in speech, and to explore how they express emotions via speech prosody. The recognition and production processes of emotional communication in Singapore English was investigated through two emotion classes – anger and happiness. An emotion perception test was conducted using acted angry and happy speech as stimuli to understand how Singapore English speakers recognized emotions in speech. The results of the perception test revealed that respondents displayed very similar patterns in their rating of emotional speech stimuli. The study also found that respondents tended to rated anger with low dominance. At the same time, a spontaneous speech corpus known as the Singapore English Spontaneous Emotional Speech Corpus (SESESC) was also developed, featuring audio recordings of 44 Singapore English speakers from the three ethnic groups playing a co-operative video game with either a familiar partner or a confederate. More than 36,000 intervals were annotated in-context of the recordings by each annotator based on a modified emotion annotation scheme that included information regarding the emotion category and arousal levels. Intervals that were labelled as conveying joy or anger with full agreement from the annotators were selected following a series of criteria to facilitate comparable prosodic analyses given the inherent dynamicity of spontaneous speech. The analysis focused on exploring the F0 movements of the selected utterances in tandem with periodic energy. Overall, it was observed that utterances conveying happiness were associated more with level contours, while angry utterances were produced with more steep falling contours especially on the utterance-final syllable. A tendency for an F0 peak to be produced on the second syllable of the utterance was also observed in both sets of data analyzed, alluding to the interactions between emotional prosody and the intonational structure of Singapore English. Finally, an emotion classification was conducted using a subset of the corpus as test data for a fine-tuned SER model. Results showed that while utterances conveying happiness could be automatically recognized to a high percentage of 86%, there was also a high percentage of misclassification of anger. This result appears to counter findings in past SER studies, which typically reported high levels of recognition accuracy for anger. Taken together, the thesis contributes an understanding of how spontaneous speech and emotional cues on the prosodic level interact even in task-oriented conversations, with further considerations of how these patterns may vary among bilinguals in multicultural contexts, so as to enable more robust speech emotion recognition systems in the future.

In this paper, we investigate cross-lingual automatic speech emotion recognition. The basic idea is that since the emotion recognition system is based on the acoustic features only, it is possible to combine data in different languages to improve the recognition accuracy. We begin with the construction of a Mandarin database of emotional speech, which is similar to the well-known Berlin Database of Emotional Speech (EMO-DB) in the composition and size. In order to reduce the variability due to different languages and different speakers, we propose to apply histogram equalization as a data normalization method. Recognition systems based on support vector machines have been evaluated on EMO-DB. Compared to the baseline system without multi-lingual databases and data normalization, the proposed system has achieved a relative improvement of 39.9% in the emotion recognition accuracy, from 86.2% to 91.7%. The accuracy is among the best known results reported on EMO-DB, if not the best.

Speech is a commonly used signal for interaction between humans, this leads to the usage of speech for human and machine interactions as well. Improvements in this interactive system reach toward speech emotion recognition (SER) system. SER gives sufficient intelligence for efficient natural communication between humans and machines. SER system classifies emotional states such as sadness, angry, neutral, and happiness from the speaker's utterances. This paper describes speech features and machine learning models that can be used for SER. For effective classification and to learn multidimensional complex data, a deep learning algorithm is used in this system. This paper also presents the preliminary results of a system with an MFCC feature and an LSTM algorithm.

Speech Emotion Recognition (SER) plays a crucial role in applications involving human-machine interaction. However, the scarcity of suitable emotional speech datasets presents a major challenge for accurate SER systems. Deep Neural Network (DNN)-based solutions currently in use require substantial labelled data for successful training. Previous studies have proposed strategies to expand the training set in this framework by leveraging available emotion speech corpora. This paper assesses the impact of a cross-corpus training extension for a SER system using self-supervised (SS) representations, namely HuBERT and WavLM. The feasibility of training systems with just a few minutes of in-domain audio is also analyzed. The experimental results demonstrate that augmenting the training set with EmoDB (German), RAVDESS, and CREMA-D (English) datasets leads to improved SER accuracy on the IEMOCAP dataset. By combining a cross-corpus training extension and SS representations, state-of-the-art performance is achieved. These findings suggest that the cross-corpus strategy effectively addresses the scarcity of labelled data and enhances the performance of SER systems.

In today's world, accurately understanding and interpreting emotions in human-computer interaction is important. In this context, this study has adopted a detailed approach to the emotion recognition problem on both speech and text data using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. First, the problem of datasets with limited number of records and unbalanced distribution across classes was addressed. For this purpose, a dataset obtained from records created as improvised in the IEMOCAP dataset was used and data augmentation methods were applied for both speech and text data. Using datasets that were balanced by applying data augmentation, single-mode emotion recognition experiments were performed with models developed for Speech Emotion Recognition (SER) and Textual Emotion Recognition (TER). Subsequently, the features obtained from these two single modalities were combined with the intermediate fusion method to provide more comprehensive emotion recognition and accuracy, and the Bimodal Emotion Recognition (BiMER) system was developed. The ResNet50-CRNN+AT model, which we obtained the highest accuracy from the three different models developed for SER, creates the speech mode of BiMER, while the Bidirectional Encoder Representations from Transformers (BERT) model used for TER creates the text mode of BiMER. In this way, BiMER was supported with data augmentation methods and the robustness and generalization ability of the model were improved, reaching 88.33% accuracy. Finally, the developed BiMER system was implemented as a real-time web application using the Flask framework, and the capacity of this application to recognize emotions interactively through the user interface was tested.

An important part of the human-computer interaction process is speech emotion recognition (SER), which has been receiving more attention in recent years. However, although a wide diversity of methods has been proposed in SER, these approaches still cannot improve the performance. A key issue in the low performance of the SER system is how to effectively extract emotion-oriented features. In this paper, we propose a novel algorithm, an autoencoder with emotion embedding, to extract deep emotion features. Unlike many previous works, instance normalization, which is a common technique in the style transfer field, is introduced into our model rather than batch normalization. Furthermore, the emotion embedding path in our method can lead the autoencoder to efficiently learn a priori knowledge from the label. It can enable the model to distinguish which features are most related to human emotion. We concatenate the latent representation learned by the autoencoder and acoustic features obtained by the openSMILE toolkit. Finally, the concatenated feature vector is utilized for emotion classification. To improve the generalization of our method, a simple data augmentation approach is applied. Two publicly available and highly popular databases, IEMOCAP and EMODB, are chosen to evaluate our method. Experimental results demonstrate that the proposed model achieves significant performance improvement compared to other speech emotion recognition systems.