Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

In this paper, we present a hybrid speech emotion recognition system exploiting both spectral and prosodic features in speech. For capturing the emotional information in the spectral domain, we propose a new spectral feature extraction method by applying a novel non-uniform subband processing, instead of the mel-frequency subbands used in Mel-Frequency Cepstral Coefficients (MFCC). For prosodic features, a set of features that are closely correlated with speech emotional states are selected. In the proposed hybrid emotion recognition system, due to the inherently different characteristics of these two kinds of features (e.g., data size), the newly extracted spectral features are modeled by Gaussian Mixture Model (GMM) and the selected prosodic features are modeled by Support Vector Machine (SVM). The final result of the proposed emotion recognition system is obtained by combining the results from these two subsystems. Experimental results show that (1) the proposed non-uniform spectral features are more effective than the traditional MFCC features for emotion recognition; (2) the proposed hybrid emotion recognition system using both spectral and prosodic features yields the relative recognition error reduction rate of 17.0% over the traditional recognition systems using only the spectral features, and 62.3% over those using only the prosodic features.

-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications

The acoustic cues that convey emotion in speech are similar to those that convey emotion in music, and recognition of emotion in both of these types of cue recruits overlapping networks in the brain. Given the similarities between music and speech prosody, developmental research is uniquely positioned to determine whether recognition of these cues develops in parallel. In the present study, we asked 60 children aged 6 to 11 years, and 51 university students, to judge the emotions of 10 musical excerpts, 10 inflected speech clips, and 10 affect burst clips. We presented stimuli intended to convey happiness, sadness, anger, fear, and pride. Each emotion was presented twice per type of stimulus. We found that recognition of emotions in music and speech developed in parallel, and adult-levels of recognition develop later for these stimuli than for affect bursts. We also found that sad stimuli were most easily recognised, followed by happiness, fear, and then anger. In addition, we found that recognition of emotion in speech and affect bursts can predict emotion recognition in music stimuli independently of age and musical training. Finally, although proud speech and affect bursts were not well recognised, children aged eight years and older showed adult-like responses in recognition of proud music.

The paper addresses the research question of automatic emotional speech recognition for Serbian. It integrates two research issues: (i) selection of an appropriate feature set, and (ii) investigation of different classification techniques. The paper reports a set of experiments with three feature sets: (i) the prosodic feature set, (ii) the spectral feature set, and (iii) the set of both spectral and prosodic features. The linear Bayes, the perceptron rule and the kNN classifier were considered in all three experiments. The experimental results show that the highest recognition accuracy of 91.5 % was obtained with the third feature set using the linear Bayes classifier.DOI: http://dx.doi.org/10.5755/j01.eee.18.9.2806

Speech emotion recognition (SER) is one of the latest challenge in human-computer interaction. In conventional SER classification methods, a single emotion label is outputted per one utterance as the estimation result. This is because conventional speech emotional databases which are used to train SER models have a single emotion label for one utterance. However, it is often the case that multiple emotions are expressed simultaneously with different intensities in human speech. In order to realize more natural SER than ever, existence of multiple emotions in one utterance should be taken into account. Therefore, we created an emotional speech database which contains multiple emotions and their intensities labels. The creation experiment was conducted by extracting speech utterance parts where emotions appear from existing video works. In addition, we evaluated the created database by performing statistical analysis on the database. As a result, 2,025 samples were obtained, of which 1,525 samples contained multiple emotions.

Speech emotion recognition (SER) has become one of the significant approaches to improve human-robot interaction. In this paper, two methods are proposed which take into consideration the size of the databases along with other aspects of the models. The first model applied K nearest neighbors (KNN) algorithms with 1-30 Gammatone frequency cepstral coefficients (GTCCs) which is mainly proposed for relatively small databases. It achieved 95.3% overall recognition accuracy on Berlin Emotional Speech database (EMODB). The second model is mainly focused on relatively large databases, which adopted 1-30 GTCCs, delta 1-30 GTCCs, delta-delta 1- 30 GTCCs, spectral features and prosodic features as the feature set and used long short-term memory (LSTM) as the classifier. An overall accuracy of 87.5% is achieved with this model when applied to Chinese emotional speech database (CASIA).

There has been a lot of progress in recent years in the fields of expert systems, artificial intelligence (AI) and human machine interface (HMI). The use of voice commands to engage with machinery or instruct it to do a certain task is becoming more common. Numerous consumer electronics have SIRI, Alexa, Cortana, and Google Assistant built in. In the field of human-device interaction, emotion recognition from speech is a complex research subject. We can't imagine modern life without machines, so naturally there's a need to create a more robust framework for human-machine communication. A number of academics are now working on speech emotion recognition (SER) in an effort to improve the interaction between humans and machines. We aimed to identify four fundamental emotions: angry, unhappy, neutral and joyful from speech in our experiment. As you can hear below, we trained and tested our model using audio data of brief Manipuri speeches taken from films. This task makes use of convolutional neural networks (CNNs) to extract functions from speech in order to recognize different moods using the Mel-frequency cepstral coefficient (MFCC).

In the recent years, many research works have been published using speech related features for speech emotion recognition, however, recent studies show that there is a strong correlation between emotional states and glottal features. In this work, Mel-frequency cepstralcoefficients (MFCCs), linear predictive cepstral coefficients (LPCCs), perceptual linear predictive (PLP) features, gammatone filter outputs, timbral texture features, stationary wavelet transform based timbral texture features and relative wavelet packet energy and entropy features were extracted from the emotional speech (ES) signals and its glottal waveforms(GW). Particle swarm optimization based clustering (PSOC) and wrapper based particle swarm optimization (WPSO) were proposed to enhance the discerning ability of the features and to select the discriminating features respectively. Three different emotional speech databases were utilized to gauge the proposed method. Extreme learning machine (ELM) was employed to classify the different types of emotions. Different experiments were conducted and the results show that the proposed method significantly improves the speech emotion recognition performance compared to previous works published in the literature.

Emotional expressions are a fundamental aspect of human communication, with speech being one of the most natural modes of interaction. Speech Emotion Recognition (SER) is a significant research topic in Natural Language Processing (NLP), aimed at identifying emotions such as satisfaction, frustration, and anger from speech audio using multiple classifiers. This article presents a method to emotion recognition from spontaneous Tunisian Dialect (TD) speech, marking the first work in the SER field to utilize spontaneous speech for emotion recognition in this dialect. The dataset was created from freely available YouTube videos across multiple domains and labeled with four perceived emotions: anger, satisfaction, frustration, and neutral. To address the data scarcity issue, we implemented data augmentation techniques, specifically Vocal Tract Length Perturbation (VTLP). The preprocessing of the speech signals involved cleaning the data from ambient and unwanted noises. We extracted and selected various spectral features, including Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCC). Subsequently, we applied several classification methods: Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Random Forest. Our experiments demonstrated that the Random Forest classifier achieved the highest F-score of 58.75%. The results were thoroughly discussed, analyzed, and compared across the five models using different feature extractions. This study provides valuable insights and advancements in the SER field, particularly for the TD, for future research directions for improving emotion recognition systems.

Recognition of emotion in speech usually uses acoustic models that ignore the spoken content. Likewise one general model per emotion is trained independent of the phonetic structure. Given sufficient data, this approach seemingly works well enough. Yet, this paper tries to answer the question whether acoustic emotion recognition strongly depends on phonetic content, and if models tailored for the spoken unit can lead to higher accuracies. We therefore investigate phoneme-, and word-models by use of a large prosodic, spectral, and voice quality feature space and Support Vector Machines (SVM). Experiments also take the necessity of ASR into account to select appropriate unit- models. Test-runs on the well-known EMO-DB database facing speaker-independence demonstrate superiority of word emotion models over today's common general models provided sufficient occurrences in the training corpus.

The objective of speech emotion recognition (SER) is to enhance man-machine interface. It can also be used to cover the physiological state of a person in critical situations. In recent time, speech emotion recognition also finds its operations in medicine and forensics. A new feature extraction technique using Teager energy operator (TEO) is proposed for the detection of stressed emotions as Teager energy-autocorrelation envelope (TEO-Auto-Env). TEO is basically designed for increasing the energies of the stressed speech signals whose energies are reduced during the speech production process and hence used in this analysis. A stressed speech emotion recognition (SSER) system is developed using TEO-Auto-Env and spectral feature combination for detecting the emotions. The spectral features considered are Mel-frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), and relative spectra-perceptual linear prediction (RASTA-PLP). EMO-DB (German), EMOVO (Italian), IITKGP (Telugu), and EMA (English) databases are used in this analysis. The classification of the emotions is carried out using the k-nearest neighborhood (k-NN) classifier for gender-dependent (GD) and speaker-independent (SI) cases. The proposed SSER system provides improved accuracy compared to the existing ones. Average recall is used for performance evaluation. The highest classification accuracy is achieved using the feature combination of TEO-Auto-Env, MFCC, and LPCC features with 91.4% (SI), 91.4% (GD-male), and 93.1%(GD-female) for EMO-DB; 68.5% (SI), 68.5% (GD-male), and 74.6% (GD-female) for EMOVO; 90.6%(SI), 91% (GD-male), and 92.3% (GD-female) for EMA; and 95.1% (GD-female) for IITKGP female database.

Hearing disabilities affect people all over the world. Nearly 360 million people have a disabling hearing loss. In the recent years, there has been a rapid increase in the number of deaf and dumb victims due to birth defects, accidents and oral diseases. Since deaf and dumb people cannot communicate with a hearing person, they have to depend on some sort of visual communication. Sign Language is not understood by the majority of hearing people. The objective of our project is to form a bridge between the hearing-impaired community and the hearing community; to further commence a two-way communication. We propose a two-way sign language translator that converts Voice to sign language and vice versa in real time. We process the video frame by frame using the OpenCV Library in Python. Furthermore, we use background subtraction method called Gaussian Mixture-based Background/Foreground Segmentation Algorithm on each frame to subtract the background. Contours of this processed image are then passed into a Deep Neural Net (DNN) to classify the frame to the corresponding written language equivalent words. For conversion from sign language to voice, we use basic Natural Language Processing and gtts to accurately conserve the grammar in the sign language. The enrolment rate and literacy among the hearing-impaired children is far below the average for the population at large. This technology will then help the normal schools better integrate the hearing-impaired community thus making education more accessible and cheaper for them. Speech Emotion Recognition (SER) is a vital aspect of human-computer interaction and emotional intelligence applications. This SER that employs Mel- frequency cepstral coefficients (MFCC) as the basis for feature extraction, followed by a Convolutional Neural Network (CNN) with a customized architecture for emotion classification. The methodology 2 involves transforming audio samples into MFCC images, enabling the application of CNNs typically used in computer vision tasks. This fusion of traditional audio analysis with deep learning techniques harnesses the power of MFCC's spectral representation and CNN's spatial pattern recognition. By modifying the CNN architecture, the model can better discern emotional cues in the MFCC images, enhancing SER performance. This strategy offers SER a viable path forward, with the ability to identify complex emotional expressions. In order to increase the accuracy of emotion recognition, it tackles the requirement for feature-rich representations and sophisticated modelling techniques. KEYWORDS: OpenCV, DNN, SER, CNN, MFCC

<title>Abstract</title> In the field of speech emotion recognition, the choice of audio features can dramatically influence the accuracy and effectiveness of classification systems. This study presents a comprehensive comparative analysis of feature significance, shedding light on how different audio characteristics contribute to the success of emotion recognition methodologies. This paper analyzes speech-based emotion recognition techniques and works with audio analyses using the Ryerson Audio-Visual Database (RAVD) of Emotional Speech and Song, a database consisting of audio analysis on raw audio files. Analysis involved features like Log-Mel Spectrograms (LMS), Mel-Frequency Cepstral Coefficients (MFCCs), pitch, and energy, after raw audio files are pre-processed. We measured the relevance of these features to the classification of emotion through a series of approaches that include Long Short-Term Memory networks, Convolutional Neural Networks, Hidden Markov Models, and Deep Neural Networks. On a 14-class classification problem that covers two genders and seven emotions, we obtained 56% accuracy by using a 4-layer 2-dimensional CNN with Log-Mel Spectrogram features. Our results show the importance of selection of good audio features while complexity is not that important for performance for emotion recognition.

Speech Emotion Recognition (SER) has emerged as a significant research area within Human–Computer Interaction (HCI), enabling intelligent systems to interpret human emotional states from spoken audio. Accurate emotion recognition from speech plays a crucial role in enhancing natural interaction between humans and machines. This paper presents a deep learning–based SER framework that combines Mel-Frequency Cepstral Coefficients (MFCC) for feature extraction with Long Short-Term Memory (LSTM) networks for temporal modelling and emotion classification. MFCC features effectively capture the spectral characteristics of speech signals, whereas LSTM networks are well-suited to modelling long-term temporal dependencies inherent in emotional speech patterns. The proposed model is trained and evaluated on the Toronto Emotional Speech Set (TESS) dataset, which covers multiple emotional categories, including happiness, sadness, anger, fear, and neutrality. Experimental results demonstrate that the proposed MFCC–LSTM approach achieves promising classification accuracy, indicating its effectiveness in recognising emotional states from speech signals. The findings highlight the potential applicability of the proposed system in real-world scenarios, including virtual assistants, call centre analytics, and mental health monitoring systems, thereby contributing to the development of emotion-aware intelligent interfaces.

Speech is a powerful means of expressing thoughts, emotions, and perspectives. However, accurately determining the emotions conveyed through speech remains a challenging task. Existing manual methods for analyzing speech to recognize emotions are prone to errors, limiting our understanding and response to individuals’ emotional states. To address diverse accents, an automated system capable of real-time emotion prediction from human speech is needed. This paper introduces a speech emotion recognition (SER) system that leverages supervised learning techniques to tackle cross-accent diversity. Distinctively, the system extracts a comprehensive set of nine speech features—Zero Crossing Rate, Mel Spectrum, Pitch, Root Mean Square values, Mel Frequency Cepstral Coefficients, chroma-stft, and three spectral features (Centroid, Contrast, and Roll-off) for refined speech signal processing and recognition. Seven machine learning models are employed, encompassing Random Forest, Logistic Regression, Decision Tree, Support Vector Machines, Gaussian Naive Bayes, K-Nearest Neighbors, ensemble learning, and four individual, hybrid deep learning models including Long short-term memory (LSTM) and 1-Dimensional Convolutional Neural Network (1D-CNN) with stratified cross-validation. Audio samples from diverse English regions are combined to train the models. The performance evaluation results of conventional machine learning and deep learning models indicate that the Random Forest-based feature selection model achieves the highest accuracy of up to 76% among the conventional machine learning models. Simultaneously, the 1D-CNN model with stratified cross-validation reaches up to 99% accuracy. The proposed framework enhances the cross-accent emotion recognition accuracy up to 86.3%, 89.87%, 90.27%, and 84.96% by margins of 14.71%, 10.15%, 9.6%, and 16.52% respectively.