A Hybrid Speech Emotion Recognition System Based on Spectral and Prosodic Features

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications

The paper addresses the research question of automatic emotional speech recognition for Serbian. It integrates two research issues: (i) selection of an appropriate feature set, and (ii) investigation of different classification techniques. The paper reports a set of experiments with three feature sets: (i) the prosodic feature set, (ii) the spectral feature set, and (iii) the set of both spectral and prosodic features. The linear Bayes, the perceptron rule and the kNN classifier were considered in all three experiments. The experimental results show that the highest recognition accuracy of 91.5 % was obtained with the third feature set using the linear Bayes classifier.DOI: http://dx.doi.org/10.5755/j01.eee.18.9.2806

In this work, spectral features extracted from sub-syllabic regions and pitch synchronous analysis are proposed for speech emotion recognition. Linear prediction cepstral coefficients, mel frequency cepstral coefficients and the features extracted from high amplitude regions of spectrum are used to represent emotion specific spectral information. These features are extracted from consonant, vowel and transition regions of each syllable to study the contribution of these regions toward recognition of emotions. Consonant, vowel and the transition regions are determined using vowel onset points. Spectral features extracted from each pitch cycle, are also used to recognize emotions present in speech. The emotions used in this study are: anger, fear, happy, neutral and sad. The emotion recognition performance using sub-syllabic speech segments are compared with the results of conventional block processing approach, where entire speech signal is processed frame by frame. The proposed emotion specific features are evaluated using simulated emotion speech corpus, IITKGP-SESC (Indian Institute of Technology, KharaGPur-Simulated Emotion Speech Corpus). The emotion recognition results obtained using IITKGP-SESC are compared with the results of Berlin emotion speech corpus. Emotion recognition systems are developed using Gaussian mixture models and auto-associative neural networks. The purpose of this study is to explore sub-syllabic regions to identify the emotions embedded in a speech signal, and if possible, to avoid processing of entire speech signal for emotion recognition without serious compromise in the performance.

Emotion recognition (ER) systems finds applications in many fields like call centres, humanoid Roberts and robotic pets, telecommunication, psychiatry, behavioral science, educational softwares, etc., In this work, the speech and facial features extracted from the video data is explored to recognize the emotions. Since both these features are compliment to each other, on combining them will result in higher performance. The features used for emotion recognition from video data are geometric and appearance based while prosodic and spectral features are employed for speech signal. Support Vector Machine (SVM) classifier is used to capture the emotion specific information. The basic aim of this work is to explore the capability of speech and facial features to provide the emotion specific information.

In this paper, spectral and prosodic features are explored for recognition of infant cry. Different types of infant cries considered in this work are wet-diaper, hunger and pain. In this work, mel-frequency cepstral coefficients (MFCC) are used to represent the spectral information, and short-time frame energies (STE) and pause duration are used for representing the prosodic information. Support Vector Machines (SVM) are used to capture the discriminative information with respect to above mentioned cries from the spectral and prosodic features. SVM models are developed seperately using spectral and prosodic features. For carrying out these studies, infant cry database collected under Telemedicine project at IIT-KGP has been used. The recognition performance of the developed SVM models using spectral and prosodic features is observed to be 61.11% and 57.41% respectively. In this work, we also examined the recognition performance by combining the spectral and prosodic information at feature and score levels. The recognition performance using feature and score level fusion is observed to be 74.07% and 80.56% respectively.

Speech is a powerful means of expressing thoughts, emotions, and perspectives. However, accurately determining the emotions conveyed through speech remains a challenging task. Existing manual methods for analyzing speech to recognize emotions are prone to errors, limiting our understanding and response to individuals’ emotional states. To address diverse accents, an automated system capable of real-time emotion prediction from human speech is needed. This paper introduces a speech emotion recognition (SER) system that leverages supervised learning techniques to tackle cross-accent diversity. Distinctively, the system extracts a comprehensive set of nine speech features—Zero Crossing Rate, Mel Spectrum, Pitch, Root Mean Square values, Mel Frequency Cepstral Coefficients, chroma-stft, and three spectral features (Centroid, Contrast, and Roll-off) for refined speech signal processing and recognition. Seven machine learning models are employed, encompassing Random Forest, Logistic Regression, Decision Tree, Support Vector Machines, Gaussian Naive Bayes, K-Nearest Neighbors, ensemble learning, and four individual, hybrid deep learning models including Long short-term memory (LSTM) and 1-Dimensional Convolutional Neural Network (1D-CNN) with stratified cross-validation. Audio samples from diverse English regions are combined to train the models. The performance evaluation results of conventional machine learning and deep learning models indicate that the Random Forest-based feature selection model achieves the highest accuracy of up to 76% among the conventional machine learning models. Simultaneously, the 1D-CNN model with stratified cross-validation reaches up to 99% accuracy. The proposed framework enhances the cross-accent emotion recognition accuracy up to 86.3%, 89.87%, 90.27%, and 84.96% by margins of 14.71%, 10.15%, 9.6%, and 16.52% respectively.

The Speech Emotion Recognition (SER) system is one of the essential human-computer interface applications. Despite the rapid advancement of technology, there is still a gap in SER research in the Arabic language corpus. The goal of this research is to build an Arabic-based SER based on a feature set that has both high performance and low computational cost. Two novel feature sets were implemented using a mix of spectral and prosodic features. An Arabic semi-natural corpus “EYASE” was adopted for testing the proposed system. Five machine learning classifiers using the different feature sets were implemented. Featureset-2 showed promising results in association with the SVM classifier. To validate the results, a survey analysis was implemented on the Arabic-Egyptian corpus “EYASE” used.

BackgroundDepression is major global public health problems among university students. Currently, the evaluation and monitoring of depression predominantly depend on subjective and self-reported methods. There is an urgent necessity to develop objective means of identifying depression. Acoustic features, which convey emotional information, have the potential to enhance the objectivity of depression assessments. This study aimed to investigate the feasibility of utilizing acoustic features for the objective and automated identification and characterization of depression among Chinese university students.MethodsA cross-sectional study was undertaken involving 103 students with depression and 103 controls matched for age, gender, and education. Participants' voices were recorded using a smartphone as they read neutral texts. Acoustic analysis and feature extraction were performed using the OpenSMILE toolkit, yielding 523 features encompassing spectral, glottal, and prosodic characteristics. These extracted acoustic features were utilized for discriminant analysis between depression and control groups. Pearson correlation analyses were conducted to evaluate the relationship between acoustic features and Patient Health Questionnaire-9 (PHQ-9) scores. Five machine learning algorithms including Linear Discriminant Analysis (LDA), Logistic Regression, Support Vector Classification, Naive Bayes, and Random Forest were used to perform the classification. For training and testing, ten-fold cross-validation was employed. Model performance was assessed using receiver operating characteristic (ROC) curve, area under the curve (AUC), precision, accuracy, recall, and F1 score. Shapley Additive exPlanations (SHAP) method was used for model interpretation.ResultsIn depression group, 32 acoustic features (25 spectral features, 5 prosodic features and 2 glottal features) showed significant alterations compared with controls. Further, 27 acoustic features (10 spectral features, 3 prosodic features, and 1 glottal features) were significantly correlated with depression severity. Among five machine learning algorithms, LDA model demonstrated the highest classification performance, with an AUC of 0.771. SHAP analysis suggested that Mel-frequency cepstral coefficients (MFCC) features contributed most to the model's classification efficacy.ConclusionsThe integration of acoustic features and LDA model demonstrates a high accuracy in distinguishing depression among Chinese university students, suggesting its potential utility in rapid and large-scale depression screening. MFCC may serve as objective and valid features for the automated identification of depression on Chinese university campuses.

The world's biggest pandemic, COVID-19, has shown its lethal impact on human life. The current diagnostics methods are reverse transcription-polymerase chain reaction (RT-PCR) and rapid diagnostic assays have several bottlenecks in terms of the nature of sample collection as it needs some laboratory experts and careful handling of the potentially infectious samples. However, one of the non-invasive ways of diagnostics is to focus on speech modality, which has been paid less attention, during the detection of COVID-19. Hence in this work, the speech features, particularly temporal and spectral features have been studied for COVID-19 detection. The temporal features used in this work are Short-Time Energy, Long-Term Log Energy Variation (LTLEV) Zero Crossing Count (ZCC) and Pitch etc. On the other hand, the spectral features used herein are Power Spectral Density, Average Power, Mel-Frequency Cepstral Coefficients, Group delay spectrum, Spectral Entropy etc. Such spectral and temporal speech features have not been analyzed in the identification of COVID-19 symptoms to the best of authors knowledge. Further, this paper has shown the impact of COVID-19 on a real time human voice, analyzed using speech processing techniques, and shown their efficacy in detecting COVID-19. These features are safe, comparatively faster, cost-effective, and require fewer complexities. Our article will motivate the scientific community to use such features for further research in the collective battle against COVID-19.

In generation of emotional speech, there are deviations in the speech production features when compared to neutral (non-emotional) speech. The objective of this study is to capture the deviations in features related to the excitation component of speech and to develop a system for automatic recognition of emotions based on these deviations. The emotions considered in this study are anger, happiness, sadness and neutral state. The study shows that there are useful features in the deviations of the excitation features, which can be exploited to develop an emotion recognition system. The excitation features used in this study are the instantaneous fundamental frequency (F_0), the strength of excitation, the energy of excitation and the ratio of the high-frequency to low-frequency band energy (beta ). A hierarchical binary decision tree approach is used to develop an emotion recognition system with neutral speech as reference. The recognition experiments showed that the excitation features are comparable or better than the existing prosody features and spectral features, such as mel-frequency cepstral coefficients, perceptual linear predictive coefficients and modulation spectral features.

Speaker recognition systems based on spectral features perform well in acoustically matched and noise-free conditions. Spectral features are unsuccessful to model information about the speaker at higher levels. Prosody which represents intonation, rhythm and stress of speech, better represents speaker characteristics at higher levels. Voice disguise is a common threat to automatic speaker identification systems. It is the action of transforming one's voice purposefully to imitate or just to conceal one's identity. This work attempts to study the effects of voice disguise on prosody based speaker verification system. For this a prosody based automatic speaker verification system is implemented using support vector machine (SVM). For the extraction of prosodic features, speech is segmented into phrase-like regions by detecting long pauses regions and further syllable-like segmentation is done at the valleys of short time energy contour. In order to segment fused syllables, vowel onset point (VOP) detection is performed in such regions. Then prosodic features are extracted and used for building SVM models. To study the effect of voice disguise, normal and masked speech were recorded simultaneously for a set of speakers in the lab environment. Prosodic features for both normal and masked speech were analyzed. Using normal speech, automatic speaker verification system is implemented and testing was done using both normal and masked speech. Through this study, it is identified that prosodic features derived from pitch and duration are almost robust against voice masking but energy features are slightly affected.

In the past decades, the area of plastic-mulched cultivation has grown rapidly both on the global and regional scope, due to its remarkable efficiency of increasing crop production. However, the rapid expansion has brought a series of ecological and environmental problems in many regions, such as the vast amount of residues from the plastic-mulched farmland bring about aesthetic pollution and adverse environmental impacts on water, air and soil of the agricultural ecosystem and the terrestrial ecosystem. It is a serious problem that will do great damage to the sustainability of agriculture. In order to alleviate problems and to find solutions, the development of efficient methods to map plastic-mulched farmland as accurately as possible is of great significance to the policy-makers and scientists. In this paper, the optical remotely sensed data, Landsat-8 OLI, have been used in monitoring the plastic-mulched farmland in a study area located in Jizhou city, Hebei Province, China, where the plastic mulching is extensively practiced in farming and interlaced with uncovered farmland. The spectral and texture features of different ground objects have been statistically analyzed on the OLI imagery in the light of the training samples. Then the spectral and texture features that carry a certain degree of explanation for plastic-mulched farmland were selected for classification. The training and testing samples (bigger than 4 pixels × 4 pixels of panchromatic fused OLI imagery) were collected from Google Earth images, and were amended by comparing to false color composite Landsat bands 7 (SWIR), 5 (NIR) and 4 (RED). At the same time, the samples were purified according the J-M distance method. The Support Vector Machines (SVM) has been used as classifier to extract the plastic-mulched farmland based on the spectral features alone, on the texture features alone and on the combined spectral and texture features respectively. The classification result was validated using confusion matrix. The overall classification accuracy for plastic-mulched farmland based on spectra features alone, on texture features alone and on combined spectral and texture features are 89.45%, 94.68% and 94.73%, respectively. Their respective Kappa coefficients are 0.85, 0.92 and 0.92. And the producer's accuracy and user's accuracy of plastic-mulched farmland are 87.30% and 63.96% on spectral features, 83.03%, and 81.37% on texture features, 83.39% and 81.37% on the combined features, respectively. The commission and omission errors for plastic-mulched farmland are 36.04% and 12.70% on spectral features, 18.63% and 16.97% on texture features, 18.63% and 16.61% on the combined features. This study shows that the plastic-mulched farmland can be extracted effectively using SVM classifier based on the Landsat-8 imagery.

Emotional expressions are a fundamental aspect of human communication, with speech being one of the most natural modes of interaction. Speech Emotion Recognition (SER) is a significant research topic in Natural Language Processing (NLP), aimed at identifying emotions such as satisfaction, frustration, and anger from speech audio using multiple classifiers. This article presents a method to emotion recognition from spontaneous Tunisian Dialect (TD) speech, marking the first work in the SER field to utilize spontaneous speech for emotion recognition in this dialect. The dataset was created from freely available YouTube videos across multiple domains and labeled with four perceived emotions: anger, satisfaction, frustration, and neutral. To address the data scarcity issue, we implemented data augmentation techniques, specifically Vocal Tract Length Perturbation (VTLP). The preprocessing of the speech signals involved cleaning the data from ambient and unwanted noises. We extracted and selected various spectral features, including Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCC). Subsequently, we applied several classification methods: Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Random Forest. Our experiments demonstrated that the Random Forest classifier achieved the highest F-score of 58.75%. The results were thoroughly discussed, analyzed, and compared across the five models using different feature extractions. This study provides valuable insights and advancements in the SER field, particularly for the TD, for future research directions for improving emotion recognition systems.

In this paper, we study the performance of different prosody and spectral features of speech on an emotion detection task. In particular, a feature selection algorithm has been used to assess the relevancy of the different features. Gaussian mixtures models have been used to model the features extracted at the frame-level, while support vector machines (SVM) and k-nearest neighbor (k-NN) methods have been used to model the features extracted at the utterance level. We use a normalization approach (T-norm) to combine the scores from the different models. The results using the above approach are reported for the Berlin emotional database corpus and the task consisted of classifying the six emotions namely - anger, happiness, neutral, sadness, boredom and anxiety. We show that the use of feature selection algorithm improves the result, while in addition the fusion of GMM and SVM results in an overall accuracy of 75.4% for the above task.