- Research Article
108
- 10.1016/j.apacoust.2023.109492
Emotional speech Recognition using CNN and Deep learning techniques
- Jun 28, 2023
- Applied Acoustics
- C Hema + 1 more +1
Emotional speech Recognition using CNN and Deep learning techniques
In several
 application, emotion  recognition from
 the speech signal has been research topic since many years. To determine the
 emotions from the speech signal, many systems have been developed. To solve the
 speaker emotion recognition problem, hybrid model is proposed to classify five
 speech emotions, including  anger,
 sadness, fear, happiness and neutral. The aim this study of was to actualize
 automatic voice and speech emotion recognition system using hybrid model taking
 Turkish sound forms and properties into consideration.  Approximately 3000 Turkish voice samples of
 words and clauses with differing lengths have been collected from 25 males
 and  25 females. In this study, an
 authentic and unique  Turkish  database has been used. Features of these
 voice samples have been obtained using Mel Frequency Cepstral Coefficients
 (MFCC) and Mel Frequency Discrete Wavelet Coefficients (MFDWC). Moreover,
 spectral features of these voice samples have been obtained  using Support Vector Machine (SVM). Feature
 vectors of the voice samples obtained have been trained with such methods as
 Gauss Mixture Model( GMM), Artifical Neural Network (ANN), Dynamic Time Warping
 (DTW), Hidden Markov Model (HMM) and hybrid model(GMM with combined SVM).  This hybrid model has been carried out by
 combining with SVM and GMM.  In first
 stage of this model, with SVM has been performed  subsets obtained vector of  spectral features. In the second  phase, a set of training and tests have been
 formed from these spectral features. In the test phase, owner of a given voice
 sample has been identified taking the trained voice samples into consideration.
 Results and performances of the algorithms employed in the study for
 classification have been also demonstrated in a comparative manner.         
Loading PDF
Emotional speech Recognition using CNN and Deep learning techniques
Emotional speech Recognition using CNN and Deep learning techniques
ANN based decision fusion for speech emotion recognition
As a hot research field, speech emotion recognition has attracted increasing attentions from both academic and business. In this paper, we proposed a method to recognize speech emotions adopting ANNs and to fuse two kinds of recognitions using different features at the decision level. Each emotional utterance is recognized by some individual recognizers firstly. Then the outputs of these recognizers were fused adopting the voting strategy. Furthermore, the dimensionality of supervectors constructed from spectral features is reduced through PCA. Experimental results demonstrated that the proposed decision fusion is effective and the dimensionality reduction is feasible. Index Terms : speech emotion recognition, ANN, decision fusion 1. Introduction Speech is a dominant tool for communication, and it is also an important and effective approach for transmitting information and human emotions. With the increasing role of speech interfaces in human-machine interaction applications, speech emotion recognition becomes more and more important recently. Speech emotion recognition is an interesting and challenging speech technology, which can be applied to broad areas, such as environment of call center [1], treatment of mental and psychological diseases [2], development of education and entertainment software [3], and so on. Speech emotion recognition deals with how to make the computer automatically recognize various emotions in speech signal by extracting and analyzing some acoustic features. A key problem of speech emotion recognition is that which kinds of speech features can be used to represent human emotions. Some researchers have investigated the relations between features and emotions. With their efforts, many speech features were found to be used for emotion recognition. Statistical features based on prosody and voice quality have been widely used in speech emotion recognition and demonstrated considerable recognition success [4, 5]. Besides statistical features, spectral or cepstral features are another effective group for describing emotional states [6, 7]. Since these features all have played significant roles in speech emotion recognition, it is necessary to explore an effective way to complementarily fuse both two kinds of features to further enhance the performance of emotion recognition.Another key issue of speech emotion recognition is how to choose an effective method to classify speech emotions. So far, many pattern classification methods have been used for speech emotion recognition [6-9], such as Support Vector Machine (SVM), k-Nearest Neighbors (k-NN), Gaussian Mixture Model (GMM), Hidden Markov Model (HMM), Artificial Neural Networks (ANN), and so on. These methods are all feasible, but their performances are different with each other seriously. The SVMs based method has been shown to be robust and performs well. But some fresh researches have indicated that ideal performance may be obtained using ANNs as well. However, it is difficult to determine which kind of ANNs is suitable for emotion recognition and it is necessary to compare its performance with the SVMs. In this paper, the ANN based decision fusion for speech emotion recognition was presented. Firstly four different ANNs were used to recognize various emotions. Then the voting scheme was adopted to fuse recognitions using two kinds of features at the decision level. Experimental results demonstrated that the proposed approach improved the performance of ANN based recognition and its accuracy was comparable with SVM based method.The remainder of this paper is organized as follows. The features used for speech emotion recognition are introduced in Section 2. The principles of PCA and ANN are briefly described in Section 3 and Section 4 respectively. The proposed decision fusion is depicted in Section 5. In Section 6, experiments and discussions of experimental results are presented. In Section 7, conclusions are drawn and future works are suggested.
Read moreA Hybrid Speech Emotion Recognition System Based on Spectral and Prosodic Features
In this paper, we present a hybrid speech emotion recognition system exploiting both spectral and prosodic features in speech. For capturing the emotional information in the spectral domain, we propose a new spectral feature extraction method by applying a novel non-uniform subband processing, instead of the mel-frequency subbands used in Mel-Frequency Cepstral Coefficients (MFCC). For prosodic features, a set of features that are closely correlated with speech emotional states are selected. In the proposed hybrid emotion recognition system, due to the inherently different characteristics of these two kinds of features (e.g., data size), the newly extracted spectral features are modeled by Gaussian Mixture Model (GMM) and the selected prosodic features are modeled by Support Vector Machine (SVM). The final result of the proposed emotion recognition system is obtained by combining the results from these two subsystems. Experimental results show that (1) the proposed non-uniform spectral features are more effective than the traditional MFCC features for emotion recognition; (2) the proposed hybrid emotion recognition system using both spectral and prosodic features yields the relative recognition error reduction rate of 17.0% over the traditional recognition systems using only the spectral features, and 62.3% over those using only the prosodic features.
Read moreSpeech emotion recognition using cepstral features extracted with novel triangular filter banks based on bark and ERB frequency scales
Speech emotion recognition using cepstral features extracted with novel triangular filter banks based on bark and ERB frequency scales
Read moreSpeech Emotion Feature Extraction Using FFT Spectrum Analysis
Speech Emotion Recognition has widely researched and applied to some appllication such as for communication with robot, E-learning system and emergency call etc.Speech emotion feature extraction is an importance key to achieve the speech emotion recognition which can be classify for personal identity. Speech emotion features are extracted into several coefficients such as Linear Predictive Coefficients (LPCs), Linear Spectral Frequency (LSF), Zero-Crossing (ZC), Mel-Frequency Cepstrum Coefficients (MFCC) [1-6] etc. There are some of research works which have been done in the speech emotion recgnition. A study of zero-crossing with peak-amplitudes in speech emotion classification is introduced in [4]. The results shown that it provides the the technique to extract the emotion feature in time-domain, which still got the problem in amplitude shifting. The emotion recognition from speech is descrpited in [5]. It used the Gaussian Mixture Model (GMM) for extractor of feature speech. The GMM is provided the good results to reduce the back ground noise, howere it still have to focus on random noise in GMM for recognition model. The speech emotion recognition using hidden markov model and support vector machine is explained in [6]. The results shown the average performance of recognition system according to the features of speech emotion still has got the error information. Thus [1-6] provides the recognition performance which still requiers more focus on speech features.
Read moreSpeech Emotion Recognition Using Feature Fusion of TEO and MFCC on Multilingual Databases
In the speech signal, emotion is considered one of the most critical elements. For the recognition of emotions, the field of speech emotion recognition came into existence. Speech Emotion Recognition (SER) is becoming an area of research interest in the last few years. A typical SER system focuses on extracting features such as pitch frequency, formant features, energy-related features, and spectral features from speech, tailing it with a classification quest to foresee different classes of emotion. The critical issue to be addressed for a successful SER system is the emotional feature extraction, which can be solved by using different feature extraction techniques. In this paper, along with Teager Energy Operator (TEO) and Mel Frequency Cepstral Coefficients (MFCC) a trailblazing feature extraction method, a fusion of MFCC and TEO as Teager-MFCC (T-MFCC) is used for the recognition of energy-based emotions. We have used three corpora of emotions in German, English, and Hindi to develop the multilingual SER system. The classification of these energy-based emotions is done by Deep Neural Network (DNN). It is found that TEO achieves a better recognition rate compared to MFCC and T-MFCC.KeywordsSpeech emotion recognitionDeep neural networkMultilingual databaseTEOMFCC
Read moreSpeech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review
Emotion recognition from speech signals plays a crucial role in Human-Machine Interaction (HMI), particularly in the development of applications such as affective computing and interactive systems. This review seeks to provide an in-depth examination of current methodologies in speech emotion recognition (SER), with a focus on databases, feature extraction techniques, and classification models. It has been done in the past using low-level descriptors (LLDs) like Mel-Frequency Cepstral Coefficients (MFCCs), linear predictive coding (LPC), and pitch-based features in methods like Support Vector Machines (SVM), Random Forests (RF), and Gaussian Mixture Models (GMM). But the development of deep learning techniques has completely changed the field. Models like convolutional neural networks (CNNs) and long short-term memory (LSTM) networks have shown that they are better at capturing the complex temporal and spectral features of speech. This paper reviews prominent speech emotion datasets, exploring their linguistic diversity, annotation processes, and emotional labels. It also analyzes the efficacy of different speech features and classifiers in handling challenges such as data imbalance, limited data availability, and cross-lingual variations. The review highlights the need for future work to address real-time processing, context-sensitive emotion detection, and the integration of multi-modal data to enhance the performance of SER systems. By consolidating recent advancements and identifying areas for further research, this paper aims to provide a clearer path for optimizing feature extraction and classification techniques in the field of emotion recognition.
Read moreSpeech Emotion Recognition using Machine Learning
-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications
Read moreSpeech Emotion Recognition System Using Gaussian Mixture Model and Improvement proposed via Boosted GMM
<p>Speech emotion recognition is an important issue which affects the human machine interaction. Automatic recognition of human emotion in speech aims at recognizing the underlying emotional state of a speaker from the speech signal. Gaussian mixture models (GMMs) and the minimum error rate classifier (i.e. Bayesian optimal classifier) are popular and effective tools for speech emotion recognition. Typically, GMMs are used to model the class-conditional distributions of acoustic features and their parameters are estimated by the expectation maximization (EM) algorithm based on a training data set. In this paper, we introduce a boosting algorithm for reliably and accurately estimating the class-conditional GMMs. The resulting algorithm is named the Boosted-GMM algorithm. Our speech emotion recognition experiments show that the emotion recognition rates are effectively and significantly boosted by the Boosted-GMM algorithm as compared to the EM-GMM algorithm.<br />During this interaction, human beings have some feelings that they want to convey to their communication partner with whom they are communicating, and then their communication partner may be the human or machine. This work dependent on the emotion recognition of the human beings from their speech signal<br />Emotion recognition from the speaker’s speech is very difficult because of the following reasons: Because of the existence of the different sentences, speakers, speaking styles, speaking rates accosting variability was introduced. The same utterance may show different emotions. Therefore it is very difficult to differentiate these portions of utterance. Another problem is that emotion expression is depending on the speaker and his or her culture and environment. As the culture and environment gets change the speaking style also gets change, which is another challenge in front of the speech emotion recognition system.</p>
Read moreSpeech Signal Imaging and Emotion Recognition Based on Symmetric-Diagonal Matrix Model
Speaking is the main medium of communication in our daily life. It can convey the thought and express the emotional state between humans. The goal of speech emotion recognition is to recognize human emotional states from speaking. The speech emotion recognition mainly includes two steps: feature extraction and classifier construction. Taking speech features usually refers to its spectral features. The speech signal input is an approximately continuous value, and the spectral features contain considerable number of information, such as speech content, rhythm, tone, intonation, and so on. However, related with the emotion speech feature extraction is still an immature research direction. Influenced by the success of the computer vision, the visualization of speech signals has become a new method for analysis emotion recognition on the acoustic features of speech. Based on the Graham Angle Field Method, this research using a variety of neural network models to extract speech feature values and recognizing speech emotions. According to the experiments, this research find that it is feasible to visualize speech signals and use the obtained results for emotion recognition. In the future, we will further optimize the network model. Combing with other acoustic features, such as speech content, rhythm. etc., completing speech emotion recognition in our real life.
Read moreEnglish
The technique of recognizing and classifying emotions expressed in language spoken using audio features is Speech Emotion Recognition (SER). Human-computer interaction must enable machines to accurately perceive and respond to human emotions. Numerous challenges, like capturing both spatial and temporal features in speech signals, impact the accuracy of emotion recognition models. Conventional emotion recognition systems heavily depend on manual feature extraction and classification, which require significant effort and often lead to errors in detection. Advances in image processing and Artificial Intelligence (AI) have introduced hybrid Deep Learning (DL) approaches to improve SER tasks. This study developed an efficient Speech Emotion Recognition (SER) system utilizing a hybrid DL model combined with an ensemble approach to accurately classify emotions expressed through speech. The models were evaluated on the CREMA dataset which contains 7,442 audio samples across six different emotions. After preprocessing and data augmentation, Mel Frequency Cepstral Coefficients (MFCC) were captured as features from speech data. The proposed models include CNN-LSTM and CNN-GRU to extract both spatial and temporal features. Outputs from these frameworks were combined using an ensemble learning approach with a Support Vector Machine (SVM) classifier as the meta-learner. Experimental results specify that the suggested model attained improved performance with an accuracy of 98.69%, precision of 98.70%, recall of 98.72% and an F1 score of 98.70%. The results highlight the effectiveness of combining advanced neural networks for achieving high performance in emotion detection from speech signals, providing valuable information for developing real-time emotion recognition systems and enhancing human-computer interaction.
Read moreEgyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features
Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features
Improved Emotion Recognition with Novel Task-Oriented Wavelet Packet Features
In this paper, a wavelet packet based adaptive filter-bank construction method is proposed for speech signal processing. On this basis, a set of acoustic features are proposed for speech emotion recognition, namely Wavelet Packet Cepstral Coefficients (WPCC). The former extends the conventional Mel-Frequency Cepstral Coefficients (MFCC) by adapting the filter-bank structure according to the decision task; while the later aims at selecting the most crucial frequency bands where the most discriminative emotion information is located. Speech emotion recognition system is constructed with the two proposed feature sets and Gaussian mixture model as classifier. Experimental results on Berlin emotional speech database show that the proposed features improve emotion recognition performance over the conventional MFCC feature. The proposed feature extraction scheme has encouraging prospects since it can be extended to 2D image processing with 2D wavelet packets and hence extended to audio-visual bimodal emotion recognition application.
Read moreInvestigation Using MLP-SVM-PCA Classifiers on Speech Emotion Recognition
Sound localization by human listeners are capable of identifying a particular speaker, by listening to the voice of the speaker over the telephone or an entrance-way out of sight. Machines are incapable of understanding and expressing emotions. Emotions play a important role in today's digital world of remote communication. Emotion recognition can be defined as an act of predicting human's emotion through their voice samples and get the accuracy of prediction thus creating a better Human-Computer Interaction (HCI). There are various states to predict human's emotion based on behaviour, expression, pitch, tone, etc. Few of the emotions are considered to recognize the emotions of a speaker behind the speech. This research was conducted to test an speech emotion recognition (SER) system based on voice samples in two-stage approach, namely feature extraction and classification engine. The first one, the key features used for classification of emotions such as extraction of Mel Frequency Cepstral Coefficients (MFCCs), Mel Spectrogram along with Chroma features. Secondly, we use the Multilayer Perceptron (MLP) classifier, elementary classifying Support Vector Machines (SVM) and dimensionality reductionPrincipal Component Analysis (PCA) as classification methods. The research work is considered on the Toronto Emotional Speech Set (TESS) dataset. The proposed approaches gives us 94.17%, 93.43% and 97.86% classification accuracy respectively.
Read moreAnalyzing the influence of different speech data corpora and speech features on speech emotion recognition: A review
Analyzing the influence of different speech data corpora and speech features on speech emotion recognition: A review