Excitation Features of Speech for Emotion Recognition Using Neutral Speech as Reference

The study of closed set text-independent speaker identification using whisper speech is presented in this paper. A new feature called temporal Teager energy based sub band cepstral coefficients (TTESBCC) is proposed. The work presented compares the performance of four feature sets: Mel frequency cepstral coefficients (MFCC), temporal energy of sub band cepstral coefficients (TESBCC), weighted instantaneous frequency (WIF) and TTESBCC. Next, outputs of three classifiers are combined and its performance is compared with that of the individual classifiers. The speaker identification system is trained using neutral speech and tested using neutral and whisper speech. The database of twenty five speakers containing speech utterances recorded in one of the Indian languages (Marathi) in the neutral and whisper environments is used for experimentation. Gaussian mixture model is used for classification. It is observed that performance of the speaker identification system degrades drastically when tested using whisper speech utterances. Fusion of classifiers enhances the speaker identification accuracy in both whisper and neutral environment.

Mel Frequency Cepstral Coefficients (MFCCs) and Perceptual linear prediction coefficients (PLPCs) are widely casted nonlinear vocal parameters in majority of the speaker identification, speaker and speech recognition techniques as well in the field of emotion recognition. Post 1980s, significant exertions are put forth on for the progress of these features. Considerations like the usage of appropriate frequency estimation approaches, proposal of appropriate filter banks, and selection of preferred features perform a vital part for the strength of models employing these features. This article projects an overview of MFCC and PLPC features for different speech applications. The insights such as performance metrics of accuracy, background environment, type of data, and size of features are inspected and concise with the corresponding key references. Adding more to this, the advantages and shortcomings of these features have been discussed. This background work will hopefully contribute to floating a heading step in the direction of the enhancement of MFCC and PLPC with respect to novelty, raised levels of accuracy, and lesser complexity.

In this work, spectral features extracted from sub-syllabic regions and pitch synchronous analysis are proposed for speech emotion recognition. Linear prediction cepstral coefficients, mel frequency cepstral coefficients and the features extracted from high amplitude regions of spectrum are used to represent emotion specific spectral information. These features are extracted from consonant, vowel and transition regions of each syllable to study the contribution of these regions toward recognition of emotions. Consonant, vowel and the transition regions are determined using vowel onset points. Spectral features extracted from each pitch cycle, are also used to recognize emotions present in speech. The emotions used in this study are: anger, fear, happy, neutral and sad. The emotion recognition performance using sub-syllabic speech segments are compared with the results of conventional block processing approach, where entire speech signal is processed frame by frame. The proposed emotion specific features are evaluated using simulated emotion speech corpus, IITKGP-SESC (Indian Institute of Technology, KharaGPur-Simulated Emotion Speech Corpus). The emotion recognition results obtained using IITKGP-SESC are compared with the results of Berlin emotion speech corpus. Emotion recognition systems are developed using Gaussian mixture models and auto-associative neural networks. The purpose of this study is to explore sub-syllabic regions to identify the emotions embedded in a speech signal, and if possible, to avoid processing of entire speech signal for emotion recognition without serious compromise in the performance.

The work on emotion recognition and models evaluation requires large corpora with recordings of emotional voices. The objective of this paper is to show a simple technique of automatic data validation that can be used in the development of a speech corpus. The paper describes an experimental study for a speech corpus development using two collections of data for vocal emotion expression with three emotions, happiness, anger, sadness and a normal (unemotional) state. In the validation step we used two classifiers, k-NN (k - Nearest Neighborhood) and SVM (Support Vector Machines), and five different sets of feature vectors based on formants F0–F4 values, MFCC (Mel-Frequency Cepstral Coefficients) and PLP (Perceptual Linear Prediction) coefficients values of the speech recording. The presented method is verified by human validation process in building an emotional recognition corpus.

In this paper, we present a hybrid speech emotion recognition system exploiting both spectral and prosodic features in speech. For capturing the emotional information in the spectral domain, we propose a new spectral feature extraction method by applying a novel non-uniform subband processing, instead of the mel-frequency subbands used in Mel-Frequency Cepstral Coefficients (MFCC). For prosodic features, a set of features that are closely correlated with speech emotional states are selected. In the proposed hybrid emotion recognition system, due to the inherently different characteristics of these two kinds of features (e.g., data size), the newly extracted spectral features are modeled by Gaussian Mixture Model (GMM) and the selected prosodic features are modeled by Support Vector Machine (SVM). The final result of the proposed emotion recognition system is obtained by combining the results from these two subsystems. Experimental results show that (1) the proposed non-uniform spectral features are more effective than the traditional MFCC features for emotion recognition; (2) the proposed hybrid emotion recognition system using both spectral and prosodic features yields the relative recognition error reduction rate of 17.0% over the traditional recognition systems using only the spectral features, and 62.3% over those using only the prosodic features.

In this work, we propose a new feature based on formants for whispered speaker verification (SV) task, where neutral data is used for enrollment and whispered recordings are used for test. Such a mismatch between enrollment and test often degrades the performance of whispered SV systems due to the difference in acoustic characteristics of whispered and neutral speech. We hypothesize that the proposed formant and formant gap (F oG) features are more invariant to the modes of speech in capturing speaker specific information compared to traditional baseline features for SV including mel frequency cepstral coefficients (MFCC) and auditory-inspired amplitude modulation features (AAMF). Whispered SV experiments with 714 speakers comprising 29232 neutral and 22932 whispered recordings reveal that the equal error rate (EER) using the proposed features is lower than that using the best baseline features by ~3.79% (absolute). It was also observed that at least four whispered recordings during enrollment are required for the baseline features to perform at par with the proposed features. However, it was found that the best performing baseline features yield an EER for neutral SV task which is ~1.88% higher than that using the proposed features.

This paper addresses the realization of a Human/Machine (H/M) interface including a system for automatic recognition of the Continuous Pathological Speech (ARSCPS) and several communication tools in order to help frail people with speech problems (Dysarthric speech) to access services providing by new technologies of information and communication (TIC) while making it easier for the doctors to achieve a first diagnosis on the patient’s disease. In addition, an ARSCPS has been improved and developed for normal and pathology voice while establishing a link with our graphic interface which is based on the box tools Hidden Markov Model Toolkit (HTK), in addition to the Hidden Models of Markov (HMM). In our work we used different techniques of feature extraction for the speech recognition system in order to improve the dysarthric speech intelligibility while developing an ARSCPS which can perform well for pathological and normal speakers. These techniques are based on the coefficients of ETSI standard Mel Frequency Cepstral Coefficient Front End (ETSI MFCC FE V2.0); Perceptual Linear Prediction coefficients (PLP); Mel Frequency Cepstral Coefficients (MFCC) and the recently proposed Power Normalized Cepstral Coefficients (PNCC) have been used as a basis for comparison. In this context we used the Nemours database which contains 11 speakers that represents dysarthric speech and 11 speakers that represents normal speech.

In the project, an emotion recognition system from speech is proposed using deep learning. The goal of this project is to classify a speech signal into one of the five emotions listed below: anger, boredom, fear, happiness, and sadness. Snippets below from numerous Marathi movies and TV shows were used to construct the dataset for Marathi language samples which include 20 audio samples for anger, 19 for boredom, 5 for fear, and 11 for happiness. The proposed system first processes a speech signal from the time domain to the frequency domain using Discrete Time Fourier Transform (DTFT). Then, data augmentation is performed which includes noise injection, stretching, shifting, and pitch scaling of the speech signal. Next, feature extraction is performed in which 5 features were selected, which include Mel Frequency Cepstral Coefficients (MFCC), Zero Crossing Rate (ZCR), Chroma STFT, Mel Spectrogram, and Root mean square value. These features were then fed to a Convolutional Neural Network (CNN). The efficiency of the suggested system employing the CNNs is supported by experimental findings. This model’s accuracy on the test data is 80.33%, and its f1 values for anger, boredom, fear, happiness, and sadness are 0.85, 0.83, 0.50, 0.62, and 0.84, respectively.

The speech recognisers use a parametric form of the signal to get the most important features in speech for the recognition task. Mel-frequency cepstral coefficients (MFCC) and Perceptual linear prediction coefficients (PLP) belong to the most commonly used methods. There is no rule to decide which one is better to use and it depends mainly on the particular conditions. The tests on taking advantage of different parts of each parametrization process to get the best results in given conditions are presented in this paper. Robust Hidden Markov model-based (HMM) Czech digit recogniser in slightly noisy environment is used for this purpose. The experiments show, that using Bark-frequency scaling, equal loudness pre-emphasis and intensity-loudness power law in the original MFCC method can bring improvement in white noise robustness for particular conditions. The results also uncovered that the LP-based methods tend to generate insertion errors in given environment.

Speaker identification refers to the process of recognizing human voice using artificial intelligence techniques. Speaker identification technologies are widely applied in voice authentication, security and surveillance, electronic voice eavesdropping, and identity verification. In the speaker identification process, extracting discriminative and salient features from speaker utterances is an important task to accurately identify speakers. Various features for speaker identification have been recently proposed by researchers. Most studies on speaker identification have utilized short-time features, such as perceptual linear predictive (PLP) coefficients and Mel frequency cepstral coefficients (MFCC), due to their capability to capture the repetitive nature and efficiency of signals. Various studies have shown the effectiveness of MFCC features in correctly identifying speakers. However, the performances of these features degrade on complex speech datasets, and therefore, these features fail to accurately identify speaker characteristics. To address this problem, this study proposes a novel fusion of MFCC and time-based features (MFCCT), which combines the effectiveness of MFCC and time-domain features to improve the accuracy of text-independent speaker identification (SI) systems. The extracted MFCCT features were fed as input to a deep neural network (DNN) to construct the speaker identification model. Results showed that the proposed MFCCT features coupled with DNN outperformed existing baseline MFCC and time-domain features on the LibriSpeech dataset. In addition, DNN obtained better classification results compared with five machine learning algorithms that were recently utilized in speaker recognition. Moreover, this study evaluated the effectiveness of one-level and two-level classification methods for speaker identification. The experimental results showed that two-level classification presented better results than one-level classification. The proposed features and classification model for identifying a speaker can be widely applied to different types of speaker datasets.

Since emotional speech can be regarded as a variation onneutral (non-emotional) speech, it is expected that a robust neu-tral speech model can be useful in contrasting different emo-tions expressed in speech. This study explores this idea by cre-ating acoustic models trained with spectral features, using theemotionally-neutral TIMIT corpus. The performance is testedwith two emotional speech databases: one recorded with a mi-crophone (acted), and another recorded from a telephone ap-plication (spontaneous). It is found that accuracy up to 78%and 65% can be achieved in the binary and category emotiondiscriminations, respectively. Raw Mel Filter Bank (MFB) out-put was found to perform better than conventional MFCC, withboth broad-band and telephone-band speech. These results sug-gest that well-trained neutral acoustic models can be effectivelyused as a front-end for emotion recognition, and once trainedwith MFB, it may reasonably work well regardless of the chan-nel characteristics.Index Terms: Emotion recognition, Neutral speech, HMMs,Mel filter bank (MFB), TIMIT

Speech is a powerful means of expressing thoughts, emotions, and perspectives. However, accurately determining the emotions conveyed through speech remains a challenging task. Existing manual methods for analyzing speech to recognize emotions are prone to errors, limiting our understanding and response to individuals’ emotional states. To address diverse accents, an automated system capable of real-time emotion prediction from human speech is needed. This paper introduces a speech emotion recognition (SER) system that leverages supervised learning techniques to tackle cross-accent diversity. Distinctively, the system extracts a comprehensive set of nine speech features—Zero Crossing Rate, Mel Spectrum, Pitch, Root Mean Square values, Mel Frequency Cepstral Coefficients, chroma-stft, and three spectral features (Centroid, Contrast, and Roll-off) for refined speech signal processing and recognition. Seven machine learning models are employed, encompassing Random Forest, Logistic Regression, Decision Tree, Support Vector Machines, Gaussian Naive Bayes, K-Nearest Neighbors, ensemble learning, and four individual, hybrid deep learning models including Long short-term memory (LSTM) and 1-Dimensional Convolutional Neural Network (1D-CNN) with stratified cross-validation. Audio samples from diverse English regions are combined to train the models. The performance evaluation results of conventional machine learning and deep learning models indicate that the Random Forest-based feature selection model achieves the highest accuracy of up to 76% among the conventional machine learning models. Simultaneously, the 1D-CNN model with stratified cross-validation reaches up to 99% accuracy. The proposed framework enhances the cross-accent emotion recognition accuracy up to 86.3%, 89.87%, 90.27%, and 84.96% by margins of 14.71%, 10.15%, 9.6%, and 16.52% respectively.

Emotion is the psychological and physiological response of a person or animal to a specific stimulus or event, and it can also have an impact on cognition, decision-making, and behavior. Research on emotion recognition has wide applications in various fields, such as dialogue and communication with machines, and in the medical field. At present, most emotion recognition systems are based on PC or embedded platforms. Compared with traditional emotion recognition platforms, FPGA has good parallel computing ability and can process multiple data streams simultaneously, making it suitable for complex pattern recognition in emotion recognition. This article chooses ZYNQ as the development platform, combined with the collaborative ability of FPGA and ARM processors, to build a complete hardware application system that can carry neural networks. Secondly, in response to the problems of complex computation and high storage resource consumption in existing neural network computing frameworks, this paper adopts a lightweight neural network MobileNetV3 and implements and improves it. The model is encapsulated and ported using the Vivado HLS development tool and C++. The hardware IP core is called on the hardware platform to achieve ZYNQ based facial emotion classification and recognition, and the power consumption of hardware resources is analyzed.

Research in emotional speech recognition is generally focused on analysis of a set of primary emotions. However it is clear that spontaneous speech, which is more intricate comparing to acted out utterances, carries information about emotional complexity or degree of their intensity. This paper refers to the theory of Robert Plutchik, who suggested the existence of eight primary emotions. All other states are derivatives and occur as combinations, mixtures or compounds of the primary emotions. During the analysis Polish spontaneous speech database containing manually created confidence labels was implemented as a training and testing set. Classification results of four primary emotions (anger, fear, joy, sadness) and their intensities have been presented. The level of intensity is determined basing on the similarity of particular emotion to neutral speech. Studies have been conducted using prosodic features and perceptual coefficients. Results have shown that the proposed measure is effective in recognition of intensity of the predicted emotion.

Emotional expressions are a fundamental aspect of human communication, with speech being one of the most natural modes of interaction. Speech Emotion Recognition (SER) is a significant research topic in Natural Language Processing (NLP), aimed at identifying emotions such as satisfaction, frustration, and anger from speech audio using multiple classifiers. This article presents a method to emotion recognition from spontaneous Tunisian Dialect (TD) speech, marking the first work in the SER field to utilize spontaneous speech for emotion recognition in this dialect. The dataset was created from freely available YouTube videos across multiple domains and labeled with four perceived emotions: anger, satisfaction, frustration, and neutral. To address the data scarcity issue, we implemented data augmentation techniques, specifically Vocal Tract Length Perturbation (VTLP). The preprocessing of the speech signals involved cleaning the data from ambient and unwanted noises. We extracted and selected various spectral features, including Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCC). Subsequently, we applied several classification methods: Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Random Forest. Our experiments demonstrated that the Random Forest classifier achieved the highest F-score of 58.75%. The results were thoroughly discussed, analyzed, and compared across the five models using different feature extractions. This study provides valuable insights and advancements in the SER field, particularly for the TD, for future research directions for improving emotion recognition systems.