Recognition of Emotion Intensity Basing on Neutral Speech Model

Recognizing emotions using natural or spontaneous speech are extremely difficult compared to doing the same for acted or elicited speeches. Speech emotion recognition for real conversation such as spontaneous speech requires linguistic information of the speech to be included in the speech emotion recognition component to achieve a high recognition rate. However, with the lack of digital speech resources of an under-resourced language, this requirement poses a problem. In this paper, speech emotion recognition of spontaneous speech in Malay language using prosodic features and Random Forest classifier is presented. We also investigate the influence of age categorized as children, young adults and middle-aged on emotion recognition. Ninety spontaneous speech sentences from 30 native speakers of Malay language are collected and classified into three emotions, which are happy, angry and sad. Results show that the spontaneous speech of middle-aged group achieved the highest accuracy rate followed by children age group and finally the young adults. While sad emotions are recognized satisfactorily across all age groups, confusions exist between happy and angry emotions.

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

Emotions are an integral part of social interactions, serving vital functions in the management of our interpersonal relationships. Cues to emotional meaning can be conveyed through a variety of modalities, from facial expressions and body gestures to the linguistic and non-linguistic, or prosodic aspects of speech. As automatic speech systems continue to grow in use, the ability to recognize emotional cues from these modalities is no longer solely expected in human communication, but also in human-computer interactions with the goal of developing more user-friendly and effective applications. This thesis explores how speech emotion recognition (SER) systems can be more robust to factors of language, culture, and context through the lens of emotional speech and emotional prosody from bilingual English-speaking Singaporeans of Chinese, Malay, or Indian ethnicity. Specifically, the thesis aims to examine how Singaporeans from the three ethnic groups may vary in their perception of emotions in speech, and to explore how they express emotions via speech prosody. The recognition and production processes of emotional communication in Singapore English was investigated through two emotion classes – anger and happiness. An emotion perception test was conducted using acted angry and happy speech as stimuli to understand how Singapore English speakers recognized emotions in speech. The results of the perception test revealed that respondents displayed very similar patterns in their rating of emotional speech stimuli. The study also found that respondents tended to rated anger with low dominance. At the same time, a spontaneous speech corpus known as the Singapore English Spontaneous Emotional Speech Corpus (SESESC) was also developed, featuring audio recordings of 44 Singapore English speakers from the three ethnic groups playing a co-operative video game with either a familiar partner or a confederate. More than 36,000 intervals were annotated in-context of the recordings by each annotator based on a modified emotion annotation scheme that included information regarding the emotion category and arousal levels. Intervals that were labelled as conveying joy or anger with full agreement from the annotators were selected following a series of criteria to facilitate comparable prosodic analyses given the inherent dynamicity of spontaneous speech. The analysis focused on exploring the F0 movements of the selected utterances in tandem with periodic energy. Overall, it was observed that utterances conveying happiness were associated more with level contours, while angry utterances were produced with more steep falling contours especially on the utterance-final syllable. A tendency for an F0 peak to be produced on the second syllable of the utterance was also observed in both sets of data analyzed, alluding to the interactions between emotional prosody and the intonational structure of Singapore English. Finally, an emotion classification was conducted using a subset of the corpus as test data for a fine-tuned SER model. Results showed that while utterances conveying happiness could be automatically recognized to a high percentage of 86%, there was also a high percentage of misclassification of anger. This result appears to counter findings in past SER studies, which typically reported high levels of recognition accuracy for anger. Taken together, the thesis contributes an understanding of how spontaneous speech and emotional cues on the prosodic level interact even in task-oriented conversations, with further considerations of how these patterns may vary among bilinguals in multicultural contexts, so as to enable more robust speech emotion recognition systems in the future.

-- Speech signals are being considered as most effective means of communication between human beings. Many researchers have found different methods or systems to identify emotions from speech signals. Here, the various features of speech are used to classify emotions. Features like pitch, tone, intensity are essential for classification. Large number of the datasets are available for speech emotion recognition. Firstly, the extraction of features from speech emotion is carried out and then another important part is classification of emotions based upon speech. Hence, different classifiers are used to classify emotions such as Happy, Sad, Anger, Surprise, Neutral, etc. Although, there are other approaches based on machine learning algorithms for identifying emotions. Speech Emotion Recognition is a current research topic because of its wide range of applications and it became a challenge in the field of speech processing too. We have carried out a brief study on Speech Emotion Analysis along with Emotion Recognition. Speech Emotion Recognition (SER) can be defined as extraction of the emotional state of the speaker from his or her speech signal. There are few universal emotions including Neutral, Anger, we have worked on different tools to be used in SER. SER is tough because emotions are subjective and annotating audio is challenging task. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. We have classified based on different type of emotions to detect from speech. Key Words: Speech Emotion Recognition, Affective Computing, Machine Learning, Deep Learning, Audio Signal Processing, Emotion Classification, Feature Extraction, Prosodic Features, Spectral Features, Mel-Frequency Cepstral Coefficients (MFCCs), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Attention Mechanisms, Multimodal Emotion Recognition, Speaker-Independent SER, Real-Time Emotion Detection, Noise-Robust Emotion Recognition, Data Augmentation, Emotion-Aware Applications

With the exception of extensive research on test anxiety, research on emotions experienced by teachers and students in classrooms is still in its infancy. In particular, little research has explored the emotional arousal of classroom participants in the teaching and learning of specific subject matter such as science (Rosiek and Beghetto 2009). However, overall, it is now clear that emotions have a profound effect on students’ and teachers’ interest, engagement, performance, and achievement. In short, emotions appear to be central to understanding teaching and learning (Pekrun and Schutz 2007). Moreover, the centrality of emotions to daily life is evident in the realization that every dimension of society is held together or ripped apart by emotional arousal (Turner 2007). Defining emotion has been an elusive challenge because it can be studied from a variety of theoretical perspectives. Emotions can be viewed as the complex of causal effects among four interrelated elements, namely, situational cues (e.g., behaviors relevant to achieving a specific task or goal), physiological changes (e.g., heart rate, blood pressure), emotion labels (e.g., happiness, anger, fear, sadness), and expressive gesture (e.g., facial expressions, hand gestures) (Turner 2007). Michalinos Zembylas’ extensive reporting of his three-year ethnographic study of science teaching and learning in an elementary teacher’s classroom not only reaffirmed the pivotal role of emotions in teaching and learning (e.g., Zembylas 2004) but also showed how emotional practices could be constructed by the teacher and her students. Yet, the social-constructivist perspective underpinning the study is limited because the arousal of emotion is not simply a labeling or socially constructed process but also involves neural transmissions and physiological functions that can be triggered automatically without conscious effort. This suggests that an integrated theory that embraces rather than ignores the biology of emotions, such as Turner’s sociology of human emotions, offers a promising way forward to researching emotions in educational contexts (Turner 2007). Most scholars agree that the four primary emotions are the following: satisfaction-happiness, aversion-fear, assertion-anger, and disappointment-sadness, which Turner (2007) presumes are hard-wired in human neuroanatomy. Some scholars add surprise and disgust to this list. These primary emotions can be experienced at different levels of intensity and in various combinations that denote the more complex emotions such as shame and guilt. Interest, boredom, and curiosity are sometimes included on lists of emotions, but other times excluded because they are considered cognitive states (Turner 2007). A series of studies conducted by Ritchie and his colleagues (see: http://profiles.murdoch.edu.au/ myprofile/stephen-ritchies/) in a variety of secondary science and teacher education classroom contexts embraced Turner’s theoretical framework that also incorporates related sociological constructs such as positive emotional energy generated from successful interactions. They accept that

Knowledge-based Framework for Intelligent Emotion Recognition in Spontaneous Speech

The acoustic cues that convey emotion in speech are similar to those that convey emotion in music, and recognition of emotion in both of these types of cue recruits overlapping networks in the brain. Given the similarities between music and speech prosody, developmental research is uniquely positioned to determine whether recognition of these cues develops in parallel. In the present study, we asked 60 children aged 6 to 11 years, and 51 university students, to judge the emotions of 10 musical excerpts, 10 inflected speech clips, and 10 affect burst clips. We presented stimuli intended to convey happiness, sadness, anger, fear, and pride. Each emotion was presented twice per type of stimulus. We found that recognition of emotions in music and speech developed in parallel, and adult-levels of recognition develop later for these stimuli than for affect bursts. We also found that sad stimuli were most easily recognised, followed by happiness, fear, and then anger. In addition, we found that recognition of emotion in speech and affect bursts can predict emotion recognition in music stimuli independently of age and musical training. Finally, although proud speech and affect bursts were not well recognised, children aged eight years and older showed adult-like responses in recognition of proud music.

Speech Emotion Recognition (SER) plays a vital role in various real-world applications, from mental health diagnostics to human-computer interaction. This research examines the integration of Stacked Long Short-Term Memory (LSTM) networks and Hidden Markov Models (HMM) for SER, addressing the limitations of traditional models like Gaussian Mixture Models (GMM) and Support Vector Machines (SVM). While GMMs and SVMs offer decent performance, their inability to model temporal dynamics limits their accuracy in detecting complex emotions. The proposed hybrid model leverages LSTM’s strength in handling sequential data and HMM’s ability to model emotional transitions, making it highly suitable for real-world noisy environments. Preprocessing techniques such as MFCC and LPCC are applied to enhance feature extraction, and benchmark datasets like IEMOCAP and RAVDESS are used for evaluation. Finally, The Systematic review highlights the superior role of the hybrid model's performance on SER and sets the stage for a significant shift in future research in addressing bias and fairness in SER systems by combining LSTM and HMM as hybrid model. In recent studies, context-aware emotion recognition models are commonly evaluated using existing datasets such as IEMOCAP, MELD, DailyDialog, and SEMAINE, which provide conversational and multimodal emotional data. Features used often include acoustic features as MFCCs, spectrograms, prosodic features, linguistic features as BERT or transformer embeddings, and visual cues such as facial landmarks or video frames. Many models also incorporate contextual embeddings from transformers, graph attention networks, or sequential memory modules to capture speaker history and emotional shifts.

The paper addresses the research question of automatic emotional speech recognition for Serbian. It integrates two research issues: (i) selection of an appropriate feature set, and (ii) investigation of different classification techniques. The paper reports a set of experiments with three feature sets: (i) the prosodic feature set, (ii) the spectral feature set, and (iii) the set of both spectral and prosodic features. The linear Bayes, the perceptron rule and the kNN classifier were considered in all three experiments. The experimental results show that the highest recognition accuracy of 91.5 % was obtained with the third feature set using the linear Bayes classifier.DOI: http://dx.doi.org/10.5755/j01.eee.18.9.2806

This article presents a probabilistic scheme for detecting the interruption point (IP) in spontaneous speech based on inter-syllable boundary-based prosodic features. Because of the high error rate in spontaneous speech recognition, a combined acoustic model considering both syllable and subsyllable recognition units, is firstly used to determine the inter-syllable boundaries and output the recognition confidence of the input speech. Based on the finding that IPs always occur at inter-syllable boundaries, a probability distribution of the prosodic features at the current potential IP is estimated. The Conditional Random Field (CRF) model, which employs the clustered prosodic features of the current potential IP and its preceding and succeeding inter-syllable boundaries, is employed to output the IP likelihood measure. Finally, the confidence of the recognized speech, the probability distribution of the prosodic features and the CRF-based IP likelihood measure are integrated to determine the optimal IP sequence of the input spontaneous speech. In addition, pitch reset and lengthening are also applied to improve the IP detection performance. The Mandarin Conversional Dialogue Corpus is adopted for evaluation. Experimental results show that the proposed IP detection approach obtains 10.56% and 6.5% more effective results than the hidden Markov model and the Maximum Entropy model respectively under the same experimental conditions. Besides, the IP detection error rate can be further reduced by 9.15% using pitch reset and lengthening information. The experimental results confirm that the proposed model based on inter-syllable boundary-based prosodic features can effectively detect the interruption point in spontaneous Mandarin speech.

Emotionally coloured speech recognition is a key technology toward achieving human-like spoken dialog systems. However, despite rapid progress in automatic speech recognition (ASR) and emotion research, much less work has examined ASR systems that recognize the verbal content of emotionally coloured speech. Approaches that exist in emotional speech recognition mostly involve adapting standard ASR models to include information about prosody and emotion. In this study, instead of adapting a model to handle emotional speech, we focus on feature transformation methods to solve the mismatch and improve the ASR performance. In this way, we can train the model with emotionally coloured speech without any explicit emotional annotation. We investigate the use of two different deep bottleneck network structures: deep neural networks (DNNs) and convolutional neural networks (CNNs). We hypothesize that the trained bottleneck features may be able to extract essential information that represents the verbal content while abstracting away from superficial differences caused by emotional variance. We also try various combinations of these two bottleneck features with feature-space speaker adaptation. Experiments using Japanese and English emotional speech data reveal that both varieties of bottleneck features and feature-space speaker adaptation successfully improve the emotional speech recognition performance.

Context-aware computing has become an essential component of modern wearable technology, enabling devices to sense, interpret, and respond to users’ environments and affective states. By integrating contextual intelligence, smart systems can adapt their own behavior to better align with user needs, enhancing efficiency, personalization, and user experience. A critical aspect of context awareness is the ability to recognize a user’s internal state, particularly emotions, which influence decision-making, communication, and overall well-being. While emotion recognition is traditionally approached through multimodal sensing—using cameras, microphones, or physiological sensors such as Electroencephalography (EEG), Electrocardiography (ECG), and Electro-dermal Activity (EDA)—these methods often require multiple sensors, leading to challenges related to power consumption, complexity, and privacy. The demand for a more efficient and unobtrusive approach to emotion and context recognition has driven research toward minimal-sensor solutions that can extract meaningful affective and contextual information with reduced hardware requirements. Restricting sensing sources of contextual information by using only limited number of sensors might come at the expense of constraining the ultimate goal of context-awareness. Such expense can be mitigated by “doing more with less” i.e. delving into the opportunities of using a single sensor for multiple purposes. What is desirable is a minimal-sensor solution capable of capturing both motion and physiological signals for both robust emotion recognition and additional context estimation. Among wearable sensors, chest-worn Inertial Measurement Units (IMUs) stand out as promising candidates for comprehensive context estimation, as they are well-positioned to capture motion patterns, and physiological vibrations, being the subtle cardio-respiratory signals. This raises the question of whether a single chest-worn IMU could serve as a minimal yet effective sensor for simultaneously estimating both emotional and contextual information, offering a practical alternative to traditional multi-sensor approaches. To address the challenge of using chest-worn IMU for multiple context sensing, I investigated the potential of using Seismocardiography (SCG) and Accelerometry-Derived Respiration (ADR) to estimate emotional states. First, I explored the existing capabilities of chest-worn IMUs in capturing cardio-respiratory vibrations and motion data, which I used as a knowledge base for embedding such IMUs in context sensing. Data plays a crucial role in this problem, therefore I developed a flexible software interface, ColEmo, to facilitate the scalable collection of emotion-related data and introduced a novel dataset, EmoWear, integrating physiological and motion data for emotion recognition and context awareness. Then, I examined the feasibility of using SCG for emotion recognition, comparing its performance with traditional physiological signals and evaluating its effectiveness as a single-sensor solution. This thesis presents a detailed analysis of the potential of chest-worn IMUs for emotion and context recognition. It reports on the design and validation of a SCG-based wearable system, the development and characteristics of the EmoWear dataset, and the implementation of the ColEmo platform for emotion data collection. The research also examines machine learning techniques applied to the SCG and ADR signals, assessing their effectiveness in estimating emotional states. My findings indicate that SCG-based emotion recognition achieves performance comparable to conventional physiological counterparts, demonstrating the feasibility of using a single IMU sensor for affective computing. The publicly-available EmoWear dataset provides a valuable resource for future research, highlighting the potential of combining motion and physiological data for enhanced context awareness. Additionally, the ColEmo software interface, which I published open-source, proves to be an effective tool for multi-modal emotion data collection, facilitating further advancements in wearable emotion recognition. The results pave the way for a minimal-sensor approach for context information estimation that can significantly reduce system complexity while maintaining high performance. These findings confirm that chest-worn IMUs present an opportunity to further improve emotion recognition in addition to the motion-based context information estimation. My research highlights the need for standardized methodologies in affective computing and demonstrates the viability of motion-based physiological sensing for emotion estimation. By using a single chest-worn IMU, this work presents an alternative to multi-sensor systems, reducing power consumption, and privacy concerns while maintaining robust performance. Future research should focus on refining IMU-based emotion recognition using deep learning techniques and larger datasets to enhance classification accuracy. Further validation in real-world settings is necessary to assess the practical applicability of the proposed system. The EmoWear dataset should be explored for additional context-awareness applications, such as activity recognition and stress monitoring, to maximize its utility. The ColEmo platform offers opportunities for expansion beyond emotion recognition, including cognitive load estimation, stress monitoring, and human-computer interaction. This research moves wearable technology forward by using fewer sensors, helping make affective computing more efficient, versatile, and respectful of privacy.

Emotional expressions are a fundamental aspect of human communication, with speech being one of the most natural modes of interaction. Speech Emotion Recognition (SER) is a significant research topic in Natural Language Processing (NLP), aimed at identifying emotions such as satisfaction, frustration, and anger from speech audio using multiple classifiers. This article presents a method to emotion recognition from spontaneous Tunisian Dialect (TD) speech, marking the first work in the SER field to utilize spontaneous speech for emotion recognition in this dialect. The dataset was created from freely available YouTube videos across multiple domains and labeled with four perceived emotions: anger, satisfaction, frustration, and neutral. To address the data scarcity issue, we implemented data augmentation techniques, specifically Vocal Tract Length Perturbation (VTLP). The preprocessing of the speech signals involved cleaning the data from ambient and unwanted noises. We extracted and selected various spectral features, including Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCC). Subsequently, we applied several classification methods: Support Vector Machine (SVM), Bidirectional Long Short-Term Memory (BiLSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Random Forest. Our experiments demonstrated that the Random Forest classifier achieved the highest F-score of 58.75%. The results were thoroughly discussed, analyzed, and compared across the five models using different feature extractions. This study provides valuable insights and advancements in the SER field, particularly for the TD, for future research directions for improving emotion recognition systems.

Speech emotion recognition (SER) has been actively studied in the recent decade and has achieved promising results. Most state-of-the-art SER methods are based on a classification approach that ultimately outputs the softmax probability over different emotion classes. On the other hand, we have recently introduced an anomalous sound detection approach to improve the SER performance of the neutral class. It uses a neutral speech detector consisting of an autoencoder that reconstructs acoustic and text features in latent space and is trained using only neutral speech data. The experimental result confirmed that the reconstruction error could be successfully used as a cue to decide whether or not the class is neutral and suggested that it could be applied to other emotion classes. In this paper, we propose an SER method based on the reconstruction of acoustic and text features in latent space, in which the reconstructor for different emotion classes, including the neutral class, is used. The proposed method selects the emotion class with the lowest normalized reconstruction error as the SER result. Unlike the classifier approach, one reconstructor is dedicated to each emotion class and trained using only the data of the target emotion class. Therefore, the reconstructor can be trained without being affected by imbalanced training data and also facilitates the application of data augmentation to only a specific emotion class. Our experimental result obtained using the IEMOCAP dataset showed that the proposed method improved the class-average weighted accuracy by 1.7% to 77.8% compared with the state-of - the-art SER methods.

In generation of emotional speech, there are deviations in the speech production features when compared to neutral (non-emotional) speech. The objective of this study is to capture the deviations in features related to the excitation component of speech and to develop a system for automatic recognition of emotions based on these deviations. The emotions considered in this study are anger, happiness, sadness and neutral state. The study shows that there are useful features in the deviations of the excitation features, which can be exploited to develop an emotion recognition system. The excitation features used in this study are the instantaneous fundamental frequency (F_0), the strength of excitation, the energy of excitation and the ratio of the high-frequency to low-frequency band energy (beta ). A hierarchical binary decision tree approach is used to develop an emotion recognition system with neutral speech as reference. The recognition experiments showed that the excitation features are comparable or better than the existing prosody features and spectral features, such as mel-frequency cepstral coefficients, perceptual linear predictive coefficients and modulation spectral features.