Cross-Lingual Vocal Emotion Recognition in Five Native Languages of Assam Using Eigenvalue Decomposition

This work investigates whether vocal emotion expression of discrete emotion (i) can be recognized cross- lingually, (ii) of surprise, which is actually a cognitive component that could be present with any emotion, can also be recognized as a distinct emotion. This study will enable us to get more information regarding nature and function of emotion. Furthermore, this work will help in developing a generalized vocal emotion recognition system, which will increase the efficiency of human-machine interaction systems. In this work, an emotional speech database consisting of short sentences of five full-blown basic emotions, full-blown surprise emotion and 'no- emotion' (i.e. neutral) is created with 140 acted utterances per speaker of five native languages of Assam. This database is validated by a Listening Test. A new feature set is proposed based on Eigen Values of Autocorrelation Matrix (EVAM) of each frame of the speech signal. The Gaussian Mixture Model (GMM) is used as the classifier. The performance of the proposed feature set is compared with Mel Frequency Cepstral Coefficients (MFCCs) at sampling frequencies of 44.1 kHz and 8.1 kHz and with additive white noise of 5 db and 0 db Signal-to- Noise Ratios (SNRs) under matched noise condition.

Emotional inferences from speech require the integration of verbal and vocal emotional expressions. We asked whether this integration is comparable when listeners are exposed to their native language and when they listen to a language learned later in life. To this end, we presented native and non-native listeners with positive, neutral and negative words that were spoken with a happy, neutral or sad tone of voice. In two separate tasks, participants judged word valence and ignored tone of voice or judged emotional tone of voice and ignored word valence. While native listeners outperformed non-native listeners in the word valence task, performance was comparable in the voice task. More importantly, both native and non-native listeners responded faster and more accurately when verbal and vocal emotional expressions were congruent as compared to when they were incongruent. Given that the size of the latter effect did not differ as a function of language proficiency, one can conclude that the integration of verbal and vocal emotional expressions occurs as readily in one's second language as it does in one's native language.

The ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

Many authors have speculated about a close relationship between vocal expression of emotions and musical expression of emotions. but evidence bearing on this relationship has unfortunately been lacking. This review of 104 studies of vocal expression and 41 studies of music performance reveals similarities between the 2 channels concerning (a) the accuracy with which discrete emotions were communicated to listeners and (b) the emotion-specific patterns of acoustic cues used to communicate each emotion. The patterns are generally consistent with K. R. Scherer's (1986) theoretical predictions. The results can explain why music is perceived as expressive of emotion, and they are consistent with an evolutionary perspective on vocal expression of emotions. Discussion focuses on theoretical accounts and directions for future research.

Introduction: Parkinson’s disease (PD) commonly causes speech impairments, including difficulties in expressing emotions through voice. Method: The objective of this study was to investigate gendered vocal expressions of fear, anger, sadness, and happiness for mild to moderate PD. Prosodic features (related to fundamental frequency (F0), intensity (I), speech rate, articulation rate, and number and duration of pauses) and acoustic correlates of voice quality (CPPS, jitter, shimmer, and HNR) were collected from 14 patients with PD (mean age = 69.93; SD = 7.12; 8 males, 6 females) and 13 healthy controls (HC) (mean age = 68.13; SD = 8.27; 5 males, 8 females) matched for age, sex, and years of education. The utterances were extracted from four emotional and one neutral text. The neutral utterance and the emotional utterances were compared. Intra-sex comparison (female with PD vs. female HC and male with PD vs. male HC) and inter-sex comparison (female vs. male both for patients with PD and for HC), were performed with the Mann–Whitney test. A Mann–Whitney test was also used to compare the different emotional conditions, considering sex and PD diagnosis as well. Results: No significant intra-sex differences were found for the neutral speech, but inter-sex differences emerged. Regarding emotional speech, females with PD featured lower MaxF0 than female HCs for happiness and higher intensity variability (SD I) for sadness. Utterances by females with PD had lower CPPS than utterances by HCs for anger and fear. Utterances by males with PD had lower minimum intensity (MinI) than utterances by male HCs when expressing fear. Conclusions: Emotional vocal expression in individuals with PD was found to be impaired and showed sex differences. These findings have the potential to significantly impact the quality of life of PD patients.

To solve problem of lacking emotional speech database with rich emotion annotation information,a Chinese dual-mode emotional speech database which contained speech and Electroglottography( EGG)information was established. Annotation and consistency detection for the established database were conducted. Firstly,we designed detailed annotation rules and methods according to characteristics of emotional speech database and selected 5 annotators labeling emotional speech database in accordance with the rules.Secondly,in order to ensure annotation quality of emotional speech database and test the integrity of annotation rules,annotators labeled parts of utterances as a test before the official annotation,the test material comprises280 sentences( seven emotions × two actors × twenty sentences). Finally,according to the speech annotation rules,we designed corresponding consistency detection algorithm. The results show that within the time error range of 5 ms,the annotation consistency for the same utterances which labeled by 5 annotators reaches more than 60% on average. When the time error range increased to 8 ms and 10 ms,consistency can be increased by 5% and 8% on average. The experiment indicates that 5 annotators are more consistent in understanding speech. The annotation rules we designed are more complete. The quality of emotional speech database is higher.

Studies of human emotion perception have linked a distributed set of brain regions to the recognition of emotion in facial, vocal, and body expressions. In particular, lesions to somatosensory cortex in the right hemisphere have been shown to impair recognition of facial and vocal expressions of emotion. Although these findings suggest that somatosensory cortex represents body states associated with distinct emotions, such as a furrowed brow or gaping jaw, functional evidence directly linking somatosensory activity and subjective experience during emotion perception is critically lacking. Using functional magnetic resonance imaging and multivariate decoding techniques, we show that perceiving vocal and facial expressions of emotion yields hemodynamic activity in right somatosensory cortex that discriminates among emotion categories, exhibits somatotopic organization, and tracks self-reported sensory experience. The findings both support embodied accounts of emotion and provide mechanistic insight into how emotional expressions are capable of biasing subjective experience in those who perceive them.

Speech signals carry various bits of information relevant to the speaker such as age, gender, accent, language, health, and emotions. Emotions are conveyed through modulations of facial and vocal expressions. This paper conducts an empirical comparison of performances between the classical classifiers: Gaussian Mixture Model (GMM), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Artificial neural networks (ANN); and the deep learning classifiers, i.e., Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Gated Recurrent Unit (GRU) in addition to the ivector approach for a text-independent speaker verification task in neutral and emotional talking environments. The deep models undergo hyperparameter tuning using the Grid Search optimization algorithm. The models are trained and tested using a private Arabic Emirati Speech Database, Ryerson Audio–Visual Database of Emotional Speech and Song dataset (RAVDESS) database, and a public Crowd-Sourced Emotional Multimodal Actors (CREMA) database. Experimental results illustrate that deep architectures do not necessarily outperform classical classifiers. In fact, evaluation was carried out through Equal Error Rate (EER) along with Area Under the Curve (AUC) scores. The findings reveal that the GMM model yields the lowest EER values and the best AUC scores across all datasets, amongst classical classifiers. In addition, the ivector model surpasses all the fine-tuned deep models (CNN, LSTM, and GRU) based on both evaluation metrics in the neutral, as well as the emotional speech. In addition, the GMM outperforms the ivector using the Emirati and RAVDESS databases.

ABSTRACTWidespread cortical-subcortical networks are involved in the recognition and discrimination of emotional contents of facial and vocal expression, whereby the cerebellum and basal ganglia are two subcortical regions implicated in these networks with limited evidence to their specific contributions. To investigate this we compared patients with circumscribed cerebellar lesions and patients with Parkinson’s disease (PD) on an approved test battery. We studied two groups with subcortical disease, focal cerebellar infarction (n = 22) and PD (n = 22), and a neurological control group with focal supratentorial ischemia (SI) (n = 16) were. Assessments were according to inpatient protocols for neuropsychological routine evaluation, including tests of memory, executive function and attention. Participants completed the Tuebingen Affect Battery, a recognized measure of recognition and discrimination of facial and vocal expression of emotion. As a result, cerebellar lesions were associated with greater impairment than PD and SI in recognition and discrimination of cues of both facial and vocal expressions of differing basic emotions. No confounding effect of other cognitive domains, particularly executive function and attention, was found. Taken together, our findings suggest a specific contribution of the cerebellum to cerebral networks that process facial and vocal emotion expression, related to rapid decisions regulating appropriate behavioral responses in social environments.

The expression of human emotion is integral to social interaction, and in virtual reality it is increasingly common to develop virtual avatars that attempt to convey emotions by mimicking these visual and aural cues, i.e. the facial and vocal expressions. However, errors in (or the absence of) facial tracking can result in the rendering of incorrect facial expressions on these virtual avatars. For example, a virtual avatar may speak with a happy or unhappy vocal inflection while their facial expression remains otherwise neutral. In circumstances where there is conflict between the avatar's facial and vocal expressions, it is possible that users will incorrectly interpret the avatar's emotion, which may have unintended consequences in terms of social influence or in terms of the outcome of the interaction. In this paper, we present a human-subjects study (N = 22) aimed at understanding the impact of conflicting facial and vocal emotional expressions. Specifically we explored three levels of emotional valence (unhappy, neutral, and happy) expressed in both visual (facial) and aural (vocal) forms. We also investigate three levels of head scales (down-scaled, accurate, and up-scaled) to evaluate whether head scale affects user interpretation of the conveyed emotion. We find significant effects of different multimodal expressions on happiness and trust perception, while no significant effect was observed for head scales. Evidence from our results suggest that facial expressions have a stronger impact than vocal expressions. Additionally, as the difference between the two expressions increase, the less predictable the multimodal expression becomes. For example, for the happy-looking and happy-sounding multimodal expression, we expect and see high happiness rating and high trust, however if one of the two expressions change, this mismatch makes the expression less predictable. We discuss the relationships, implications, and guidelines for social applications that aim to leverage multimodal social cues.

Experimental studies indicate that recognition of emotions, particularly negative emotions, decreases with age. However, there is no consensus at which age the decrease in emotion recognition begins, how selective this is to negative emotions, and whether this applies to both facial and vocal expression. In the current cross-sectional study, 607 participants ranging in age from 18 to 84 years (mean age = 32.6 +/- 14.9 years) were asked to recognize emotions expressed either facially or vocally. In general, older participants were found to be less accurate at recognizing emotions, with the most distinctive age difference pertaining to a certain group of negative emotions. Both modalities revealed an age-related decline in the recognition of sadness and -- to a lesser degree -- anger, starting at about 30 years of age. Although age-related differences in the recognition of expression of emotion were not mediated by personality traits, 2 of the Big 5 traits, openness and conscientiousness, made an independent contribution to emotion-recognition performance. Implications of age-related differences in facial and vocal emotion expression and early onset of the selective decrease in emotion recognition are discussed in terms of previous findings and relevant theoretical models.

Cochlear implant (CI) listeners were found to have great difficulty with vocal emotion recognition because of the limited spectral cues provided by CI devices. Previous studies have shown that the modulation spectral features of temporal envelopes may be important cues for vocal emotion recognition of noise-vocoded speech (NVS) as simulated CIs. In this paper, the feasibility of vocal emotion conversion on a modulation spectrogram for simulated CIs for correctly recognizing vocal emotion is confirmed. A method based on a linear prediction scheme is proposed to modify the modulation spectrogram and its features of neutral speech to match that of emotional speech. The logic of this approach is that if vocal emotion perception of NVS is based on the modulation spectral features, NVS with similar modulation spectral features of emotional speech will be recognized as the same emotion. As a result, it was found that the modulation spectrogram of neutral speech can be successfully converted to that of emotional speech. The results of the evaluation experiment showed the feasibility of vocal emotion conversion on the modulation spectrogram for simulated CIs. The vocal emotion enhancement on the modulation spectrogram was also further discussed.

Accuracy and intensity of posed emotional expressions in unmedicated schizophrenia patients: Vocal and facial channels

From many years there has been gaining interest in the field of SER using Matlab. SER states the emotional state by analyzing the input speech. SER has a simple pattern and also including feature extraction, feature matching, classification and database. Here, from the input speech by using algorithm features are extracted by using some models the feature matching takes place. By this process we use to analyze the characteristics of the input speech signal. Hence, the system recognize the state of emotion. The system states some the of emotions: Angry, Boredom, Anxiety, Disgust,Happiness, Neutral, Sadness. The main purpose of this paper is to give survey on two of the algorithms using HMM model with different speech emotion databases. There are several audio features for extracting are available. And also various classifiers are available. The most popular models are Hidden Markov model(HMM), Vector Quantization(VQ), Gaussian Mixture model(GMM), Deep Neural Networks (DNN), Neural Networks(NN) and Artificial Neural Networks(ANN). There are several speech emotion databases included Berlin database. Hence, we reviewed some of the models will be discussed in this paper.

From children to teenagers and then to adults, individuals' emotional expression ability has undergone significant changes. From the compulsory education stage to the general senior middle school stage, students' cognitive and emotional needs will change greatly. Students of different school stages show significant differences in their vocal emotional expression, which not only affects their social skills but also has important guiding significance for educational practice. Therefore, this study focuses on the vocal emotional expression of students at different school stages. This study explores the differences in vocal emotional expression among students at different school stages (primary school, junior high school, and senior high school), with a focus on the expression characteristics of different emotional types and the situation of inconsistent internal and external emotions. The research results show that in terms of emotional types, students at different school stages have a relatively high accuracy rate in expressing sadness, while the accuracy rate for expressing anger is the lowest, and the accuracy rate for expressing anger is significantly lower than that for other emotions. In terms of school stages, the overall accuracy of emotional expression among students improves with the increase of school stages, but the differences between school stages are not significant. In the task of expressing inconsistent internal and external emotions, the study found that senior high school students have significantly better emotional conversion ability than primary and junior high school students, indicating that as the school stage increases, students' adaptability in complex emotional expression tasks gradually improves.