Communication of emotions in vocal expression and music performance: different channels, same code?

Studies of human emotion perception have linked a distributed set of brain regions to the recognition of emotion in facial, vocal, and body expressions. In particular, lesions to somatosensory cortex in the right hemisphere have been shown to impair recognition of facial and vocal expressions of emotion. Although these findings suggest that somatosensory cortex represents body states associated with distinct emotions, such as a furrowed brow or gaping jaw, functional evidence directly linking somatosensory activity and subjective experience during emotion perception is critically lacking. Using functional magnetic resonance imaging and multivariate decoding techniques, we show that perceiving vocal and facial expressions of emotion yields hemodynamic activity in right somatosensory cortex that discriminates among emotion categories, exhibits somatotopic organization, and tracks self-reported sensory experience. The findings both support embodied accounts of emotion and provide mechanistic insight into how emotional expressions are capable of biasing subjective experience in those who perceive them.

The ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

Accuracy and intensity of posed emotional expressions in unmedicated schizophrenia patients: Vocal and facial channels

From children to teenagers and then to adults, individuals' emotional expression ability has undergone significant changes. From the compulsory education stage to the general senior middle school stage, students' cognitive and emotional needs will change greatly. Students of different school stages show significant differences in their vocal emotional expression, which not only affects their social skills but also has important guiding significance for educational practice. Therefore, this study focuses on the vocal emotional expression of students at different school stages. This study explores the differences in vocal emotional expression among students at different school stages (primary school, junior high school, and senior high school), with a focus on the expression characteristics of different emotional types and the situation of inconsistent internal and external emotions. The research results show that in terms of emotional types, students at different school stages have a relatively high accuracy rate in expressing sadness, while the accuracy rate for expressing anger is the lowest, and the accuracy rate for expressing anger is significantly lower than that for other emotions. In terms of school stages, the overall accuracy of emotional expression among students improves with the increase of school stages, but the differences between school stages are not significant. In the task of expressing inconsistent internal and external emotions, the study found that senior high school students have significantly better emotional conversion ability than primary and junior high school students, indicating that as the school stage increases, students' adaptability in complex emotional expression tasks gradually improves.

The expression of human emotion is integral to social interaction, and in virtual reality it is increasingly common to develop virtual avatars that attempt to convey emotions by mimicking these visual and aural cues, i.e. the facial and vocal expressions. However, errors in (or the absence of) facial tracking can result in the rendering of incorrect facial expressions on these virtual avatars. For example, a virtual avatar may speak with a happy or unhappy vocal inflection while their facial expression remains otherwise neutral. In circumstances where there is conflict between the avatar's facial and vocal expressions, it is possible that users will incorrectly interpret the avatar's emotion, which may have unintended consequences in terms of social influence or in terms of the outcome of the interaction. In this paper, we present a human-subjects study (N = 22) aimed at understanding the impact of conflicting facial and vocal emotional expressions. Specifically we explored three levels of emotional valence (unhappy, neutral, and happy) expressed in both visual (facial) and aural (vocal) forms. We also investigate three levels of head scales (down-scaled, accurate, and up-scaled) to evaluate whether head scale affects user interpretation of the conveyed emotion. We find significant effects of different multimodal expressions on happiness and trust perception, while no significant effect was observed for head scales. Evidence from our results suggest that facial expressions have a stronger impact than vocal expressions. Additionally, as the difference between the two expressions increase, the less predictable the multimodal expression becomes. For example, for the happy-looking and happy-sounding multimodal expression, we expect and see high happiness rating and high trust, however if one of the two expressions change, this mismatch makes the expression less predictable. We discuss the relationships, implications, and guidelines for social applications that aim to leverage multimodal social cues.

Experimental studies indicate that recognition of emotions, particularly negative emotions, decreases with age. However, there is no consensus at which age the decrease in emotion recognition begins, how selective this is to negative emotions, and whether this applies to both facial and vocal expression. In the current cross-sectional study, 607 participants ranging in age from 18 to 84 years (mean age = 32.6 +/- 14.9 years) were asked to recognize emotions expressed either facially or vocally. In general, older participants were found to be less accurate at recognizing emotions, with the most distinctive age difference pertaining to a certain group of negative emotions. Both modalities revealed an age-related decline in the recognition of sadness and -- to a lesser degree -- anger, starting at about 30 years of age. Although age-related differences in the recognition of expression of emotion were not mediated by personality traits, 2 of the Big 5 traits, openness and conscientiousness, made an independent contribution to emotion-recognition performance. Implications of age-related differences in facial and vocal emotion expression and early onset of the selective decrease in emotion recognition are discussed in terms of previous findings and relevant theoretical models.

Introduction: Parkinson’s disease (PD) commonly causes speech impairments, including difficulties in expressing emotions through voice. Method: The objective of this study was to investigate gendered vocal expressions of fear, anger, sadness, and happiness for mild to moderate PD. Prosodic features (related to fundamental frequency (F0), intensity (I), speech rate, articulation rate, and number and duration of pauses) and acoustic correlates of voice quality (CPPS, jitter, shimmer, and HNR) were collected from 14 patients with PD (mean age = 69.93; SD = 7.12; 8 males, 6 females) and 13 healthy controls (HC) (mean age = 68.13; SD = 8.27; 5 males, 8 females) matched for age, sex, and years of education. The utterances were extracted from four emotional and one neutral text. The neutral utterance and the emotional utterances were compared. Intra-sex comparison (female with PD vs. female HC and male with PD vs. male HC) and inter-sex comparison (female vs. male both for patients with PD and for HC), were performed with the Mann–Whitney test. A Mann–Whitney test was also used to compare the different emotional conditions, considering sex and PD diagnosis as well. Results: No significant intra-sex differences were found for the neutral speech, but inter-sex differences emerged. Regarding emotional speech, females with PD featured lower MaxF0 than female HCs for happiness and higher intensity variability (SD I) for sadness. Utterances by females with PD had lower CPPS than utterances by HCs for anger and fear. Utterances by males with PD had lower minimum intensity (MinI) than utterances by male HCs when expressing fear. Conclusions: Emotional vocal expression in individuals with PD was found to be impaired and showed sex differences. These findings have the potential to significantly impact the quality of life of PD patients.

Several studies have investigated the encoding and perception of emotional expressivity in music performance. A relevant question concerns how the ability to communicate emotions in music performance is acquired. In accordance with recent theories on the embodiment of emotion, we suggest here that both the expression and recognition of emotion in music might at least in part rely on knowledge about the sounds of expressive body movements. We test this hypothesis by drawing parallels between musical expression of emotions and expression of emotions in sounds associated with a non-musical motor activity: walking. In a combined production-perception design, two experiments were conducted, and expressive acoustical features were compared across modalities. An initial performance experiment tested for similar feature use in walking sounds and music performance, and revealed that strong similarities exist. Features related to sound intensity, tempo and tempo regularity were identified as been used similarly in both domains. Participants in a subsequent perception experiment were able to recognize both non-emotional and emotional properties of the sound-generating walkers. An analysis of the acoustical correlates of behavioral data revealed that variations in sound intensity, tempo, and tempo regularity were likely used to recognize expressed emotions. Taken together, these results lend support the motor origin hypothesis for the musical expression of emotions.

ABSTRACTWidespread cortical-subcortical networks are involved in the recognition and discrimination of emotional contents of facial and vocal expression, whereby the cerebellum and basal ganglia are two subcortical regions implicated in these networks with limited evidence to their specific contributions. To investigate this we compared patients with circumscribed cerebellar lesions and patients with Parkinson’s disease (PD) on an approved test battery. We studied two groups with subcortical disease, focal cerebellar infarction (n = 22) and PD (n = 22), and a neurological control group with focal supratentorial ischemia (SI) (n = 16) were. Assessments were according to inpatient protocols for neuropsychological routine evaluation, including tests of memory, executive function and attention. Participants completed the Tuebingen Affect Battery, a recognized measure of recognition and discrimination of facial and vocal expression of emotion. As a result, cerebellar lesions were associated with greater impairment than PD and SI in recognition and discrimination of cues of both facial and vocal expressions of differing basic emotions. No confounding effect of other cognitive domains, particularly executive function and attention, was found. Taken together, our findings suggest a specific contribution of the cerebellum to cerebral networks that process facial and vocal emotion expression, related to rapid decisions regulating appropriate behavioral responses in social environments.

There exists substantial cultural variation in how emotions are expressed, but there is also considerable evidence for universal properties in facial and vocal affective expressions. This is the first empirical effort examining the perception of vocal emotional expressions across cultures with little common exposure to sources of emotion stimuli, such as mass media. Shuar hunter-horticulturalists from Amazonian Ecuador were able to reliably identify happy, angry, fearful and sad vocalizations produced by American native English speakers by matching emotional spoken utterances to emotional expressions portrayed in pictured faces. The Shuar performed similarly to English speakers who heard the same utterances in a content-filtered condition. These data support the hypothesis that vocal emotional expressions of basic affective categories manifest themselves in similar ways across quite disparate cultures.

This chapter provides a comprehensive overview of the functional roles of the orbitofrontal cortex (OFC) and the basal ganglia (BG) in the processing of vocal emotional expressions in humans. We suggest that the two structures may interface during the integration of verbal and non-verbal vocal emotional expressions during late, cognitive processing stages as evidenced by recent event-related brain potential (ERP) and behavioural patient evidence. In accordance with our recent network model on emotional speech processing (Schirmer and Kotz 2006), we propose that the BG and the OFC are critically involved in the integration and evaluation of vocal emotional expressions. However, the overview will also reveal that the OFC is functionally differentiated and may thus respond to emotional expressions during both early automatic and late controlled, evaluative processing stages. Which processing stages the OFC responds to may critically hinge on the nature of stimulus and task type. As both structural and functional subcomponents of the orbito-striatal interface appear to be comparable in humans and non-human primates, we finally speculate that this system may be involved in the processing of emotional vocalizations across different species.

The present study used a cross-fostering procedure to investigate the effects of early traumatic experience on vocal expressions of emotions in rhesus macaques (Macaca mulatta). The subjects of the study were 12 juvenile females: six were born to abusive mothers and reared by nonabusive controls, and six were born to controls and reared by abusive mothers. The cross-fostering took place within 24-48 hr after birth. Vocalizations were recorded from the subjects in their social groups during their first 2 years of life. Abusive mothers maltreated their adopted daughters in the first 2-3 months after birth with patterns similar to those previously shown with their biological offspring. Abused females produced proportionally more noisy screams compared to controls. While controls used noisy screams during contact aggression and tonal screams during non-contact aggression, the screams from the abused animals appeared to be distributed equally across contexts. Acoustical analyses revealed that the screams of the abused females were less modulated and had lower fundamental frequencies compared to the screams of controls. Taken together, these results suggest that traumatic experience in the first few months of life can have long-term effects on vocal emotional expression in rhesus macaques.

Emotional inferences from speech require the integration of verbal and vocal emotional expressions. We asked whether this integration is comparable when listeners are exposed to their native language and when they listen to a language learned later in life. To this end, we presented native and non-native listeners with positive, neutral and negative words that were spoken with a happy, neutral or sad tone of voice. In two separate tasks, participants judged word valence and ignored tone of voice or judged emotional tone of voice and ignored word valence. While native listeners outperformed non-native listeners in the word valence task, performance was comparable in the voice task. More importantly, both native and non-native listeners responded faster and more accurately when verbal and vocal emotional expressions were congruent as compared to when they were incongruent. Given that the size of the latter effect did not differ as a function of language proficiency, one can conclude that the integration of verbal and vocal emotional expressions occurs as readily in one's second language as it does in one's native language.

This work investigates whether vocal emotion expressions of full-blown discrete emotions can be recognized cross-lingually. This study will enable us to get more information regarding nature and function of emotion. Furthermore, this work will help in developing a generalized vocal emotion recognition system, which will increase the efficiency required for human-machine interaction systems. An emotional speech database was created with 140 simulated utterances (20 per emotion) per speaker, consisting of short sentences of six full-blown discrete basic emotions and one 'no-emotion' (i.e. neutral) in five native languages (not dialects) of Assam. A new feature set is proposed based on Eigenvalues of Autocorrelation Matrix (EVAM) of each frame of utterance. The Gaussian Mixture Model is used as classifier. The performance of EVAM feature set is compared at two sampling frequencies (44.1 kHz and 8.1 kHz) and with additive white noise with signal-to-noise ratios of 0 db, 5 db, 10 db and 20 db.

It has been the subject of much debate in the study of vocal expression of emotions whether posed expressions (e.g., actor portrayals) are different from spontaneous expressions. In the present investigation, we assembled a new database consisting of 1877 voice clips from 23 datasets, and used it to systematically compare spontaneous and posed expressions across 3 experiments. Results showed that (a) spontaneous expressions were generally rated as more genuinely emotional than were posed expressions, even when controlling for differences in emotion intensity, (b) there were differences between the two stimulus types with regard to their acoustic characteristics, and (c) spontaneous expressions with a high emotion intensity conveyed discrete emotions to listeners to a similar degree as has previously been found for posed expressions, supporting a dose–response relationship between intensity of expression and discreteness in perceived emotions. Our conclusion is that there are reliable differences between spontaneous and posed expressions, though not necessarily in the ways commonly assumed. Implications for emotion theories and the use of emotion portrayals in studies of vocal expression are discussed.