Feasibility of vocal emotion conversion on modulation spectrogram for simulated cochlear implants

Chatterjee et al. (2015) reported that cochlear implant (CI) listeners have difficulty recognizing vocal emotions due to the limited spectral cues provided by CI devices. Researches on vocal emotion perception of CI listeners have been studying ways to simulate responses of CI listeners by using noise-vocoded speech (NVS) as CI simulations with normal-hearing (NH) listeners. However, it is still unclear whether the results of CI simulations with NH listeners are reliable with regards to CI listeners. This study aims to clarify whether CI listeners can perceive vocal emotion the same way as NH listeners with NVS do. Vocal-emotion recognition experiments were carried out by having both NH and CI listeners listen to original emotional speech and its NVS. The results for CI listeners revealed that they recognized sadness and hot anger more easily than joy and cold anger in both original emotional speech and NVS conditions. Moreover, the results for NH listeners with NVS showed the same trend. The results sugg...

Our recent study found that cochlear implant (CI) users’ quality of life in auditory, psychological, and social functioning were predicted by vocal emotion rather than sentence recognition scores. To eventually improve vocal emotion recognition with CIs, this study investigated the acoustic cues for vocal emotion recognition by CI users with CI alone or bimodal fitting, as compared to normal-hearing (NH) listeners. Sentence duration, mean fundamental frequency (F0), and F0 range were individually normalized for emotional utterances of each talker and sentence. In two other conditions, emotional utterances were presented backward in time and with upside-down F0 contours, respectively. Perceptual results showed significant effects of subject group, cue condition, talker, and emotion. Time-reversed utterances worsened NH listeners’ recognition of all emotions except sad, while upside-down F0 contours worsened that of angry and happy. Vocal emotion recognition with CI alone only degraded with time-reversed utterances. Time-reversed utterances worsened bimodal CI users’ recognition of angry and neutral, while upside-down F0 contours worsened that of angry and happy. Bimodal CI users and NH listeners were also affected by mean F0 and F0 range normalization when recognizing happy. We conclude that natural F0 contours should be faithfully encoded with CIs for better vocal emotion recognition.

The ability to recognize emotion is important to wellbeing and building relationships with others, making this skill important in adolescence. Research investigating adolescents’ ability to recognize facial and vocal emotion expressions has reported differing conclusions about the pattern of emotion recognition across this developmental period. This systematic review aimed to clarify the pattern of recognition for facial and vocal emotion expressions, and the relationship of performance to different task and emotion expression characteristics. A comprehensive and systematic search of the literature was conducted using six databases. To be eligible for inclusion, studies had to report data for adolescents between 11 and 18 years of age and measure accuracy of the recognition of emotion cues in either the face or voice. A total of 2333 studies were identified and 47 met inclusion criteria. The majority of studies focused on facial emotion recognition. Overall, early, mid-, and late-adolescents showed a similar pattern of recognition for both facial and vocal emotion expressions with the exception of Sad facial expressions. Sex of the participant also had minimal impact on the overall recognition of different emotions. However, analysis showed considerable variability according to task and emotion expression characteristics. Future research needs to increase focus on recognition of complex emotions, and low-intensity emotion expressions as well as the influence of the inclusion of Neutral as a response option.

A cochlear implant (CI) is a hearing device introduced in the 1980s for profoundly deaf subjects who gained little or no benefit from powerful hearing aids. This device comprises an electrode array inserted in the cochlea, connected to an internal receiver, and an externally worn speech processor. The CI transforms acoustic signals into electrical currents which directly stimulate the auditory nerve. Since the early 1990s, cochlear implantation in children has been developing rapidly. Although it is still difficult to predict how a child will perform with a cochlear implant, the success of cochlear implantation can no longer be denied. In this paper, some recent papers and reports, and the results of the various Nijmegen cochlear implant studies, are reviewed. Issues about selection, examinations, surgery and the outcome are discussed. Overall, our results were comparable with those of other authors. It can be concluded that cochlear implantation is an effective treatment for postlingually deaf as well as prelingually (congenital or acquired) deaf children with profound bilateral sensorineural deafness.

Introduction: Recently, studies have suggested a role of motor symptom asymmetry on impaired emotional recognition abilities in Parkinson’s disease with a greater vulnerability in patients with a predominance of left-sided symptoms. However, none of them explored the interaction between motor symptom asymmetry and dopamine replacement therapy in different stages of the disease. Methodology: We explored the recognition of vocal emotion (i.e., emotional prosody) in 15 newly diagnosed Parkinson’s disease patients in the early stages of the disease, 15 patients in the advanced stages of the disease and 15 healthy controls. The early patients were studied in two conditions: ON and OFF dopaminergic replacement therapy and both Parkinson’s disease groups (early and advanced) were divided into two subgroups according to the asymmetry of motor symptoms. Results: The analyses revealed two patterns of results. First, as predicted, we observed a reduction in performance for the recognition of vocal emotions in patients with a predominance of left-sided symptoms as compared to both healthy controls and predominantly right-sided symptom patients. Second, in the early stages of the disease, we observed a deleterious effect of dopatherapy on the recognition of vocal emotions for the patients with left-predominant symptoms, and the inverse pattern (i.e., a positive effect of dopatherapy) for the patients with right-predominant symptoms. Conclusions: Our results bring to knowledge the differential effects of disease duration, dopaminergic replacement therapy and motor symptom asymmetry on vocal emotion recognition in Parkinson’s disease.

This work investigates whether vocal emotion expression of discrete emotion (i) can be recognized cross- lingually, (ii) of surprise, which is actually a cognitive component that could be present with any emotion, can also be recognized as a distinct emotion. This study will enable us to get more information regarding nature and function of emotion. Furthermore, this work will help in developing a generalized vocal emotion recognition system, which will increase the efficiency of human-machine interaction systems. In this work, an emotional speech database consisting of short sentences of five full-blown basic emotions, full-blown surprise emotion and 'no- emotion' (i.e. neutral) is created with 140 acted utterances per speaker of five native languages of Assam. This database is validated by a Listening Test. A new feature set is proposed based on Eigen Values of Autocorrelation Matrix (EVAM) of each frame of the speech signal. The Gaussian Mixture Model (GMM) is used as the classifier. The performance of the proposed feature set is compared with Mel Frequency Cepstral Coefficients (MFCCs) at sampling frequencies of 44.1 kHz and 8.1 kHz and with additive white noise of 5 db and 0 db Signal-to- Noise Ratios (SNRs) under matched noise condition.

Clinical tests of cochlear implant (CI) outcomes in sentence recognition cannot fully reflect CI users' self-reported quality of life (QoL). Here, vocal emotion recognition scores, speech reception thresholds (SRTs), and demographic factors were tested as predictors of QoL scores assessed with the Nijmegen Cochlear Implant Questionnaire in postlingually deafened adult CI users. After correction for multiple comparisons, vocal emotion recognition scores were significantly correlated with QoL scores in all subdomains (social interaction, self-esteem, etc.), while SRTs and duration of CI use were not. Vocal emotion recognition may thus be used in clinic to accurately and broadly predict QoL with CIs.

Wind noise can be a nuisance or a debilitating masker for cochlear implant users in outdoor environments. Previous studies indicated that wind noise at the microphone/hearing aid output had high levels of low-frequency energy and the amount of noise generated is related to the microphone directionality. Currently, cochlear implants only offer either directional microphones or omnidirectional microphones for users at-large. As all cochlear implants utilize pre-emphasis filters to reduce low-frequency energy before the signal is encoded, effective wind noise reduction algorithms for hearing aids might not be applicable for cochlear implants. The purposes of this study were to investigate the effect of microphone directionality on speech recognition and perceived sound quality of cochlear implant users in wind noise and to derive effective wind noise reduction strategies for cochlear implants. A repeated-measure design was used to examine the effects of spectral and temporal masking created by wind noise recorded through directional and omnidirectional microphones and the effects of pre-emphasis filters on cochlear implant performance. A digital hearing aid was programmed to have linear amplification and relatively flat in-situ frequency responses for the directional and omnidirectional modes. The hearing aid output was then recorded from 0 to 360° at flow velocities of 4.5 and 13.5 m/sec in a quiet wind tunnel. Sixteen postlingually deafened adult cochlear implant listeners who reported to be able to communicate on the phone with friends and family without text messages participated in the study. Cochlear implant users listened to speech in wind noise recorded at locations that the directional and omnidirectional microphones yielded the lowest noise levels. Cochlear implant listeners repeated the sentences and rated the sound quality of the testing materials. Spectral and temporal characteristics of flow noise, as well as speech and/or noise characteristics before and after the pre-emphasis filter, were analyzed. Correlation coefficients between speech recognition scores and crest factors of wind noise before and after pre-emphasis filtering were also calculated. Listeners obtained higher scores using the omnidirectional than the directional microphone mode at 13.5 m/sec, but they obtained similar speech recognition scores for the two microphone modes at 4.5 m/sec. Higher correlation coefficients were obtained between speech recognition scores and crest factors of wind noise after pre-emphasis filtering rather than before filtering. Cochlear implant users would benefit from both directional and omnidirectional microphones to reduce far-field background noise and near-field wind noise. Automatic microphone switching algorithms can be more effective if the incoming signal were analyzed after pre-emphasis filters for microphone switching decisions.

In order to improve students’ learning effect, more and more universities favor foreign language teachers who are native speakers of English. Based on the analysis and summary of the research status of emotion recognition, this paper proposes that, in college English classroom teaching, foreign language teachers can reduce the communication barriers with Chinese students through emotion recognition. Based on literature review and actual situation investigation, this study identified four influencing factors on emotion recognition of foreign language teachers, namely, interactive action, facial expression, vocal emotion, and body posture. In our opinion, in the teaching process, teachers can adjust the four factors of emotion recognition to achieve better teaching effect. Further, improve students’ learning efficiency. Analytic Hierarchy Process (AHP) is chosen as the research method in this study. After building the analysis model, we collected the questionnaire using the Questionnaire Star, and finally got 12 valid data. After determining the importance of different factors by pairwise comparison, we draw the following conclusions: the influence degree of emotion recognition factors of foreign language teachers is in descending order, interactive action (43%), facial expression (28%), vocal emotion (21%), and body posture (9%). Our research adds to the body of knowledge on emotion recognition among college English teachers. Furthermore, this research assists students in improving their grasp of course content based on the emotions of foreign English lecturers. Based on the findings, we recommend that foreign language teachers in college English classrooms alter their interactive behaviors, facial expressions, and vocal emotions in response to various instructional materials and emphases.

Intervention for a Child with Auditory Neuropathy/Dys-synchrony

Speech recognition outcomes with a cochlear implant (CI) are highly variable. One factor suggested to correlate with CI-aided speech recognition is frequency-to-place mismatch, or the discrepancy between the natural tonotopic organization of the cochlea and the electric frequency allocation of the CI electrodes within the patient's cochlea. To evaluate the association between frequency-to-place mismatch and speech recognition outcomes in a large cohort of postlingually deafened adult CI users, while controlling for various clinical factors known to be associated with those outcomes. This retrospective cohort study used data from a CI program at a tertiary medical center and included CIs from postlingually deafened adult CI users. After excluding patients whose data were not logged, patients with implantations occurring between 2016 and 2023 were included in the analysis. The data were extracted in November 2023. Results of the Consonant-Nucleus-Consonant (CNC) monosyllabic word recognition test measured in the CI-aided alone condition 1 month, 3 months, 6 months, and 12 months after activation served as the main outcome. The independent variables included frequency-to-place mismatch, electrode array type, mean modiolar distance, electrode position, age at implantation, biological sex, contralateral hearing abilities, time since CI activation (test interval), and daily device use. In 498 CIs from 447 postlingually deafened adults (mean [SD] age, 63.1 [17.1] years; 271 [54.4%] CIs with male users), frequency-to-place mismatch was negatively correlated with CI-aided speech recognition outcomes, but the association was only significant for precurved electrode arrays and not straight electrode arrays. In the linear mixed effects model for straight electrode arrays, only test interval (β = 1.14 [95% CI, 0.90-1.38]) and daily device use (β = 0.90 [95% CI, 0.42-1.38]) were correlated with the improvement of word recognition over the first year of device use. In the model for precurved electrode arrays, mismatch at 1500 Hz (β = -0.011 [95% CI, -0.011 to -0.006]), scalar location (β = 16.37 [95% CI, 9.01 to 23.74]), test interval (β = 1.18 [95% CI, 1.18-1.41]) and daily device use (β = 1.65 [95% CI, 1.15-2.14]) all were significantly associated with the improvement of word recognition over the first year of device use. In this cohort study of postlingually deafened adult CI users, including both straight and precurved electrode arrays, daily device use and time since CI activation were found to be significantly associated with improved CI-aided speech recognition outcomes. Frequency-to-place mismatch at 1500 Hz and scalar location were associated with word recognition only for precurved arrays. These findings suggest that, for patients with straight arrays, any interference produced by frequency-to-place mismatch may be overcome by adaptation, which could be supported by daily device use. However, for patients with precurved arrays, daily device use may not be sufficient to completely overcome electrode placement factors, including scalar location and mismatch.

The ability to recognize speech that is degraded spectrally is a critical skill for successfully using a cochlear implant (CI). Previous research has shown that toddlers with normal hearing can successfully recognize noise-vocoded words as long as the signal contains at least eight spectral channels [Newman and Chatterjee. (2013). J. Acoust. Soc. Am. 133(1), 483-494; Newman, Chatterjee, Morini, and Remez. (2015). J. Acoust. Soc. Am. 138(3), EL311-EL317], although they have difficulty with signals that only contain four channels of information. Young children with CIs not only need to match a degraded speech signal to a stored representation (word recognition), but they also need to create new representations (word learning), a task that is likely to be more cognitively demanding. Normal-hearing toddlers aged 34 months were tested on their ability to initially learn (fast-map) new words in noise-vocoded stimuli. While children were successful at fast-mapping new words from 16-channel noise-vocoded stimuli, they failed to do so from 8-channel noise-vocoded speech. The level of degradation imposed by 8-channel vocoding appears sufficient to disrupt fast-mapping in young children. Recent results indicate that only CI patients with high spectral resolution can benefit from more than eight active electrodes. This suggests that for many children with CIs, reduced spectral resolution may limit their acquisition of novel words.

Determine the impact of cochlear implantation on quality of life (QOL) and determine the correlation between QOL and speech recognition ability. Two authors independently searched PubMed, Medline, Scopus, and the Cumulative Index to Nursing and Allied Health Literature to identify studies reporting hearing-specific or cochlear implant (CI)-specific QOL outcomes before and after cochlear implantation, and studies reporting correlations between QOL and speech recognition after cochlear implantation. Data from the included articles were obtained independently by two authors. Standardized mean difference (SMD) for each measure and pooled effects were determined to assess improvement in QOL before and after cochlear implantation. From 14 articles with 679 CI patients who met the inclusion criteria, pooled analyses of all hearing-specific QOL measures revealed a very strong improvement in QOL after cochlear implantation (SMD = 1.77). Subset analysis of CI-specific QOL measures also showed very strong improvement (SMD = 1.69). Thirteen articles with 715 patients met the criteria to evaluate associations between QOL and speech recognition. Pooled analyses showed a low positive correlation between hearing-specific QOL and word recognition in quiet (r = 0.213), sentence recognition in quiet (r = 0.241), and sentence recognition in noise (r = 0.238). Subset analysis of CI-specific QOL showed similarly low positive correlations with word recognition in quiet (r = 0.213), word recognition in noise (r = 0.241), and sentence recognition in noise (r = 0.255). Using hearing-specific and CI-specific measures of QOL, patients report significantly improved QOL after cochlear implantation. However, widely used clinical measures of speech recognition are poor predictors of patient-reported QOL with CIs. Laryngoscope, 128:982-990, 2018.

Previous research in cochlear implant (CI) recipients with bilateral severe-to-profound sensorineural hearing loss showed improvements in speech recognition in noise using remote wireless microphone systems. However, to our knowledge, no previous studies have addressed the benefit of these systems in CI recipients with single-sided deafness. The objective of this study was to evaluate the potential improvement in speech recognition in noise for distant speakers in single-sided deaf (SSD) CI recipients obtained using the digital remote wireless microphone system, Roger. In addition, we evaluated the potential benefit in normal hearing (NH) participants gained by applying this system. Speech recognition in noise for a distant speaker in different conditions with and without Roger was evaluated with a two-way repeated-measures design in each group, SSD CI recipients, and NH participants. Post hoc analyses were conducted using pairwise comparison t-tests with Bonferroni correction. Eleven adult SSD participants aided with CIs and eleven adult NH participants were included in this study. All participants were assessed in 15 test conditions (5 listening conditions × 3 noise levels) each. The listening conditions for SSD CI recipients included the following: (I) only NH ear and CI turned off, (II) NH ear and CI (turned on), (III) NH ear and CI with Roger 14, (IV) NH ear with Roger Focus and CI, and (V) NH ear with Roger Focus and CI with Roger 14. For the NH participants, five corresponding listening conditions were chosen: (I) only better ear and weaker ear masked, (II) both ears, (III) better ear and weaker ear with Roger Focus, (IV) better ear with Roger Focus and weaker ear, and (V) both ears with Roger Focus. The speech level was fixed at 65 dB(A) at 1 meter from the speech-presenting loudspeaker, yielding a speech level of 56.5 dB(A) at the recipient's head. Noise levels were 55, 65, and 75 dB(A). Digitally altered noise recorded in school classrooms was used as competing noise. Speech recognition was measured in percent correct using the Oldenburg sentence test. In SSD CI recipients, a significant improvement in speech recognition was found for all listening conditions with Roger (III, IV, and V) versus all no-Roger conditions (I and II) at the higher noise levels (65 and 75 dB[A]). NH participants significantly benefited from the application of Roger in noise for higher levels, too. In both groups, no significant difference was detected between any of the different listening conditions at 55 dB(A) competing noise. There was also no significant difference between any of the Roger conditions III, IV, and V across all noise levels. The application of the advanced remote wireless microphone system, Roger, in SSD CI recipients provided significant benefits in speech recognition for distant speakers at higher noise levels. In NH participants, the application of Roger also produced a significant benefit in speech recognition in noise.

To evaluate the quality of life after cochlear implantation in adult cochlear implant recipients and to assess the applicability of Nijmegen Cochlear implant Questionnaire (NCIQ). Thirty-two postlingually deaf patients, 19 males and 13 females, underwent cochlear implantation with the age at operation of 35.27 (18-66) during the period from 1996 to the end of 2006. NCIQ was used to investigate the application of the equipment, hearing ability, and emotional exchange ability. All of the patients used their cochlear implants for 16 h every day on average with an experience in using such implant of more than 6 months. Improvement was seen in sound recognition, speech recognition, confidence, daily life, social exchange, and emotional behavior ability at different degrees. The improvement of confidence was very obvious. The speech communication ability of those with a hearing deprivation time longer than 5 years was lower than that of those with a shorter hearing deprivation time. Age and educational level did not influence the efficiency. Cochlear implantation is a cost effective intervention in the postlingually deaf adults. NCIQ test can be used to evaluate the quality of life among postlingually deaf adult cochlear implant recipients in China.