Mutual information based speech intelligibility prediction and its application to hearing aid fitting

Aided cortical response, speech intelligibility, consonant perception and functional performance of young children using conventional amplification or nonlinear frequency compression

This article presents an overview of twelve existing objective speech quality and intelligibility prediction tools. Two classes of algorithms are presented, namely intrusive and non-intrusive, with the former requiring the use of a reference signal, while the latter does not. Investigated metrics include both those developed for normal hearing listeners, as well as those tailored particularly for hearing impaired (HI) listeners who are users of assistive listening devices (i.e., hearing aids, HAs, and cochlear implants, CIs). Representative examples of those optimized for HI listeners include the speech-to-reverberation modulation energy ratio, tailored to hearing aids (SRMR-HA) and to cochlear implants (SRMR-CI); the modulation spectrum area (ModA); the hearing aid speech quality (HASQI) and perception indices (HASPI); and the PErception MOdel - hearing impairment quality (PEMO-Q-HI). The objective metrics are tested on three subjectively-rated speech datasets covering reverberation-alone, noise-alone, and reverberation-plus-noise degradation conditions, as well as degradations resultant from nonlinear frequency compression and different speech enhancement strategies. The advantages and limitations of each measure are highlighted and recommendations are given for suggested uses of the different tools under specific environmental and processing conditions.

Objective: This study aimed to 1) investigate the influence of nonlinear frequency compression (NLFC) in hearing aids on intelligibility of speech produced by children with hearing loss; and 2) examine whether clinicians’ or parents’ judgments might be correlated with those of inexperienced listeners. Methods: Twenty-seven adult listeners with normal hearing who reported no experience listening to speech produced by people with hearing loss were asked to judge the intelligibility of speech samples of eight hearingimpaired children under four aided conditions. Also, the parents and the clinicians who provided services to the children provided ratings. The children were enrolled in a four-period multi-site trial that was aimed to compare the effects of conventional processing with NLFC in hearing aids on children’s performance. In that study, the children were familiarized with each of four hearing-aid setting for at least six weeks before they were evaluated using a range of tests, including the production of 20 sentences. The current study used the recorded sentences as stimuli for intelligibility judgments. Each listener heard sentences produced by two child-talkers, 40 from each talker. The stimuli were presented to listeners at 65 dB SPL via headphones. Four child-talkers received ratings from eight listeners and four from seven listeners. Results: Group-level results indicate that speech intelligibility was rated to be better by inexperienced listeners when children used NLFC than when they did not. Three child-talkers showed a significant advantage with NLFC activation. These results are consistent with the estimated audible bandwidth of hearing aids for individual talkers. Significant positive correlations for intelligibility ratings between inexperienced listeners and clinicians were found, but neither correlated with ratings from parents. Conclusions: The use of NLFC improved intelligibility of speech produced by children, on average, as rated by inexperienced listeners. Clinicians’ judgment of children’s speech production is a clinically viable tool for evaluating the effectiveness of amplification for children.

The primary goal of nonlinear frequency compression (NFC) and other frequency-lowering strategies is to increase the audibility of high-frequency sounds that are not otherwise audible with conventional hearing aid (HA) processing due to the degree of hearing loss, limited HA bandwidth, or a combination of both factors. The aim of the present study was to compare estimates of speech audibility processed by NFC with improvements in speech recognition for a group of children and adults with high-frequency hearing loss. Monosyllabic word recognition was measured in noise for 24 adults and 12 children with mild to severe sensorineural hearing loss. Stimuli were amplified based on each listener's audiogram with conventional processing (CP) with amplitude compression or with NFC and presented under headphones using a software-based HA simulator. A modification of the speech intelligibility index (SII) was used to estimate audibility of information in frequency-lowered bands. The mean improvement in SII was compared with the mean improvement in speech recognition. All but 2 listeners experienced improvements in speech recognition with NFC compared with CP, consistent with the small increase in audibility that was estimated using the modification of the SII. Children and adults had similar improvements in speech recognition with NFC. Word recognition with NFC was higher than CP for children and adults with mild to severe hearing loss. The average improvement in speech recognition with NFC (7%) was consistent with the modified SII, which indicated that listeners experienced an increase in audibility with NFC compared with CP. Further studies are necessary to determine whether changes in audibility with NFC are related to speech recognition with NFC for listeners with greater degrees of hearing loss, with a greater variety of compression settings, and using auditory training.

To evaluate sound quality preferences of participants wearing hearing aids with different strengths of nonlinear frequency compression (NFC) processing versus no NFC processing. Two analysis methods, one without and one with a qualifier as to the magnitude of preferences, were compared for their percent agreement to differentiate a small difference in perceived sound quality as a result of applied NFC processing. A single-blind design was used with participants unaware of the presence or strength of NFC processing (independent variable). The National Acoustic Laboratories-Nonlinear 2 (NAL-NL2) prescription of amplification was chosen because audibility is intentionally not prescribed in the presence of larger sensorineural hearing loss thresholds. A lack of prescribed audibility, when present, was deemed an objective qualifier for NFC. NFC is known to improve the input bandwidth available to listeners when high-frequency audibility is not otherwise available and increasing strengths of NFC were examined. Experimental condition 3 (EC3) was stronger than the manufacturer default (EC2). More aggressive strengths (e.g., EC4 and EC5), however, were expected to include excessive distortion and even reduce the output bandwidth that had been prescribed as audible by NAL-NL2 (EC1). A total of 14 male Veterans with severe high-frequency sensorineural hearing loss. Participant sound quality preference ratings (dependent variable) without a qualifier as to the magnitude of preference were analyzed based on binomial probability theory, as is traditional with paired comparison data. The ratings with a qualifier as to the magnitude of preference were analyzed based on the nonparametric statistic of the Wilcoxon signed rank test. The binomial probability analysis method identified a sound quality preference as well as the nonparametric probability test method. As the strength of NFC increased, more participants preferred the EC with less NFC. Fourteen of 14 participants showed equal preference between EC1 and EC2 perhaps, in part, because EC2 showed no objective improvement in audibility for six of the 14 participants (42%). Thirteen of the 14 participants showed no preference between NAL-NL2 and EC3, but all participants had an objective improvement in audibility. With more NFC than EC3, more and more participants preferred the other EC with less NFC in the paired comparison. By referencing the recommended sensation levels of amplitude compression (e.g., NAL-NL2) in the ear canal of hearing aid wearers, the targeting of NFC parameters can likely be optimized with respect to improvements in effective audibility that may contribute to speech recognition without adversely impacting sound quality. After targeting of NFC parameters, providers can facilitate decisions about the use of NFC parameters (strengths of processing) via sound quality preference judgments using paired comparisons.

Speech intelligibility is usually assessed using subjective tests, but several objective measures of speech intelligibility have also been proposed. This study compares intelligibility of IEEE sentences in collocated speech-shaped noise, recorded through a low-cost amplifier and three current-issue NHS hearing aids with single-channel noise reduction settings switched on and off. Results from twenty-one normal hearing listeners indicates that single-channel noise reduction algorithms can significantly improve intelligibility of noisy speech (p < 0.05), but no differences can be seen between different hearing aid models currently available on the NHS. The low-cost, off-the shelf hearing amplifier performed significantly worse than the NHS prescribed devices. Three objective speech intelligibility measures were applied to the recordings. None of the objective metrics was found to accurately predict the outcomes seen in this study for all hearing aid conditions, though general trends can be predicted (r = 0.83 overall, but for any discrete signal-to-noise ratio across all conditions, maximum r = 0.64). Machine-learning approaches were used to refit parameters from the best performing of these metrics. Although performance was improved, problems in predicting particular conditions remain, suggesting that objective measures do not encompass all relevant factors. [Action on Hearing Loss (England/Wales Registered Charity Number 207720).]

Speech intelligibility prediction of noisy and processed noisy speech is important in a number of application domains such as hearing instruments and forensics. Most available objective intelligibility measures employ either a signal-to-noise ratio (SNR)-based or correlation-based comparison between frequency bands of the clean and the processed speech. In this paper, we approach the speech intelligibility prediction from the angle of information theory and show that an information theoretic concept provides a unified viewpoint on both the SNR and the correlation based approaches. Two objective intelligibility measures are introduced based on estimated mutual information between the clean speech and the processed speech in the time and the frequency subband domain. Our proposed measures show high correlation with subjective intelligibility measure (i.e. word correct scores) and comparative results with the short-term objective intelligibility measure (STOI).

The performance of Kalman filters (KF) in speech enhancement is highly dependent on the estimation accuracy of linear predictive coefficients (LPCs). To address the significant estimation bias of LPCs in complex noisy environments, this paper proposes a time-frequency dual-domain Conformer-based Kalman filter speech enhancement method—CAKFNet. The model adopts an encoder-decoder architecture: during the encoding phase, a Dilated DenseNet module is introduced to exponentially expand the receptive field and enhance local feature continuity, thereby achieving multi-scale feature aggregation. Subsequently, dual Conformer modules in both time and frequency domains jointly model the long-term dependencies of speech signals and local convolutional features, fully capturing spatio-temporal correlations. The decoding stage employs a decoupled dual-branch architecture to separately regress the LPC power spectrum (LPC-PS) of clean speech and noise. Building upon this, an autocorrelation matrix is generated via inverse Fourier transform. The Levinson-Durbin recursive algorithm is then employed to compute the clean speech and noise LPC coefficients required for the Augmented Kalman Filter (AKF), along with their prediction error variance, ultimately achieving speech enhancement. The proposed method undergoes systematic experimental evaluation on the publicly available VoiceBank + DEMAND dataset. Results demonstrate that CAKFNet effectively reduces spectral distortion in LPC estimation. Objective metrics, including PESQ and STOI, show significant improvements over baseline methods, while subjective listening tests confirm substantial enhancements in speech quality and intelligibility.

The primary objective of this study is to evaluate the benefits of nonlinear frequency compression (NLFC) hearing aids in the nonimplanted ears of children with unilateral cochlear implants (CIs). It is hypothesized that speech perception performance will benefit from complementary auditory cues provided by the CI and the hearing aid, particularly with the increased access to high-frequency sounds provided by NLFC. Eleven children using unilateral CIs with usable residual hearing in the nonimplanted ears were enrolled in the study and fitted with NLFC hearing aids. The test protocol included consonant-nucleus-consonant words in quiet, the Hearing in Noise Test sentences presented in speech noise and two-talker maskers, and a consonant identification task. Subjects were tested in a CI-alone condition as well as bimodally, with and without NLFC enabled. The results support previous work in adults and children, demonstrating the beneficial effects of bimodal listening. Frequency compression did not significantly affect performance for the children enrolled in this study, although some preferred using NLFC. The results yield suggestions regarding test methods for pediatric bimodal listeners, and considerations regarding validation and audibility of the compressed signal. Hearing aid use in the contralateral ear of unilaterally implanted children is beneficial. Children and young adults who are fitted bimodally should be tested both in quiet and in complex listening situations to determine bimodal benefit. In the current test battery, the inclusion of frequency compression in the hearing aid fitting does not seem to provide significant improvement beyond standard hearing aid fittings or any bilateral interference symptoms for this group of bimodal listeners.

To determine whether digital signal processing (DSP) hearing aids produce conducted radio frequency interference that can affect the use of personal FM systems, to quantify the nature of any such interference, and to discuss practical remedies. Sixteen DSP hearing aids were used. Measurements were made of the spectral characteristics of any conducted radio frequency interference produced by each aid with FM shoe and 40 cm direct audio input (DAI) lead when the DAI facility was enabled. Measurements were made with the aid, shoe, and lead inside an electrically screened chamber. The effect of DAI lead length was also investigated with one of the hearing aids. Finally, some subjective listening tests were carried out by using different FM systems coupled to a number of the aids. All but four of the DSP hearing aids tested produced readily measurable interference, with some much worse than others. Levels of interference were high enough with some hearing aids to be likely to significantly impede signal perception when the radio frequency of the interference coincided with the radio frequency of the FM system. This usually occurs intermittently as a result of the processor design of most DSP hearing aids. The listening tests suggested that when personal FM systems are in use with some DSP hearing aids, the interference would be audible, unpleasant, and detrimental to audio quality. DSP hearing aids without low electromagnetic interference processors should not be fitted to clients if personal FM systems are expected to be used. Manufacturers of DSP aids should be encouraged to use low electromagnetic interference processors in their DSP hearing aid design. Meanwhile, FM systems should be used with DSP hearing aids in such a way as to ensure high received radio signal levels, and FM receivers should be switched off when not in use.

Objectives: To determine whether non-linear frequency compression (NLFC) is effective for hearing-impaired adults in a clinical setting. To determine whether benefit from NLFC is related to duration of NLFC experience or severity of high-frequency hearing loss. Design: Participants were fitted with Phonak frequency compression hearing aids as part of their standard clinical care, using the manufacturer's default fitting settings. Participants had been using NLFC for between 1 and 121 weeks at the time of testing. Speech recognition thresholds in noise and consonant recognition in quiet were measured with and without NLFC enabled. Study sample: Forty-six experienced adult hearing-aid users. Results: Consonant recognition in quiet, but not speech recognition in noise was significantly better with NLFC enabled. There was no significant correlation between duration of frequency compression experience and benefit. Benefit for consonant recognition was negatively correlated with mean audiometric thresholds from 2–6 kHz. Conclusions: NLFC was beneficial for consonant recognition but not speech recognition in noise. There was no evidence to support the idea that a long period of acclimatization is necessary to gain full benefit. The relation between benefit and high-frequency thresholds might be explained by the poor audibility of compressed information for some listeners with severe loss.

In conventional single-channel speech enhancement, typically the noisy spectral amplitude is modified while the noisy phase is used to reconstruct the enhanced signal. Several recent attempts have shown the effectiveness of utilizing an improved spectral phase for phase-aware speech enhancement and consequently its positive impact on the perceived speech quality. In this paper, we present a harmonic phase estimation method relying on fundamental frequency and signal-to-noise ratio (SNR) information estimated from noisy speech. The proposed method relies on SNR-based time-frequency smoothing of the unwrapped phase obtained from the decomposition of the noisy phase. To incorporate the uncertainty in the estimated phase due to unreliable voicing decision and SNR estimate, we propose a binary hypothesis test assuming speech-present and speech-absent classes representing high and low SNRs. The effectiveness of the proposed phase estimation method is evaluated for both phase-only enhancement of noisy speech and in combination with an amplitude-only enhancement scheme. We show that by enhancing the noisy phase both perceived speech quality as well as speech intelligibility are improved as predicted by the instrumental metrics and justified by subjective listening tests.

Individuals with hearing impairments face challenges in their ability to comprehend speech, particularly in noisy environments. This study explores the effectiveness of audio-visual speech enhancement (AVSE) in improving the intelligibility of vocoded speech in cochlear implant (CI) simulations. We propose a speech enhancement framework called Self-Supervised Learning-based AVSE (SSL-AVSE), which uses visual cues such as lip and mouth movements along with corresponding speech. Features are extracted using the AV-HuBERT model and refined through a bidirectional LSTM. Experiments were conducted using the Taiwan Mandarin speech with video (TMSV) dataset. Objective evaluations showed improvements in PESQ from 1.43 to 1.67 and in STOI from 0.70 to 0.74. NCM scores increased by up to 87.2% over the noisy baseline. Subjective listening tests further demonstrated maximum gains of 45.2% in speech quality and 51.9% in word intelligibility. SSL-AVSE consistently outperforms audio-only speech enhancement (AOSE) and conventional AVSE baselines. Listening tests with statistically significant results confirm its effectiveness. In addition to its strong performance, SSL-AVSE demonstrates cross-lingual generalization: although it was pretrained on English data, it performs effectively on Mandarin speech. This finding highlights the robustness of the features extracted by a pretrained foundation model and their applicability across languages. To the best of our knowledge, no prior work has explored the application of AVSE to CI simulations. This study provides the first evidence that incorporating visual information can significantly improve the intelligibility of vocoded speech in CI scenarios.

BackgroundHigh-frequency consonants form an area of difficulty among children with hearing impairment. Many technological setups have emerged in order to improve their audibility, among them nonlinear frequency compression (NLFC). Therefore, the following study was carried out in order to evaluate change in production of high-frequency fricatives by hearing-impaired children utilizing NLFC hearing aids. This was through spectrographic analysis of F2 onset of transition elicited by fricatives in a prevocalic context, and results were compared to typically developing (TD) children. A change in mid-frequency sounds due to high-frequency compression was also tested through production of sustained vowels in isolation.ResultsA preliminary prospective case-control study involved 9 patients with NLFC hearing aids, evaluated at 1- and at 6-month post-fitting, without receiving speech therapy. Spectrographic analysis of F2 formant onset in [a] and [i] vowels when combined with [s] and [f] fricatives in a CV (consonant-vowel) pattern, presented auditorily and audio-visually, was observed and analyzed. One patient was excluded due to noncompliance in attendance. In the 8 patients who completed the study, very high-frequency level of F2 onset in CV was notable in second versus first evaluation in NLFC group, to the extent of surpassing the typical value in TD control group with [i] vowel. Contrastingly, F2 of sustained vowels in isolation took a trend of a lower value after 6 months of NLFC hearing aid use, as compared to the first evaluation. Consequently, the gap between sustained vowel F2 level in TD and NLFC groups widened.ConclusionMean high-frequency range of F2 onset of transition in CV patterns of voiceless fricatives [s] and [f] with [i] and [a] vowels 6 months after NLFC fitting reflects an enhancement in the frequency range production of these sounds. A shift toward a lower F2 frequency range in sustained mid-frequency vowels, on the other hand, remains to be carefully investigated as a sequel of nonlinear frequency compression in the Egyptian hearing-impaired children.

The purpose of this research was twofold: firstly, to develop a music perception test (MPT) for hearing-aid users, and secondly, to evaluate the influence of non-linear frequency compression (NFC) on music perception with the use of the self-compiled test. This article focuses on the description of the development and validation of the MPT. To date, the main direction in frequency-lowering hearing-aid studies has been in relation to speech perception abilities. As hearing-aid technology has improved, interest has grown in musical perception as a dimension that could improve hearing-aid users' quality of life. The MPT was designed to evaluate different aspects of rhythm, timbre, pitch and melody. The development of the MPT could be described as design-based. Phase 1 of the study included test development and recording, while phase 2 entailed presentation of stimuli to normal hearing listeners (n = 15) and hearing-aid users (n = 4). Based on the findings of phase 2, item analysis was performed to eliminate or change stimuli that resulted in high error rates. During phase 3 the adapted version of the test was performed on a smaller group of normal hearing listeners (n = 4) and 20 hearing-aid users. Results proved that adults with normal hearing as well as adults using hearing aids were able to complete all the sub-tests of the MPT, although hearing-aid users scored lower on the various sub-tests than normal hearing listeners. For the rhythm section of the MPT normal hearing listeners scored on average 93.8% versus 75.5% of hearing-aid users; for the timbre section the scores were 83% versus 62.3% respectively. Normal hearing listeners obtained an average score of 86.3% for the pitch section and 88.2% for the melody section, compared with the 70.8% and 61.9% respectively obtained by hearing-aid users. This implies that the MPT can be used successfully for assessment of music perception in hearing-aid users within the South African context and may therefore result in more effective hearing-aid fittings taking place. The test can be used as a counselling tool to assist audiologists and patients in understanding the problems they experience regarding music perception, and might be used for future musical training in areas where participants experience problems in customising individual fittings.