Toward a Model of Auditory-Visual Speech Intelligibility

Audio-visual speech perception in infants and toddlers with Down syndrome, fragile X syndrome, and Williams syndrome

<p>Objectives: Elementary school classrooms that meet the acoustic requirements for near-optimum speech recognition are extremely scarce. Poor classroom acoustics may become a barrier to speech understanding as children enter school. The purpose of this study was threefold: 1) to quantify the extent to which reverberation, lexical difficulty, and presentation mode affect speech recognition in noise, 2) to examine to what extent auditory-visual (AV) integration assists with the recognition of speech in noisy and reverberant environments typical of elementary school classrooms, 3) to understand the relationship between developing mechanisms of multisensory integration and the concurrently developing linguistic and cognitive abilities.</p> <p>Design: Twenty-seven typically developing children and 9 young adults participated. Participants repeated short sentences reproduced by 10 speakers on a 30” HDTV and/or over loudspeakers located around the listener in a simulated classroom environment. Signal-to-noise ratio (SNR) for 70 (SNR70) and 30 (SNR30) percent correct performance were measured using an adaptive tracking procedure. Auditory-visual integration was assessed via the SNR difference between AV and auditory-only (AO) conditions, labeled speech-reading benefit (SRB). Linguistic and cognitive aptitude was assessed using the NIH-Toolbox: Cognition Battery (NIH-TB: CB).</p> <p>Results: Children required more favorable SNRs for equivalent performance when compared to adults. Participants benefited from the reduction in lexical difficulty, and in most cases the reduction in reverberation time. Reverberation affected children’s speech recognition in AO condition and adults in AV condition. At SNR30, SRB was greater than that at SNR70. Adults showed marginally significant increase in AV integration relative to children. Adults also showed increase in SRB for lexically hard versus easy words, at high level of reverberation. Development of linguistic and cognitive aptitude accounts for approximately 35% of the variance in AV integration, with crystalized and fluid cognition composite scores identified as strongest predictors.</p> <p>Conclusions: The results of this study add to the body of evidence in support of children requiring more favorable SNRs to perform the same speech recognition tasks as adults in simulated listening environments akin to school classrooms. Our findings shed light on the development of AV integration for speech recognition in noise and reverberation during the school years, and provide insight into the balance of cognitive and linguistic underpinnings necessary for AV integration of degraded speech.</p>

Artificial auditory devices such as cochlear implants (CIs) and auditory brainstem implants (ABIs) have become standard means to manage profound sensorineural hearing loss. However, because of their structural limitations compared to the cochlea and the cochlear nucleus, the generated auditory sensations are still imperfect. Recipients need postoperative auditory rehabilitation. To improve these rehabilitation programs, this study evaluated the effects of bimodal (audio–visual) training under seven experimental conditions of distorted speech sound, named noise-vocoded speech sound (NVSS), which is similarly processed with a speech processor of CI/ABI. Word intelligibilities under the seven conditions of two-band noise-vocoded speech were measured for auditory (A), visual (V) and auditory–visual (AV) modalities after a few hours of bimodal (AV) training. The experiment was performed with 56 subjects with normal hearing. Performance of A and AV word recognition was significantly different under the seven auditory conditions. The V word intelligibility was not influenced by the condition of combined auditory cues. However, V word intelligibility was correlated with AV word recognition under all frequency conditions. Correlation between A and AV word intelligibilities was ambiguous. These findings suggest the importance of visual cues in AV speech perception under extremely degraded auditory conditions, and underscore the importance of the possible effectiveness of bimodal audio–visual training in postoperative rehabilitation for patients with postlingual deafness who have undergone artificial auditory device implantation.

This study examined the effect of auditory-visual (AV) speech stimuli on working memory in older adults with poorer-hearing (PH) in comparison to age- and education-matched older adults with better hearing (BH). Participants completed a working memory n-back task (0- to 2-back) in which sequences of digits were presented in visual-only (i.e., speech-reading), auditory-only (A-only), and AV conditions. Auditory event-related potentials (ERP) were collected to assess the relationship between perceptual and working memory processing. The behavioral results showed that both groups were faster in the AV condition in comparison to the unisensory conditions. The ERP data showed perceptual facilitation in the AV condition, in the form of reduced amplitudes and latencies of the auditory N1 and/or P1 components, in the PH group. Furthermore, a working memory ERP component, the P3, peaked earlier for both groups in the AV condition compared to the A-only condition. In general, the PH group showed a more robust AV benefit; however, the BH group showed a dose-response relationship between perceptual facilitation and working memory improvement, especially for facilitation of processing speed. Two measures, reaction time and P3 amplitude, suggested that the presence of visual speech cues may have helped the PH group to counteract the demanding auditory processing, to the level that no group differences were evident during the AV modality despite lower performance during the A-only condition. Overall, this study provides support for the theory of an integrated perceptual-cognitive system. The practical significance of these findings is also discussed.

Visual--auditory (VA) and visual--proprioceptive (VP) localization conflict paradigms were varied to explore the comparability of the conflict situations. In experiment 1 various attempts were made to decrease the dominance of visual information over proprioceptive and auditory target information. Pairing auditory with proprioceptive information against conflicting visual information did not lessen the visual dominance, nor did dimming the visual field. A 'cognitive' manipulation, in which the subject was led to doubt the reliability of the visual information, reduced visual dominance over audition but not visual dominance over proprioception. This difference between the two conflict situations was further explored and corroborated in experiment 2. In experiment 3 no attempt was made to lead the subject to believe that paired discrepant targets were related, and the visual dominance of audition was strong while the visual dominance of proprioception did not occur. The apparent differences between the VA and VP conflict situations are discussed with regard to the feability of generating a unitary explanation of localization conflict results. Several further factors are discussed that must be explored before undertaking such a unitary formulation.

Cochlear implant (CI) users in tone language environments report great difficulty in perceiving lexical tone. This study investigated the augmentation of simulated cochlear implant audio by visual (facial) speech information for tone. Native speakers of Mandarin and Australian English were asked to discriminate between minimal pairs of Mandarin tones in five conditions: Auditory-Only, Auditory-Visual, CI-simulated Auditory-Only, CI-simulated Auditory-Visual, and Visual-Only (silent video). Discrimination in CI-simulated audio conditions was poor compared with normal audio, and varied according to tone pair, with tone pairs with strong non-F0 cues discriminated the most easily. The availability of visual speech information also improved discrimination in the CI-simulated audio conditions, particularly on tone pairs with strong durational cues. In the silent Visual-Only condition, both Mandarin and Australian English speakers discriminated tones above chance levels. Interestingly, tone-naïve listeners outperformed native listeners in the Visual-Only condition, suggesting firstly that visual speech information for tone is available, and may in fact be under-used by normal-hearing tone language perceivers, and secondly that the perception of such information may be language-general, rather than the product of language-specific learning. This may find application in the development of methods to improve tone perception in CI users in tone language environments.

This work is concerned with using deep neural networks for estimating binary masks within a speech enhancement framework. We first examine the effect of supplementing the audio features used in mask estimation with visual speech information. Visual speech is known to be robust to noise although not necessarily as discriminative as audio features, particularly at higher signal-to-noise ratios. Furthermore, most DNN approaches to mask estimate use the cross-entropy (CE) loss function which aims to maximise classification accuracy. However, we first propose a loss function that aims to maximise the hit minus false-alarm (HIT-FA) rate of the mask, which is known to correlate more closely to speech intelligibility than classification accuracy. We then extend this to a hybrid loss function that combines both the CE and HIT-FA loss functions to provide a balance between classification accuracy and HIT-FA rate of the resulting masks. Evaluations of the perceptually motivated loss functions are carried out using the GRID and larger RM-3000 datasets and show improvements to HIT-FA rate and ESTOI across all noises and SNRs tested. Tests also found that supplementing audio with visual information into a single bimodal audio-visual system gave best performance for all measures and conditions tested.

In audio-visual automatic speech recognition (AVASR) both acoustic and visual modalities of speech are used to identify what a person is saying. In this paper we propose a basic AVASR system implemented using SciPy, an open source Python library for scientific computing. AVASR research draws from the fields of signal processing, computer vision and machine learning, all of which are active fields of development in the SciPy community. As such, AVASR researchers using SciPy are able to benefit from a wide range of tools available in SciPy. The performance of the system is tested using the Clemson University audio- visual experiments (CUAVE) database. We find that visual speech information is in itself not sufficient for automatic speech recognition. However, by integrating visual and acoustic speech information we are able to obtain better performance than what is possible with audio-only ASR.

In studying speech intelligibility in traffic noise, I was able to reproduce a sound field with considerable reality by reproducing both speech and traffic noises by the speaker method using a CD. As a result, the following conclusions were reached. 1) The influence of noise on speech intelligibility is similar for the single syllable and adult disyllable in normal persons. 2) The influence of the kinds of noises and changes with time in the sound pressure level on the intelligibility of the single syllable does not easily manifest itself in patients with sensorineural hearing loss. 3) The speech intelligibility depends largely on the blocking effect of the mid-high frequency components of the noise in both normal persons and patients with sensorineural hearing loss. 4) The intelligibility of the second sound of the adult disyllable is good in both normal persons and patients with sensorineural hearing loss. Furthermore, normal persons can understand a significant disyllable word to some extent by means of "uncertain understanding" even when they cannot hear each vowel and consonant clearly. 5) The intelligibility of the vowel is good in both normal persons and patients with sensorineural hearing loss. 6) As for the intelligibility classified by consonant in normal persons, the intelligibility of the nasal sounds (m, n) and voiced consonants (b, d, g, r, z) becomes low in the case of the S/N ratio being poor. 7) In patients with sensorineural hearing loss, the intelligibility of the bulldozer noise was the lowest, followed by motorway noise among the noises applied. The frequency spectrum distribution of the former consisted mainly of mid-high frequencies of 1 kHz or more. A serrated change with an amplitude of about 6 dB, 2 Hz was noted at the sound pressure level. A sine wave-like change with an amplitude of about 7 dB, 1 Hz was recognized at the sound pressure level of the latter. These are considered to be factors which account for a decline in intelligibility. The present study is part of an experiment I conducted at a sound field using TY-89. The results of this experiment suggest that the frequency spectrum intelligibility of noise as well as time-dependent changes in the sound pressure level of the noise exert an influence on communication disruption in patients with sensorineural hearing loss.

This work examines whether visual speech infor- mation can be effective within audio masking-based speaker separation to improve the quality and intelligibility of the target speech. Two visual-only methods of generating an audio mask for speaker separation are first developed. These use a deep neural network to map visual speech features to an audio feature space from which both visually-derived binary masks and visually- derived ratio masks are estimated, before application to the speech mixture. Secondly, an audio ratio masking method forms a baseline approach for speaker separation which is extended to exploit visual speech information to form audio-visual ratio masks. Speech quality and intelligibility tests are carried out on the visual-only, audio-only and audio-visual masking methods of speaker separation at mixing levels from -10dB to +10dB. These reveal substantial improvements in the target speech when applying the visual-only and audio-only masks, but with highest performance occurring when combining audio and visual information to create the audio-visual masks.

Recent influential models of audiovisual speech perception suggest that visual speech aids perception by generating predictions about the identity of upcoming speech sounds. These models place stock in the assumption that visual speech leads auditory speech in time. However, it is unclear whether and to what extent temporally-leading visual speech information contributes to perception. Previous studies exploring audiovisual-speech timing have relied upon psychophysical procedures that require artificial manipulation of cross-modal alignment or stimulus duration. We introduce a classification procedure that tracks perceptually relevant visual speech information in time without requiring such manipulations. Participants were shown videos of a McGurk syllable (auditory /apa/ + visual /aka/ = perceptual /ata/) and asked to perform phoneme identification (/apa/ yes-no). The mouth region of the visual stimulus was overlaid with a dynamic transparency mask that obscured visual speech in some frames but not others randomly across trials. Variability in participants' responses (~35% identification of /apa/ compared to ~5% in the absence of the masker) served as the basis for classification analysis. The outcome was a high resolution spatiotemporal map of perceptually relevant visual features. We produced these maps for McGurk stimuli at different audiovisual temporal offsets (natural timing, 50-ms visual lead, and 100-ms visual lead). Briefly, temporally-leading (~130ms) visual information did influence auditory perception. Moreover, several visual features influenced perception of a single speech sound, with the relative influence of each feature depending on both its temporal relation to the auditory signal and its informational content.

Speech perception under natural conditions entails integration of auditory and visual information. Understanding how visual and auditory speech information are integrated requires detailed descriptions of the nature and processing of visual speech information. To understand better the process of gathering visual information, we studied the distribution of face-directed fixations of humans performing an audiovisual speech perception task to characterise the degree of asymmetrical viewing and its relationship to speech intelligibility. Participants showed stronger gaze fixation asymmetries while viewing dynamic faces, compared to static faces or face-like objects, especially when gaze was directed to the talkers' eyes. Although speech perception accuracy was significantly enhanced by the viewing of congruent, dynamic faces, we found no correlation between task performance and gaze fixation asymmetry. Most participants preferentially fixated the right side of the faces and their preferences persisted while viewing horizontally mirrored stimuli, different talkers, or static faces. These results suggest that the asymmetrical distributions of gaze fixations reflect the participants' viewing preferences, rather than being a product of asymmetrical faces, but that this behavioural bias does not predict correct audiovisual speech perception.

ObjectiveWe aimed to study the effect of background noise on speech auditory evoked responses in both normal-hearing (NH) individuals and those with sensorineural hearing loss (SNHL) and correlate these neurophysiological results with subjective measures.BackgroundUnderstanding speech in noisy situations is one of the most complex activities encountered in everyday life. It is dependent on cognitive factors and sound processing at peripheral, subcortical, and cortical levels. These make perception of speech in noise one of the most complex aspects of human communication.Patients and methodsFifty NH individuals and 40 individuals with mild-to-moderate SNHL were subjected to the Speech, Spatial, and Qualities of Hearing Scale questionnaire, the speech perception in noise test, speech auditory brainstem response, and cortical auditory evoked responses using speech syllable/da/in quiet condition and at three different signal-to-noise ratios (SNRs) (+10, 0, and − 10).ResultsSpeech auditory brainstem response results in NH participants showed delayed latencies of both transients (waves V, A, C, and O) and frequency-following response with reduction in amplitudes of onset (V and A) and F wave only in noisy relative to quiet condition, whereas in SNHL delayed latencies and decreased amplitudes of both transients and fractional flow reserve (FFR) were observed. The effect of noise can be seen in all waves of cortical auditory evoked responses – that is, with decreasing SNR response, latency increased and response amplitude decreased in both SNHL and NH.ConclusionBackground noise affects both transients and sustained components of speech in NH and SNHL. P1, N1, P2, and N2 components are all affected by SNR, despite their distinct neural generators; this raises the possibility that the effects demonstrated here are reflective of subcortical processing that is propagated to higher levels of the auditory system. cocktail party, speech auditory brainstem response, speech-in-noise perception

An exploratory model of speech intelligibility for healthy aging based on phonatory and articulatory measures

Influence of native language phonetic system on audio-visual speech perception