Using Visual Speech Information in Masking Methods for Audio Speaker Separation

Recent influential models of audiovisual speech perception suggest that visual speech aids perception by generating predictions about the identity of upcoming speech sounds. These models place stock in the assumption that visual speech leads auditory speech in time. However, it is unclear whether and to what extent temporally-leading visual speech information contributes to perception. Previous studies exploring audiovisual-speech timing have relied upon psychophysical procedures that require artificial manipulation of cross-modal alignment or stimulus duration. We introduce a classification procedure that tracks perceptually relevant visual speech information in time without requiring such manipulations. Participants were shown videos of a McGurk syllable (auditory /apa/ + visual /aka/ = perceptual /ata/) and asked to perform phoneme identification (/apa/ yes-no). The mouth region of the visual stimulus was overlaid with a dynamic transparency mask that obscured visual speech in some frames but not others randomly across trials. Variability in participants' responses (~35% identification of /apa/ compared to ~5% in the absence of the masker) served as the basis for classification analysis. The outcome was a high resolution spatiotemporal map of perceptually relevant visual features. We produced these maps for McGurk stimuli at different audiovisual temporal offsets (natural timing, 50-ms visual lead, and 100-ms visual lead). Briefly, temporally-leading (~130ms) visual information did influence auditory perception. Moreover, several visual features influenced perception of a single speech sound, with the relative influence of each feature depending on both its temporal relation to the auditory signal and its informational content.

This work is concerned with using deep neural networks for estimating binary masks within a speech enhancement framework. We first examine the effect of supplementing the audio features used in mask estimation with visual speech information. Visual speech is known to be robust to noise although not necessarily as discriminative as audio features, particularly at higher signal-to-noise ratios. Furthermore, most DNN approaches to mask estimate use the cross-entropy (CE) loss function which aims to maximise classification accuracy. However, we first propose a loss function that aims to maximise the hit minus false-alarm (HIT-FA) rate of the mask, which is known to correlate more closely to speech intelligibility than classification accuracy. We then extend this to a hybrid loss function that combines both the CE and HIT-FA loss functions to provide a balance between classification accuracy and HIT-FA rate of the resulting masks. Evaluations of the perceptually motivated loss functions are carried out using the GRID and larger RM-3000 datasets and show improvements to HIT-FA rate and ESTOI across all noises and SNRs tested. Tests also found that supplementing audio with visual information into a single bimodal audio-visual system gave best performance for all measures and conditions tested.

Cochlear implant (CI) users in tone language environments report great difficulty in perceiving lexical tone. This study investigated the augmentation of simulated cochlear implant audio by visual (facial) speech information for tone. Native speakers of Mandarin and Australian English were asked to discriminate between minimal pairs of Mandarin tones in five conditions: Auditory-Only, Auditory-Visual, CI-simulated Auditory-Only, CI-simulated Auditory-Visual, and Visual-Only (silent video). Discrimination in CI-simulated audio conditions was poor compared with normal audio, and varied according to tone pair, with tone pairs with strong non-F0 cues discriminated the most easily. The availability of visual speech information also improved discrimination in the CI-simulated audio conditions, particularly on tone pairs with strong durational cues. In the silent Visual-Only condition, both Mandarin and Australian English speakers discriminated tones above chance levels. Interestingly, tone-naïve listeners outperformed native listeners in the Visual-Only condition, suggesting firstly that visual speech information for tone is available, and may in fact be under-used by normal-hearing tone language perceivers, and secondly that the perception of such information may be language-general, rather than the product of language-specific learning. This may find application in the development of methods to improve tone perception in CI users in tone language environments.

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

In audio-visual automatic speech recognition (AVASR) both acoustic and visual modalities of speech are used to identify what a person is saying. In this paper we propose a basic AVASR system implemented using SciPy, an open source Python library for scientific computing. AVASR research draws from the fields of signal processing, computer vision and machine learning, all of which are active fields of development in the SciPy community. As such, AVASR researchers using SciPy are able to benefit from a wide range of tools available in SciPy. The performance of the system is tested using the Clemson University audio- visual experiments (CUAVE) database. We find that visual speech information is in itself not sufficient for automatic speech recognition. However, by integrating visual and acoustic speech information we are able to obtain better performance than what is possible with audio-only ASR.

Speech perception under natural conditions entails integration of auditory and visual information. Understanding how visual and auditory speech information are integrated requires detailed descriptions of the nature and processing of visual speech information. To understand better the process of gathering visual information, we studied the distribution of face-directed fixations of humans performing an audiovisual speech perception task to characterise the degree of asymmetrical viewing and its relationship to speech intelligibility. Participants showed stronger gaze fixation asymmetries while viewing dynamic faces, compared to static faces or face-like objects, especially when gaze was directed to the talkers' eyes. Although speech perception accuracy was significantly enhanced by the viewing of congruent, dynamic faces, we found no correlation between task performance and gaze fixation asymmetry. Most participants preferentially fixated the right side of the faces and their preferences persisted while viewing horizontally mirrored stimuli, different talkers, or static faces. These results suggest that the asymmetrical distributions of gaze fixations reflect the participants' viewing preferences, rather than being a product of asymmetrical faces, but that this behavioural bias does not predict correct audiovisual speech perception.

When a target-speech/masker mixture is processed with the signal-separation technique, ideal binary mask (IBM), intelligibility of target speech is remarkably improved in both normal-hearing listeners and hearing-impaired listeners. Intelligibility of speech can also be improved by filling in speech gaps with un-modulated broadband noise. This study investigated whether intelligibility of target speech in the IBM-treated target-speech/masker mixture can be further improved by adding a broadband-noise background. The results of this study show that following the IBM manipulation, which remarkably released target speech from speech-spectrum noise, foreign-speech, or native-speech masking (experiment 1), adding a broadband-noise background with the signal-to-noise ratio no less than 4 dB significantly improved intelligibility of target speech when the masker was either noise (experiment 2) or speech (experiment 3). The results suggest that since adding the noise background shallows the areas of silence in the time-frequency domain of the IBM-treated target-speech/masker mixture, the abruption of transient changes in the mixture is smoothed and the perceived continuity of target-speech components becomes enhanced, leading to improved target-speech intelligibility. The findings are useful for advancing computational auditory scene analysis, hearing-aid/cochlear-implant designs, and understanding of speech perception under "cocktail-party" conditions.

A significant proportion of speech communication occurs when speakers and listeners are within face-to-face proximity of one other. In noisy and reverberant environments with multiple sound sources, auditory-visual (AV) speech communication takes on increased importance because it offers the best chance for successful communication. This chapter reviews AV processing for speech understanding by normal-hearing individuals. Auditory, visual, and AV factors that influence intelligibility, such as the speech spectral regions that are most important for AV speech recognition, complementary and redundant auditory and visual speech information, AV integration efficiency, the time window for auditory (across spectrum) and AV (cross-modality) integration, and the modulation coherence between auditory and visual speech signals are each discussed. The knowledge gained from understanding the benefits and limitations of visual speech information as it applies to AV speech perception is used to propose a signal-based model of AV speech intelligibility. It is hoped that the development and refinement of quantitative models of AV speech intelligibility will increase our understanding of the multimodal processes that function every day to aid speech communication, as well guide advances in future generation hearing aids and cochlear implants for individuals with sensorineural hearing loss.

Influence of native language phonetic system on audio-visual speech perception

Audio-visual speech enhancement (AV-SE) is the task of improving speech quality and intelligibility in a noisy environment using audio and visual information from a talker. Recently, deep learning techniques have been adopted to solve the AV-SE task in a supervised manner. In this context, the choice of the target, i.e. the quantity to be estimated, and the objective function, which quantifies the quality of this estimate, to be used for training is critical for the performance. This work is the first that presents an experimental study of a range of different targets and objective functions used to train a deep-learning-based AV-SE system. The results show that the approaches that directly estimate a mask perform the best overall in terms of estimated speech quality and intelligibility, although the model that directly estimates the log magnitude spectrum performs as good in terms of estimated speech quality.

The intelligibility and quality of speech produced by current text-to-speech (TTS) engines was assessed using well-established tests of speech intelligibility. TTS engines included Microsoft’s Whistler, AT&T’s Watson, and Elan Informatique’s ProVerbe. The speech intelligibility tests varied in complexity and in the degree to which syntax and lexical knowledge could aid perception. Stimuli included CVC words, semantically meaningful sentences, semantically anomalous sentences, and paragraphs representing different styles of discourse. Intelligibility was evaluated using transcription accuracy for isolated word and sentence items, as well as ratings of the clarity of articulation for word and paragraph items. Perceptual errors on word items also were analyzed by phoneme and word position (initial, medial, or final). Additional ratings were used to determine the perceived ‘‘naturalness’’ of meaningful sentences and paragraph items. Across levels of speech complexity, clear and consistent differences between TTS engines were obtained in terms of both intelligibility and naturalness. Possible explanations of the results will be discussed, as will potential areas where text-to-speech synthesis can generally be improved.

Speech shadowing experiments were conducted to test whether alignment (inadvertent imitation) to voice onset time (VOT) can be influenced by visual speech information. Experiment 1 examined whether alignment would occur to auditory /pa/ syllables manipulated to have 3 different VOTs. Nineteen female participants were asked to listen to 180 syllables over headphones and to say each syllable out loud quickly and clearly. In Experiment 2, visual speech tokens composed of a face articulating /pa/ syllables at 2 different rates were dubbed onto the audio /pa/ syllables of Experiment 1. Sixteen new female participants were asked to listen to and watch (over a video monitor) 180 syllables and to say each syllable out loud quickly and clearly. Results of Experiment 1 showed that the 3 VOTs of the audio /pa/ stimuli influenced the VOTs of the participants' produced syllables. Results of Experiment 2 revealed that both the visible syllable rate and audio VOT of the audiovisual /pa/ stimuli influenced the VOTs of the participants' produced syllables. These results show that, like auditory speech, visual speech information can induce speech alignment to a phonetically relevant property of an utterance.

Benefits of intensive speech treatment have been documented for a range of speech signs in English speakers with Parkinson's Disease (PD). However, the answer to a critical question that whether the same treatment benefits speech variables including intelligibility in Mandarin speakers is still unclear. In order to develop a targeted speech treatment for Mandarin speakers with PD, we reviewed the efficacy of intensive speech treatment to improve vocal loudness and functional communication and discuss possible explanations for efficacy on Mandarin speakers with PD. Literatures about intensive speech treatment for Mandarin speakers with PD were retrieved from PubMed, Web of Science, Embase, China National Knowledge Infrastructure (CNKI), Wanfang and Weipu Database for Chinese Technical Periodicals (VIP) Database. Search strategy was (voice therapy OR speech therapy OR voice treatment OR speech treatment OR voice training OR speech training OR voice rehabilitation OR speech rehabilitation OR Lee Silverman voice treatment OR intensive speech treatment) and (Parkinson's disease) and (Mandarin speakers OR Chinese OR Chinese people). Five randomized controlled trials were selected and possible explanations for efficacy on individuals with PD are discussed. Further research directions are suggested. The existing evidence from treatment efficacy studies of intensive speech treatment provides support for improving vocal loudness, speech intelligibility, pitch and rate in Mandarin speakers with PD. Our future research will continue to work to conduct a large sample multicenter randomized controlled trial to provide high quality evidence and understand the basic mechanisms accompanying treatment-related change.

EnAbstract Background Improved articulatory placement through speech therapy may eliminate compensatory errors, improve velopharyngeal function, minimize perception of hypernasality, and improve speech intelligibility. Aim The aim of this work was to study the effect of preoperative speech therapy on the speech intelligibility in patients with residual velopharyngeal insufficiency in order to provide a better speech outcome in such patients. Patients and methods This study included 41 patients diagnosed with residual velopharyngeal insufficiency after cleft palate repair. Patients were divided into two groups: group I included 22 patients scheduled to undergo speech therapy 6 months preoperatively, and group II included 19 patients who did not receive preoperative speech therapy. Both groups underwent speech therapy for 6 months postoperatively. Evaluation was carried out through auditory perceptual assessment (APA) including the type and degree of open nasality, consonant precision, compensatory articulatory mechanisms (glottal articulation and pharyngealization of fricatives), audible nasal emission of air, and overall intelligibility of speech. All these elements are graded along a five-point scale in which 0 is normal and 4 is severe affection. Documentation of APA is performed by high fidelity speech and voice audio recording and endoscopy. Formal speech intelligibility testing was carried out using the Arabic Speech Intelligibility test, which is designed to provide an estimation of the overall speech intelligibility of children by providing a total score in percentage. Nasometry was performed for all patients using a Kay nasometer, which provides the ‘nasalance score’. Results On comparing the APA assessment of group I after 6 months of speech therapy postoperatively with the preoperative data, a highly significant decrease with regard to all parameters was revealed; however, in group II, the same comparison revealed a highly significant decrease as regards the degree of open nasality and nasal emission of air and a significant difference in terms of glottal articulation, pharyngealization of fricatives, and overall speech intelligibility. On comparing the two groups postoperatively, a significant difference in glottal articulation, pharyngealization of fricatives, and overall speech intelligibility, being more improved in group I, was revealed. Nasometry showed a significant difference between the postoperative results after speech therapy compared with the preoperative results, with a nonsignificant difference between the two groups postoperatively. In group I, the results of the Arabic Speech Intelligibility test showed a highly significant decrease in the number of patients with unintelligible speech and poor speech intelligibility, a significant decrease in number of patients with fair speech intelligibility, and a highly significant increase in the number of patients with good and excellent speech intelligibility. In group II, there was a significant decrease in the number of patients with unintelligible speech and poor speech intelligibility and a significant increase in the number of patients with fair, good, and excellent speech intelligibility. Conclusion and recommendations Speech therapy before surgery for residual velopharyngeal insufficiency can improve the results of postoperative therapy, with a better speech intelligibility outcome. Therefore, it is recommended to schedule a speech therapy program before secondary repair of the velopharyngeal valve in order to attain better speech intelligibility.

Parental reports of communication, listening, and behavior in children receiving a clinical diagnosis of specific language impairment (SLI) or auditory processing disorder (APD) were compared with direct tests of intelligence, memory, language, phonology, literacy, and speech intelligibility. The primary aim was to identify whether there were differences between these characteristics in children with SLI or APD. Normally hearing children who were clinically diagnosed with SLI (n = 22) or APD (n = 19), and a quasirandom sample of mainstream school (MS) children (n = 47) aged 6-13 years, underwent tests of verbal and nonverbal IQ, digit span, nonsense word repetition, Spoonerisms, reading, grammar, and sentence and VCV nonword intelligibility. Parents completed questionnaires on the children's communication, listening, and behavior. There was generally no difference between the performance of the children with SLI and the children with APD on the questionnaire and test measures, and both groups consistently and significantly underperformed compared with the children in the MS group. Speech intelligibility in both noise and quiet was unimpaired in the SLI and APD groups. Despite clinical diagnoses of SLI or APD, the 2 groups of children had very similar behavioral and parental report profiles, suggesting that the children were differentially diagnosed based on their referral route rather than on actual differences.