Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing

It is generally accepted that adults use visual cues to improve speech intelligibility in noisy environments, but findings regarding visual speech benefit in children are mixed. We explored factors that contribute to audiovisual (AV) gain in young children's speech understanding. We examined whether there is an AV benefit to speech-in-noise recognition in children in first grade and if visual salience of phonemes influences their AV benefit. We explored if individual differences in AV speech enhancement could be explained by vocabulary knowledge, phonological awareness, or general psychophysical testing performance. Thirty-seven first graders completed online psychophysical experiments. We used an online single-interval, four-alternative forced-choice picture-pointing task with age-appropriate consonant-vowel-consonant words to measure auditory-only, visual-only, and AV word recognition in noise at -2 and -8 dB SNR. We obtained standard measures of vocabulary and phonological awareness and included a general psychophysical test to examine correlations with AV benefits. We observed a significant overall AV gain among children in first grade. This effect was mainly attributed to the benefit at -8 dB SNR, for visually distinct targets. Individual differences were not explained by any of the child variables. Boys showed lower auditory-only performances, leading to significantly larger AV gains. This study shows AV benefit, of distinctive visual cues, to word recognition in challenging noisy conditions in first graders. The cognitive and linguistic constraints of the task may have minimized the impact of individual differences of vocabulary and phonological awareness on AV benefit. The gender difference should be studied on a larger sample and age range.

To investigate the nature of facial information involved in the integration of audiovisual speech perception, we examined the influence of facial views on the McGurk effect under two auditory conditions. While the speech perception of most of the audiovisual syllables used was little affected by the facial views and the auditory noise, a stronger McGurk effect was obtained for the 3/4-view image uttering labial sounds presented with auditory syllables of alveolars under the auditory noise. However, the facial view did not affect the visual identification of labials in the same way. These results suggest that the information about labial or nonlabial is not the only facial information involved in the McGurk effect. It appears some other information available only in the 3/4-view image might also be involved in the McGurk effect. Implications for the processing of visual, auditory and audiovisual speech are discussed.

International audience

Somatosensory contribution to audio-visual speech processing

Event Abstract Back to Event Development of audiovisual integration of print and speech in normal and poor readers: An ERP and fMRI study Silvia Brem1, 2*, Ulla Richardson1, Silvia Bach1, 2, Christoph Hofstetter2, Ernst Martin3, 4 and Daniel Brandeis2, 4, 5 1 University of Jyväskylä, Agora Center, Finland 2 University of Zurich, Department of Child and Adolescent Psychiatry, Switzerland 3 University Childrens Hospital Zurich, MR-Center, Switzerland 4 University of Zurich, Center for Integrative Human Physiology, Switzerland 5 Central Institute of Mental Health, Department of Child and Adolescent Psychiatry, Germany The integration of multisensory information from speech and print is highly automatized in good readers but remains deficient in poor readers (Blau et al. 2009). Recent studies consistently point to the crucial involvement of the superior temporal sulcus (STS) in integration (van Atteveldt et al. 2004) and conceptual matching (Hocking and Price 2008) of incoming audiovisual information starting around 380ms (Raij et al. 2000) after presentation. In this study we aimed to track emergence and development of audiovisual integration from kindergarten to adulthood in normal and poor readers with functional magnetic resonance imaging (fMRI) and event-related potentials (ERPs). Non-reading kindergartners (n=14, 5.7-7.1y), normal (n=31, 7.8-9.2y) and poor reading (n=15, 7.8-8.9y) 2nd graders as well as normal reading adults (n=15, 21.9-34.5y) were examined with fMRI and ERPs. Conceptual matching effects determined by the statistical difference between incongruous and congruous audiovisual conditions are reported for each age group and for the differences between groups. While kindergarten children did not show conceptual matching effects, normal reading 2nd graders and adults activated inferior frontal and superior temporal brain regions more to incongruous than congruous items. The inferior frontal activation of the 2nd graders was furthermore correlated with reading performance. ERPs pointed to a mismatch effect in the form of a pronounced centro-parietal negativity starting at around 380ms in adults, which was delayed in normal reading 2nd graders. In contrast to adults, 2nd graders showed an additional earlier condition difference around 140-220ms. In summary our results show that incongruous presentation of speech and print evokes a conceptual mismatch effect in fMRI and ERPs as soon as children’s basic reading skills allow matching written to spoken information. The modulation of the matching effect by children’s reading skills points to deficient integration of audiovisual information in poor readers. Conference: EARLI SIG22 - Neuroscience and Education, Zurich, Switzerland, 3 Jun - 5 Jun, 2010. Presentation Type: Poster Presentation Topic: Language and reading Citation: Brem S, Richardson U, Bach S, Hofstetter C, Martin E and Brandeis D (2010). Development of audiovisual integration of print and speech in normal and poor readers: An ERP and fMRI study. Front. Neurosci. Conference Abstract: EARLI SIG22 - Neuroscience and Education. doi: 10.3389/conf.fnins.2010.11.00010 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 28 May 2010; Published Online: 28 May 2010. * Correspondence: Silvia Brem, University of Jyväskylä, Agora Center, Jyväskylä, Finland, sbrem@kjpd.uzh.ch Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Silvia Brem Ulla Richardson Silvia Bach Christoph Hofstetter Ernst Martin Daniel Brandeis Google Silvia Brem Ulla Richardson Silvia Bach Christoph Hofstetter Ernst Martin Daniel Brandeis Google Scholar Silvia Brem Ulla Richardson Silvia Bach Christoph Hofstetter Ernst Martin Daniel Brandeis PubMed Silvia Brem Ulla Richardson Silvia Bach Christoph Hofstetter Ernst Martin Daniel Brandeis Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Content and temporal cues have been shown to interact during audio-visual (AV) speech identification. Typically, the most reliable unimodal cue is used more strongly to identify specific speech features; however, visual cues are only used if the AV stimuli are presented within a certain temporal window of integration (TWI). This suggests that temporal cues denote whether unimodal stimuli belong together, that is, whether they should be integrated. It is not known whether temporal cues also provide information about the identity of a syllable. Since spoken syllables have naturally varying AV onset asynchronies, we hypothesize that for suboptimal AV cues presented within the TWI, information about the natural AV onset differences can aid in speech identification. To test this, we presented low-intensity auditory syllables concurrently with visual speech signals, and varied the stimulus onset asynchronies (SOA) of the AV pair, while participants were instructed to identify the auditory syllables. We revealed that specific speech features (e.g., voicing) were identified by relying primarily on one modality (e.g., auditory). Additionally, we showed a wide window in which visual information influenced auditory perception, that seemed even wider for congruent stimulus pairs. Finally, we found a specific response pattern across the SOA range for syllables that were not reliably identified by the unimodal cues, which we explained as the result of the use of natural onset differences between AV speech signals. This indicates that temporal cues not only provide information about the temporal integration of AV stimuli, but additionally convey information about the identity of AV pairs. These results provide a detailed behavioral basis for further neuro-imaging and stimulation studies to unravel the neurofunctional mechanisms of the audio-visual-temporal interplay within speech perception.

Speech comprehension is improved by viewing a speaker's face, especially in adverse hearing conditions, a principle known as inverse effectiveness. However, the neural mechanisms that help to optimize how we integrate auditory and visual speech in such suboptimal conversational environments are not yet fully understood. Using human EEG recordings, we examined how visual speech enhances the cortical representation of auditory speech at a signal-to-noise ratio that maximized the perceptual benefit conferred by multisensory processing relative to unisensory processing. We found that the influence of visual input on the neural tracking of the audio speech signal was significantly greater in noisy than in quiet listening conditions, consistent with the principle of inverse effectiveness. Although envelope tracking during audio-only speech was greatly reduced by background noise at an early processing stage, it was markedly restored by the addition of visual speech input. In background noise, multisensory integration occurred at much lower frequencies and was shown to predict the multisensory gain in behavioral performance at a time lag of ∼250 ms. Critically, we demonstrated that inverse effectiveness, in the context of natural audiovisual (AV) speech processing, relies on crossmodal integration over long temporal windows. Our findings suggest that disparate integration mechanisms contribute to the efficient processing of AV speech in background noise. The behavioral benefit of seeing a speaker's face during conversation is especially pronounced in challenging listening environments. However, the neural mechanisms underlying this phenomenon, known as inverse effectiveness, have not yet been established. Here, we examine this in the human brain using natural speech-in-noise stimuli that were designed specifically to maximize the behavioral benefit of audiovisual (AV) speech. We find that this benefit arises from our ability to integrate multimodal information over longer periods of time. Our data also suggest that the addition of visual speech restores early tracking of the acoustic speech signal during excessive background noise. These findings support and extend current mechanistic perspectives on AV speech perception.

Dynamics of processing invisible faces in the brain: Automatic neural encoding of facial expression information

Interpersonal negotiation is an essential component of social-economic interactions, yet the concept is not entirely gender-neutral. Male stereotypes - dominance, assertiveness, and rationality - help shape the negotiator role, giving men a perceived advantage in negotiations. This study adapts electroencephalogram hyperscanning and an iterated ultimatum game to investigate the gender differences in dyadic economic negotiation, focusing on event-related potentials and source-localized phase-locked interbrain synchronization (IBS). Behaviorally, dyads with male proposers (M-dyads) achieve better negotiation outcomes than dyads with female proposers (F-dyads) in goal-driven negotiation contexts, reflecting differences in negotiation strategies. Neurally, M-dyads exhibit lower P200 amplitude in the frontal cortex and significantly higher IBS within social brain networks, both of which are strongly associated with reciprocal negotiation behaviors. Notably, IBS between the temporoparietal junction (TPJ), medial prefrontal cortex (mPFC), and superior temporal sulcus (STS), alongside P200 amplitude, serves as a strong predictor of reciprocal behaviors, underscoring the role of TPJ, mPFC, and STS in interpersonal coordination. Moreover, significant mediation effects highlighted the bridging role of IBS between gender composition and negotiation outcomes. These findings provide neurobehavioral accounts of how dyad gender composition influences negotiation outcomes, suggesting that stronger IBS within key social brain regions underlies effective interpersonal coordination.

Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise. Participants reported the intelligibility of each sentence with a button press and trials were sorted post-hoc into those that were more or less intelligible. Response patterns were measured in regions of the pSTG/S identified with an independent localizer. Noisy audiovisual sentences with very similar physical properties evoked very different response patterns depending on their intelligibility. When a noisy audiovisual sentence was reported as intelligible, the pattern was nearly identical to that elicited by clear audiovisual sentences. In contrast, an unintelligible noisy audiovisual sentence evoked a pattern like that of visual-only sentences. This effect was less pronounced for noisy auditory-only sentences, which evoked similar response patterns regardless of intelligibility. The successful integration of visual and auditory speech produces a characteristic neural signature in pSTG/S, highlighting the importance of this region in generating the perceptual benefit of visual speech.

Event-Related Potentials (ERPs)

Viewing our interlocutor facilitates speech perception, unlike for instance when we telephone. Several neural routes and mechanisms could account for this phenomenon. Using magnetoencephalography, we show that when seeing the interlocutor, latencies of auditory responses (M100) are the shorter the more predictable speech is from visual input, whether the auditory signal was congruent or not. Incongruence of auditory and visual input affected auditory responses approximately 20 ms after latency shortening was detected, indicating that initial content-dependent auditory facilitation by vision is followed by a feedback signal that reflects the error between expected and received auditory input (prediction error). We then used functional magnetic resonance imaging and confirmed that distinct routes of visual information to auditory processing underlie these two functional mechanisms. Functional connectivity between visual motion and auditory areas depended on the degree of visual predictability, whereas connectivity between the superior temporal sulcus and both auditory and visual motion areas was driven by audiovisual (AV) incongruence. These results establish two distinct mechanisms by which the brain uses potentially predictive visual information to improve auditory perception. A fast direct corticocortical pathway conveys visual motion parameters to auditory cortex, and a slower and indirect feedback pathway signals the error between visual prediction and auditory input.

Recognising the identity of voices is a key ingredient of communication. Visual mechanisms support this ability: recognition is better for voices previously learned with their corresponding face (compared to a control condition). This so‐called ‘face‐benefit’ is supported by the fusiform face area (FFA), a region sensitive to facial form and identity. Behavioural findings indicate that the face‐benefit increases in noisy listening conditions. The neural mechanisms for this increase are unknown. Here, using functional magnetic resonance imaging, we examined responses in face‐sensitive regions while participants recognised the identity of auditory‐only speakers (previously learned by face) in high (SNR −4 dB) and low (SNR +4 dB) levels of auditory noise. We observed a face‐benefit in both noise levels, for most participants (16 of 21). In high‐noise, the recognition of face‐learned speakers engaged the right posterior superior temporal sulcus motion‐sensitive face area (pSTS‐mFA), a region implicated in the processing of dynamic facial cues. The face‐benefit in high‐noise also correlated positively with increased functional connectivity between this region and voice‐sensitive regions in the temporal lobe in the group of 16 participants with a behavioural face‐benefit. In low‐noise, the face‐benefit was robustly associated with increased responses in the FFA and to a lesser extent the right pSTS‐mFA. The findings highlight the remarkably adaptive nature of the visual network supporting voice‐identity recognition in auditory‐only listening conditions.

Face specific broadband electrocorticographic spectral power change in the rhinal cortex

Gaze direction affects face perception in humans