Footstep sounds increase sensitivity to point-light walking when visual cues are weak

Superior colliculus (SC) neurons have the ability to synthesize information from different sensory modalities, resulting in enhancements (or depressions) of their activity. This physiological capacity is, in turn, closely tied to changes in overt attentive and orientation responses. The present study shows that, in contrast to more altricial species, many deep layer SC neurons in the rhesus monkey are multisensory at birth. Such neurons can respond to stimuli from different sensory modalities, and all convergence patterns seen in the adult are represented. Nevertheless, these neurons cannot yet synthesize their multisensory inputs. Rather, they respond to combinations of cross-modal stimuli much like they respond to their individual modality-specific components. This immature property of multisensory neurons is in contrast to many of the surprisingly sophisticated modality-specific response properties of these neurons and of their modality-specific neighbors. Thus, although deep SC neurons in the newborn have longer latencies and larger receptive fields than their adult counterparts, they are already highly active and are distributed in the typical adult admixture of visual, auditory, somatosensory, and multisensory neurons. Furthermore, the receptive fields of these neurons are already ordered into well organized topographic maps, and the different receptive fields of the same multisensory neurons show a good degree of cross-modal spatial register. These data, coupled with those from cat, suggest that the capacity to synthesize multisensory information does not simply appear in SC neurons at a prescribed maturational stage but rather develops only after substantial experience with cross-modal cues.

Corticofugal projections to the thalamus reveal 2 axonal morphologies, each associated with specific physiological attributes. These determine the functional characteristics of thalamic neurons. It is not clear, however, whether such features characterize the corticofugal projections that mediate multisensory integration in superior colliculus (SC) neurons. The cortico-collicular projections from cat anterior ectosylvian sulcus (AES) are derived from its visual, auditory, and somatosensory representations and are critical for multisensory integration. Following tracer injections into each subdivision, 2 types of cortico-collicular axons were observed. Most were categorized as type I and consisted of small-caliber axons traversing long distances without branching, bearing mainly small boutons. The less frequent type II had thicker axons, more complex branching patterns, larger boutons, and more complex terminal boutons. Following combinatorial injections of 2 different fluorescent tracers into defined AES subdivisions, fibers from each were seen converging onto individual SC neurons and indicate that such convergence, like that in the corticothalamic system, is mediated by 2 distinct morphological types of axon terminals. Nevertheless, and despite the conservation of axonal morphologies across different subcortical systems, it is not yet clear if the concomitant physiological attributes described in the thalamus are directly applicable to multisensory integration.

The present studies were initiated to explore the basis for the response suppression that occurs in cat superior colliculus (SC) neurons when two spatially disparate stimuli are presented simultaneously or in close temporal proximity to one another. Of specific interest was examining the possibility that suppressive regions border the receptive fields (RFs) of unimodal and multisensory SC neurons and, when activated, degrade the neuron's responses to excitatory stimuli. Both within- and cross-modality effects were examined. An example of the former is when a response to a visual stimulus within its RF is suppressed by a second visual stimulus outside the RF. An example of the latter is when the response to a visual stimulus within the visual RF is suppressed when a stimulus from a different modality (e. g., auditory) is presented outside its (i.e., auditory) RF. Suppressive regions were found bordering visual, auditory, and somatosensory RFs. Despite significant modality-specific differences in the incidence and effectiveness of these regions, they were generally quite potent regardless of the modality. In the vast majority (85%) of cases, responses to the excitatory stimulus were degraded by >/=50% by simultaneously stimulating the suppressive region. Contrary to expectations and previous speculations, the effects of activating these suppressive regions often were quite specific. Thus powerful within-modality suppression could be demonstrated in many multisensory neurons in which cross-modality suppression could not be generated. However, the converse was not true. If an extra-RF stimulus inhibited center responses to stimuli of a different modality, it also would suppress center responses to stimuli of its own modality. Thus when cross-modality suppression was demonstrated, it was always accompanied by within-modality suppression. These observations suggest that separate mechanisms underlie within- and cross-modality suppression in the SC. Because some modality-specific tectopetal structures contain neurons with suppressive regions bordering their RFs, the within-modality suppression observed in the SC simply may reflect interactions taking place at the level of one input channel. However, the presence of modality-specific suppression at the level of one input channel would have no effect on the excitation initiated via another input channel. Given the modality-specificity of tectopetal inputs, it appears that cross-modality interactions require the convergence of two or more modality-specific inputs onto the same SC neuron and that the expression of these interactions depends on the internal circuitry of the SC. This allows a cross-modality suppressive signal to be nonspecific and to degrade any and all of the neuron's excitatory inputs.

A compelling example of auditory-visual multisensory integration is the McGurk effect, in which an auditory syllable is perceived very differently depending on whether it is accompanied by a visual movie of a speaker pronouncing the same syllable or a different, incongruent syllable. Anatomical and physiological studies in human and non-human primates have suggested that the superior temporal sulcus (STS) is involved in auditory-visual integration for both speech and non-speech stimuli. We hypothesized that the STS plays a critical role in the creation of the McGurk percept. Because the location of multisensory integration in the STS varies from subject to subject, the location of auditory-visual speech processing in the STS was first identified in each subject with fMRI. Then, activity in this region of the STS was disrupted with single-pulse TMS as subjects rated their percept of McGurk and non-McGurk stimuli. Across three experiments, TMS of the STS significantly reduced the likelihood of the McGurk percept, but did not interfere with perception of non-McGurk stimuli. TMS of the STS was only effective at disrupting the McGurk effect in a narrow temporal window from 100 ms before auditory syllable onset to 100 ms after onset, and TMS of a control location did not influence perception of McGurk or control stimuli. These results demonstrate that the STS plays a critical role in the McGurk effect and auditory-visual integration of speech.

Multimodal integration is crucial for human interaction, in particular for social communication, which relies on integrating information from various sensory modalities. Recently a third visual pathway specialized in social perception was proposed, which includes the right superior temporal sulcus (STS) playing a key role in processing socially relevant cues and high-level social perception. Importantly, it has also recently been proposed that the left STS contributes to audiovisual integration of speech processing. In this article, we propose that brain areas along the right STS that support multimodal integration for social perception and cognition can be considered homologs to those in the left, language-dominant hemisphere, sustaining multimodal integration of speech and semantic concepts fundamental for social communication. Emphasizing the significance of the left STS in multimodal integration and associated processes such as multimodal attention to socially relevant stimuli, we underscore its potential relevance in comprehending neurodevelopmental conditions characterized by challenges in social communication such as autism spectrum disorder (ASD). Further research into this left lateral processing stream holds the promise of enhancing our understanding of social communication in both typical development and ASD, which may lead to more effective interventions that could improve the quality of life for individuals with atypical neurodevelopment.

Late development of audio-visual integration in the vertical plane

Mature multisensory superior colliculus (SC) neurons integrate information across the senses to enhance their responses to spatiotemporally congruent cross-modal stimuli. The development of this neurotypic feature of SC neurons requires experience with cross-modal cues. In the absence of such experience the response of an SC neuron to congruent cross-modal cues is no more robust than its response to the most effective component cue. This "default" or "naive" state is believed to be one in which cross-modal signals do not interact. The present results challenge this characterization by identifying interactions between visual-auditory signals in male and female cats reared without visual-auditory experience. By manipulating the relative effectiveness of the visual and auditory cross-modal cues that were presented to each of these naive neurons, an active competition between cross-modal signals was revealed. Although contrary to current expectations, this result is explained by a neuro-computational model in which the default interaction is mutual inhibition. These findings suggest that multisensory neurons at all maturational stages are capable of some form of multisensory integration, and use experience with cross-modal stimuli to transition from their initial state of competition to their mature state of cooperation. By doing so, they develop the ability to enhance the physiological salience of cross-modal events thereby increasing their impact on the sensorimotor circuitry of the SC, and the likelihood that biologically significant events will elicit SC-mediated overt behaviors.SIGNIFICANCE STATEMENT The present results demonstrate that the default mode of multisensory processing in the superior colliculus is competition, not non-integration as previously characterized. A neuro-computational model explains how these competitive dynamics can be implemented via mutual inhibition, and how this default mode is superseded by the emergence of cooperative interactions during development.

Humans are remarkably adept at understanding speech, even when it is contaminated by noise. Multisensory integration may explain some of this ability: combining independent information from the auditory modality (vocalizations) and the visual modality (mouth movements) reduces noise and increases accuracy. Converging evidence suggests that the superior temporal sulcus (STS) is a critical brain area for multisensory integration, but little is known about its role in the perception of noisy speech. Behavioral studies have shown that perceptual judgments are weighted by the reliability of the sensory modality: more reliable modalities are weighted more strongly, even if the reliability changes rapidly. We hypothesized that changes in the functional connectivity of STS with auditory and visual cortex could provide a neural mechanism for perceptual reliability-weighting. To test this idea, we performed five blood oxygenation level dependent (BOLD) fMRI and behavioral experiments in 34 healthy subjects. We found increased functional connectivity between the STS and auditory cortex when the auditory modality was more reliable (less noisy) and increased functional connectivity between the STS and visual cortex when the visual modality was more reliable, even when the reliability changed rapidly during presentation of successive words. This finding matched the results of a behavioral experiment in which the perception of incongruent audiovisual syllables was biased toward the more reliable modality, even with rapidly changing reliability. Changes in STS functional connectivity may be an important neural mechanism underlying the perception of noisy speech.

The environment often engages multiple sensory channels simultaneously. When events or objects stimulate more than one sensory organ, they are more likely to be detected and more efficiently processed than events stimulating just one sensory modality (e.g. Stein & Stanford, 2008). Such crossmodal enhancement requires that the sensory inputs arising from one event are correctly assigned to that object (termed the ‘correspondence problem’; Ernst & Bulthoff, 2004). Multisensory research has demonstrated that features that are shared by all sensory systems, such as space, time or meaning are used for crossmodal binding (for recent reviews see Driver & Noesselt, 2008; Stein & Stanford, 2008). For example, in a cocktail party setting, matching of the temporal dynamics of lip movements and of speech sound allows us to assign the correct message to each speaker. Furthermore, this ability of crossmodal binding assists in orienting spatial attention to a particular conversation partner while ignoring other sources of distracting sensory information, such as other peoples’ conversations. The binding of synchronous auditory and visual (speech) signals seems to happen automatically and pre-attentively (see Driver, 1996; Bertelson et al., 2000; Vroomen et al., 2001a). The voluntary orienting of spatial attention to one particular speaker or, more generally, to an event is an example of endogenous attention allocation. By contrast, stimulus-driven orienting of spatial attention, elicited by the saliency of the stimulus, has been termed exogenous attention. Due to their relatively high salience, such pre-attentively detected, congruent cross-modal stimuli may function as exogenous attentional cues (Spence & Driver, 2000; Stekelenburg et al., 2004; Senkowski et al., 2008). In this issue, Fairhall & Macaluso (2009) provide evidence for the operation of both mechanisms: (i) pre-attentively crossmodal matching and (ii) enhancement of crossmodal interaction by endogenous attention. The authors presented the lips of two speaking faces, one in the left and one in the right visual hemifield. The centrally presented auditory stream was matched with the lip movements of one speaker only. In such a situation people generally perceive the auditory speech signal as originating from the location of the matching lips. This phenomenon is well known as the ‘ventriloquist effect’. Critically, Fairhall and Macaluso also asked participants to perform a visual detection task in one hemifield only. This task served to systematically manipulate the focus of visual spatial attention. The auditory stream did not have any significance to the participants and thus could be ignored. This paradigm allowed the authors to compare brain responses as assessed with functional magnetic resonance imaging (fMRI) to multisensory, matching audio-visual speech sequences that occurred either at the location of visual spatial attention (‘congruent condition’) or on the opposite side (‘incongruent condition’) while keeping physical stimulation constant. The authors report that visual attention enhances activity in brain regions well accepted as multisensory integration areas (Driver & Noesselt, 2008), such as the superior temporal sulcus (STS) and the superior colliculus (SC), only when the auditory track matched the attended lip movements. Similar effects were observed in several visual brain regions, including the primary visual cortex. These results suggest that the processing of congruent cross-modal stimuli within the locus of endogenous visual spatial attention is enhanced. Furthermore, the data of Fairhall & Macaluso are compatible with the view that binding of audio-visual speech can occur pre-attentively. When audio-visual matching speech was presented on the opposite side of the locus of visual-spatial attention (incongruent condition), participants performed worse in the visual detection task than when matching stimuli were presented on the same side as the focus of participant’s endogenous visual spatial attention (congruent condition). These behavioural results, together with findings from earlier studies (Spence & Driver, 2000; Vroomen et al., 2001b), suggest that the audio-visual binding process that elicited a ventriloquist illusion towards the opposite side of the visual attention focus took place outside the locus of endogenous attention and resulted in an exogenously attentional shift towards the task irrelevant side, which in turn interfered with the visual detection task. In summary, the results described by Fairhall and Macaluso suggest that endogenous (visual) spatial attention amplifies the processing of congruent cross-modal input that is likely integrated and detected pre-attentively. Fairhall and Macaluso did not attempt to link the enhanced activity in multisensory and early visual cortical regions to behavioural data such as, for example, improved speech comprehension. The question of whether activity changes in multisensory regions due to endogenous spatial attention are accompanied by altered multisensory performance (Alsius et al., 2005) remains to be investigated in future work. Moreover, other top-down influences on multisensory integration such as plausibility and meaning (Kitagawa & Spence, 2005; Hein et al., 2007) will need to be investigated in more detail to extend our understanding of the conditions when and which multisensory binding processes are subject to voluntary modulation. European Journal of Neuroscience, Vol. 29, pp. 1245–1246, 2009 doi:10.1111/j.1460-9568.2009.06715.x

The superior colliculus (SC) plays an important role in integrating visual, auditory and somatosensory information, and in guiding the orientation of the eyes, ears and head. Previously we have shown that cats with unilateral SC lesions showed a preferential loss of multisensory orientation behaviors for stimuli contralateral to the lesion. Surprisingly, this behavioral loss was seen even under circumstances where the SC lesion was far from complete. To assess the physiological changes induced by these lesions, we employed single unit electrophysiological methods to record from individual neurons in both the intact and damaged SC following behavioral testing in two animals. In the damaged SC of these animals, multisensory neurons were preferentially reduced in incidence, comprising less than 25% of the sensory-responsive population (as compared with 49% on the control side). In those multisensory neurons that remained following the lesion, receptive fields were nearly twofold larger, and less than 25% showed normal patterns of multisensory integration, with those that did being found in areas outside of the lesion. These results strongly suggest that the multisensory behavioral deficits seen following SC lesions are the combined result of a loss of multisensory neurons and a loss of multisensory integration in those neurons that remain.

Auditory feedback may provide the nervous system with valuable temporal (e. g., footstep sounds) and spatial (e.g., external reference sounds) information that can assist in the control of upright walking. As such, hearing loss may directly contribute to declines in mobility among older adults. Our purpose was to examine the impact of auditory feedback on the control of walking in older adults. Twenty older adults (65–86 years) with no diagnosed hearing loss walked on a treadmill for three sound conditions: Baseline, Ear Plugs, and White Noise. We hypothesized that in response to reduced temporal auditory feedback during the Ear Plugs and White Noise conditions, participants would adapt shorter and faster steps that are traditionally believed to increase mechanical stability. This hypothesis was not supported. Interestingly, we observed increases in step length (p = 0.047) and step time (p = 0.026) during the Ear Plugs condition vs. Baseline. Taking longer steps during the Ear Plugs condition may have increased ground reaction forces, thus allowing participants to sense footsteps via an occlusion effect. As a follow-up, we performed a Pearson's correlation relating the step length increase during the Ear Plugs condition to participants' scores on a clinical walking balance test, the Functional Gait Assessment. We found a moderate negative relationship (rho = −0.44, p = 0.055), indicating that participants with worse balance made the greatest increases in step length during the Ear Plugs condition. This trend suggests that participants may have actively sought auditory feedback with longer steps, sacrificing a more mechanically stable stepping pattern. We also hypothesized that reduced spatial localization feedback during the Ear Plugs and White Noise conditions would decrease control of center of mass (COM) dynamics, resulting in an increase in lateral COM excursion, lateral margin of stability, and maximum Lyapunov exponent. However, we found no main effects of auditory feedback on these metrics (p = 0.580, p = 0.896, and p = 0.056, respectively). Overall, these results suggest that during a steady-state walking task, healthy older adults can maintain walking control without auditory feedback. However, increases in step length observed during the Ear Plugs condition suggest that temporal auditory cues provide locomotor feedback that becomes increasingly valuable as balance deteriorates with age.

Audiovisual integration increases the intentional step synchronization of side-by-side walkers

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Naturally hyperglycemic, birds maintain twice the plasma-glucose concentration of mammals at equivalent size and are relatively insensitive to insulin. Most avian species possess mechanisms to enhance fatty acid (FA) transport and oxidation using pathways comparable to those in diabetic humans. Recent evidence suggests that perturbation of myo-inositol (MI), a carbocyclic sugar derived from glucose-6-phosphate, plays a role in mammalian hyperglycemic regulation. While MI is not typically utilized for energy, it serves as a mediator in signal transduction, regulating multiple metabolic processes. We hypothesized that MI plays a key role in regulating avian metabolism and body mass regulations. We investigated this at two levels. At the organismal level, Anna’s Hummingbirds (Calypte anna) were supplemented with either control nectar (Nektar plus) or MI enhanced nectar (5% MI), to explore the role of dietary MI in body mass regulation. Hummingbirds body masses were continuously monitored for 7 days. Additionally, we designed a choice experiment, presenting the birds with 2 feeders (control and either 1%, 3% or 5% MI enhanced), to explore hummingbirds MI preferences. We recorded hummingbirds body masses before and after a 2-hour trial, as well as their nectar consumption. At the cellular level, we assessed MI's effect on mitochondrial fuel utilization using fibroblast cells from Japanese Quail (Coturnix japonica), incubated for 2 hours with control media or MI enhanced media (5% MI), and assayed utilizing extracellular flux assays. Results: For the chronic exposure experiment, a split plot ANOVA with treatment as the between-subject factor; day as a within-subject factor and initial body mass as a covariate, revealed that MI significantly decreased evening body mass (F 1,6 =12.862, p=0.012). For the choice experiments, split plot ANOVAs with either treatment or body mass change as the between-subject factor and trial number as a within-subject factor, revealed that Ingested MI significantly and negatively affected changes in body mass. (F 1,18 4 =5.768, p=0.017). We also found a significant effect of seasonality on the birds’ choice (F 2,3 =60.392, p=0.004), and that birds avoid higher MI concentrations (F 2,8 =5.754, p=0.028) in most seasons, except for post-migration season, when they arrive at higher body mass to the wintering grounds. Cellular assays showed a significantly higher external FA oxidation (t 3 = -3.42, p = 0.042), and lower Pyruvate dehydrogenase (PDH) activity (t 2 =17.96, P < 0.001) for MI treated cells. Our findings show that dietary MI reduces fat accumulation in Anna’s Hummingbirds. MI consumption by hummingbirds varies with seasonal changes in body mass, consistent with a regulatory role. Furthermore, MI enhances FA oxidation in avian cells, via effects on PDH complexes. Adi Domer is supported by a long term Human Frontiers Science Program Postdoctoral fellowship This abstract was presented at the American Physiology Summit 2025 and is only available in HTML format. There is no downloadable file or PDF version. The Physiology editorial board was not involved in the peer review process.

The modality convergence patterns, sensory response properties, and principles governing multisensory integration in the superior colliculus (SC) of the alert cat were found to have fundamental similarities to those in anesthetized animals. Of particular interest was the observation that, in a manner indistinguishable from the anesthetized animal, combinations of two different sensory stimuli significantly enhanced the responses of SC neurons above those evoked by either unimodal stimulus. These observations are consistent with the speculation that there is a functional link among multisensory integration in individual SC neurons and cross-modality attentive and orientation behaviors.