Late development of audio-visual integration in the vertical plane

When alertness fades: Drowsiness-induced visual dominance and oscillatory recalibration in audiovisual integration.

Multisensory integration (MSI) and spatial attention are both mechanisms through which the processing of sensory information can be facilitated. Studies on the interaction between spatial attention and MSI have mainly focused on the interaction between endogenous spatial attention and MSI. Most of these studies have shown that endogenously attending a multisensory target enhances MSI. It is currently unclear, however, whether and how exogenous spatial attention and MSI interact. In the current study, we investigated the interaction between these two important bottom-up processes in two experiments. In Experiment 1 the target location was task-relevant, and in Experiment 2 the target location was task-irrelevant. Valid or invalid exogenous auditory cues were presented before the onset of unimodal auditory, unimodal visual, and audiovisual targets. We observed reliable cueing effects and multisensory response enhancement in both experiments. To examine whether audiovisual integration was influenced by exogenous spatial attention, the amount of race model violation was compared between exogenously attended and unattended targets. In both Experiment 1 and Experiment 2, a decrease in MSI was observed when audiovisual targets were exogenously attended, compared to when they were not. The interaction between exogenous attention and MSI was less pronounced in Experiment 2. Therefore, our results indicate that exogenous attention diminishes MSI when spatial orienting is relevant. The results are discussed in terms of models of multisensory integration and attention.

Multisensory integration (MSI) and spatial attention are both mechanisms through which the processing of sensory information can be facilitated. Studies on the interaction between spatial attention and MSI have mainly focused on the interaction between endogenous spatial attention and MSI. Most of these studies have shown that endogenously attending a multisensory target enhances MSI. It is currently unclear, however, whether and how exogenous spatial attention and MSI interact. In the current study, we investigated the interaction between these two important bottom-up processes in two experiments. In Experiment 1 the target location was task-relevant, and in Experiment 2 the target location was task-irrelevant. Valid or invalid exogenous auditory cues were presented before the onset of unimodal auditory, unimodal visual, and audiovisual targets. We observed reliable cueing effects and multisensory response enhancement in both experiments. To examine whether audiovisual integration was influenced by exogenous spatial attention, the amount of race model violation was compared between exogenously attended and unattended targets. In both Experiment 1 and Experiment 2, a decrease in MSI was observed when audiovisual targets were exogenously attended, compared to when they were not. The interaction between exogenous attention and MSI was less pronounced in Experiment 2. Therefore, our results indicate that exogenous attention diminishes MSI when spatial orienting is relevant. The results are discussed in terms of models of multisensory integration and attention.

  Regarding the atypical characteristics of cognition and information processing in individuals with autism spectrum disorder (ASD), recent focus has been centered around fundamental processing, such as multisensory integration (MSI). Experimental studies have reported atypical MSI, especially audio-visual integration, in both children and adults with ASD using social (e.g., faces and voices) and nonsocial stimuli (e.g., flashes and beeps). Furthermore, there has been a gradual increase in the understanding of the behavioral (e.g., higher temporal resolution) and neural mechanisms (e.g., impaired phase alignment of neuro-oscillations) underlying atypical MSI in ASD. Previous studies have proposed that prominent deficits in social cognition and interactions (i.e., higher-order functions) are influenced and/or induced by atypicalities in MSI (i.e., lower-order function). Thus, interventions targeting MSI may promote social cognition, likely resulting in better outcomes in adulthood in individuals with ASD. This chapter describes current knowledge regarding multisensory processing in ASD and future perspectives on relevant research and practices. We highlight the value of focusing on MSI for understanding the clinical characteristics of ASD and possible interventions targeting MSI for this population.  

Sustained integration of sensory inputs over increased temporal delays is associated with reduced cognitive and physical functioning in older adults and adverse outcomes such as falls. Here, we explored the relationship between multisensory integration and a clinically relevant measure of balance/postural control; Sit-to-Stand Time, the efficiency with which an older adult can transition between a seated and a standing posture. We investigated whether temporal multisensory integration was associated with performance on the Five-Times Sit-to-Stand Test (FTSST) in a large sample of 2556 older adults (mean age = 63.62 years, SD = 7.50; 55% female) drawn from The Irish Longitudinal Study on Ageing (TILDA). K-means clustering was applied to FTSST data, yielding three clusters characterised by fast (mean = 10.88 s; n = 1122), medium (mean = 14.34 s; n = 1133) and slow (mean = 18.97 s; n = 301) sit-to-stand times. At wave 3 of TILDA, older adults participated in the Sound Induced Flash Illusion (SIFI), a measure of the precision of temporal audio–visual integration, which included three audio–visual stimulus onset asynchronies (SOAs): 70, 150 and 230 ms. Older adults with the slowest sit-to-stand times were more susceptible to the SIFI at the longest SOA (230 ms) compared to the shortest SOA (70 ms) relative to those with the fastest times (p = 0.02). Older adults who take longer to repeatedly transition from a seated to a standing posture exhibit an expanded temporal binding window for audio–visual events, supporting a link between multisensory perception and balance/postural control in ageing.

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

The human brain is an optimization of multisensory information integration system. The study of audiovisual information integration can helps to reveal processing mechanism of human brain. It was very important to explore multi-sensor data fusion theory model. In the present study, we designed two experiments to investigate the effects of temporal-spatial unpredictable auditory information on temporal-spatial coincident audiovisual integration. Our results showed temporal-spatial unpredictable auditory information increased the strength of attention, and significantly improved visual detection rates. In addition, through the comparison with the results in another study we speculate that the temporal unpredictable auditory information has less of no effects on audiovisual integration.

We are continually bombarded by information arriving to each of our senses; however, the brain seems to effortlessly integrate this separate information into a unified percept. Although multisensory integration has been researched extensively using simple computer tasks and stimuli, much less is known about how multisensory integration functions in real-world contexts. Additionally, several recent studies have demonstrated that multisensory integration varies tremendously across naturalistic stimuli. Virtual reality can be used to study multisensory integration in realistic settings because it combines realism with precise control over the environment and stimulus presentation. In the current study, we investigated whether multisensory integration as measured by the redundant signals effects (RSE) is observable in naturalistic environments using virtual reality and whether it differs as a function of target and/or environment cue-richness. Participants detected auditory, visual, and audiovisual targets which varied in cue-richness within three distinct virtual worlds that also varied in cue-richness. We demonstrated integrative effects in each environment-by-target pairing and further showed a modest effect on multisensory integration as a function of target cue-richness but only in the cue-rich environment. Our study is the first to definitively show that minimal and more naturalistic tasks elicit comparable redundant signals effects. Our results also suggest that multisensory integration may function differently depending on the features of the environment. The results of this study have important implications in the design of virtual multisensory environments that are currently being used for training, educational, and entertainment purposes.

Reaching with one arm to the other: Coordinating touch, proprioception, and action during infancy

Auditory cross-modal reorganization in cochlear implant users indicates audio-visual integration.

One of the classic examples of multisensory integration in humans occurs when speech sounds are combined with the sight of corresponding articulatory gestures. Despite the longstanding assumption that this kind of audiovisual binding operates in an attention-free mode, recent findings (Alsius et al. in Curr Biol, 15(9):839-843, 2005) suggest that audiovisual speech integration decreases when visual or auditory attentional resources are depleted. The present study addressed the generalization of this attention constraint by testing whether a similar decrease in multisensory integration is observed when attention demands are imposed on a sensory domain that is not involved in speech perception, such as touch. We measured the McGurk illusion in a dual task paradigm involving a difficult tactile task. The results showed that the percentage of visually influenced responses to audiovisual stimuli was reduced when attention was diverted to a tactile task. This finding is attributed to a modulatory effect on audiovisual integration of speech mediated by supramodal attention limitations. We suggest that the interactions between the attentional system and crossmodal binding mechanisms may be much more extensive and dynamic than it was advanced in previous studies.

Attention-deficit/hyperactivity disorder (ADHD) is a common neurodevelopmental disorder in children. Previous studies have shown that children with ADHD have impaired processing of emotional stimuli, but it is unclear whether their ability to integrate multimodal emotional stimuli is impaired and at which processing pathway this impairment exists. The present study investigated the ability of children with ADHD to integrate emotional audiovisual stimuli under different emotional conditions, and the effect of audiovisual integration on IOR to reveal the impaired processing pathway of their emotional audiovisual integration. Fifty-eight school-age children (29 with ADHD and 29 matched typically developing (TD) children) performed an emotional valence discrimination task with a cue-target paradigm. The results showed that children with ADHD did not exhibit audiovisual integration of emotional stimuli in all experimental conditions. In addition, the IOR effect was significantly smaller for audiovisual targets than for visual targets under the negative but not the neutral emotion condition in children with ADHD, whereas this effect was present in all emotion conditions in TD children. These results indicate that the ability to integrate emotional audiovisual information is impaired in children with ADHD and this impairment exists in both bottom-up and top-down pathways. Additionally, although presenting emotional auditory stimuli at the same time as emotional faces reduced IOR both in children with ADHD and TD, the manner of reduction differed. These findings provide new evidence of emotional processing deficits and multimodal integration deficits in children with ADHD, and help provide support for children in educational settings.

The ability of integrating into a unified percept sensory inputs deriving from different sensory modalities, but related to the same external event, is called multisensory integration and might represent an efficient mechanism of sensory compensation when a sensory modality is damaged by a cortical lesion. This hypothesis has been discussed in the present dissertation. Experiment 1 explored the role of superior colliculus (SC) in multisensory integration, testing patients with collicular lesions, patients with subcortical lesions not involving the SC and healthy control subjects in a multisensory task. The results revealed that patients with collicular lesions, paralleling the evidence of animal studies, demonstrated a loss of multisensory enhancement, in contrast with control subjects, providing the first lesional evidence in humans of the essential role of SC in mediating audio-visual integration. Experiment 2 investigated the role of cortex in mediating multisensory integrative effects, inducing virtual lesions by inhibitory theta-burst stimulation on temporo-parietal cortex, occipital cortex and posterior parietal cortex, demonstrating that only temporo-parietal cortex was causally involved in modulating the integration of audio-visual stimuli at the same spatial location. Given the involvement of the retino-colliculo-extrastriate pathway in mediating audio-visual integration, the functional sparing of this circuit in hemianopic patients is extremely relevant in the perspective of a multisensory-based approach to the recovery of unisensory defects. Experiment 3 demonstrated the spared functional activity of this circuit in a group of hemianopic patients, revealing the presence of implicit recognition of the fearful content of unseen visual stimuli (i.e. affective blindsight), an ability mediated by the retino-colliculo-extrastriate pathway and its connections with amygdala. Finally, Experiment 4 provided evidence that a systematic audio-visual stimulation is effective in inducing long-lasting clinical improvements in patients with visual field defect and revealed that the activity of the spared retino-colliculo-extrastriate pathway is responsible of the observed clinical amelioration, as suggested by the greater improvement observed in patients with cortical lesions limited to the occipital cortex, compared to patients with lesions extending to other cortical areas, found in tasks high demanding in terms of spatial orienting. Overall, the present results indicated that multisensory integration is mediated by the retino-colliculo-extrastriate pathway and that a systematic audio-visual stimulation, activating this spared neural circuit, is able to affect orientation towards the blind field in hemianopic patients and, therefore, might constitute an effective and innovative approach for the rehabilitation of unisensory visual impairments.

Chapter 7 - Variants of the Spontaneous Recognition Procedure Assessing Multisensory Integration Reveal Behavioral Alterations in Rodent Models of Psychiatric and Neurological Disorders

Many studies evaluate the multisensory integration to investigate how the human brain combines senses information and weighted each of senses in different situations. The effect of sensory training on multisensory integration is subject to a few of research. The purpose of the present study is to evaluate and compare the effect of both modalities, i.e., visual and proprioception training in multisensory integration. To achieve to this aim, a set of experiment was designed in which the subject was asked to move his/her hand in a circle and estimate its position through a new designed setup. The experiment was performed on 8 subjects with trained proprioception and 8 subjects with visual training. Results of the experiments have three important points: 1) learning rate of visual is significantly more than that of proprioception; 2) the mean of visual and proprioceptive errors are decreased by training but statistical analysis shows this decreasing is significant for proprioceptive error and non-significant for visual error, and 3) visual errors in training phase even in the beginning of it, is much less than the main test stage because in the main test the subject have to focus on two sense.