Visual influences on auditory spatial processing

Characteristics of neurons in auditory cortex of monkeys performing a simple auditory task.

Reliability and Representational Bandwidth in the Auditory Cortex

Spatial hearing, which largely relies on binaural time/level cues, is a challenge for patients with asymmetric hearing. The degree of the deficit is largely variable, and better sound localization performance is frequently reported. Studies on the compensatory mechanism revealed that monaural level cues and monoaural spectral cues contribute to variable behavior in those patients who lack binaural spatial cues. However, changes in the monaural level cues have not yet been separately investigated. In this study, the use of the level cue in sound localization was measured using stimuli of 1 kHz at a fixed level in patients with single-sided deafness (SSD), the most severe form of asymmetric hearing. The mean absolute error (MAE) was calculated and related to the duration/age onset of SSD. To elucidate the biological correlate of this variable behavior, sound localization ability was compared with the cortical volume of the parcellated auditory cortex. In both SSD patients (n = 26) and normal controls with one ear acutely plugged (n = 23), localization performance was best on the intact ear side; otherwise, there was wide interindividual variability. In the SSD group, the MAE on the intact ear side was worse than that of the acutely plugged controls, and it deteriorated with longer duration/younger age at SSD onset. On the impaired ear side, MAE improved with longer duration/younger age at SSD onset. Performance asymmetry across lateral hemifields decreased in the SSD group, and the maximum decrease was observed with the most extended duration/youngest age at SSD onset. The decreased functional asymmetry in patients with right SSD was related to greater cortical volumes in the right posterior superior temporal gyrus and the left planum temporale, which are typically involved in auditory spatial processing. The study results suggest that structural plasticity in the auditory cortex is related to behavioral changes in sound localization when utilizing monaural level cues in patients with SSD.

Audiovisual information reaches the brain via both sustained and transient input channels, representing signals’ intensity over time or changes thereof, respectively. To date, it is unclear to what extent transient and sustained input channels contribute to the combined percept obtained through multisensory integration. Based on the results of two novel psychophysical experiments, here we demonstrate the importance of the transient (instead of the sustained) channel for the integration of audiovisual signals. To account for the present results, we developed a biologically-inspired, general-purpose model for multisensory integration, the Multisensory Correlation Detectors, which combines correlated input from unimodal transient channels. Besides accounting for the results of our psychophysical experiments, this model could quantitatively replicate several recent findings in multisensory research, as tested against a large collection of published datasets. In particular, the model could simultaneously account for the perceived timing of audiovisual events, multisensory facilitation in detection tasks, causality judgments, and optimal integration. All-in-all, this study demonstrates that several phenomena in multisensory research that were previously considered unrelated, all stem from the integration of correlated input from unimodal transient channels.

With the exception of low‐frequency characteristic frequency (CF) units, the ipsilaterally evoked discharge of the majority of S‐segment cells can be inhibited with simultaneous stimulation of the contralateral ear. The contralateral inhibitory CF is usually identical to the ipsilateral excitatory CF. The width of the contralateral inhibitory tuning curve is similar to the width of the ipsilateral excitatory tuning curve. Partial inhibition of the ipsilaterally evoked discharge increases the interspike intervals and increases the variability of the discharge. Inhibition, like excitation, is tonically acting. The contralateral inhibitory input with equally intense tones (measured in terms of decibels above threshold) is either about as effective as the excitatory input or less effective, rarely more effective. The primary factor determining the spike output from S‐segment cells is not the absolute intensity level of the stimulus but rather the relative intensity difference of the stimulus at the two ears. [Work supported in part by National Institutes of Health, U. S. Department of Health, Education, and Welfare.]

Functional specialization in non-primary auditory cortex of the cat: Areal and laminar contributions to sound localization

Objective To explore the possible mechanism of miR-34a/Bcl-2 apoptotic pathway in the mouse model of age-related hearing loss. Methods A C57BL/6 mouse model of age-related hearing loss was conducted, and 4-week-old and 12-month-old mice were considered as the objects. The auditory brainstem response (ABR) was used to test the hearing function. The TdT-mediated dUTP nick end labeling (TUNEL) was used to observe the apoptosis of neuron in auditory cortex. The mRNA and protein levels of miR-34a, Bcl-2 and caspase 3 were detected by real-time PCR and Western bloting, respectively. Results The ABR showed that the hearing threshold level at 4, 8, 16, 32 kHz were higher in 12-month-old mice than in 4-week-old mice〔(80.0±2.5) dBHL vs. (32.0±4.5) dBHL, (74.0±3.5) dBHL vs. (51.0±1.2) dBHL, (86.0±4.6) dBHL vs. (51.0±3.5) dBHL, (87.0±6.6) dBHL vs. (56.0±1.5) dBHL, all P<0.05〕. Compared with 4-week-old mice, the total number of neurons in auditory cortex was decreased, the number of apoptosis neurons was increased, the expressions of miR-34a (t=6.02, P=0.001), Bax (t=6.51, P=0.012) and Caspase-3 (P=0.023) rised, and the expression of Bcl-2 (t=7.12, P=0.032) declined in 12-month-old mice. Conclusions The activation of miR-34a/Bcl-2 apoptotic pathway may be one of the mechanisms of age-related hearing loss. Key words: Presbycusis; Apoptosis

Introduction to dual sensory loss issue.

It is relatively easy for a human listener to attend to a particular speaker at a cocktail party in the presence of other speakers, music and environmental sounds. To perform this task, the human listener needs to separate the target speech from a mixture of multiple concurrent sources reflected by various surfaces. This process is referred to as auditory scene analysis. While humans excel at this task using only two ears, machine separation based on two-microphone recordings has proven to be extremely challenging. By incorporating the mechanisms underlying the perception of sound by human listeners, computational auditory scene analysis (CASA) offers a new approach to sound segregation. Binaural hearing – hearing with two ears – employs the difference in sound source locations to improve sound segregation. In this chapter, we describe the principles of binaural processing and review the state-of-the-art in binaural CASA, particularly for speech segregation.

The ability of integrating into a unified percept sensory inputs deriving from different sensory modalities, but related to the same external event, is called multisensory integration and might represent an efficient mechanism of sensory compensation when a sensory modality is damaged by a cortical lesion. This hypothesis has been discussed in the present dissertation. Experiment 1 explored the role of superior colliculus (SC) in multisensory integration, testing patients with collicular lesions, patients with subcortical lesions not involving the SC and healthy control subjects in a multisensory task. The results revealed that patients with collicular lesions, paralleling the evidence of animal studies, demonstrated a loss of multisensory enhancement, in contrast with control subjects, providing the first lesional evidence in humans of the essential role of SC in mediating audio-visual integration. Experiment 2 investigated the role of cortex in mediating multisensory integrative effects, inducing virtual lesions by inhibitory theta-burst stimulation on temporo-parietal cortex, occipital cortex and posterior parietal cortex, demonstrating that only temporo-parietal cortex was causally involved in modulating the integration of audio-visual stimuli at the same spatial location. Given the involvement of the retino-colliculo-extrastriate pathway in mediating audio-visual integration, the functional sparing of this circuit in hemianopic patients is extremely relevant in the perspective of a multisensory-based approach to the recovery of unisensory defects. Experiment 3 demonstrated the spared functional activity of this circuit in a group of hemianopic patients, revealing the presence of implicit recognition of the fearful content of unseen visual stimuli (i.e. affective blindsight), an ability mediated by the retino-colliculo-extrastriate pathway and its connections with amygdala. Finally, Experiment 4 provided evidence that a systematic audio-visual stimulation is effective in inducing long-lasting clinical improvements in patients with visual field defect and revealed that the activity of the spared retino-colliculo-extrastriate pathway is responsible of the observed clinical amelioration, as suggested by the greater improvement observed in patients with cortical lesions limited to the occipital cortex, compared to patients with lesions extending to other cortical areas, found in tasks high demanding in terms of spatial orienting. Overall, the present results indicated that multisensory integration is mediated by the retino-colliculo-extrastriate pathway and that a systematic audio-visual stimulation, activating this spared neural circuit, is able to affect orientation towards the blind field in hemianopic patients and, therefore, might constitute an effective and innovative approach for the rehabilitation of unisensory visual impairments.

This review discusses hearing performance in primates and selective pressures that may influence it. The hearing sensitivity and sound-localization abilities of primates, as indicated by behavioral tests, are reviewed and compared to hearing and sound localization among mammals in general. Primates fit the mammalian pattern with small species hearing higher frequencies than larger species in order to use spectral/intensity cues for sound localization. In this broader comparative context, the restricted high-frequency hearing of humans is not unusual. All of the primates tested so far are able to hear frequencies below 125 Hz, placing them among the majority of mammals. Sound-localization acuity has been determined for only three primates, and here also they have relatively good localization acuity (with a minimum audible angle roughly similar to other mammals such as cats, pigs, and opossums). This is in keeping with the pattern among mammals in general, in which species with narrow fields of best vision, such as a fovea, are better localizers than those with broad fields of best vision. Multiple lines of evidence support the view that sound localization is the selective pressure on smaller primates and on other mammals with short interaural distances for hearing high frequencies.

Binaural hearing, the ability to detect small differences in the timing and level of sounds at the two ears, underpins the ability to localize sound sources along the horizontal plane, and is important for decoding complex spatial listening environments into separate objects – a critical factor in ‘cocktail-party listening’. For human listeners, the most important spatial cue is the interaural time difference (ITD). Despite many decades of neurophysiological investigations of ITD sensitivity in small mammals, and computational models aimed at accounting for human perception, a lack of concordance between these studies has hampered our understanding of how the human brain represents and processes ITDs. Further, neural coding of spatial cues might depend on factors such as head-size or hearing range, which differ considerably between humans and commonly used experimental animals. Here, using magnetoencephalography (MEG) in human listeners, and electro-corticography (ECoG) recordings in guinea pig—a small mammal representative of a range of animals in which ITD coding has been assessed at the level of single-neuron recordings—we tested whether processing of ITDs in human auditory cortex accords with a frequency-dependent periodic code of ITD reported in small mammals, or whether alternative or additional processing stages implemented in psychoacoustic models of human binaural hearing must be assumed. Our data were well accounted for by a model consisting of periodically tuned ITD-detectors, and were highly consistent across the two species. The results suggest that the representation of ITD in human auditory cortex is similar to that found in other mammalian species, a representation in which neural responses to ITD are determined by phase differences relative to sound frequency rather than, for instance, the range of ITDs permitted by head size or the absolute magnitude or direction of ITD.

Stimulus localization and particularly directional hearing can be considered as methods for investigating neural activity and they have proven to be useful tools for research in physiology and psychology. Human directional hearing techniques have been reflected upon way back by Von Bekesy in Austrian forests [1]. For example, he observed that some of the roads took a perfectly straight course through deep, dark woods. He could not imagine how such straight roads had been cut through the forest when the usual optical methods used by road surveyors would seem to be useless in this case. Further some of these roads were very old and probably built before the introduction of the theodolite. Many of these roads were laid out by an acoustic method. How did they do it? A man stationed at the starting point noted the direction of the sound produced by someone at the other end blowing a horn. The first man then walked toward the sound source, marking the threes on the way. It turned out that this method produced a straight line from start to finish [1]. From this observation Bekesy was motivated to perform a series of studies on stimuli localization not limited to hearing but also to vibration sensations on the skin, electrical pulses on the tongue and odors through the nose as well. Strikingly, his results showed an underlying ubiquitous mechanism present in the different stimuli localization modalities. For instance, the effect on localization of the time delay between two stimuli on the skin, the tongue, the two nostrils in the nose and the two ears, presented the same dynamics [2-4]. These results were quite exciting because it showed that, in humans, the senses work similarly for stimuli localization although the basic underlying neural pathways are not the same. It was this kind of general principle on stimuli localization that motivated us in the search for more general principles related to how senses interact to generate multisensory perceptions but with a special emphasis on auditory stimulation. This is known as multisensory integration and its study is very important because it is the foundation of how humans bind all the information coming from the senses to generate a coherent percept. We began by studying something that we called cross-modal stochastic resonance. This consists

The environment often engages multiple sensory channels simultaneously. When events or objects stimulate more than one sensory organ, they are more likely to be detected and more efficiently processed than events stimulating just one sensory modality (e.g. Stein & Stanford, 2008). Such crossmodal enhancement requires that the sensory inputs arising from one event are correctly assigned to that object (termed the ‘correspondence problem’; Ernst & Bulthoff, 2004). Multisensory research has demonstrated that features that are shared by all sensory systems, such as space, time or meaning are used for crossmodal binding (for recent reviews see Driver & Noesselt, 2008; Stein & Stanford, 2008). For example, in a cocktail party setting, matching of the temporal dynamics of lip movements and of speech sound allows us to assign the correct message to each speaker. Furthermore, this ability of crossmodal binding assists in orienting spatial attention to a particular conversation partner while ignoring other sources of distracting sensory information, such as other peoples’ conversations. The binding of synchronous auditory and visual (speech) signals seems to happen automatically and pre-attentively (see Driver, 1996; Bertelson et al., 2000; Vroomen et al., 2001a). The voluntary orienting of spatial attention to one particular speaker or, more generally, to an event is an example of endogenous attention allocation. By contrast, stimulus-driven orienting of spatial attention, elicited by the saliency of the stimulus, has been termed exogenous attention. Due to their relatively high salience, such pre-attentively detected, congruent cross-modal stimuli may function as exogenous attentional cues (Spence & Driver, 2000; Stekelenburg et al., 2004; Senkowski et al., 2008). In this issue, Fairhall & Macaluso (2009) provide evidence for the operation of both mechanisms: (i) pre-attentively crossmodal matching and (ii) enhancement of crossmodal interaction by endogenous attention. The authors presented the lips of two speaking faces, one in the left and one in the right visual hemifield. The centrally presented auditory stream was matched with the lip movements of one speaker only. In such a situation people generally perceive the auditory speech signal as originating from the location of the matching lips. This phenomenon is well known as the ‘ventriloquist effect’. Critically, Fairhall and Macaluso also asked participants to perform a visual detection task in one hemifield only. This task served to systematically manipulate the focus of visual spatial attention. The auditory stream did not have any significance to the participants and thus could be ignored. This paradigm allowed the authors to compare brain responses as assessed with functional magnetic resonance imaging (fMRI) to multisensory, matching audio-visual speech sequences that occurred either at the location of visual spatial attention (‘congruent condition’) or on the opposite side (‘incongruent condition’) while keeping physical stimulation constant. The authors report that visual attention enhances activity in brain regions well accepted as multisensory integration areas (Driver & Noesselt, 2008), such as the superior temporal sulcus (STS) and the superior colliculus (SC), only when the auditory track matched the attended lip movements. Similar effects were observed in several visual brain regions, including the primary visual cortex. These results suggest that the processing of congruent cross-modal stimuli within the locus of endogenous visual spatial attention is enhanced. Furthermore, the data of Fairhall & Macaluso are compatible with the view that binding of audio-visual speech can occur pre-attentively. When audio-visual matching speech was presented on the opposite side of the locus of visual-spatial attention (incongruent condition), participants performed worse in the visual detection task than when matching stimuli were presented on the same side as the focus of participant’s endogenous visual spatial attention (congruent condition). These behavioural results, together with findings from earlier studies (Spence & Driver, 2000; Vroomen et al., 2001b), suggest that the audio-visual binding process that elicited a ventriloquist illusion towards the opposite side of the visual attention focus took place outside the locus of endogenous attention and resulted in an exogenously attentional shift towards the task irrelevant side, which in turn interfered with the visual detection task. In summary, the results described by Fairhall and Macaluso suggest that endogenous (visual) spatial attention amplifies the processing of congruent cross-modal input that is likely integrated and detected pre-attentively. Fairhall and Macaluso did not attempt to link the enhanced activity in multisensory and early visual cortical regions to behavioural data such as, for example, improved speech comprehension. The question of whether activity changes in multisensory regions due to endogenous spatial attention are accompanied by altered multisensory performance (Alsius et al., 2005) remains to be investigated in future work. Moreover, other top-down influences on multisensory integration such as plausibility and meaning (Kitagawa & Spence, 2005; Hein et al., 2007) will need to be investigated in more detail to extend our understanding of the conditions when and which multisensory binding processes are subject to voluntary modulation. European Journal of Neuroscience, Vol. 29, pp. 1245–1246, 2009 doi:10.1111/j.1460-9568.2009.06715.x

of congruent colors and shapes did impact taste perception. Participants associated angular shapes and the green color with sourness. The study highlights the role of visual cues in shaping taste experiences.Whereas the previous study altered passive perception, Parra et al. 5 explored how a conversational agent, MyndFood, can actively promote mindful cooking and eating. In a study with 40 participants, those who interacted with the mindful version of the agent reported greater enjoyment and food awareness while preparing sushi compared to those who interacted with a non-mindful version. The findings suggest that conversational agents can enhance the cooking and eating experience by promoting mindfulness.Addressing the different challenge of accessibility, Welewatta et al. 6 developed SEMA, a multisensory system that uses tactile, auditory, and somatosensory cues to help visually impaired students experience and interpret paintings. Tested with 22 visually impaired participants, the system showed strong potential, with 92.6% of users preferring it, and could enhance art experiences for the visually impaired community.Finally, turning to human-machine interaction in critical settings, Evangelou et al. 7 studied how mid-air haptic technology affects user agency in gesture-based interactions, particularly in automotive settings. They found that mid-air haptics increased users' sense of agency compared to visual feedback, regardless of the meaning behind the feedback. The study involved 36 participants who completed a time perception task using gesture poses with different sensory feedback types. The results showed a positive association between mid-air haptics and user agency, trust, and usability, highlighting its potential benefits in automotive contexts.Taken together, these five articles illustrate the breadth and potential of multisensory research at the frontier of disciplines.They demonstrate how technology can both model and personalize our environments to improve cognitive performance (Bartoletti et al. 3 ), while also revealing how specific visual cues can alter our core perception of taste (Guberman et al. 4 ). These studies pave the way for concrete applications, whether by enriching daily experiences like cooking (Parra et al. 5 ) or by compensating for sensory deficits to make art accessible to the visually impaired (Welewatta et al. 6 ).Finally, they highlight the importance of sensory feedback in enhancing user agency and trust in complex interactions, such as in-car gesture controls (Evangelou et al. 7 ). These contributions, spanning cognition, perception, accessibility, and human-computer interaction, confirm that multisensory integration is a critical field for shaping future human experiences.