Auditory attention and the representation of spatial information.

Auditory spatial attention serves important functions in auditory source separation and selection. Although auditory spatial attention mechanisms have been generally investigated, the neural substrates encoding spatial information acted on by attention have not been identified in the human neocortex. We performed functional magnetic resonance imaging experiments to identify cortical regions that support auditory spatial attention and to test 2 hypotheses regarding the coding of auditory spatial attention: 1) auditory spatial attention might recruit the visuospatial maps of the intraparietal sulcus (IPS) to create multimodal spatial attention maps; 2) auditory spatial information might be encoded without explicit cortical maps. We mapped visuotopic IPS regions in individual subjects and measured auditory spatial attention effects within these regions of interest. Contrary to the multimodal map hypothesis, we observed that auditory spatial attentional modulations spared the visuotopic maps of IPS; the parietal regions activated by auditory attention lacked map structure. However, multivoxel pattern analysis revealed that the superior temporal gyrus and the supramarginal gyrus contained significant information about the direction of spatial attention. These findings support the hypothesis that auditory spatial information is coded without a cortical map representation. Our findings suggest that audiospatial and visuospatial attention utilize distinctly different spatial coding schemes.

Developments of head-coupled control/display systems have focused primarily on the display of three dimensional visual information, as the visual system is the optimal sensory channel for the aquisition of spatial information in humans. The auditory system improves the efficiency of vision, however, by obtaining spatial information about relevant objects outside of the visual field of view. This auditory information can be used to direct head and eye movements. Head-coupled display systems, can also benefit from the addition of auditory spatial information, as it provides a natural method of signaling the location of important events outside of the visual field of view. This symposium will report on current efforts in the developments of head-coupled display systems, with an emphasis on the auditory spatial component. The first paper “Virtual Interface Environment Workstations”, by Scott S. Fisher, will report on the development of a prototype virtual environment. This environment consists of a head-mounted, wide-angle, stereoscopic display system which is controlled by operator position, voice, and gesture. With this interface, an operator can virtually explore a 360 degree synthesized environment, and viscerally interact with its components. The second paper, “A Virtual Display System For Conveying Three-Dimensional Acoustic Information” by Elizabeth M. Wenzel, Frederic L. Wightman and Scott H. Foster, will report on the development of a method of synthetically generating three-dimensional sound cues for the above-mentioned interface. The development of simulated auditory spatial cues is limited to some extent, by our knowlege of auditory spatial processing. The remaining papers will report on two areas of auditory space perception that have recieved little attention until recently. “Perception of Real and Simulated Motion in the Auditory Modality”, by Thomas Z. Strybel, will review recent research on auditory motion perception, because a natural acoustic environment must contain moving sounds. This review will consider applications of this knowledge to head-coupled display systems. The last paper, “Auditory Psychomotor Coordination”, will examine the interplay between the auditory, visual and motor systems. The specific emphasis of this paper is the use of auditory spatial information in the regulation of motor responses so as to provide efficient application of the visual channel.

Social presence is crucial for smooth communications in virtual reality (VR). Current telecommunication systems rarely submit spatial auditory information originating from remote people. However, such information may enhance social presence in VR. In this study, we constructed a dynamic binaural synthesis system and investigated the effect of spatial auditory information of a remote partner on a participant’s behavior using the social Simon effect (SSE). The SSE is a spatial stimulus–response compatibility effect between two persons. The SSE occurs when one perceives that their partner is present. Several studies have confirmed the SSE in actual environments. We presented partner sounds diotically (i.e., without spatial information) to one group or binaurally (i.e., with spatial information) to another group through headphones without providing visual information about the partner. The results showed that the SSE was induced only in the binaural group in the current auditory VR (Experiment 1), whereas both groups exhibited the SSE in an actual environment (Experiment 2). These results suggest that the auditory spatial information of remote people is sufficient to induce the SSE and has a potential to enhance social presence.

Position-aware image captioning with spatial relation

Hearing and Looking

The problem of sound source separation and localization is a challenging task not commonly addressed in today’s humanoid robotics projects. Additional (prior to visual) spatial auditory information is required to control humanoid head’s attention more accurately. This is done in order to augment human-robot interaction. Humanoid head robot operates in noisy, dynamic environment. System capable of handling random noise in order to separate sound sources, and localize them in robot’s coordinate system is presented.

Spatial memory is mainly studied through the visual sensory modality: navigation tasks in humans rarely integrate dynamic and spatial auditory information. In order to study how a spatial scene can be memorized on the basis of auditory and idiothetic cues only, we constructed an auditory equivalent of the Morris water maze, a task widely used to assess spatial learning and memory in rodents. Participants were equipped with wireless headphones, which delivered a soundscape updated in real time according to their movements in 3D space. A wireless tracking system (video infrared with passive markers) was used to send the coordinates of the subject's head to the sound rendering system. The rendering system used advanced HRTF-based synthesis of directional cues and room acoustic simulation for the auralization of a realistic acoustic environment. Participants were guided blindfolded in an experimental room. Their task was to explore a delimitated area in order to find a hidden auditory target, i.e., a sound that was only triggered when walking on a precise location of the area. The position of this target could be coded in relationship to auditory landmarks constantly rendered during the exploration of the area. The task was composed of a practice trial, 6 acquisition trials during which they had to memorize the localization of the target, and 4 test trials in which some aspects of the auditory scene were modified. The task ended with a probe trial in which the auditory target was removed. The configuration of searching paths allowed observing how auditory information was coded to memorize the position of the target. They suggested that space can be efficiently coded without visual information in normal sighted subjects. In conclusion, space representation can be based on sensorimotor and auditory cues only, providing another argument in favor of the hypothesis that the brain has access to a modality-invariant representation of external space.

Differential Neural Activation for Updating Rule versus Stimulus Information in Working Memory

Multiple visuospatial working memory buffers: Evidence from spatiotemporal patterns of brain activity

Load effects in auditory selective attention: Evidence for distinct facilitation and inhibition mechanisms

We have been investigating a tabletop interface at which spatial auditory information is presented as well as visual information. This paper describes a case study of using this platform in a classroom to enrich learning materials. The hundred waka poems by one hundred poets, called Hyakunin-Isshu, which is the famous Japanese poetry anthology and known as an intellectual game at home and school, is selected as a subject. Digital cards are spread out at random on the tabletop, where each of them has the second half of a poem. The user makes trials of taking a card to be matched with the first half of the poem that is read by a reciter (speech synthesis software). Auditory cue is given as well, as a hint at the position where the right answer is placed when the learner cannot find the one. Development of the learning material is achieved through user testing with people who are specialized in education. This multimedia framework would help the learner keep interest in learning Hyakunin-Isshu or other school subjects in general.KeywordsComputer-based instructioncard gameinteractive tablemultimodal interfaceauditory hint

Monocular patching may induce ipsilateral “where” spatial bias

Does auditory attention shift in the direction of an upcoming saccade?

Recent efforts to mimic real-life situations in experiments aim to improve the ecological validity of research. Acoustically, this often involves using binaural reproduction to create realistic environments. While studies have shown that simplified acoustic presentations, such as white noise, affect children’s auditory selective attention without impacting adults, other research suggests that these effects might be overestimated in monaural scenarios. This underscores the need for more realistic approaches. The current study introduced spatialized, non-stationary classroom noise with and without speech, comparing it to white noise and a silent baseline in a child-appropriate experiment on auditory selective attention in a virtual reality classroom. Findings from adult participants, who were tested to validate the paradigm, indicated significantly higher error rates for realistic classroom noise compared to white noise and silence. Trials with intelligible speech as a distractor were particularly disruptive to auditory attention. Participants also reported higher mental demand, perceived effort, and task difficulty with complex noise types. These results emphasize the importance of using complex noise scenarios in auditory cognition research to draw conclusions applicable to real-life situations. This approach provides a more accurate understanding of how realistic classroom noise affects auditory selective attention.

Differential cortical hemodynamics during standard and reversed visually guided navigation: An fNIRS-based investigation.