Spatial localization under conflict conditions: is there a single explanation?

Path integration: Effect of curved path complexity and sensory system on blindfolded walking

A significant proportion of speech communication occurs when speakers and listeners are within face-to-face proximity of one other. In noisy and reverberant environments with multiple sound sources, auditory-visual (AV) speech communication takes on increased importance because it offers the best chance for successful communication. This chapter reviews AV processing for speech understanding by normal-hearing individuals. Auditory, visual, and AV factors that influence intelligibility, such as the speech spectral regions that are most important for AV speech recognition, complementary and redundant auditory and visual speech information, AV integration efficiency, the time window for auditory (across spectrum) and AV (cross-modality) integration, and the modulation coherence between auditory and visual speech signals are each discussed. The knowledge gained from understanding the benefits and limitations of visual speech information as it applies to AV speech perception is used to propose a signal-based model of AV speech intelligibility. It is hoped that the development and refinement of quantitative models of AV speech intelligibility will increase our understanding of the multimodal processes that function every day to aid speech communication, as well guide advances in future generation hearing aids and cochlear implants for individuals with sensorineural hearing loss.

In synchronized trampolining, two gymnasts perform the same routine at the same time. While trained gymnasts are thought to coordinate their own movements with the movements of another gymnast by detecting relevant movement information, the question arises how visual and auditory information contribute to the emergence of synchronicity between both gymnasts. Therefore the aim of this study was to examine the role of visual and auditory information in the emergence of coordinated behaviour in synchronized trampolining. Twenty female gymnasts were asked to synchronize their leaps with the leaps of a model gymnast, while visual and auditory information was manipulated. The results revealed that gymnasts needed more leaps to reach synchronicity when only either auditory (12.9 leaps) or visual information (10.8 leaps) was available, as compared to when both auditory and visual information was available (8.1 leaps). It is concluded that visual and auditory information play significant roles in synchronized trampolining, whilst visual information seems to be the dominant source for emerging behavioural synchronization, and auditory information supports this emergence.

The current study used cross-modal oddball tasks to examine cardiac and behavioral responses to changing auditory and visual information. When instructed to press the same button for auditory and visual oddballs, auditory dominance was found with cross-modal presentation slowing down visual response times more than auditory response times (Experiment 1). When instructed to make separate responses to auditory and visual oddballs, visual dominance was found with cross-modal presentation decreasing auditory discrimination, and participants also made more visual-based than auditory-based errors on cross-modal trials (Experiment 2). Experiment 3 increased task demands while requiring a single button press and found evidence of auditory dominance, suggesting that it is unlikely that increased task demands can account for the reversal in Experiment 2. Auditory processing speed was the best predictor of auditory dominance, with auditory dominance being stronger in participants who were slower at processing the sounds, whereas auditory and visual processing speed and baseline heart rate variability did not predict visual dominance. Examination of cardiac responses that were time-locked with stimulus onset showed cross-modal facilitation effects, with auditory and visual discrimination occurring earlier in the course of processing in the cross-modal condition than in the unimodal conditions. The current findings showing that response demand manipulations reversed modality dominance and that time-locked cardiac responses show cross-modal facilitation, not interference, suggest that auditory and visual dominance effects may both be occurring later in the course of processing, not from disrupted encoding.

The aim of this research was to verify whether perceived distance anisotropy exists even when visual information are available, that is, to examine interaction of non-visual (vestibular and proprioceptive) and visual information in depth perception. An experiment was done in a forest, where there is a large number of equally distributed depth cues. Partiticipants were instructed to equalize the distance of the stimuli towards horizon with the distance of the standard towards zenith. As a stimuli we used paper circles, 30cm in diameter, and participants performed their task from standing and lying position. Results have shown that estimated distances towards horizon were longer than physically identical distances towards zenith, which means that distances towards horizon are being perceived as shorter. Shorten age of perceived distances towards horizon was smaller from lying position. These findings point out significance of vestibular and proprioceptive information on percieved distance anisotropy, regardless of presence of visual information.

Research in visuo-motor coupling has shown that the matching of visual and proprioceptive information is important for calibrating movement. Many state-of-the art virtual reality (VR) systems, commonly known as immersive virtual environments (IVE), are created for training users in tasks that require accurate manual dexterity. Unfortunately, these systems can suffer from technical limitations that may force de-coupling of visual and proprioceptive information due to interference, latency, and tracking error. We present an empirical evaluation of how visually distorted movements affects users' reach to near field targets using a closed-loop physical reach task in an IVE. We specifically examined the recalibration of movements when the visually reached distance is scaled differently than the physically reached distance. Subjects were randomly assigned to one of three visual feedback conditions during which they reached to target while holding a tracked stylus: i) Condition 1 (-20% gain condition) in which the visual stylus appeared at 80% of the distance of the physical stylus, ii) Condition 2 (0% or no gain condition) in which the visual stylus was co-located with the physical stylus, and iii) Condition 3 (+20% gain condition) in which the visual stylus appeared at 120% of the distance of the physical stylus. In all conditions, there is evidence of visuo-motor calibration in that users' accuracy in physically reaching to the target locations improved over trials. During closed-loop physical reach responses, participants generally tended to physically reach farther in condition 1 and closer in condition 3 to the perceived location of the targets, as compared to condition 2 in which participants' physical reach was more accurate to the perceived location of the target.

Three chronometric experiments, each comparing vision and kinesthesis, were conducted to study visual dominance. The time required to switch attention from vision and from kinesthesis was equal, while switching to kinesthesis was faster than switching to vision (Experiment 1). Responses to a combined visual-kinesthetic stimulus were slower than responses to a kinesthetic stimulus alone when the subject was expecting the bimodal stimulus. The visual dominance effect was shown to depend on the subject knowing the modality of the stimulus in advance (Experiment 2). When subjects were instructed to attend one modality they had equal difficulty with conflicting visual and kinesthetic information (Experiment 3). These findings suggest that visual dominance results from a bias to attend vision when that modality seems adequate for the task. Te bias to attend vision may develop to overcome a deficiency in visual alerting.

Although speech perception is usually considered an auditory process, studies have shown that visual information provided by the movements of a talker’s mouth and face strongly influences what an observer perceives even when the auditory signal is clear and unambiguous. The now classic “McGurk effect” is an excellent example: the auditory syllable /bi/ presented in synchrony with a videotape of a talker saying the syllable /gi/ is usually perceived as /di/, a syllable not presented to either modality. This phenomenon has important implications for theories of speech perception which must account for how and why auditory and visual signals are integrated during phonetic processing. The purpose of the current talk is to describe some of the research that has contributed to our understanding of how visual and auditory information are used in phonetic perception. Specifically, this paper focuses on three issues: (1) what factors influence the integration of auditory and visual information; (2) the nature of the auditory and visual information used in phonetic perception; and (3) the interaction of the two sources of information during phonetic processing. The implications of these findings for theories of speech perception are discussed.

Memory for proprioceptive and multisensory targets is partially coded relative to gaze

Although it has been documented that musical training enhances multisensory integration, there is not yet a consensus as to how musical training influences the visual dominance effect in sensory dominance. The present study adopted the Colavita visual dominance paradigm, presenting auditory stimuli concurrent with visual stimuli, to investigate the visual dominance effect between music majors and nonmusic majors and compared the reaction time and response proportion of the two kinds of participants in the bimodal trials. The results showed that the proportion of simultaneous responses in bimodal trials of music majors is higher than that of nonmusic majors; the nonmusic majors show a greater difference between the proportion of “Visual-Auditory” trials and “Auditory-Visual” trials compared with the music majors; the ΔRT of the two responses of the nonsimultaneous bimodal trials of nonmusic majors is longer than that of music majors. The results indicated that musically trained individuals have an enhanced ability to bind visual and auditory information and show a lesser Colavita effect, that is, a reduced visual dominance effect, than their nonmusic major peers. We conclude that musical training extends beyond the field of vision or auditory domain, improves audiovisual integration, and reduces the visual dominance effect.

Animals can adopt different navigation strategies according to the environment and the task they have to solve. The process of selecting the appropriate navigation strategy in mammals involves parallel pathways. The hippocampus has been thought to be the potential basis for a first type of navigation strategy in which a representation of the space is required. This representation uses spatially tuned neurons (place cells) found in the rat hippocampus. A second, different, navigation strategy that involves the dorsal striatum in the basal ganglia, is used by rats if the target can be identified by a visible cue. This work presents a computational model of the rat hippocampus and its interactions with the basal ganglia, able to reproduce neurobehavioral experiments regarding self-localisation, integration of sensory and proprioceptive information, and automatic selection of appropriate navigation strategies.

One of the requirements for being a successful musical conductor is to be able to locate sounds instantaneously in time and space. Because this requires the integration of auditory and visual information, the purpose of this study was to examine multisensory processing in conductors and a matched set of control subjects. Subjects participated in a series of behavioral tasks, including pitch discrimination, temporal-order judgment (TOJ), and target localization. Additionally, fMRI scans were done on a subset of subjects who performed a multisensory TOJ task. Analyses of behavioral data indicate that, in the auditory realm, conductors were more accurate in both pitch discrimination and TOJs as well as in locating targets in space. Furthermore, these same subjects also demonstrated a benefit from the combination of auditory and visual information that was not observed in control subjects when locating visual targets. Finally, neural substrates in BA 37, 39/40 were identified as potential areas underlying the conductors' superior multisensory TOJs. Data collection and analyses are ongoing and will lead to an improved understanding of multisensory integration in a complex, musical behavior.

Two experiments examined the role of attention in visual dominance during motor learning. On the movement task 10 acquisition trials were given; each included a movement presentation and a blindfolded reproduction. After completion of acquisition trials and a 5-min. interval, subjects were given 5 retention trials with reproduction attempts only. In Exp. 1, subjects receiving only kinesthetic information during movement presentation reproduced criterion movement length more accurately than subjects receiving visual and kinesthetic information. Other subjects, presented both visual and kinesthetic stimuli for the movement, were given instructions to ignore vision and focus on kinesthesis. These subjects exhibited no effects of visual dominance in reproductions. In Exp. 2, subjects were presented visual and kinesthetic stimuli during half of the movement presentations and only kinesthetic stimuli during the other half. They did not exhibit the effects of visual dominance in reproductions. Such effects in motor learning may be modified by manipulation of attention or an alternating presentation of specific sensory stimuli.

Both visual and body-based (vestibular and proprioceptive) information contribute to spatial updating, or the way a navigator keeps track of self-position during movement. Research has tested the relative contributions of these sources of information and found mixed results, with some studies demonstrating the importance of body-based information, especially for translation, and some demonstrating the sufficiency of visual information. Here, we invoke an individual differences approach to test whether some individuals may be more dependent on certain types of information compared to others. Movement experts tend to be dependent on motor processes in small-scale spatial tasks, which can help or hurt performance, but it is unknown if this effect extends into large-scale spatial tasks like spatial updating. In the current study, expert dancers and non-dancers completed a virtual reality point-to-origin task with three locomotion methods that varied the availability of body-based and visual information for translation: walking, joystick, and teleporting. We predicted decrements in performance in both groups as self-motion information was reduced, and that dancers would show a larger cost. Surprisingly, both dancers and non-dancers performed with equal accuracy in walking and joystick and were impaired in teleporting, with no large differences between groups. We found slower response times for both groups with reductions in self-motion information, and minimal evidence for a larger cost for dancers. While we did not see strong dance effects, more participation in spatial activities related to decreased angular error. Together, the results suggest a flexibility in reliance on visual or body-based information for translation in spatial updating that generalizes across dancers and non-dancers, but significant decrements associated with removing both of these sources of information.

Many researchers have taken the Colavita effect to represent a paradigm case of visual dominance. Broadly defined, the effect occurs when people fail to respond to an auditory target if they also have to respond to a visual target presented at the same time. Previous studies have revealed the remarkable resilience of this effect to various manipulations. In fact, a reversal of the Colavita visual dominance effect (i.e., auditory dominance) has never been reported. Here, we present a series of experiments designed to investigate whether it is possible to reverse the Colavita effect when the target stimuli consist of repetitions embedded in simultaneously presented auditory and visual streams of stimuli. In line with previous findings, the Colavita effect was still observed for an immediate repetition task, but when an n-1 repetition detection task was used, a reversal of visual dominance was demonstrated. These results suggest that masking from intervening stimuli between n-1 repetition targets was responsible for the elimination and reversal of the Colavita visual dominance effect. They further suggest that varying the presence of a mask (pattern, conceptual, or absent) in the repetition detection task gives rise to different patterns of sensory dominance (i.e., visual dominance, an elimination of the Colavita effect, or even auditory dominance).