Coordinates of Human Visual and Inertial Heading Perception

The superior colliculus (SC) is a multilayered structure in the mammalian midbrain. Even though the behavioral function of the SC has been studied for well over 30 yr, it continues to be a source of surprising new findings. Traditionally it has been assumed that the primate SC selects and generates

1. In the companion paper we reported that the predominant signal of the population of neurons in the lateral intraparietal area (area LIP) of the monkey's posterior parietal cortex (PPC) encode the next intended saccadic eye movement during the delay period of a memory-saccade task. This result predicts that, should be monkey change his intention of what the next saccade will be, LIP activity should change accordingly to reflect the new plan. We tested this prediction by training monkeys to change their saccadic plan on command and recording the activity of LIP neurons across plan changes. 2. We trained rhesus monkeys (Macaca mulatta) to maintain fixation on a light spot as long as this spot remained on. During this period we briefly presented one, two, or three peripheral visual stimuli in sequence, each followed by a delay (memory period, M). After the final delay the fixation spot was extinguished, and the monkey had to quickly make a saccade to the location of the last target to have appeared. The monkey could not predict which stimuli, nor how many, would appear on each trial. He thus had to plan a saccade to each stimulus as it appeared and change his saccade plan whenever a stimulus appeared at a different location. 3. We recorded the M period activity of 81 area LIP neurons (from 3 hemispheres of 2 monkeys) in this task. We predicted that, if a neuron's activity reflected the monkey's planned saccade, its activity should be high while the monkey planned a saccade in the neuron's motor field (MF), and low while the planned saccade was in the opposite direction. The activity of most of the neurons in our sample changed in accordance with our hypothesis as the monkey's planned saccade changed. 4. In one condition the monkey was instructed by visual stimuli to change his plan from a saccade in the neuron's preferred direction to a saccade planned in the opposite direction. In this condition activity decreased significantly (P < 0.05) in 65 (80%) of 81 neurons tested. These neurons' activity changed to reflect the new saccade plan even though the cue for this change was not presented in their RF. 5. As a control we randomly interleaved, among trials requiring a plan change, trials in which the monkey had to formulate two consecutive plans to make a saccade in the neuron's preferred direction. The activity remained unchanged (P < 0.05) in 22 of 31 neurons tested (79%), indicating that the neurons continued to encode the same saccade plan. 6. In a variant of the task, the cue to the location of the required saccade was either a light spot or a noise burst from a loudspeaker. Of 22 neurons tested in this task, 16 (73%) showed activity changes consistent with plan changes cued by visual or auditory stimuli. 7. Alterations in the monkey's intentions, even in the absence of overt behavior, are manifested in altered LIP activity. These activity changes could be induced whether visual or auditory cues were used to indicate the required plan changes. Most LIP neurons thus do not encode only the locations of visual stimuli, but also the intention to direct gaze to specific locations, independently of whether a gaze shift actually occurs.

Saccades are rapid eye movements that turn both eyes in the same direction. Brainstem omnipause neurons gate saccades. Most natural shifts of the fixation point are between targets lying in different directions and at different distances requiring a combination of saccades and vergence. We investigated whether the visual stimulus, the saccadic command or the vergence command turns off omnipause neurons during gaze shifts. Using the scleral search coil technique, eye movements were measured in seven normal subjects, as they made voluntary, disjunctive gaze shifts comprising saccades and vergence movements. Conjugate oscillations of small amplitude and high frequency were identified during the vergence movement that followed the initial saccade. These oscillations, which are an indication that the omnipause neurons are turned off, corresponded to the saccade onset, and not the vergence eye movement or the visual stimulus. These data were used to test and modify a model of the human saccade-vergence system.

In this article results of several published studies are synthesized in order to address the neural system for the determination of eye and head movement amplitudes of horizontal eye/head gaze shifts with arbitrary initial head and eye positions. Target position, initial head position, and initial eye position span the space of physical parameters for a planned eye/head gaze saccade. The principal result is that a functional mechanism for determining the amplitudes of the component eye and head movements must use the entire space of variables. Moreover, it is shown that amplitudes cannot be determined additively by summing contributions from single variables. Many earlier models calculate amplitudes as a function of one or two variables and/or restrict consideration to best-fit linear formulae. Our analysis systematically eliminates such models as candidates for a system that can generate appropriate movements for all possible initial conditions. The results of this study are stated in terms of properties of the response system. Certain axiom sets for the intrinsic organization of the response system obey these properties. We briefly provide one example of such an axiomatic model. The results presented in this article help to characterize the actual neural system for the control of rapid eye/head gaze shifts by showing that, in order to account for behavioral data, certain physical quantities must be represented in and used by the neural system. Our theoretical analysis generates predictions and identifies gaps in the data. We suggest needed experiments.

A head movement propensity model for animating gaze shifts and blinks of virtual characters

The effect of visual stimulus significance on saccade latency and accuracy in gaze shifts was investigated. Stimulus significance refers to the temporal proximity of an imperative stimulus. It was hypothesized that stimulus significance serves as a cue that influences the observer's expectancy for a forthcoming event. Twenty-one undergraduate students participated in two experiments involving monitoring of visual events. Stimuli were a sequence of 1,000 single-digit integers presented one at a time at 9 degrees of visual angle to the left or right of a fixation point in random order. The subject was required to make a manual response after the presentation of a sequence of two odd integers. The probability of the subsequent integer being odd, a "target," after the presentation of an odd integer was 0.75. The integer after a target was always an even integer. When stimulus location was unpredictable (Experiment 1), latency of visually guided saccades to the target was shorter (p < .01) than to the nontarget stimuli. Under the condition of predictable stimulus location (Experiment 2), anticipatory saccades to the target occurred earlier (p < .01), more frequently (p < .01), and more accurately (p < .05). Our results suggest that both timing and accuracy of gaze shift are affected by the observer's expectation of future events.

In this paper, we show that human saccadic eye movements toward a visual target are generated with a reduced latency when this target is spatially and temporally aligned with an irrelevant auditory nontarget. This effect gradually disappears if the temporal and/or spatial alignment of the visual and auditory stimuli are changed. When subjects are able to accurately localize the auditory stimulus in two dimensions, the spatial dependence of the reduction in latency depends on the actual radial distance between the auditory and the visual stimulus. If, however, only the azimuth of the sound source can be determined by the subjects, the horizontal target separation determines the strength of the interaction. Neither saccade accuracy nor saccade kinematics were affected in these paradigms. We propose that, in addition to an aspecific warning signal, the reduction of saccadic latency is due to interactions that take place at a multimodal stage of saccade programming, where the perceived positions of visual and auditory stimuli are represented in a common frame of reference. This hypothesis is in agreement with our finding that the saccades often are initially directed to the average position of the visual and the auditory target, provided that their spatial separation is not too large. Striking similarities with electrophysiological findings on multisensory interactions in the deep layers of the midbrain superior colliculus are discussed.

Heading direction is determined from visual and vestibular cues. Both sensory modalities have been shown to have better direction discrimination for headings near straight ahead. Previous studies of visual heading estimation have not used the full range of stimuli, and vestibular heading estimation has not previously been reported. The current experiments measure human heading estimation in the horizontal plane to vestibular, visual, and spoken stimuli. The vestibular and visual tasks involved 16 cm of platform or visual motion. The spoken stimulus was a voice command speaking a heading angle. All conditions demonstrated direction dependent biases in perceived headings such that biases increased with headings further from the fore-aft axis. The bias was larger with the visual stimulus when compared with the vestibular stimulus in all 10 subjects. For the visual and vestibular tasks precision was best for headings near fore-aft. The spoken headings had the least bias, and the variation in precision was less dependent on direction. In a separate experiment when headings were limited to ±45°, the biases were much less, demonstrating the range of headings influences perception. There was a strong and highly significant correlation between the bias curves for visual and spoken stimuli in every subject. The correlation between visual-vestibular and vestibular-spoken biases were weaker but remained significant. The observed biases in both visual and vestibular heading perception qualitatively resembled predictions of a recent population vector decoder model (Gu et al., 2010) based on the known distribution of neuronal sensitivities.

1. Monkeys were trained to look to brief visual targets presented in a completely darkened room. On some trials, after the visual target disappeared but before a saccade to the target could be initiated, the eyes were driven to another position in the orbit by electrical stimulation of the superior colliculus. Retinocentric models of the saccadic system predict that a saccade with a predetermined distance and direction based entirely on retinal error will occur. If this were the case, gaze would miss the target location by a distance and direction equal to the vector of the stimulation-induced movement. Spatial models assume that the retinal error signal will be combined with information about the change in eye position produced by stimulation and predict that the animal will look to the position of the target in space. 2. Results confirm the predictions of spatial models. Animals compensated for the stimulation-induced perturbation by looking to the position of the target in space. The result predicted by retinocentric models-a saccade with a direction and amplitude based on retinal error alone-was never observed. 3. The eye movement that compensated for the change in eye position produced by stimulation was a saccade, not a passive, lowvelocity movement to an orbital position of mechanical equilibrium established by a tonic pattern of motoneuron activation specified by the visual target. This indicates that a new saccade command, based on stored information about the location of the retinal image and information about the new position of the eyes, had been issued. Computation of the vector of the compensatory saccade does not necessarily increase the latency of target acquisition. The interval between the end of a stimulation-induced saccade and the beginning of the compensatory saccade was frequently 20 ms or less, permitting the animal to acquire the target with a normal latency.

How multisensory cues affect processing in early sensory brain areas is not well understood. The inferior colliculus (IC) is an early auditory structure that is visually responsive (Porter KK, Metzger RR, Groh JM. Proc Natl Acad Sci USA 104: 17855-17860, 2007; Bulkin DA, Groh JM. Front Neural Circuits 6: 61, 2012; Bulkin DA, Groh JM. J Neurophysiol 107: 785-795, 2012), but little is known about how visual signals affect the IC's auditory representation. We explored how visual cues affect both spiking and local field potential (LFP) activity in the IC of two monkeys performing a task involving saccades to auditory, visual, or combined audiovisual stimuli. We confirm that LFPs are sensitive to the onset of fixation lights and the onset of visual targets presented during steady fixation. The LFP waveforms evoked by combined audiovisual stimuli differed from those evoked by sounds alone. In single-unit spiking activity, responses were weak when visual stimuli were presented alone, but visual stimuli could modulate the activity evoked by sounds in a stronger way. Such modulations could involve either increases or decreases in activity, and whether increases or decreases were observed was variable and not obviously correlated with the responses evoked by visual or auditory stimuli alone. These findings indicate that visual stimuli shape the IC's auditory representation in flexible ways that differ from those observed previously in multisensory areas.NEW & NOTEWORTHY We find that the inferior colliculus, a primarily auditory brain area, displays distinct population-level responses to visual stimuli. We also find that visual cues can influence the auditory responses of individual neurons. Together, the results provide insight into how relatively early sensory areas may play a role in combining multiple sensory modalities to refine the perception of complex environments.

Conventional explanations of the vestibulo-ocular reflex (VOR) and eye and head movements are revisited by considering two alternative frameworks addressing the question of how the brain controls motor actions. Traditionally, biomechanical and/or computational frameworks reflect the views of several prominent scholars of the past, including Helmholtz and von Holst, who assumed that the brain directly specifies the desired motor outcome and uses efference copy to influence perception. However, empirical studies resulting in the theory of referent control of action and perception (an extension of the equilibrium-point hypothesis) revealed that direct specification of motor outcome is inconsistent with nonlinear properties of motoneurons and with the physical principle that the brain can control motor actions only indirectly, by changing or maintaining the values of neurophysiological parameters that influence, but can remain independent of, biomechanical variables. Some parameters are used to shift the origin (referent) points of spatial frames of reference (FRs) or system of coordinates in which motor actions emerge without being predetermined. Parameters are adjusted until the emergent motor actions meet the task demands. Several physiological parameters and spatial FRs have been identified, supporting the notion of indirect, referent control of movements. Instead of integration of velocity-dependent signals, position-dimensional referent signals underlying head motion can likely be transmitted to motoneurons of extraocular muscles. This would produce compensatory eye movement preventing shifts in gaze during head rotation, even after bilateral destruction of the labyrinths. The referent control framework symbolizes a shift in the paradigm for the understanding of VOR and eye and head movement production.

Whenever we shift our gaze, any location information encoded in the retinocentric reference frame that is predominant in the visual system is obliterated. How is spatial memory retained across gaze changes? Two different explanations have been proposed: Retinocentric information may be transformed into a gaze-invariant representation through a mechanism consistent with gain fields observed in parietal cortex, or retinocentric information may be updated in anticipation of the shift expected with every gaze change, a proposal consistent with neural observations in LIP. The explanations were considered incompatible with each other, because retinocentric update is observed before the gaze shift has terminated. Here, we show that a neural dynamic mechanism for coordinate transformation can also account for retinocentric updating. Our model postulates an extended mechanism of reference frame transformation that is based on bidirectional mapping between a retinocentric and a body-centered representation and that enables transforming multiple object locations in parallel. The dynamic coupling between the two reference frames generates a shift of the retinocentric representation for every gaze change. We account for the predictive nature of the observed remapping activity by using the same kind of neural mechanism to generate an internal representation of gaze direction that is predictively updated based on corollary discharge signals. We provide evidence for the model by accounting for a series of behavioral and neural experimental observations.

Optic flow neurons in area PEc integrate eye and head position signals

Multisensory integration is a powerful mechanism for maximizing sensitivity to sensory events. We examined its effects on auditory localization in healthy human subjects. The specific objective was to test whether the relative intensity and location of a seemingly irrelevant visual stimulus would influence auditory localization in accordance with the inverse effectiveness and spatial rules of multisensory integration that have been developed from neurophysiological studies with animals [Stein and Meredith, 1993 The Merging of the Senses (Cambridge, MA: MIT Press)]. Subjects were asked to localize a sound in one condition in which a neutral visual stimulus was either above threshold (supra-threshold) or at threshold. In both cases the spatial disparity of the visual and auditory stimuli was systematically varied. The results reveal that stimulus salience is a critical factor in determining the effect of a neutral visual cue on auditory localization. Visual bias and, hence, perceptual translocation of the auditory stimulus appeared when the visual stimulus was supra-threshold, regardless of its location. However, this was not the case when the visual stimulus was at threshold. In this case, the influence of the visual cue was apparent only when the two cues were spatially coincident and resulted in an enhancement of stimulus localization. These data suggest that the brain uses multiple strategies to integrate multisensory information.

A moving visual field can induce the feeling of self-motion or vection. Illusory motion from static repeated asymmetric patterns creates a compelling visual motion stimulus, but it is unclear if such illusory motion can induce a feeling of self-motion or alter self-motion perception. In these experiments, human subjects reported the perceived direction of self-motion for sway translation and yaw rotation at the end of a period of viewing set visual stimuli coordinated with varying inertial stimuli. This tested the hypothesis that illusory visual motion would influence self-motion perception in the horizontal plane. Trials were arranged into 5 blocks based on stimulus type: moving star field with yaw rotation, moving star field with sway translation, illusory motion with yaw, illusory motion with sway, and static arrows with sway. Static arrows were used to evaluate the effect of cognitive suggestion on self-motion perception. Each trial had a control condition; the illusory motion controls were altered versions of the experimental image, which removed the illusory motion effect. For the moving visual stimulus, controls were carried out in a dark room. With the arrow visual stimulus, controls were a gray screen. In blocks containing a visual stimulus there was an 8s viewing interval with the inertial stimulus occurring over the final 1s. This allowed measurement of the visual illusion perception using objective methods. When no visual stimulus was present, only the 1s motion stimulus was presented. Eight women and five men (mean age 37) participated. To assess for a shift in self-motion perception, the effect of each visual stimulus on the self-motion stimulus (cm/s) at which subjects were equally likely to report motion in either direction was measured. Significant effects were seen for moving star fields for both translation (p = 0.001) and rotation (p<0.001), and arrows (p = 0.02). For the visual motion stimuli, inertial motion perception was shifted in the direction consistent with the visual stimulus. Arrows had a small effect on self-motion perception driven by a minority of subjects. There was no significant effect of illusory motion on self-motion perception for either translation or rotation (p>0.1 for both). Thus, although a true moving visual field can induce self-motion, results of this study show that illusory motion does not.