Artificial Looming Yields Improved Performance over Lateral Motion: Implications for Stereoscopic Display Techniques

Natural scene statistics have played an increasingly important role in both our understanding of the function and evolution of the human vision system, and in the development of modern image processing applications. Because range (egocentric distance) is arguably the most important thing a visual system must compute (from an evolutionary perspective), the joint statistics between image information (color and luminance) and range information are of particular interest. It seems obvious that where there is a depth discontinuity, there must be a higher probability of a brightness or color discontinuity too. This is true, but the more interesting case is in the other direction--because image information is much more easily computed than range information, the key conditional probabilities are those of finding a range discontinuity given an image discontinuity. Here, the intuition is much weaker; the plethora of shadows and textures in the natural environment imply that many image discontinuities must exist without corresponding changes in range. In this paper, we extend previous work in two ways--we use as our starting point a very high quality data set of coregistered color and range values collected specifically for this purpose, and we evaluate the statistics of perceptually relevant chromatic information in addition to luminance, range, and binocular disparity information. The most fundamental finding is that the probabilities of finding range changes do in fact depend in a useful and systematic way on color and luminance changes; larger range changes are associated with larger image changes. Second, we are able to parametrically model the prior marginal and conditional distributions of luminance, color, range, and (computed) binocular disparity. Finally, we provide a proof of principle that this information is useful by showing that our distribution models improve the performance of a Bayesian stereo algorithm on an independent set of input images. To summarize, we show that there is useful information about range in very low-level luminance and color information. To a system sensitive to this statistical information, it amounts to an additional (and only recently appreciated) depth cue, and one that is trivial to compute from the image data. We are confident that this information is robust, in that we go to great effort and expense to collect very high quality raw data. Finally, we demonstrate the practical utility of these findings by using them to improve the performance of a Bayesian stereo algorithm.

Following another pedestrian is common in everyday locomotion, and may contribute to the collective behavior of crowds. Last year we modeled the behavioral strategy for speed control in 1D following (Rio, Rhea, & Warren 2010); this year we investigate the visual information used to regulate speed. Previously, we modeled data on the follower's acceleration in response to the leader's change in speed by (a) nulling change in distance, (b) nulling the difference in speed, and (c) combining speed and distance using a ratio (Gazis, Herman, & Rothery 1961) or a linear combination (Helly, 1959; Anderson & Sauer, 2007). The data are well-accounted for by the relative speed model. Here, we manipulate the visual angle and binocular disparity of a virtual ‘leader.’ The participant followed a virtual target pole (0.2 m diameter) viewed stereoscopically in a head-mounted display (63°H × 53°V) in the 12 × 12 m VENLab. A change in target speed over a 3s interval was specified by (1) a 50% increase, decrease, or 0% change in visual angle, achieved by shrinking or expanding the pole, and (2) a 50% decrease, increase, or 0% change in binocular disparity. The visual angle and disparity conditions were fully crossed, so that the changes could be congruent or incongruent. Participants' mean speed during the manipulation showed that they relied entirely on the change in visual angle (p < .001) and ignored the changes in disparity (ns). We combine these results to derive a speed control law for following, in which the follower matches the leader's speed by nulling change in their visual angle. Future work will expand the following model to the general 2D case, in which the leader changes direction as well as speed (Rhea, Cohen, & Warren 2009), and will investigate its role in crowd behavior (Rio, Bonneaud, & Warren 2010).

Stereopsis is an ability to gauge depth from a pair of two-dimensional retinal images. Because the eyes are separated laterally, the two retinal images differ slightly from each other, with images of objects shifted mainly along the horizontal dimension. This shift is known as binocular disparity. Previous studies suggest that binocular disparity information may be encoded by simple cells with varying degrees of dissimilarity between left and right receptive field (RF) profiles [1]. Furthermore, there is an anisotropy in the distribution of RF dissimilarities (as quantified by RF phase difference) with respect to the neuron’s preferred orientation. Neurons that prefer near vertical orientation (which encode horizontal disparities) could have similar or dissimilar RF profiles. However, those that prefer near horizontal orientation (which encode vertical disparities) have highly similar RF profiles for the two eyes. Therefore, the visual system appears to employ a highly efficient encoding scheme tailored for statistical properties of the binocular visual environment.

Binocular disparity is the main depth cue that makes stereoscopic images appear 3D. However, in many scenarios, the range of depth that can be reproduced by this cue is greatly limited and typically fixed due to constraints imposed by displays. For example, due to the low angular resolution of current automultiscopic screens, they can only reproduce a shallow depth range. In this work, we study the motion parallax cue, which is a relatively strong depth cue, and can be freely reproduced even on a 2D screen without any limits. We exploit the fact that in many practical scenarios, motion parallax provides sufficiently strong depth information that the presence of binocular depth cues can be reduced through aggressive disparity compression. To assess the strength of the effect we conduct psycho-visual experiments that measure the influence of motion parallax on depth perception and relate it to the depth resulting from binocular disparity. Based on the measurements, we propose a joint disparity-parallax computational model that predicts apparent depth resulting from both cues. We demonstrate how this model can be applied in the context of stereo and multiscopic image processing, and propose new disparity manipulation techniques, which first quantify depth obtained from motion parallax, and then adjust binocular disparity information accordingly. This allows us to manipulate the disparity signal according to the strength of motion parallax to improve the overall depth reproduction. This technique is validated in additional experiments.

Seeing in three dimensions: the neurophysiology of stereopsis

Although it is known that stereo-acuity declines with increasing eccentricity of the targets, it is not clear how we use disparity information in the peripheral visual field for pattern perception. To examine this question, we investigated pattern perception by restricting the area that presented stereo or luminance information in the periphery. We measured the reaction time for recognising a letter defined by binocular disparity or by luminance in a random-dot display. We restricted the area containing the specific information (disparity or luminance) using an eye-contingent window technique. Disparity or luminance information was thus present only inside a window centred on the fixation point. Observers viewed the display with free eye movement. The magnitudes of the disparity and luminance contrast were chosen so as to give the same reaction times when the area containing the pattern information was not restricted. Eye movements were measured by a limbus-tracking system and the signal was fed into a computer for real-time control of the window position. The reaction time increased as the window size decreased. The increase in reaction time, however, was steeper for the stimuli defined by disparity than for the stimuli defined by luminance. We conclude that disparity information in the periphery is used for recognising a pattern and is more effective than luminance information for a given window size.

Binocular disparity information is an important source of 3D perception. Neurons sensitive to binocular disparity are found in almost all major visual areas in nonhuman primates. In area V4, disparity processes are suggested for the purposes of 3D-shape representation and fine disparity perception. However, whether neurons in V4 are sensitive to disparity-defined edges used in shape representation is not clear. Additionally, a functional organization for disparity edges has not been demonstrated so far. With intrinsic signal optical imaging, we studied functional organization for disparity edges in the monkey visual areas V1, V2, and V4. We found that there is an orientation map in V4 activated by edges purely defined by binocular disparity. This map is consistent with the orientation map obtained with regular luminance-defined edges, indicating a cue-invariant edge representation in this area. In contrast, such a map is much weaker in V2 and totally absent in V1. These findings reveal a hierarchical processing of 3D shape along the ventral pathway and the important role that V4 plays in shape-from-disparity detection.

Adaptation to Natural Binocular Disparities in Primate V1 Explained by a Generalized Energy Model

With the purpose of mapping those anatomical structures participating in the processing and analysis of form, colour and disparity information, we have measured, with positron emission tomography and [15O]butanol, regional cerebral blood flow (rCBF) as an indicator of regional cerebral metabolic activity in 13 right-handed male volunteers during visual discrimination of colour, form and disparity information. The brain images were anatomically standardized using a computerized brain atlas and statistically significant changes were localized by cluster analysis. The changes in rCBF between specific activation and reference states were measured and the volumes of changes were determined, as were the loci and volumes of areas commonly activated by two or three different tasks. Each of the tasks activated over a dozen distinct and separate fields in the cortex--in the occipital, parietal, temporal and frontal lobes as well as the cerebellum. A number of overlapping fields were commonly activated in two tasks (four in the form and colour tasks, five in the form and disparity tasks, and eleven in the colour and disparity tasks), and two field overlaps were present in all three tasks (in the right superior frontal and left lingual gyri). These findings indicate that, in a visual discrimination task, the processing and analysis of single visual submodalities take place in a number of cortical fields in the human brain. As the same visual submodality is processed and analysed by numerous fields and the same field may participate in the processing of different submodalities, a divergence-convergence pattern of information processing is present in the human brain. This observation supports a hypothesis based on earlier studies in primates, namely that information processing in the visual system requires the concerted activation of a relatively large number of fields of functional networks in the brain.

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594–6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222–2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749–1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments.NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by “border ownership” neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.

A stereoscope displays 2-D images with binocular disparities (stereograms), which fuse to form a 3-D stereoscopic object. But a stereoscopic object creates a conflict between vergence and accommodation. Also, motion in depth of a stereoscopic object simulated solely from change in target vergence produces anomalous motion parallax and anomalous changes in perspective. We describe a new instrument, which overcomes these problems. We call it the dichoptiscope. It resembles a mirror stereoscope, but instead of stereograms, it displays identical 2-D or 3-D physical objects to each eye. When a pair of the physical, monocular objects is fused, they create a dichoptic object that is visually identical to a real object. There is no conflict between vergence and accommodation, and motion parallax is normal. When the monocular objects move in real depth, the dichoptic object also moves in depth. The instrument allows the experimenter to control independently each of several cues to motion in depth. These cues include changes in the size of the images, changes in the vergence of the eyes, changes in binocular disparity within the moving object, and changes in the relative disparity between the moving object and a stationary object.

Do relative binocular disparities guide our movements in depth? In order to find out we asked subjects to move a 'cursor' to a target within a simulated horizontal plane at eye height. They did so by moving a computer mouse. We determined how quickly subjects responded to the target jumping in depth. We found that it took subjects about 200 ms to respond to changes in binocular disparity. Subjects responded just as quickly if the cursor was temporarily only visible to one eye near the time that the target jumped in depth, and less vigorously, though just as quickly, if the cursor jumped rather than the target, so the fastest binocular responses cannot be based directly on the relative retinal disparity between the target and the cursor. Subjects reacted faster to changes in the target's height in the visual field than to changes in binocular disparity, but did not react faster to changes in image size. These results suggest that binocular vision mainly improves people's everyday movements by giving them a better sense of the distances of relevant objects, rather than by relative retinal disparities being used to directly guide the movement. We propose that relative disparities only guide parts of very slow movements that require extreme precision.

Contributions to the subject of binocular vision

Depth perception is an important component of many augmented reality applications. It is, however, subject to multiple error sources. In this study, we investigated depth judgments with a stereoscopic video see-through head-mounted display for the purpose of designing depth cueing for systems that operate in an individual’s action space. In the experiment, we studied the use of binocular disparity and relative size to improve relative depth judgments of augmented objects above the ground plane. The relative size cue was created by adding auxiliary augmentations to the scene according to constraints described in the section on the underlying theory. The results showed that binocular disparity and relative size improved depth judgments over the distance range. This indicates that for accurate depth judgments, additional depth cues should be used to facilitate stereoscopic perception within an individual’s action space.

Stereoscopic displays present different images to the two eyes and thereby create a compelling 3D sensation. However, such displays cause a host of perceptual and ergonomic problems. These problems occur because some of the presented depth cues (i.e., perspective and binocular disparity) specify the intended 3D scene while focus cues (blur and accommodation) specify the fixed distance of the display itself. These problems are particularly problematic in VR and AR displays. We have developed a stereoscopic display that circumvents these problems. Using the display, we investigated how incorrect focus cues affect visual perception, visual performance, and, most importantly, visual comfort. With respect to perception, we showed that the ability to perceive correct depth ordering is significantly improved when focus cues are correct. With respect to visual performance, we showed that the ability to binocularly fuse stimuli is substantially improved when the vergence‐accommodation conflict is minimized. Finally, with respect to visual comfort, we showed that comfort is significantly increased when the vergence‐accommodation conflict is zero. I offer suggestions on ways to create nearly correct focus cues in practical stereoscopic displays, particularly VR and AR.