Neural mechanisms of three-dimensional vision

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594–6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222–2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749–1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments.NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by “border ownership” neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.

Reliable estimation of three-dimensional (3D) surface orientation is critical for recognizing and interacting with complex 3D objects in our environment. Human observers maximize the reliability of their estimates of surface slant by integrating multiple depth cues. Texture and binocular disparity are two such cues, but they are qualitatively very different. Existing evidence suggests that representations of surface tilt from each of these cues coincide at the single-neuron level in higher cortical areas. However, the cortical circuits responsible for 1) integration of such qualitatively distinct cues and 2) encoding the slant component of surface orientation have not been assessed. We tested for cortical responses related to slanted plane stimuli that were defined independently by texture, disparity, and combinations of these two cues. We analyzed the discriminability of functional MRI responses to two slant angles using multivariate pattern classification. Responses in visual area V3B/KO to stimuli containing congruent cues were more discriminable than those elicited by single cues, in line with predictions based on the fusion of slant estimates from component cues. This improvement was specific to congruent combinations of cues: incongruent cues yielded lower decoding accuracies, which suggests the robust use of individual cues in cases of large cue conflicts. These data suggest that area V3B/KO is intricately involved in the integration of qualitatively dissimilar depth cues.

Stereoscopic 3D (S3D) geometric distortions can be introduced by mismatches among image capture, display, and viewing configurations. In previous work of S3D geometric models, geometric distortions have been analyzed from a third-person perspective based on the binocular depth cue (i.e., binocular disparity). A third-person perspective is different from what the viewer sees since monocular depth cues (e.g., linear perspective, occlusion, and shadows) from different perspectives are different. However, depth perception in a 3D space involves both monocular and binocular depth cues. Geometric distortions that are solely predicted by the binocular depth cue cannot describe what a viewer really perceives. In this paper, we combine geometric models and retinal disparity models to analyze geometric distortions from the viewer's perspective where both monocular and binocular depth cues are considered. Results show that binocular and monocular depth-cue conflicts in a geometrically distorted S3D space. Moreover, user-initiated head translations averting from the optimal viewing position in conventional S3D displays can also introduce geometric distortions, which are inconsistent with our natural 3D viewing condition. The inconsistency of depth cues in a dynamic scene may be a source of visually induced motions sickness.

Neurons in the macaque Anterior Intraparietal area (AIP) encode depth structure in random-dot stimuli defined by gradients of binocular disparity, but the importance of binocular disparity in real-world objects for AIP neurons is unknown. We investigated the effect of binocular disparity on the responses of AIP neurons to images of real-world objects during passive fixation. We presented stereoscopic images of natural and man-made objects in which the disparity information was congruent or incongruent with disparity gradients present in the real-world objects, and images of the same objects where such gradients were absent. Although more than half of the AIP neurons were significantly affected by binocular disparity, the great majority of AIP neurons remained image selective even in the absence of binocular disparity. AIP neurons tended to prefer stimuli in which the depth information derived from binocular disparity was congruent with the depth information signaled by monocular depth cues, indicating that these monocular depth cues have an influence upon AIP neurons. Finally, in contrast to neurons in the inferior temporal cortex, AIP neurons do not represent images of objects in terms of categories such as animate-inanimate, but utilize representations based upon simple shape features including aspect ratio.

In many situations, we perceive 3-D depth feeling from 2-D images. Among various depth cues, monocular depth cues make us perceive depth from single-view images. In this paper, we propose several methods to enhance monocular depth cues and improve depth perception of single-view images. The proposed methods deal with four depth cues: linear perspective, aerial perspective, focus, and shadow effects. Once we have a 2-D colour image and its corresponding depth information, we can modify its depth cues and enhance 3-D depth perception in the 2-D image. Experimental results show that the proposed methods can enhance depth perception of single-view images without any considerable visual distortions.

Mirror spectacles which enhance binocular disparity by optically doubling the normal separation between the eyes were used to create conditions of combined perceptual and oculomotor conflict. Apparent depth and distance, as well as tonic accommodation, tonic vergence, and accommodative-vergence gain (response AC/A ratio), were assessed immediately before and after a 30 min exposure period of naturalistic viewing with the spectacles. Wearing the spectacles produced an increase in tonic vergence, and perceptual aftereffects consisting of increased apparent distance and depth. The results indicate that oculomotor conflict associated with enhanced interocular separation may be resolved through adaptation of tonic vergence, rather than through alteration of accommodative-vergence gain. The results also demonstrate that perceptual conflict between disparity and multiple veridical depth cues does not necessarily produce adaptive modification of the relationship between binocular disparity and apparent depth.

Binocular disparity is the main depth cue that makes stereoscopic images appear 3D. However, in many scenarios, the range of depth that can be reproduced by this cue is greatly limited and typically fixed due to constraints imposed by displays. For example, due to the low angular resolution of current automultiscopic screens, they can only reproduce a shallow depth range. In this work, we study the motion parallax cue, which is a relatively strong depth cue, and can be freely reproduced even on a 2D screen without any limits. We exploit the fact that in many practical scenarios, motion parallax provides sufficiently strong depth information that the presence of binocular depth cues can be reduced through aggressive disparity compression. To assess the strength of the effect we conduct psycho-visual experiments that measure the influence of motion parallax on depth perception and relate it to the depth resulting from binocular disparity. Based on the measurements, we propose a joint disparity-parallax computational model that predicts apparent depth resulting from both cues. We demonstrate how this model can be applied in the context of stereo and multiscopic image processing, and propose new disparity manipulation techniques, which first quantify depth obtained from motion parallax, and then adjust binocular disparity information accordingly. This allows us to manipulate the disparity signal according to the strength of motion parallax to improve the overall depth reproduction. This technique is validated in additional experiments.

When a three dimensional scene is projected to the two dimensional receptive field of a camera or a biological vision system, all depth information is lost. Even without a knowledgebase, i. e. without knowing what object can be seen, it is possible to reconstruct the depth information. Beside stereoscopic depth cues, also a number of moncular depth cues can be used. One of the most important monocular depth cues ist the occlusion of object boundaries. Therefore one of the elaborated tasks for the low level image processing stage of a vision system is the completion of cluttered or occluded object boundaries and the depth assignment of overlapped boundaries. We describe a method for depth ordering and figure-ground segregation from monocular depth cues, namely the arrangement of so-called illusory contours at junctions in the edge map of an image. Therefore, a computational approach to the perception of illusory contours, based on the tensor voting technique, is introduced and compared with an alternative contour completion realized by spline interpolation. While most approaches assume, that the position of junctions and the orientations of associated contours are already known, we also consider the preprocessing steps that are necessary for a robust perception task. This implies the anisotropic diffusion of the input image in order to simplify the image contents while preserving the edge information.

Reaching movements towards stereoscopically presented virtual objects have been reported to be imprecise. This might be a problem for touch interaction with virtual environments. Estimating the distance to an object in personal space relies on binocular disparity and other depth cues but previous work on the influence of familiar size for reaching and grasping has produced conflicting results. We presented a virtual tennis ball and manipulated binocular disparity as well as the size of the tennis ball. The results suggest that depth information from binocular disparity and from familiar size is combined for reaching movements towards virtual objects. However, subjects differed in the weight they assigned to each depth cue.

<p id="C2">Binocular disparity, a critical cue to stereopsis, is defined as the small horizontal displacement between the two corresponding images projected onto the retina of the two eyes. The study of binocular disparity can be dated back to the early 18th century. Recent studies on binocular disparity have advanced our understanding in two aspects. The first is using electrophysiological and brain imaging technique to investigate the functional specialization in disparity processing in the dorsal and ventral visual pathways, which reveals hierarchical and parallel processing principles in the visual system. The second is about learning-induced plasticity. Future research needs to combine brain imaging, neuromodulation and other cutting-edge techniques to investigate the neural mechanisms underlying binocular disparity, its learning effect, and the interaction between binocular disparity and other depth clues. On the application side, future research needs to optimize training paradigms (e.g., with virtual reality technique) for rehabilitation and enhancement in the binocular disparity performance.

INTRODUCTION Traditionally, the perception of three-dimensional (3D) scenes and objects was assumed to be the result of the reconstruction of 3D surfaces from available depth cues, such as binocular disparity, motion parallax, texture, and shading (Marr, 1982). In this approach, the role of ‰gure-ground organization was kept to a minimum. Figure-ground organization refers to ‰nding objects in the image, diŠerentiating them from the background, and grouping contour elements into two-dimensional (2D) shapes representing the shapes of the 3D objects. According to Marr, solving ‰gureground organization was not necessary. e depth cues could provide the observer with local estimates of surface orientation at many points. ese local estimates were then integrated into the estimates of the visible parts of 3D shapes. Marr called this a 2.5D representation. If the objects in the scene are familiar, their entire 3D shapes can be identi‰ed (recognized) byCONTENTS Introduction 113 Figure-Ground Organization 114 Using Simplicity Principle to Recover 3D Shapes 118 Summary and Discussion 123matching the shapes in the observer's memory to the 2.5D sketch. If they are not familiar, the observer will have to view the object from more than one viewing direction and integrate the visible surfaces into the 3D shape representations.

Depth Perception of Surgeons in Minimally Invasive Surgery.

Infants' distance perception from linear perspective and texture gradients

The generation of three-dimensional (3D) videos from monoscopic two-dimensional (2D) videos has received a lot of attention in the last few years. Current conversion techniques are based on generating an estimated depth map for each frame from different depth cues, and then using Depth Image Based Rendering (DIBR) to synthesize the additional views. Efficient interactive techniques have been developed in which multiple depth factors (monocular depth cues) are utilized to estimate the depth map using machine-learning algorithms. The challenge with such methods is that they cannot be used for real-time conversion. We address this problem by proposing an effective scheme that generates high quality depth maps for scenic views in real-time. In our work, we use three depth cues, haze, vertical edges, and sharpness to estimate a sparse depth map. We then obtain the full depth map from the sparse map using an edge-aware interpolation method. Performance evaluations show that our method outperforms the existing state-of-the-art 2D-to3D conversion methods.

In the natural world, a number of visual cues indicate that an item is quickly approaching the perceiver. Binocular disparity is one cue for depth, and it has been demonstrated that abrupt changes in disparity, artificially unaccompanied by correlated depth cues, are capable of causing the perception of looming for the observer. An experiment involving 38 undergraduates, using a computer-controlled stereoscopic display, examined the ability of above-threshold changes in disparity (artificial looming) to facilitate response time and accuracy for observers engaged in an object-enumeration task within a cluttered display. Compared with performance using the same stimuli without disparity information (lateral motion), participants were more accurate regardless of the disparity level (9, 12, 24, or 48 minutes of arc) and faster at the two lowest levels of disparity. Participants showed the classic subitizing function, suggesting that target stimuli presented with motion information were segregated from otherwise identical distractor items. It is proposed that binocular disparity information can act as a valid location cuing method in stereoscopic computer displays in which form and color information are to be preserved.