Stereo vision for planetary rovers: Stochastic modeling to near real-time implementation

NASA scenarios for lunar and planetary missions include robotic vehicles that function in both teleoperated and semi-autonomous modes. Under teleoperation, on-board stereo cameras may provide 3-D scene information to human operators via stereographic displays; likewise, under semi-autonomy, machine stereo vision may provide 3-D information for obstacle avoidance. In the past, the slow speed of machine stereo vision systems has posed a hurdle to the semi- autonomous scenario; however, recent work at JPL and other laboratories has produced stereo vision systems with high reliability and near real-time performance for low-resolution image pairs. In particular, JPL has taken a significant step by achieving the first autonomous, cross- country robotic traverses (of up to 100 meters to use stereo vision, with all computing on- board the vehicle. This paper describes the stereo vision system, including the underlying statistical model and the details of the implementation. The statistical and algorithmic aspects employ random field models of the disparity map, Bayesian formulations of single-scale matching, and area-based image comparisons. The implementations builds Laplacian image pyramids and produces disparity maps from the 60X64 level of the pyramids at rates of up to two seconds per image pair. All vision processing is done on one 68020 augmented with Datacube image processing boards. The author argues that the overall approach provides a unifying paradigm for practical, domain-independent stereo ranging.

Stereo vision is an important computer vision technique which estimates depth information from two images - similar to how human vision works. However, the computational intensity of stereo vision deters implementation in real-time systems. Recently, massively parallel processors called Graphics Processing Units or GPUs are gaining traction as an acceleration platform to offload compute-intensive tasks to. This paper presents the development of a GPU-accelerated stereo vision system called DepthStream. The DepthStream algorithm uses a combined absolute difference and census cost, cross-based cost aggregation and bitwise fast region voting to achieve error rates up to 3.7× lower than Block-Matching based algorithms in mainstream libraries. Leveraging the parallel processing power that GPUs offer enables the DepthStream algorithm to achieve processing times up to 3.12× faster than Semi-Global Matching based implementations in the same libraries. The study also includes measurements of the accuracy of the estimated distances of the developed stereo vision system. Results show that the estimated depths of the DepthStream stereo vision system are within 4% of the true value for distances less than 6m. Statistical analysis also indicate that there is no significant difference between the stereo vision systems estimated distances and the actual value.

Depth estimation for dynamic scenes is a challenging and relevant problem in computer vision. Although this problem can be tackled by means of ToF cameras or stereo vision systems, each of the two systems alone has its own limitations. In this paper a framework for the fusion of 3D data produced by a ToF camera and a stereo vision system is proposed. Initially, depth data acquired by the ToF camera are up-sampled to the spatial resolution of the stereo vision images by a novel up-sampling algorithm based on image segmentation and bilateral filtering. In parallel a dense disparity field is obtained by a stereo vision algorithm. Finally, the up-sampled ToF depth data and the disparity field provided by stereo vision are synergically fused by enforcing the local consistency of depth data. The depth information obtained with the proposed framework is characterized by the high resolution of the stereo vision system and by an improved accuracy with respect to the one produced by both subsystems. Experimental results clearly show how the proposed method is able to outperform the compared fusion algorithms.

Self-supervised learning as an enabling technology for future space exploration robots: ISS experiments on monocular distance learning

Recovery of depth information for 3-D modeling of surfaces and objects is a central problem in low-level computer vision. A number of applications related to reconstruction, representation, and recognition of scenes and objects in the three-dimensional geometry of the real world rely on the correct estimation of range data. In stereo vision, depth information is obtained from triangulation of corresponding points in a pair of stereo images. The automatic determination of local similarity between two data sets or collections of pixels is fundamental in a depth-algorithm from stereo algorithm for numerical evaluation of disparities. In this paper, the use of the Cepstrum transformation as a disparity measurement technique between corresponding windows of different block sizes is described. This measurement process is embedded within a coarse to fine stereopsis algorithm, providing an initial depth map with the depth information encoded as gray levels. In real stereo imagery the features to be used as information in the matching procedure are usually sparsely and irregularly distributed over the images. For that reason, obtention of a dense array of depth data is a difficult task in passive methods. This paper makes emphasis in surface recovery of smooth curved surfaces with application to depth estimation of medical images. Examples and experimental results using random dot stereograms and real imagery are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Computer stereo vision has been paid high attention in areas such as industry measurement and automatic control. Calibration of stereo vision system is a key step to implement vision measurement. Nonlinear calibration method involves too many parameters and too complex function, so the processing of optimization is difficult and slow. In this paper, two digital cameras are used to set up stereo vision system which provides convenient experimental operation. The binocular stereo vision system is calibrated using linear calibration model. The results indicate that the method is stable and fast with high precision and practicability.

Significant obstacle location with ultra-wide FOV LWIR stereo vision system

Stereo vision system is a practical method for depth gathering of objects and features in an environment. This paper presents the depth evaluation in selected region of disparity mapping for stereo vision mobile robot using block matching algorithm. This region is a reference sight of the stereo camera and stereo vision baseline is based on horizontal configuration. The block matching technique is briefly described with the performance of its output. The disparity mapping is generated by the algorithm with the reference to the left image coordinate. The algorithm uses Sum of Absolute Differences (SAD) which is developed using Matlab software.

Methods for determining the three-dimensional (3-D) form of complex objects in a scene from a pair of images using the principles of stereo vision are described. A pair of stereo images are epipolar- or raster-scan-aligned to restrict the search to one dimension along the horizontal lines as a first step in location of image feature points. A correlation window of size 5/spl times/5 around a pixel on the left image is superimposed on a similar window centered around every pixel along the same scan line of the right image. A normalized cross-correlation value for every pair of windows is determined. The image points in the right image which give the maximum correlation value are chosen as the corresponding points for the associated points in the left image. Once the corresponding points are found, the disparity is determined. This initial disparity map is noisy. A smoothing or interpolation algorithm has been developed, greatly increasing the number of feature points that can be identified. Sobel Edge detection is employed to locate discontinuities. Disparity values at edge points are added back into the smoothed disparity map, and the modified smoothed disparity map is used for 3-D rendering.

In this paper, the stereo vision technique with two cameras has been employed to locate object's position in 3D space. It is necessary to use many algorithms to derive object positions, including stereo calibration, rectification, correspondence and triangulation. Two tracking algorithms have been applied to identify and track objects alternately: template matching and Camshift. To evaluate the proposed approach, the stereo vision system is installed on a cubic space which is the working space of a tiny robot arm. The real positions in this coordinate system are then compared to the received positions from the cameras. The end-effector of the robot arm will track the object depending on received positions. Experiment results show that the robot arm can effectively track the indicated objects, and thus the tracking errors between actual and received positions are less than 3% which are reasonably acceptable.

Using stereo vision and marks pasted on wall to solve the guidance and the environment learning problem for a mobile robot is proposed. A circular window is used to detect corner points as features in the scene. From the perspective transformation matrices of the two cameras and matching point-pairs in images the 3-D coordinates can be determined. With ground projections of these space points and the constraint of minimum diameter of the mobile robot, a few separated polygons are used to represent the occupied space of the surrounding obstacles. With the corner points of the mark the location of the mobile robot can be determined not only by the stereo vision system but also by each of the two cameras with plane-motion constraint. Experimental results show that the depth measurement error is less than 2%, the location determination of the mobile robot working on the ground plane is less than 5%, and the map of the occupied space can be used in further path-planning or collision avoidance.

When designing a multi-vision stereo vision network, the camera’s positional information has a critical impact on the measurement accuracy. In order to solve the problem of optimizing the camera pose during network design, in this paper, we split the multi-vision stereo vision system into binocular stereo vision system. Based on the mathematical model of binocular stereo vision system, the error function of the positional parameters is constructed. By simulation analyzing the function models the field of view angle, the angle between the optical axis and the baseline, the baseline length, the law of the influence of each parameter on the measurement accuracy and the optimal range of values are obtained. Then we compare the actual measurement results with the simulation analysis results. The results show that as the field of view angle, optical axis and baseline angle, baseline length increases, the measurement error first decreases and then increases, and the increase is gradually accelerated. When each parameter is located in the optimal range, the measurement error meets the precision engineering measurement accuracy requirement of 0.05mm, and the minimum error is reduced by 0.114mm, 0.120mm and 0.061mm compared to the maximum error. In the network design, the optimization of the pose parameters can effectively obtain better pose information of the camera and thus improve the accuracy of the measurement.

The stereo vision system is one of the most important sensors for 3D reconstruction of the environment and scene understanding. It plays an essential role for the automated driving and ADAS applications. Although recently great progress has been made in this field both in the density and accuracy, the stereo vision in the challenging environment, such as rain, snow and low light conditions, are still open problems and need to be improved. To address this issue, we propose a novel real-time stereo vision algorithm, we transform the rough confidence score of the matching cost to the outlier probability rate and use it to guide the multi-path Viterbi method to estimate the disparity. Our proposed algorithm costs less than 40ms and improves the accuracy of stereo results in difficult scenarios. Real-world experiments show that the proposed method can significantly improve the disparity results in the various challenging environment.

The focus settings and physical characteristics of lenses significantly affect distortion correction, particularly under blurred imaging conditions, thereby limiting the measurement accuracy of existing stereo vision systems. Moreover, traditional calibration processes for stereo vision systems are complex due to the need for precise focus distance adjustments and careful calibration board selection. In this paper, we propose a camera parametric model based on weighted radial constraints that accounts for both radial and non-central lens distortion. This model aims to improve calibration accuracy while simplifying the calibration workflow. Additionally, we introduce a stereo vision correction method that utilizes speckle patterns as calibration images and eliminates the requirement for precise camera focusing. This approach is more user-friendly than conventional planar chessboard techniques, as it avoids frequent lens adjustments for optimal focus. Experimental results demonstrate improvements in accuracy of 17.2% and 15.6% over Zhang’s and Tsai’s calibration methods, respectively.

A stereo vision system that is required to support high-level object based tasks in a tele-operated environment is described. Stereo vision is computationally expensive, due to having to find corresponding pixels. Correlation is a fast, standard way to solve the correspondence problem. This paper analyses the behaviour of correlation based stereo to find ways to improve its quality while maintaining its realtime suitability. Three methods are suggested. Two of them aim to improve the disparity image especially at depth discontinuities, while one targets the identification of possible errors in general. Results are given on real stereo images with ground truth. A comparison with five standard correlation methods shows that improvements of simple stereo correlation are possible in real-time on current computer hardware.