Disparity prediction between adjacent frames for dynamic scenes

Real-time 3D sensing plays a critical role in robotic navigation, video surveillance and human-computer interaction, etc. When computing 3D structures of dynamic scenes from stereo sequences, spatiotemporal stereo and scene flow methods can produce temporally coherent disparity. However, most existing methods do not utilize the previous disparity map sufficiently to compute the next disparity map, and the searching space of correspondences limits the speed of disparity computation for each image pair. This paper proposes an effective scheme to predict disparity maps from stereo sequences. In particular, we apply a robust 3D registration algorithm based on the angular-invariant feature to estimate the ego-motion of the stereo rig between consecutive frames, and present the transformation between consecutive disparity maps. The scheme can produce a sequence of temporally coherent disparity maps rapidly. We apply the new scheme to real outdoor scenes, and thorough empirical studies indicate the effectiveness of the new scheme for practical applications.KeywordsImage PairPattern AnalConsecutive FrameIterative Close PointStereo MatchThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Temporal consistent real-time stereo for intelligent vehicles

This chapter presents a technique for estimating the three-dimensional displacement vector field that describes the motion of each visible scene point. This displacement field consists of the change in image position, also known as optical flow, and the change in disparity and is called the scene flow. The technique presented uses two consecutive image pairs from a stereo sequence. The main contribution is to decouple the disparity (3D position) and the scene flow (optical flow and disparity change) estimation steps. Thus we present a technique to estimate a dense scene flow field using a variational approach from the gray value images and a given stereo disparity map.Although the two subproblems disparity and scene flow estimation are decoupled, we enforce the scene flow to yield consistent displacement vectors in all four stereo images involved at two time instances. The decoupling strategy has two benefits: Firstly, we are independent in choosing a disparity estimation technique, which can yield either sparse or dense correspondences, and secondly, we can achieve frame rates of 5 fps on standard consumer hardware. This approach is then expanded to real-world displacements, and two metrics are presented that define likelihoods of movement with respect to the background. Furthermore, an evaluation approach is presented to compare scene flow algorithms on long image sequences, using synthetic data as ground truth.KeywordsRoot Mean SquareDisparity EstimationLead VehicleStereo Image PairStereo AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Recently, stereo models based on coarse-to-fine approaches have drastically alleviated the memory footprint and speed limitations of complex network models. However, previous designs used a uniformly defined range for disparity estimation, which ignored the difficulty of pixel matching in different regions and introduced many unnecessary matching candidates. In this paper, we construct Adaptive Thin Volume Network (ATVNet) to improve accuracy and reduce computation time. Firstly, multi-scale feature maps are extracted to construct a combined cost volume, then the rough disparity map is derived from the low-resolution volume regression. Finally, the Disparity Correction Range (DCR) module adaptively adjusts the matching space through uncertainty estimation, which constructs thin volumes for gradually retrieving a more accurate high-resolution disparity map. With the gradual improvement of volume resolution and the adaptive reduction of correction range, the disparity map is recovered in a coarser-to-fine strategy. When training and testing on the Scene Flow and KITTI 2015 datasets, our ATVNet model exceeds the mainstream end-to-end model in accuracy and efficiency.

This paper presents a human gesture detection and tracking system using real-time stereo matching. A disparity map is obtained from stereo matching based on general-purpose computing of graphics processing units in real-time, and then, 3-D foreground blobs are generated using depth information gathered from the map. The distribution of the 3-D foreground blobs, combined with detection of the face and torso, is applied to determine the position of the human body. A skeleton model for the upper body is successively fitted to a median axis in the area that has more 3-D blobs from the shoulder area to the hands. The position and color information obtained from the 3-D blobs for a robust tracking of arm and hand gestures are examined. The reconstructed trajectory is then classified into one of eight gesture reference sets.

Scene flow is a description of real world motion in 3D that contains more information than optical flow. Because of its complexity there exists no applicable variant for real-time scene flow estimation in an automotive or commercial vehicle context that is sufficiently robust and accurate. Therefore, many applications estimate the 2D optical flow instead. In this paper, we examine the combination of top-performing state-of-the-art optical flow and stereo disparity algorithms in order to achieve a basic scene flow. On the public KITTI Scene Flow Benchmark we demonstrate the reasonable accuracy of the combination approach and show its speed in computation.

In this paper, a new stereo object tracking system using the disparity motion vector(DMV) is proposed. Here, the DMV is defined as a disparity difference between two time-sequential disparity maps of T-1 and T frames and it can be sequentially estimated from the disparity maps which are extracted from the input stereo sequence. Here, the disparity maps are obtained by using the feature-based matching algorithm, in which the block size and the search range for the feature-based matching are given by N x × N y = 1 × 1 and S x = ±32, respectively. And then, the DMV can be estimated by calculating the motion difference between two consecutive disparity maps. Basically, the DMV has a relatively large change of disparity values in the target region because of the target's movement, whereas it has very low difference values in the background region; there are almost no changes by comparing with the target region. As a result, the DMV can provide us with the motional information of a moving target by showing a large disparity difference in the target area. That is, the dynamic relationship between these disparity vectors of T-1 and T frames seems to be very similar to that between the motion vectors of T-1 and T frames in the sequence of the conventional 2D video images. Accordingly, the target area and its location coordinates can be detected by using these DMV maps. Basing on these locational data of a moving target, the pan/tilt embedded in the stereo camera system can be controlled and as a result, real-time stereo tracking of a moving target can be achieved. From some experiments with 9 frames of the stereo sequence having 256×256 pixels, it is shown that the proposed DMV-based stereo object tracking system can track the moving target with a relatively low error ratio of about 3.02 % on average. This good experimental result finally suggests a possibility of implementing the DMV-based stereo object tracking system.

This paper addresses the problem of matching stereo images acquired by a stereo system mounted aboard an intelligent vehicle. The main idea behind the new method consists in involving temporal matching between a current stereo pair and its preceding one to achieve the spatial matching of the former stereo by involving the matching results obtained at the last frame. The proposed method is achieved in three main steps. First, an edge based disparity map is derived from the disparity map of the preceding frame, we call it the assisting disparity map (ADM). Second, for each scan-line, a set of local ranges and global ranges are deduced from the ADM to keep only potential matching candidates. Third, the matching is done on the basis of dynamic programming algorithm by involving in the resulting local and global ranges we get from the last step. The proposed approach has been tested on both real and synthetic stereo sequences and the results demonstrate its effectiveness.

In this paper, we propose a new disparity-based obstacle detection method for intelligent vehicle. The disparity maps captured by the stereo vision system ZED is utilized to detect obstacles in outdoor environment. This method consists of several steps namely, histogram calculation based on the disparity map, peaks finding using Non-Maximum Suppression (NMS), disparity slicing and obstacle detection using contours. The first step is to obtain the distribution of the pixels in the disparity map. The second step is performed to get the peaks in the histogram after the binary processing on the disparity map. However, there are always many noise objects in the binary image. The real obstacle area needs to be separated from the noise obstacles. Thus, the third step is performed to filter the binary image based on a statistics by column and then segment the detected obstacle area. The last step is to calculate the obstacle including its size, distance and position on the disparity map. A series of experiments prove that the proposed method is accurate at detecting obstacles and calculating the required information based on stereovision techniques.

We propose a continuous optimization method for solving dense 3D scene flow problems from stereo imagery. As in recent work, we represent the dynamic 3D scene as a collection of rigidly moving planar segments. The scene flow problem then becomes the joint estimation of pixel-to-segment assignment, 3D position, normal vector and rigid motion parameters for each segment, leading to a complex and expensive discrete-continuous optimization problem. In contrast, we propose a purely continuous formulation which can be solved more efficiently. Using a fine superpixel segmentation that is fixed a-priori, we propose a factor graph formulation that decomposes the problem into photometric, geometric, and smoothing constraints. We initialize the solution with a novel, high-quality initialization method, then independently refine the geometry and motion of the scene, and finally perform a global non-linear refinement using Levenberg-Marquardt. We evaluate our method in the challenging KITTI Scene Flow benchmark, ranking in third position, while being 3 to 30 times faster than the top competitors (x37 [10] and x3.75 [24]).

Innovative applications in rapidly evolving domains such as robotic navigation and autonomous (driverless) vehicles rely on binocular computer vision systems that meet stringent response time and accuracy requirements. A key problem in these vision systems is stereo matching, which involves matching pixels from two input images in order to construct the output, a 3D map. Building upon the existing local stereo matching algorithms, this paper proposes a novel stereo matching algorithm that is based on a weighted guided filtering foundation. The proposed algorithm consists of three main steps; each step is designed with the goal of improving accuracy. First, the matching costs are computed using a unique combination of complementary methods (absolute difference, Census, and gradient algorithms) to reduce errors. Second, the costs are aggregated using an adaptive weighted guided image filtering method. Here, the regularization parameters are adjusted adaptively using the Canny method, further reducing errors. Third, a disparity map is generated using the winner-take-all strategy; this map is subsequently refined using a densification method to reduce errors. Our experimental results indicate that the proposed algorithm provides a higher level of accuracy in comparison to a collection of the existing state-of-the-art local algorithms.

Stereo vision finds a wide range of applications in automotive, object detection, robot navigation, agriculture mapping, and others. Stereo matching is a stereo algorithm targeted to identify the corresponding pixels from two or more images. This study shows the implementation of stereo matching using the Sum of Absolute Difference (SAD) algorithm to extract the object's depth or disparity from stereo images. Our key objective revolves around the implementation of the stereo matching algorithm for a small field-programmable gate array (FPGA)—requiring relatively few resources while maintaining the processing speed as well as disparity map. For meeting this requirement, we used small window buffers to compute the stereo matching. The occluded pixels were reduced by introducing secondary consistency checking implementation. From the results of experiments performed using the Zynq UltraScale+ ZCU102 FPGA board with SDSoC compiler, the processing speed for computing the stereo matching algorithm with a 4×4 window buffer was 0.038 s for an image size of 486×720 pixels and 0.051 s for 375×1242 pixels resolution. The proposed design needed 1% each of BRAM and FF and 7% of LUT. An 18% reduction in the pixel errors has been observed when employing the secondary consistency matching on the post-processing.

Depth estimation has vital importance in various applications like Advance driver assistance systems, Robot navigation, 3D movies, etc. Depth information is represented using disparity maps which are generated from various stereo correspondence algorithms. This paper proposes an implementation of Semi global block matching algorithm on raspberry pi using python. The proposed algorithm generates a disparity map by block wise matching and smoothness constraint. The novel algorithm is compared with other algorithms on the standard dataset. It provides an efficient accuracy over local methods. The real time implementation of SGM algorithm on the system uses a block size of 21 × 21 for images having resolution 1280×720P. The system provides faster processing time for the proposed algorithm.

A hybrid mapping framework is presented in this work. The goal is to obtain better computational efficiency than pure metrical mapping techniques and better accuracy as well as usability for robot guidance and navigation compared to the topological mapping. Image sequences acquired in an environment by manually driving a robot are used to build a hierarchical map representation by using an image sequence partitioning (ISP) technique that uses local image features. The hierarchical map built can be understood as a topological map with nodes corresponding to certain regions in the environment. Each node in turn is made up of a set of images acquired in that region. These maps are further augmented with metrical information at those nodes which correspond to image subsequences acquired while the robot is turning as a part of its trajectory. Metrical information becomes invaluable during autonomous robot navigation through these places. Hence, we call the resulting maps hybrid since they primarily contain topological information and metrical information at places that are important for navigation. Experimental results obtained on a sequence acquired in an outdoor environment are provided to demonstrate our approach.

Recent works have demonstrated superior results for depth estimation from a stereo pair of images using convolutional neural networks. However, these methods require large amounts of computational resources and are not suited to real-time applications on edge devices. In this work, we propose a novel method for real-time stereo matching on edge devices, which consists of an efficient backbone for feature extraction, an attention-aware feature aggregation, and a cascaded 3D CNN architecture for multi-scale disparity estimation. The efficient backbone is designed to generate multi-scale feature maps with constrained computational power. The multi-scale feature maps are further adaptively aggregated via the proposed attention-aware feature aggregation module to improve representational capacity of features. Multi-scale cost volumes are constructed using aggregated feature maps and regularized using a cascaded 3D CNN architecture to estimate disparity maps in anytime settings. The network infers a disparity map at low resolution and then progressively refines the disparity maps at higher resolutions by calculating the disparity residuals. Because of the efficient extraction and aggregation of informative features, the proposed method can achieve accurate depth estimation in real-time inference. Experimental results demonstrated that the proposed method processed stereo image pairs with resolution 1242 \(\times \) 375 at 12–33 fps on an NVIDIA Jetson TX2 module and achieved competitive accuracy in depth estimation. The code is available at https://github.com/JiaRenChang/RealtimeStereo.