Codebook based stereo matching for natural user interface

Outlier detection and disparity refinement in stereo matching

The paper focuses on estimating disparity map that can handle outliers. Outliers are the major challenging problem in stereo matching. The results obtained from depth maps are significantly affected by outliers regions that are present in the image. The stereo images are pre-processed using homomorphic filter. The Semi-global block matching method is used to estimate disparity information. Left to right consistency check method is applied to identify outliers. Outliers identified are classified as occlusion and mismatches and then they are refined. The proposed method is robust against illumination differences and recovers disparities at the object edges, occluded regions. The performance of the technique used is analyzed by percentage of bad matching pixel and root mean square error. Experimental results generated determine that the proposed algorithm gives more stable disparity maps.

Traditionally, the problem of stereo matching has been addressed either by a sparse feature-based approach or a dense area-based approach. This technique still suffers from a lack in accuracy or long computation time needed to match stereo images. In this paper, we employ color segments on scan lines of stereo images as primitives for matching, and introduce the widely used fuzzy set theory to the matching processes in order to handle ambiguities in matching. During the stage of accurate pixel correspondences of matched color segments, we use the technique of dynamic time warping (DTW), which is used in speech recognition. We found that DTW works robustly and rapidly regardless of the number of points to be matched. Covering more image space than point elements and providing more information than gray level features, color segments are less sensitive to noise and photometric variations and are thus more easily matched, and can reduce the false matches. Furthermore, a post-processing step can find more matches and alleviate the influence of under-segmentation or over-segmentation. A fast median filter can be applied to the results to further remove outliers. Experiments have been performed to demonstrate the accuracy and efficiency of the algorithm.

Estimating the disparity and normal direction of one pixel simultaneously, instead of only disparity, also known as 3D label methods, can achieve much higher subpixel accuracy in the stereo matching problem. However, it is extremely difficult to assign an appropriate 3D label to each pixel from the continuous label space R3 while maintaining global consistency because of the infinite parameter space. In this paper, we propose a novel algorithm called PatchMatch-based superpixel cut to assign 3D labels of an image more accurately. In order to achieve robust and precise stereo matching between local windows, we develop a bilayer matching cost, where a bottom-up scheme is exploited to design the two layers. The bottom layer is employed to measure the similarity between small square patches locally by exploiting a pretrained convolutional neural network, and then, the top layer is developed to assemble the local matching costs in large irregular windows induced by the tangent planes of object surfaces. To optimize the spatial smoothness of local assignments, we propose a novel strategy to update 3D labels. In the procedure of optimization, both segmentation information and random refinement of PatchMatch are exploited to update candidate 3D label set for each pixel with high probability of achieving lower loss. Since pairwise energy of general candidate label sets violates the submodular property of graph cut, we propose a novel multilayer superpixel structure to group candidate label sets into candidate assignments, which thereby can be efficiently fused by α-expansion graph cut. Extensive experiments demonstrate that our method can achieve higher subpixel accuracy in different data sets, and currently ranks first on the new challenging Middlebury 3.0 benchmark among all the existing methods.

Many vision applications have been formulated as Markov Random Field (MRF) problems. Although many of them are discrete labeling problems, continuous formulation often achieves great improvement on the qualities of the solutions in some applications such as stereo matching and optical flow. In continuous formulation, however, it is much more difficult to optimize the target functions. In this paper, we propose a new method called fusion move driven Markov Chain Monte Carlo method (MCMC-F) that combines the Markov Chain Monte Carlo method and the fusion move to solve continuous MRF problems effectively. This algorithm exploits powerful fusion move while it fully explore the whole solution space. We evaluate it using the stereo matching problem. We empirically demonstrate that the proposed algorithm is more stable and always finds lower energy states than the state-of-the art optimization techniques.

In this paper, we address the problem of stereo matching. Edge information is sufficiently explored in our algorithm to compose the Global Edge Constraint (GEC). By using the GEC, an optimal correlation window for each specific match (i.e. seed match) can be estimated and pixels within the window would be quickly matched with propagation. We evaluate our algorithm with real stereo pairs and both high accuracy disparity maps and a handling of occlusions can be achieved. Experimental results show the good performance of the proposed GEC in stereo matching.

Stereo matching plays an important role in many applications, such as Advanced Driver Assistance Systems, 3D reconstruction, navigation, etc. However it is still an open problem with many difficult. Most difficult are often occlusions, object boundaries, and low or repetitive textures. In this paper, we propose a method for processing the stereo matching problem. We propose an efficient convolutional neural network to measure how likely the two patches matched or not and use the similarity as their stereo matching cost. Then the cost is refined by stereo methods, such as semiglobal maching, subpixel interpolation, median filter, etc. Our architecture uses large image patches which makes the results more robust to texture-less or repetitive textures areas. We experiment our approach on the KITTI2015 dataset which obtain an error rate of 4.42% and only needs 0.8 second for each image pairs.

In order to solve the stereo matching problem better, the accuracy of the algorithm in the stereo matching task needs to be improved. To tackle the problem, we proposed a Dual Spatial Pyramid Pooling Stereo Matching Network with Attention Mechanism (DSANet), a stereo matching network with better multi-scale feature extraction capability based on the basic structure of Pyramid Stereo Matching Network (PSMNet). The network consists of two main modules: the dual spatial pyramid pooling module and the attention module. The former module extracts multi-scale features from different branches at the same time. After concatenating these different features, the latter module will optimize the fused features and generate the cost volume with different branches multi-scale feature. The proposed approach was evaluated on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) 2015 dataset. Our experiment results on the KITTI 2015 dataset show that the mismatch rate of our network is 3.02%. Compared with the basic structure of PSMNet, the proposed method improves the effect of stereo matching.

A fast stereo matching algorithm suitable for embedded real-time systems

In stereo vision, disparity map after stereo matching using projection model can get the depth information and 3D information of the original image. In order to improve the accuracy and speed of stereo matching, improving the quality of disparity map becomes the core problem of stereo matching. Firstly, the paper extracted edge information of binocular images using Canny, Scharr and Sobel edge detection algorithm. On this basis, the paper use SGBM algorithm(Semi-global block Matching) and DP algorithm(Dynamic Programming)and CSCA algorithm(Cross-scale Cost Aggregation) respectively calculated the disparity map and its error rate. After analysis, the combination of Canny detection algorithm and CSCA algorithm can eliminate a large number of number of irrelevant information and reduce the amount of data processing. The real scene image is selected to validate the method .Based on the above research results, the method improved the quality of disparity map.

In this paper, we study the problem of stereo matching from a pair of images with different resolutions, e.g., those acquired with a tele-wide camera system. Due to the difficulty of obtaining ground-truth disparity labels in diverse real-world systems, we start from an unsupervised learning perspective. However, resolution asymmetry caused by unknown degradations between two views hinders the effectiveness of the generally assumed photometric consistency. To overcome this challenge, we propose to impose the consistency between two views in a feature space instead of the image space, named feature-metric consistency. Interestingly, we find that, although a stereo matching network trained with the photometric loss is not optimal, its feature extractor can produce degradation-agnostic and matching-specific features. These features can then be utilized to formulate a feature-metric loss to avoid the photometric inconsistency. Moreover, we introduce a self-boosting strategy to optimize the feature extractor progressively, which further strengthens the feature-metric consistency. Experiments on both simulated datasets with various degradations and a self-collected real-world dataset validate the superior performance of the proposed method over existing solutions.

In binocular stereo vision, since color and distance information can be obtained simultaneously by a passive method which will not affect the environment, it is expected to be commonly used in the vision of future mobile robots However, it is necessary to solve difficult problems such at finding corresponding points in the right and left images Many stereo algorithms dealing with this problem have been proposed so far, all with their own weaknesses. This paper considers the position reversal problem, in which the positions of the corresponding points change places on the right and left scan lines. This is a subject in which there has been little progress despite its importance in obstacle detection by mobile robots. It has been considered that the position reversal problem cannot be processed correctly in plain dynamic programming for global matching unless the assumption that the positions of the corresponding points do not change places is forced. In the method proposed in this paper, four states are set in in the search for the corresponding points. The search using dynamic programming is performed in two stages so that the position reversal problem can be solved while the above assumption is maintained. Then, matching also becomes possible in the case of objects for which the positions of the corresponding points change places. The proposed method is experimentally applied to the search problem in which the positions of the corresponding points change places inside scan lines, and its effectiveness is verified.

The belief propagation (BP) technique is successful in image stereo matching problem. However, when we consider stereo matching for videos, directly applying the BP algorithm frame by frame results in unsatisfactory temporally inconsistent disparity maps. In this paper, we present the temporally consistent belief propagation for video stereo matching. We introduce a temporal term in traditional BP objective function and propose an adaptive weighting scheme to account for this temporal term. We show that the proposed algorithm performs favorably against previous methods in the stereo video datasets. Furthermore, the proposed method can solve problems induced by previous methods like error propagation from previously occluded regions.

We propose a new approach to stereo matching for obstacle detection in the autonomous navigation framework. An accurate but slow reconstruction of the 3D scene is not needed; rather, it is more important to have a fast localization of the obstacles to avoid them. All the methods in the literature, based on a punctual stereo matching, are ineffective in realistic contexts because they are either computationally too expensive, or unable to deal with the presence of uniform patterns, or of perturbations between the left and right images. Our idea is to face the stereo matching problem as a matching between homologous regions. The stereo images are represented as graphs and a graph matching is computed to find homologous regions. Our method is strongly robust in a realistic environment, requires little parameter tuning, and is adequately fast, as experimentally demonstrated in a comparison with the best algorithms in the literature.

Thermal-visible stereo matching at night based on Multi-Modal Autoencoder