A non-local cost aggregation method for stereo matching

Matching cost aggregation is one of the oldest and still popular methods for stereo correspondence. While effective and efficient, cost aggregation methods typically aggregate the matching cost by summing/averaging over a user-specified, local support region. This is obviously only locally-optimal, and the computational complexity of the full-kernel implementation usually depends on the region size. In this paper, the cost aggregation problem is re-examined and a non-local solution is proposed. The matching cost values are aggregated adaptively based on pixel similarity on a tree structure derived from the stereo image pair to preserve depth edges. The nodes of this tree are all the image pixels, and the edges are all the edges between the nearest neighboring pixels. The similarity between any two pixels is decided by their shortest distance on the tree. The proposed method is non-local as every node receives supports from all other nodes on the tree. The proposed method can be naturally extended to the time domain for enforcing temporal coherence. Unlike previous methods, the non-local property guarantees that the depth edges will be preserved when the temporal coherency between all the video frames are considered. A non-local weighted median filter is also proposed based on the non-local cost aggregation algorithm. It has been demonstrated to outperform all local weighted median filters on disparity/depth upsampling and refinement.

Human beings process stereoscopic correspondence across multiple scales. However, this bio-inspiration is ignored by state-of-the-art cost aggregation methods for dense stereo correspondence. In this paper, a generic cross-scale cost aggregation framework is proposed to allow multi-scale interaction in cost aggregation. We firstly reformulate cost aggregation from a unified optimization perspective and show that different cost aggregation methods essentially differ in the choices of similarity kernels. Then, an inter-scale regularizer is introduced into optimization and solving this new optimization problem leads to the proposed framework. Since the regularization term is independent of the similarity kernel, various cost aggregation methods can be integrated into the proposed general framework. We show that the cross-scale framework is important as it effectively and efficiently expands state-of-the-art cost aggregation methods and leads to significant improvements, when evaluated on Middlebury, KITTI and New Tsukuba datasets.

This paper presents a new nonlocal cost aggregation method for stereo matching. The minimum spanning tree (MST) employs color difference as the sole component to build the weight function, which often leads to failure in achieving satisfactory results in some boundary regions with similar color distributions. In this paper, a modified initial cost is used. The erroneous pixels are often caused by two pixels from object and background, which have similar color distribution. And then inner color correlation is employed as a new component of the weight function, which is determined to effectively eliminate them. Besides, the segmentation method of the tree structure is also improved. Thus, a more robust and reasonable tree structure is developed. The proposed method was tested on Middlebury datasets. As can be expected, experimental results show that the proposed method outperforms the classical nonlocal methods.

Although convolutional neural network (CNN)-based stereo matching methods have become increasingly popular thanks to their robustness, they primarily have been focused on the matching cost computation. By leveraging CNNs, we present a novel method for matching cost aggregation to boost the stereo matching performance. Our insight is to learn the convolution kernel within CNN architecture for cost aggregation in a fully convolutional manner. Tailored to cost aggregation problem, our method differs from handcrafted methods in terms of its convolutional aggregation through optimally learned CNNs. First, the matching cost is aggregated with cost volume unary network, and then optimized with explicit disparity boundary, estimated through disparity boundary pairwise network, within a global energy minimization. Experiments demonstrate that our method outperforms conventional hand-crafted aggregation methods.

The stereo matching <span>process is one of the key areas that impact the stereo vision technologies which are commonly used in the application of three-dimensional reconstructions. The accuracy of the depth information used in three-dimensional reconstruction is directly proportional to the accuracy of the disparity obtained from stereo matching. The challenging issue in the stereo matching process is to determine the accurate corresponding point between the left image and right image, especially for image pairs that have different exposure such as different illumination and image pair with less texture region. In order to increase the accuracy of disparity value, a new stereo matching algorithm is proposed based on the combination of Sum of absolute different and census transform at matching cost computation. guided filter was used in the matching cost aggregation in order to remove noise and preserve the edge of the image. In the optimization step, the winner take all strategy is used to select the minimum matching cost. Finally, a median filter is applied to the initial disparity map for refinement purposes. The experimental results show that the algorithm is effective in reducing the error and improving the accuracy of the disparity map in different illumination regions, less textured regions and different environmental exposure.</span>

Stereo matching is one of the most active research fields in computer vision. The paper introduces the categories and the performance index of stereo matching and introduces three high-speed and state-of-the-art stereo matching algorithms with different cost aggregation: fast bilateral stereo (FBS), binary stereo matching (BSM), and a non-local cost aggregation method (NLCA). By comparing the performance in terms of both quality and speed, we concluded that FSB deals with the effects of noise well; BSM is suitable for embedded devices and has a good performance with radiometric differences; NLCA combines the efficiency with the accuracy of state-of-the-art algorithms.

This paper presents a local stereo matching algorithm based on the window construction method using local edge detection. In order to improve performance of window-based cost aggregation computation, a new rule called Dissimilar Intensity Support technique is proposed to distinguish support pixels with dissimilar intensities from those with similar intensity for each centered pixel. According to the proposed rule, support pixels away from centered pixel with dissimilar intensities are used to compute cost aggregation for enhancing intensity variation in local support region. Initial disparity is obtained by classical Winner-Take-All method. Then disparity refinement based on pixels near center with similar intensity as well as smooth disparity is introduced to rectify mismatch. The experimental results based on Middlebury stereo benchmark show that the proposed algorithm has good performance especially in low textured region with only few parameters.

Many stereo vision algorithms for intelligent vehicles use a cost aggregation strategy because of its efficiency. We propose Adaptive Cost Aggregation Table (ACAT), an algorithm that is a global cost aggregation method which uses every cost in the whole image to estimate each disparity. The proposed algorithm works on conditional random fields to use locally variant information. ACAT aggregates cost adaptively, considering local intensity information. However, computational complexity does not increase compared to standard Semi-Global Matching (SGM) and Cost Aggregation Table (CAT) algorithm. We compared the proposed algorithm to other cost aggregation algorithms in analysis of the KITTI dataset. Disparity results of ACAT were more accurate than those of SGM and CAT, for f at areas, for discontinuous areas and in occlusion areas.

This paper presents a novel hardware-oriented stereo vision system based on 1-D cost aggregation. Many researchers have implemented hardware efficient stereo matching to realize real-time systems. However, such methods require a large amount of memory. We proposed a system that is based on a hardware-software hybrid architecture for memory reduction. It consisted of grayscale 1-D cost aggregation HW and 2-D disparity refinement SW. The 1-D processing reduced the size of RAM in our HW to 266 kb with an input image size of 1024×768. We achieved the average error rate for the Middlebury datasets as 6.24%. The processing time was 56.6 ms for the 1024×768 images and an average of 8.6 ms for the Middlebury datasets which have an average size of 400×380. Using the resolution of Middlebury datasets, our system can perform real-time depth-aided image processing.

Stereo matching is a difficult and challenging task due to many uncontrollable factors that affect the results. These factors include the non-ideal radiometric conditions and the presence of weak-textured regions. A stereo matching algorithm including cost computation, cost aggregation, disparity computation, and disparity refinement is proposed to overcome these limitations. First, census, gradient, and HSL measures are combined as the pixel-wise matching cost to reduce the radiometric distortions. Second, a second-order smoothness constraint based on angle direction priors is utilized to improve the matching accuracy in the weak-textured regions. Then, the matching cost and smoothness constraint are applied to semi-global matching. Third, a winner-takes-all strategy is adopted to calculate the initial disparity maps. Finally, a coarse-to-fine simple linear iterative clustering superpixel segmentation is proposed to split the image into some regions with similar colors, adjacent pixels, and similar disparity planes. Through the segmentation result, boundary lines are classified into occluded, hinge, and coplanar lines. Slanted plane iterative optimization is performed to obtain the optimized disparity maps according to the segmentation result and boundary lines classification. The algorithm is evaluated in the Middlebury and KITTI datasets. The experimental results show that the method has good performance under non-ideal radiometric conditions and in the weak-textured regions. The disparity maps obtained by the method have a lower mismatching rate compared with other algorithms.

Computational complexity of the local stereo matching method is affected by disparity ranges. In case of depth estimation in sequential stereo images, high computational complexity is a problem in terms of real-time processing. In this paper, we propose a temporal correlation based stereo matching method in sequential images. Using temporal information in a sequential stereo matching method provides inaccurate disparity ranges, since the estimated depth map accuracy is gradually degraded. To preserve the depth map quality in temporal stereo matching procedure, we adopt the guided image filtering for matching cost aggregation. Since the guided image filtering has a similar structure with bilateral filter, it preserves an object boundary region even in restricted disparity search ranges. Inaccurately estimated disparity values from the temporal correlation are compensated by filtering based cost aggregation method. From the experiment results, we check that the proposed depth map acquisition method preserves the depth map quality in temporal domain stereo matching.

Stereo vision is one of the most researched areas to develop human like vision capability into machines for the purpose of automatic navigation and reconstruction of the real world from images. Point correspondence matching for disparity map calculation is a vital research issue of stereo vision system. Window-based cost aggregation methods used for the correspondence problem have attracted researches as it can be implemented in real time. In this paper, a new hybrid cost aggregation strategy for similarity evaluation based on adaptive weight and color segmentation of stereo image is proposed. In this strategy, pixels which lie on same segment and are spatially closer are given higher weight. Experimental results show that the proposed method is effective in improving overall disparity map and at depth discontinuity.

Matching cost aggregation plays a critical role for the stereo matching task. Existing CNN-based methods commonly use 3D convolutions to aggregate matching costs from a local 3D space. However, the high computational cost of 3D convolutions limits their applications. Traditional methods show that a slanted support window in 3D space can help to aggregate matching costs from informative regions, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> the surfaces of objects. Motivated by this idea, we propose a SPNet with differentiable slanted plane aggregation. Our slanted plane aggregation layers aggregate matching costs from a learnable slanted plane in a local 3D space to reduce computational and memory costs. Experimental results show that our slanted plane aggregation layers can learn to fit the surfaces of objects and effectively aggregate matching costs. Comparison with previous stereo matching methods shows that our network achieves competitive performance with higher efficiency.

A nonlocal finite element method for torsional statics and dynamics of circular nanostructures

This paper proposes a new hybrid method between the learning-based and handcrafted methods for a stereo matching algorithm. The main purpose of the stereo matching algorithm is to produce a disparity map. This map is essential for many applications, including three-dimensional (3D) reconstruction. The raw disparity map computed by a convolutional neural network (CNN) is still prone to errors in the low texture region. The algorithm is set to improve the matching cost computation stage with hybrid CNN-based combined with truncated directional intensity computation. The difference in truncated directional intensity value is employed to decrease radiometric errors. The proposed method’s raw matching cost went through the cost aggregation step using the bilateral filter (BF) to improve accuracy. The winner-take-all (WTA) optimization uses the aggregated cost volume to produce an initial disparity map. Finally, a series of refinement processes enhance the initial disparity map for a more accurate final disparity map. This paper verified the performance of the algorithm using the Middlebury online stereo benchmarking system. The proposed algorithm achieves the objective of generating a more accurate and smooth disparity map with different depths at low texture regions through better matching cost quality.