Adaptive cost aggregation table on conditional random fields for intelligent vehicles

Recent research activity on stereo matching has proved the efficacy of local approaches based on advanced cost aggregation strategies in accurately retrieving 3D information. However, accuracy is typically achieved at expense of computational efficiency, with best methods being far from meeting real-time requirements. On the other side, basic real-time local algorithms relying on a rectangular correlation window suffer from significant ambiguity along depth borders and untextured areas. This work proposes a novel local approach aimed at maximizing the speed-accuracy trade-off by means of an efficient segmentation-based cost aggregation strategy.

Matching cost aggregation is one of the oldest and still popular methods for stereo correspondence. While effective and efficient, cost aggregation methods typically aggregate the matching cost by summing/averaging over a user-specified, local support region. This is obviously only locally-optimal, and the computational complexity of the full-kernel implementation usually depends on the region size. In this paper, the cost aggregation problem is re-examined and a non-local solution is proposed. The matching cost values are aggregated adaptively based on pixel similarity on a tree structure derived from the stereo image pair to preserve depth edges. The nodes of this tree are all the image pixels, and the edges are all the edges between the nearest neighboring pixels. The similarity between any two pixels is decided by their shortest distance on the tree. The proposed method is non-local as every node receives supports from all other nodes on the tree. As can be expected, the proposed non-local solution outperforms all local cost aggregation methods on the standard (Middlebury) benchmark. Besides, it has great advantage in extremely low computational complexity: only a total of 2 addition/subtraction operations and 3 multiplication operations are required for each pixel at each disparity level. It is very close to the complexity of unnormalized box filtering using integral image which requires 6 addition/subtraction operations. Unnormalized box filter is the fastest local cost aggregation method but blurs across depth edges. The proposed method was tested on a MacBook Air laptop computer with a 1.8 GHz Intel Core i7 CPU and 4 GB memory. The average runtime on the Middlebury data sets is about 90 milliseconds, and is only about 1.25× slower than unnormalized box filter. A non-local disparity refinement method is also proposed based on the non-local cost aggregation method.

Matching cost aggregation is one of the oldest and still popular methods for stereo correspondence. While effective and efficient, cost aggregation methods typically aggregate the matching cost by summing/averaging over a user-specified, local support region. This is obviously only locally-optimal, and the computational complexity of the full-kernel implementation usually depends on the region size. In this paper, the cost aggregation problem is re-examined and a non-local solution is proposed. The matching cost values are aggregated adaptively based on pixel similarity on a tree structure derived from the stereo image pair to preserve depth edges. The nodes of this tree are all the image pixels, and the edges are all the edges between the nearest neighboring pixels. The similarity between any two pixels is decided by their shortest distance on the tree. The proposed method is non-local as every node receives supports from all other nodes on the tree. The proposed method can be naturally extended to the time domain for enforcing temporal coherence. Unlike previous methods, the non-local property guarantees that the depth edges will be preserved when the temporal coherency between all the video frames are considered. A non-local weighted median filter is also proposed based on the non-local cost aggregation algorithm. It has been demonstrated to outperform all local weighted median filters on disparity/depth upsampling and refinement.

For common binocular stereo matching algorithms in computer vision, it is not easy to obtain high precision and high matching speed at the same time. In this paper, an improved binocular stereo matching algorithm based on Minimum Spanning Tree (MST) cost aggregation is proposed. Firstly, the performance of the parallel algorithm can be improved by reducing the height of the tree. Then, an improved Root to Leaf (L2R) cost aggregation algorithm is proposed. By combining stereo matching technology with parallel computing technology, the above method can realize synchronous parallel computing at the algorithm level. Experimental results show that the improved algorithm has high accuracy and high matching speed for binocular stereo vision.

Stereo matching is one of the most active research fields in computer vision. The paper introduces the categories and the performance index of stereo matching and introduces three high-speed and state-of-the-art stereo matching algorithms with different cost aggregation: fast bilateral stereo (FBS), binary stereo matching (BSM), and a non-local cost aggregation method (NLCA). By comparing the performance in terms of both quality and speed, we concluded that FSB deals with the effects of noise well; BSM is suitable for embedded devices and has a good performance with radiometric differences; NLCA combines the efficiency with the accuracy of state-of-the-art algorithms.

Computational complexity of the local stereo matching method is affected by disparity ranges. In case of depth estimation in sequential stereo images, high computational complexity is a problem in terms of real-time processing. In this paper, we propose a temporal correlation based stereo matching method in sequential images. Using temporal information in a sequential stereo matching method provides inaccurate disparity ranges, since the estimated depth map accuracy is gradually degraded. To preserve the depth map quality in temporal stereo matching procedure, we adopt the guided image filtering for matching cost aggregation. Since the guided image filtering has a similar structure with bilateral filter, it preserves an object boundary region even in restricted disparity search ranges. Inaccurately estimated disparity values from the temporal correlation are compensated by filtering based cost aggregation method. From the experiment results, we check that the proposed depth map acquisition method preserves the depth map quality in temporal domain stereo matching.

The box filter is widely used in image processing. Its computational complexity is low, and many other filters can be realized by using different size box filters. The implementation method of the box filter that is widely used in the software programs requires an array to store data, the size of which is filter size × image width. This array often limits the feasibility of the box filter on FPGA, because its size is proportional to the image width. In this paper, we show an implementation method of the box filter that requires less memory. Two types of wide-width but shallow memory are required in this method, but these memory can be efficiently realized on FPGA using distributed RAMs and block RAMs. In this method, the image is scanned in zigzag. The performance is decreased because of the necessity to overlap the zigzag scan, but it is fast enough for practical use. This zigzag scan enable to reduce the memory size to store data, but extra line buffers are required. Our approach is specially effective for applications that calculate cross-correlations for finding the best matching, because the overhead caused by the extra line buffers becomes relatively small in these applications. We show the effectiveness of our method using a stereo vision program based on the cost aggregation with guided filter.

Stereo vision is one of the most researched areas to develop human like vision capability into machines for the purpose of automatic navigation and reconstruction of the real world from images. Point correspondence matching for disparity map calculation is a vital research issue of stereo vision system. Window-based cost aggregation methods used for the correspondence problem have attracted researches as it can be implemented in real time. In this paper, a new hybrid cost aggregation strategy for similarity evaluation based on adaptive weight and color segmentation of stereo image is proposed. In this strategy, pixels which lie on same segment and are spatially closer are given higher weight. Experimental results show that the proposed method is effective in improving overall disparity map and at depth discontinuity.

Stereo vision is used in many application areas, such as robot-assisted manufacturing processes. Recently, many different efficient stereo matching algorithms based on deep learning have been developed to solve the limitations of traditional correspondence point analysis, among others. The challenges include texture-poor objects or non-cooperative objects. One of these end-to-end learning algorithms is the Adaptive Aggregation Network (AANet/AANet+), which is divided into five steps: feature extraction, cost volume construction, cost aggregation, disparity computation and disparity refinement. By combining different components, it is easy to create an individual stereo matching model. Our goal is to develop efficient learning methods for robot-assisted manufacturing processes for cross-domain data streams. The aim is to improve recognition tasks and process optimisation. To achieve this, we have investigated the AANet+ in terms of usability and efficiency on our own test-dataset with different measurement setups (passive stereo system). Input of the AANet+ are rectified stereo pairs of the test-dataset and a pre-trained model. Instead of generating our own training dataset, we used two pre-trained models based on the KITTI-2015 and SceneFlow datasets. Our research has shown that the pretrained model based on the Scene Flow dataset predicts disparities with better object delimination. Due to the Out-of-Distribution inputs, only reliable disparity predictions of the AANet are possible for test data sets with parallel measurement setup. We compared the results with two traditional stereo matching algorithms (SemiGlobal block matching and DAISY). Compared to the traditionally computed disparity maps, the AANet+ is able to robustly detect texture-poor objects and optically non-cooperative objects.

Interactive image segmentation has gained much popularity since it can enhance the segmentation performance with human interactions. In this paper, we propose a novel framework that can effectively estimate the binary labels based on Conditional Random Field (CRF) models combining region-level information with pixels-level information. The user scribbles are treated as soft constraints. We use pixel-layer with local CRF and region-layer with fully connected CRF to model the graph. Then we combine the pixel information and region information together to calculate the pixels' likelihood term. In contrast to using relations between the pixel-layer and the region-layer during the probability computation, we calculate the probabilities of pixels and regions separately to reduce computational complexity. After the likelihood computation, a graph cut based model is used to refine the pixel-level label predictions of the image in order to avoid label inconsistency in regions and capture the boundary details. Due to the less relationship between layers and utilizing the refinement, the proposed model can provide high accuracy and user inputs insensitive results. Our experiments show that the proposed approach can efficiently and accurately segment the images compared with other state of art methods with only a few scribbles. Graph cut based fine tuning improves our segmentation accuracy and our method needs less computational cost than traditional multi-layer-based methods.

Conditional Random Fields (CRFs) have proven to per form well on natural language processing tasks like name transliteration, concept tagging or grapheme-to-phoneme (g2p) conversion. The aim of this paper is to propose some extension to the state-of-the-art CRF systems for these tasks. Since the number of features can grow rapidly, a method for features selection is very helpful to boost performance. A combination of L1 and L2 regularization (elastic net) has been adopted and implemented within the Rprop optimization algorithm. Usually, dependencies on the target side are limited to bigram dependencies since the computational complexity grows exponentially with the history length. We present a modified CRF decoding where a conventional language model on target side is integrated into the CRF search process. Thus, larger contexts can be taken into account. Besides these two main parts, the already published margin-extension to the CRF training criterion has been adopted.

Efficient local stereo matching algorithm based on fast gradient domain guided image filtering

In this paper we present a novel conditional random field (CRF) model based on Laplacian mixtures for image labeling. Nature images posses many spatial regularities that can be efficiently modeled by probabilistic graphical models such as CRF. Usually hundreds of features and several types of feature functions are used together which increases computational complexity and makes the training difficult to converge. We propose a new Laplacian mixture CRF model, which simplifies the training and inference process without losing labeling accuracy. The belief propagation inference and stochastic gradient descent training are formulated accordingly for the new model. The experimental results demonstrate that the new approach achieves better classification accuracy than the baseline CRF and comparable results with the state-of-the-art complex models.

Stereo matching algorithms are useful for estimating a dense depth characteristic of a scene by finding corresponding points from stereo images of the scene. Several factors such as occlusion, noise, and illumination inconsistencies in the scene affect the disparity estimates and make this process challenging. Algorithms developed to overcome these challenges can be broadly categorized as learning-based and non-learning based disparity estimation algorithms. The learning-based approaches are more accurate but computationally expensive. In contrary, non-learning based algorithms are widely used and are computationally efficient algorithms. In this paper, we propose a new stereo matching algorithm using guided image filtering (GIF)-based cost aggregation. The main contribution of our approach is a cost calculation framework which is a hybrid of cross-correlation between stereo-image pairs and scene segmentation (HCS). The performance of our HCS technique was compared with state-ofthe- art techniques using version 3 of the benchmark Middlebury dataset. Our results confirm the effective performance of the HCS technique.

Stereo matching is a difficult and challenging task due to many uncontrollable factors that affect the results. These factors include the non-ideal radiometric conditions and the presence of weak-textured regions. A stereo matching algorithm including cost computation, cost aggregation, disparity computation, and disparity refinement is proposed to overcome these limitations. First, census, gradient, and HSL measures are combined as the pixel-wise matching cost to reduce the radiometric distortions. Second, a second-order smoothness constraint based on angle direction priors is utilized to improve the matching accuracy in the weak-textured regions. Then, the matching cost and smoothness constraint are applied to semi-global matching. Third, a winner-takes-all strategy is adopted to calculate the initial disparity maps. Finally, a coarse-to-fine simple linear iterative clustering superpixel segmentation is proposed to split the image into some regions with similar colors, adjacent pixels, and similar disparity planes. Through the segmentation result, boundary lines are classified into occluded, hinge, and coplanar lines. Slanted plane iterative optimization is performed to obtain the optimized disparity maps according to the segmentation result and boundary lines classification. The algorithm is evaluated in the Middlebury and KITTI datasets. The experimental results show that the method has good performance under non-ideal radiometric conditions and in the weak-textured regions. The disparity maps obtained by the method have a lower mismatching rate compared with other algorithms.