Regularised Projection of an Image Domain Triangulation for Disparity Map Refinement

Depth information can be obtained using stereo matching algorithms, which compute the horizontal displacement (disparity) of the corresponding points and convert to depth information using the triangular relation. However, the matching process is challenging with presence of textureless regions. This paper proposes a novel disparity refinement method for stereo matching based on Semi-global Matching (SGM) algorithm for texture less images. To be brief, the SGM algorithm is a high-performance matching algorithm of a stereo image pair and it reaches a trade off between matching quality and computing complexity. The main contribution comes from the improvement of matching quality on texture less regions. At the end of SGM, a right-to-left consistency check is performed to remove the invalid pixels in the disparity map. The proposed method is added after the right-to-left consistent check. We assume that the texture less regions in the original stereo pairs are planar. We employ edge detection to extract the textureless regions. The fitting method is employed in horizontal and vertical directions respectively. If the distance of two adjacent edge pixels is sufficiently large, the corresponding line segment will be considered as texturelss. For every texturelss line segment, the RANSAC algorithm is performed on the corresponding line segment in the disparity map. We use the well-known Middlebury dataset to compare our method with the normal SGM and other matching algorithms. It shows that our method performs well for most texture less stereo pairs.

The Census transform is very useful in stereo matching algorithms. But the traditional Census transform cost too much computation, and its accuracy and robustness are not satisfying. In this paper, a modified semi-global matching (SGM) algorithm based on the fast Census transform is proposed to improve the quality of the range map and decrease the computation. In addition, we modify the disparity refinement to enhance the quality of the estimated range map. Our experiments show that the proposed method performs better than the original SGM algorithm in terms of the accuracy and the anti-noise capability.

Image segments are often used as a constraint in stereo matching. However, both over-segmentation and under-segmentation can lead to disparity degradation in some regions. To obtain an accurate disparity map, a modified semi-global matching (SGM) algorithm is proposed which is based on adaptive window models. Introducing the object notion and build a new global energy function to optimise segments and estimate a disparity map jointly. The effective and efficient block coordinate descent approach is used to optimise the global energy function by merging small segments. The authors’ demonstrate the performance of the proposed algorithm on the KITTI and Middlebury benchmarks. The results show that the authors’ algorithm outperforms many state-of-the-art methods and confirm the effectiveness of approach.

Initializing and accelerating Stereo-DIC computation using semi-global matching with geometric constraints

Mobile stereo matching systems are becoming more important in many applications such as auto-driving and autonomous robots. However, to maintain its low power consumption, mobile platforms have only limited hardware resources. Accurate stereo matching methods require a high computational complexity, and it is difficult to maintain both acceptable accuracy and processing speed on the mobile platforms. To solve this trade-off, in this paper, we propose a novel acceleration approach for a well-known matching algorithm Zero-means Normalized Cross Correlation (ZNCC), and show its effectiveness on a Jetson TX2 embedded GPU. By combining our new approach, Z2- ZNCC, with the Semi-Global Matching (SGM) algorithm, our system achieves a low error rate of 7.76% while keeping 28 fps for 1242×375 pixels images with the maximum disparity of 128 on the KITTI 2015 dataset. This performance is higher than previous state-of-the-art system on the same hardware platform.

The primary objective of this study is to provide a methodology to generate a dense point cloud of digital surface model (DSM) and digital terrain model (DTM) from 0.6 m GSD stereo images acquired by CARTOSAT-2E satellite of the Indian Space Research Organization. These products are required for many high-resolution applications such as mapping of watersheds and watercourses; river flood modeling; analysis of flood depth, landslide, forest structure, and individual trees; design of highway and canal alignment. The proposed method consists of several processes such as orienting the stereo images, DEM point cloud extraction using the semi-global matching, and DSM to DTM filtering. The stereo model is built by performing aero triangulation and block adjustment using the ground control points. The semi-global matching algorithm is used on the epipolar images to generate the DSM in the form of dense point cloud corresponding to one height point for each pixel. The planimetric and height accuracies are evaluated using orthoimages and DSM and found to be within 3-pixel (~ 2 m) range. A method for extracting DTM by ground point filtering, to discriminate the probable ground points and the non-ground points, is provided by using discrete cosine transformation interpolation. This robust method uses a weight function to differentiate the noise points from the ground points. The overall classification efficiency kappa (κ) value averages at 0.92 for ground point classification/DTM extraction. The results of benchmarking of the GPS-aided GEO augmented navigation GPS receiver by operating it over IGS station, in static mode for collecting the checkpoints, also are presented.

Precise depth estimation is a key kernel function to realizing autonomous navigation on micro-aerial vehicles (MAVs). The state-of-the-art semi-global matching (SGM) algorithm has become favored for its high accuracy. In particular, it effectively handles low texture regions due to its global optimization of the disparity between a left and right image over the entire frame. However, SGM involves massively parallel computation (∼2TOP/s) and extremely high bandwidth memory access (38.6Tb/s) for 30fps HD resolution. This leads to ∼20s runtime for an HD image pair on a 3GHz CPU [1] requiring ∼386MB memory and >35W power consumption. Together, these factors place it well outside the realm of MAVs. Prior ASIC implementations have used either simpler local methods [2] or aggressively truncated global algorithms [3] that produce a depth map with significantly inferior quality or limited disparity range (32 or 64 pixels) and therefore fail to support standard automotive scene benchmarks [2–5]. In addition, due to the high memory requirement of SGM, prior methods [3–4] have used external DRAM to store intermediate computation, significantly reducing performance and efficiency.

The Semi-Global Matching (SGM) algorithms and their hardware accelerators, which emphasize stereo matching rather than occlusion filling, have been developed in the last few years. However, filling occlusions is indispensable for many real-world applications. This work presents a pixel-level pipeline architecture for the post-processing of SGM, which refines disparity through a left-right check, and multi-directional occlusion filling refinement. The hardware architecture based on optimization algorithms is on the Stratix-IV platform, and it consumes about 5720 LUTs, 12961 registers, and 2.15M bits of on-chip memory. The maximum working frequency can reach up to 95.15 MHz for the 640×480 resolution video and 128 disparity range with the power dissipation of 1.46 W and 320 frames per second processing speed.

In this study, we proposed an automatic approach to extract DSMs from VHR satellite imagery. This study aims to apply the histogram of orientated phase congruency (HOPC) descriptor and SIFT-Flow algorithm to match multi-view VHR satellite images densely and extract detailed DSMs competitive with ones generated from Light Detection and Ranging (LiDAR) point clouds. The HOPC descriptor relies on detecting features by calculating the phase congruency model, which allows the image features to be highly robust against illumination changes. Moreover, we developed an automatic algorithm to extract precise and homogenously distributed tie-points and subsequently conduct bias correction of the Rational Polynomial Coefficients (RPC). We tested our proposed algorithm on four WorldViw-3 datasets captured from different viewpoints, and compared the results with the DSMs obtained by the Semi-Global Matching (SGM) algorithm. The experimental results indicated the proficiency of our proposed method in generating accurate and reliable DSMs using VHR satellite data.

Abstract. This paper describes an efficient implementation of the semi-global matching (SGM) algorithm on multi-core processors that allows a nearly arbitrary number of path directions for the cost aggregation stage. The scanlines for each orientation are discretized iteratively once, and the regular substructures of the obtained template are reused and shifted to concurrently sum up the path cost in at most two sweeps per direction over the disparity space image. Since path overlaps do not occur at any time, no expensive thread synchronization will be needed. To further reduce the runtime on high counts of path directions, pixel-wise disparity gating is applied, and both the cost function and disparity loop of SGM are optimized using current single instruction multiple data (SIMD) intrinsics for two major CPU architectures. Performance evaluation of the proposed implementation on synthetic ground truth reveals a reduced height error if the number of aggregation directions is significantly increased or when the paths start with an angular offset. Overall runtime shows a speedup that is nearly linear to the number of available processors.

Stereo dense image matching (DIM) is a key technique in generating dense 3D point clouds at low cost, among which semi-global matching (SGM) is one of the best compromise between the matching accuracy and the time cost. Most commercial or open-source DIM software packages therefore adopt SGM as the core algorithm for the 3D point generation, which computes matching results in 2D image space by simply aggregating the matching results of multi-directional 1D paths. However, such aggregations of SGM did not consider the disparity consistency between adjacent pixels in 2D image space, which will finally decrease the matching accuracy. To achieve higher-accuracy while keep the high time efficiency of SGM, this paper proposes an improved SGM with a novel matching aggregation optimization constraint. The core algorithm formulates the matching aggregation as the optimization of a global energy function, and a local solution of the energy function is utilized to impose the disparity consistency between adjacent pixels, which is capable of removing noises in the matching aggregation results and increasing the final matching accuracy at low time cost. Experiments on aerial image dataset show that the proposed method outperformed the traditional SGM method and another improved SGM method. Compared with the traditional SGM, our proposed method can increase the average matching accuracy by at most 11%. Therefore, our proposed method can applied in some smart 3D applications, e.g. 3D change detection, city-scale reconstruction, and global survey mapping.

SGM-based seamline determination for urban orthophoto mosaicking

Binocular stereo vision, which is an attractive topic in the computer vision area, mainly focuses on extracting depth information through a stereo matching algorithm. The semi-global matching algorithm (SGM) is well-studied but is hard to be applied in real-time because of the high computational cost. In this work, we report an FPGA-based stereo vision system for the SGM. The system contains a camera controller, a module for image rectification, and the SGM accelerator. The SGM accelerator module can extract the depth information with very high speed by a parallel pixel-level pipeline architecture. The left-right check module further ensures the quality of the disparity map. We implement the system on the Stratix-V platform with a 14.9% error rate and it consumes about 58K LUTs, 59K registers, and 5.2M bits of on-chip memory. The maximum working frequency can reach up to 135.96 MHz for the XGA (1024×768) video and 128 disparity range with the power dissipation of 4.302 W and the processing speed of 173 XGA-frames per second.

Efficiency and accuracy of semi-global matching (SGM) make it outperform many stereo matching algorithms and is widely used under challenging occasions. However, SGM only incorporates information along a scanline in each pass and lacks interaction between scanlines, resulting in streak artifacts in the disparity image. We introduce a local edge-aware filtering method to SGM to enhance the interaction of neighboring scanlines, since streak artifacts can be avoided. We use bilateral weights based on intensity similarity and spatial affinity between pixels to build connections among scanlines. In each pass, we recursively estimate the aggregated cost of SGM and compute the weighted average of aggregated costs for pixels in the orthogonal direction to obtain the output of our method along each scanline. As one-dimensional bilateral filtering is used in our method, the extra computation is linear to image resolution and label space, which is a small fraction of that needed by SGM. We present ablation studies using stereo pairs under both constrained and natural conditions to verify the effectiveness of our method. Extensive experiments on Middlebury and Karlsruhe Institute of Technology and Toyota Technology Institute datasets demonstrate that our method removes all streak artifacts, improves the quality of the disparity image, and outperforms many other non-local cost aggregation approaches.

Human segmentation plays an important role in vision analysis due to its importance for applications such as 3D pose estimation, human behavior analysis, body parameter estimation and image compositing. In this paper, we present a new human segmentation method that can segment human from the complex background environment without using background subtraction like algorithms and motion estimation algorithms. Initially, semi-global stereo matching algorithm was used to get the coarse disparity map. Then GrabCut was used on the disparity space image to get the human's coarse silhouette. After some erosion morphology operations, the silhouette obtained from the disparity space image was used as the input marks for the GrabCut, which was used on the color image to get accurate human segmentation. Results show that the proposed method can have a good performance.