Post-Processing Refinement for Semi-Global Matching Algorithm Based on Real-Time FPGA

Binocular stereo vision, which is an attractive topic in the computer vision area, mainly focuses on extracting depth information through a stereo matching algorithm. The semi-global matching algorithm (SGM) is well-studied but is hard to be applied in real-time because of the high computational cost. In this work, we report an FPGA-based stereo vision system for the SGM. The system contains a camera controller, a module for image rectification, and the SGM accelerator. The SGM accelerator module can extract the depth information with very high speed by a parallel pixel-level pipeline architecture. The left-right check module further ensures the quality of the disparity map. We implement the system on the Stratix-V platform with a 14.9% error rate and it consumes about 58K LUTs, 59K registers, and 5.2M bits of on-chip memory. The maximum working frequency can reach up to 135.96 MHz for the XGA (1024×768) video and 128 disparity range with the power dissipation of 4.302 W and the processing speed of 173 XGA-frames per second.

Depth information can be obtained using stereo matching algorithms, which compute the horizontal displacement (disparity) of the corresponding points and convert to depth information using the triangular relation. However, the matching process is challenging with presence of textureless regions. This paper proposes a novel disparity refinement method for stereo matching based on Semi-global Matching (SGM) algorithm for texture less images. To be brief, the SGM algorithm is a high-performance matching algorithm of a stereo image pair and it reaches a trade off between matching quality and computing complexity. The main contribution comes from the improvement of matching quality on texture less regions. At the end of SGM, a right-to-left consistency check is performed to remove the invalid pixels in the disparity map. The proposed method is added after the right-to-left consistent check. We assume that the texture less regions in the original stereo pairs are planar. We employ edge detection to extract the textureless regions. The fitting method is employed in horizontal and vertical directions respectively. If the distance of two adjacent edge pixels is sufficiently large, the corresponding line segment will be considered as texturelss. For every texturelss line segment, the RANSAC algorithm is performed on the corresponding line segment in the disparity map. We use the well-known Middlebury dataset to compare our method with the normal SGM and other matching algorithms. It shows that our method performs well for most texture less stereo pairs.

Mobile stereo matching systems are becoming more important in many applications such as auto-driving and autonomous robots. However, to maintain its low power consumption, mobile platforms have only limited hardware resources. Accurate stereo matching methods require a high computational complexity, and it is difficult to maintain both acceptable accuracy and processing speed on the mobile platforms. To solve this trade-off, in this paper, we propose a novel acceleration approach for a well-known matching algorithm Zero-means Normalized Cross Correlation (ZNCC), and show its effectiveness on a Jetson TX2 embedded GPU. By combining our new approach, Z2- ZNCC, with the Semi-Global Matching (SGM) algorithm, our system achieves a low error rate of 7.76% while keeping 28 fps for 1242×375 pixels images with the maximum disparity of 128 on the KITTI 2015 dataset. This performance is higher than previous state-of-the-art system on the same hardware platform.

Initializing and accelerating Stereo-DIC computation using semi-global matching with geometric constraints

The Census transform is very useful in stereo matching algorithms. But the traditional Census transform cost too much computation, and its accuracy and robustness are not satisfying. In this paper, a modified semi-global matching (SGM) algorithm based on the fast Census transform is proposed to improve the quality of the range map and decrease the computation. In addition, we modify the disparity refinement to enhance the quality of the estimated range map. Our experiments show that the proposed method performs better than the original SGM algorithm in terms of the accuracy and the anti-noise capability.

Image segments are often used as a constraint in stereo matching. However, both over-segmentation and under-segmentation can lead to disparity degradation in some regions. To obtain an accurate disparity map, a modified semi-global matching (SGM) algorithm is proposed which is based on adaptive window models. Introducing the object notion and build a new global energy function to optimise segments and estimate a disparity map jointly. The effective and efficient block coordinate descent approach is used to optimise the global energy function by merging small segments. The authors’ demonstrate the performance of the proposed algorithm on the KITTI and Middlebury benchmarks. The results show that the authors’ algorithm outperforms many state-of-the-art methods and confirm the effectiveness of approach.

Precise depth estimation is a key kernel function to realizing autonomous navigation on micro-aerial vehicles (MAVs). The state-of-the-art semi-global matching (SGM) algorithm has become favored for its high accuracy. In particular, it effectively handles low texture regions due to its global optimization of the disparity between a left and right image over the entire frame. However, SGM involves massively parallel computation (∼2TOP/s) and extremely high bandwidth memory access (38.6Tb/s) for 30fps HD resolution. This leads to ∼20s runtime for an HD image pair on a 3GHz CPU [1] requiring ∼386MB memory and >35W power consumption. Together, these factors place it well outside the realm of MAVs. Prior ASIC implementations have used either simpler local methods [2] or aggressively truncated global algorithms [3] that produce a depth map with significantly inferior quality or limited disparity range (32 or 64 pixels) and therefore fail to support standard automotive scene benchmarks [2–5]. In addition, due to the high memory requirement of SGM, prior methods [3–4] have used external DRAM to store intermediate computation, significantly reducing performance and efficiency.

The primary objective of this study is to provide a methodology to generate a dense point cloud of digital surface model (DSM) and digital terrain model (DTM) from 0.6 m GSD stereo images acquired by CARTOSAT-2E satellite of the Indian Space Research Organization. These products are required for many high-resolution applications such as mapping of watersheds and watercourses; river flood modeling; analysis of flood depth, landslide, forest structure, and individual trees; design of highway and canal alignment. The proposed method consists of several processes such as orienting the stereo images, DEM point cloud extraction using the semi-global matching, and DSM to DTM filtering. The stereo model is built by performing aero triangulation and block adjustment using the ground control points. The semi-global matching algorithm is used on the epipolar images to generate the DSM in the form of dense point cloud corresponding to one height point for each pixel. The planimetric and height accuracies are evaluated using orthoimages and DSM and found to be within 3-pixel (~ 2 m) range. A method for extracting DTM by ground point filtering, to discriminate the probable ground points and the non-ground points, is provided by using discrete cosine transformation interpolation. This robust method uses a weight function to differentiate the noise points from the ground points. The overall classification efficiency kappa (κ) value averages at 0.92 for ground point classification/DTM extraction. The results of benchmarking of the GPS-aided GEO augmented navigation GPS receiver by operating it over IGS station, in static mode for collecting the checkpoints, also are presented.

In this study, we proposed an automatic approach to extract DSMs from VHR satellite imagery. This study aims to apply the histogram of orientated phase congruency (HOPC) descriptor and SIFT-Flow algorithm to match multi-view VHR satellite images densely and extract detailed DSMs competitive with ones generated from Light Detection and Ranging (LiDAR) point clouds. The HOPC descriptor relies on detecting features by calculating the phase congruency model, which allows the image features to be highly robust against illumination changes. Moreover, we developed an automatic algorithm to extract precise and homogenously distributed tie-points and subsequently conduct bias correction of the Rational Polynomial Coefficients (RPC). We tested our proposed algorithm on four WorldViw-3 datasets captured from different viewpoints, and compared the results with the DSMs obtained by the Semi-Global Matching (SGM) algorithm. The experimental results indicated the proficiency of our proposed method in generating accurate and reliable DSMs using VHR satellite data.

Most of the recent top-ranked stereovision algorithms are based on some oversegmentation of the stereo pair images. Each image segment is modelled as a 3D plane, transposing the stereovision problem into a 3-parameters plane estimation problem. Trying to find the set of planes that best fits the input data while regularising on 3D adjacent planes, a global energy is minimised using different optimisation strategies. The regularisation term is important as it enforces the reconstructed scene consistency. In this paper, we propose a disparity map refinement strategy based on a triangular mesh of the image domain. This mesh is projected along the rays of the camera, each triangle being modelled by a slanted plane. Considering triangles instead of free form superpixels, i.e. adding a geometrical constraint on the shape of segment boundaries, allows to formulate simpler regularisation terms that only involve the mesh vertex disparities in the computations. Different curvature regularisation terms of the literature are explored and a new one is proposed, with better properties with regard to the usual coplanarity assumption between adjacent planes. We demonstrate the performance gain of the proposed method compared to the popular Semi-Global Matching (SGM) algorithm.

Abstract. This paper describes an efficient implementation of the semi-global matching (SGM) algorithm on multi-core processors that allows a nearly arbitrary number of path directions for the cost aggregation stage. The scanlines for each orientation are discretized iteratively once, and the regular substructures of the obtained template are reused and shifted to concurrently sum up the path cost in at most two sweeps per direction over the disparity space image. Since path overlaps do not occur at any time, no expensive thread synchronization will be needed. To further reduce the runtime on high counts of path directions, pixel-wise disparity gating is applied, and both the cost function and disparity loop of SGM are optimized using current single instruction multiple data (SIMD) intrinsics for two major CPU architectures. Performance evaluation of the proposed implementation on synthetic ground truth reveals a reduced height error if the number of aggregation directions is significantly increased or when the paths start with an angular offset. Overall runtime shows a speedup that is nearly linear to the number of available processors.

SGM-based seamline determination for urban orthophoto mosaicking

Efficiency and accuracy of semi-global matching (SGM) make it outperform many stereo matching algorithms and is widely used under challenging occasions. However, SGM only incorporates information along a scanline in each pass and lacks interaction between scanlines, resulting in streak artifacts in the disparity image. We introduce a local edge-aware filtering method to SGM to enhance the interaction of neighboring scanlines, since streak artifacts can be avoided. We use bilateral weights based on intensity similarity and spatial affinity between pixels to build connections among scanlines. In each pass, we recursively estimate the aggregated cost of SGM and compute the weighted average of aggregated costs for pixels in the orthogonal direction to obtain the output of our method along each scanline. As one-dimensional bilateral filtering is used in our method, the extra computation is linear to image resolution and label space, which is a small fraction of that needed by SGM. We present ablation studies using stereo pairs under both constrained and natural conditions to verify the effectiveness of our method. Extensive experiments on Middlebury and Karlsruhe Institute of Technology and Toyota Technology Institute datasets demonstrate that our method removes all streak artifacts, improves the quality of the disparity image, and outperforms many other non-local cost aggregation approaches.

Abstract. The demand for 3D models of various scales and precisions is strong for a wide range of applications, among which cultural heritage recording is particularly important and challenging. In this context, dense image matching is a fundamental task for processes which involve image-based reconstruction of 3D models. Despite the existence of commercial software, the need for complete and accurate results under different conditions, as well as for computational efficiency under a variety of hardware, has kept image-matching algorithms as one of the most active research topics. Semi-global matching (SGM) is among the most popular optimization algorithms due to its accuracy, computational efficiency, and simplicity. A challenging aspect in SGM implementation is the determination of smoothness constraints, i.e. penalties P1, P2 for disparity changes and discontinuities. In fact, penalty adjustment is needed for every particular stereo-pair and cost computation. In this work, a novel formulation of self-adjusting penalties is proposed: SGM penalties can be estimated solely from the statistical properties of the initial disparity space image. The proposed method of self-adjusting penalties (SGM-SAP) is evaluated using typical cost functions on stereo-pairs from the recent Middlebury dataset of interior scenes, as well as from the EPFL Herz-Jesu architectural scenes. Results are competitive against the original SGM estimates. The significant aspects of self-adjusting penalties are: (i) the time-consuming tuning process is avoided; (ii) SGM can be used in image collections with limited number of stereo-pairs; and (iii) no heuristic user intervention is needed.

Human segmentation plays an important role in vision analysis due to its importance for applications such as 3D pose estimation, human behavior analysis, body parameter estimation and image compositing. In this paper, we present a new human segmentation method that can segment human from the complex background environment without using background subtraction like algorithms and motion estimation algorithms. Initially, semi-global stereo matching algorithm was used to get the coarse disparity map. Then GrabCut was used on the disparity space image to get the human's coarse silhouette. After some erosion morphology operations, the silhouette obtained from the disparity space image was used as the input marks for the GrabCut, which was used on the color image to get accurate human segmentation. Results show that the proposed method can have a good performance.