Collaborative semi-global stereo matching.

Depth information can be obtained using stereo matching algorithms, which compute the horizontal displacement (disparity) of the corresponding points and convert to depth information using the triangular relation. However, the matching process is challenging with presence of textureless regions. This paper proposes a novel disparity refinement method for stereo matching based on Semi-global Matching (SGM) algorithm for texture less images. To be brief, the SGM algorithm is a high-performance matching algorithm of a stereo image pair and it reaches a trade off between matching quality and computing complexity. The main contribution comes from the improvement of matching quality on texture less regions. At the end of SGM, a right-to-left consistency check is performed to remove the invalid pixels in the disparity map. The proposed method is added after the right-to-left consistent check. We assume that the texture less regions in the original stereo pairs are planar. We employ edge detection to extract the textureless regions. The fitting method is employed in horizontal and vertical directions respectively. If the distance of two adjacent edge pixels is sufficiently large, the corresponding line segment will be considered as texturelss. For every texturelss line segment, the RANSAC algorithm is performed on the corresponding line segment in the disparity map. We use the well-known Middlebury dataset to compare our method with the normal SGM and other matching algorithms. It shows that our method performs well for most texture less stereo pairs.

Stereo dense image matching (DIM) is a key technique in generating dense 3D point clouds at low cost, among which semi-global matching (SGM) is one of the best compromise between the matching accuracy and the time cost. Most commercial or open-source DIM software packages therefore adopt SGM as the core algorithm for the 3D point generation, which computes matching results in 2D image space by simply aggregating the matching results of multi-directional 1D paths. However, such aggregations of SGM did not consider the disparity consistency between adjacent pixels in 2D image space, which will finally decrease the matching accuracy. To achieve higher-accuracy while keep the high time efficiency of SGM, this paper proposes an improved SGM with a novel matching aggregation optimization constraint. The core algorithm formulates the matching aggregation as the optimization of a global energy function, and a local solution of the energy function is utilized to impose the disparity consistency between adjacent pixels, which is capable of removing noises in the matching aggregation results and increasing the final matching accuracy at low time cost. Experiments on aerial image dataset show that the proposed method outperformed the traditional SGM method and another improved SGM method. Compared with the traditional SGM, our proposed method can increase the average matching accuracy by at most 11%. Therefore, our proposed method can applied in some smart 3D applications, e.g. 3D change detection, city-scale reconstruction, and global survey mapping.

A new experimental method for measuring strain fields in highly deformable materials has been developed. This technique is based on an in-house developed Digital Image Correlation (DIC) system capable of accurately capturing localized or non-uniform strain distributions. Thanks to the implemented algorithm based on a Semi-Global Matching (SGM) approach, it is possible to constraint the regularity of the displacement field in order to significantly improve the reliability of the evaluated strains, especially in highly deformable materials. Being originally introduced for Digital Surface Modelling from stereo pairs, SGM is conceived for performing a one-dimensional search of displacements between images, but here a novel implementation for 2D displacement solution space is introduced. SGM approach is compared with the previously in-house developed implementation based on a local Least Squares Matching (LSM) approach. A comparison with the open source code Ncorr and with some FEM results is also presented. The investigation using the present DIC method focuses on 2D full-field strain maps of plain and notched specimens under tensile loading made of two different highly deformable materials: hot mix asphalt and thermoplastic composites for 3D-printing applications. In the latter specimens, an elliptical hole is introduced to assess the potentiality of the method in experimentally capturing high strain gradients in mixed-mode fracture situations.

A hybrid method for multiple viewpoint projection (MVP) holography using semi-global matching (SGM) and virtual view generation (VVG) is proposed to effectively synthesize a hologram. The stereo correspondence using the SGM method performs much better matching than local methods do and is almost as accurate as global matching methods. The VVG based on the disparity images can generate virtual viewpoint images. This method can conquer the tremendous information of the generated projection images, which improves the process speed and reduces redundant information of the image acquisition process. The method of generating virtual projection images is adopted to avoid applying the camera (or lens) array and the depth camera, so that the image acquisition system complex structure of MVP holography is simplified. The image in-painting method is applied to optimize the virtual projection images and improve synthesis speed and reconstruction image quality. It is possible to acquire and real-time display 3D objects outside the laboratory. In order to challenge the tremendous information of a hologram, several experimental schemes are performed. Experimental results have verified our method validity.

For urban planning purposes updated information on land-use and urban structures and change dynamics are essential. Satellite Earth observation is a very effective information source, to provide actual representations of the situation ‘as is’ in regular time steps. The Pleiades constellation offers very high resolution imagery with a triple-stereo acquisition. The intrinsic combination of thematic and height information is an additional asset provided by recent VHR stereo imagery. Digital surface models (DSM) can be derived through stereo matching algorithms, one of which is semi-global matching (SGM). SGM integrates local and global matching methods with a short processing time and accurate surface generation. We use the generated DSM for extracting 3-D features which enrich object-based class models of composite urban objects. These multi-purpose 3-D object feature datasets can be used for various planning and monitoring purposes, revealing height-related characteristics of urban objects (blocks, neighbourhoods, parks, industrial sites, etc.). In addition object-wise aggregated height information can be used to better assess the adherence of actual building activities to zoning or land development plans, e.g. with respect to maximum or average height in recreational areas etc. As compared to detailed and potentially more accurate DSMs generated from LiDAR data, a (3-)stereo DSM has the advantage of recording height and spectral information instantaneously, so not to lose any information in high dynamic urban settings due to a time gap. Height and colour information, when integrated by spatial context and multi-scale OBIA methods, enable a wealth of new applications in urban feature characterisation and 3D city model applications.

The Census transform is very useful in stereo matching algorithms. But the traditional Census transform cost too much computation, and its accuracy and robustness are not satisfying. In this paper, a modified semi-global matching (SGM) algorithm based on the fast Census transform is proposed to improve the quality of the range map and decrease the computation. In addition, we modify the disparity refinement to enhance the quality of the estimated range map. Our experiments show that the proposed method performs better than the original SGM algorithm in terms of the accuracy and the anti-noise capability.

Dense matching plays an important role in many fields, such as DEM (digital evaluation model) producing, robot navigation and 3D environment reconstruction. Traditional approaches may meet the demand of accuracy. But the calculation time and out puts density is hardly be accepted. Focus on the matching efficiency and complex terrain surface matching feasibility an aerial image dense matching method based on optical flow field is proposed in this paper. First, some high accurate and uniformed control points are extracted by using the feature based matching method. Then the optical flow is calculated by using these control points, so as to determine the similar region between two images. Second, the optical flow field is interpolated by using the multi-level B-spline interpolation in the similar region and accomplished the pixel by pixel coarse matching. Final, the results related to the coarse matching refinement based on the combined constraint, which recognizes the same points between images. The experimental results have shown that our method can achieve per-pixel dense matching points, the matching accuracy achieves sub-pixel level, and fully meet the three-dimensional reconstruction and automatic generation of DSM-intensive matching’s requirements. The comparison experiments demonstrated that our approach’s matching efficiency is higher than semi-global matching (SGM) and Patch-based multi-view stereo matching (PMVS) which verifies the feasibility and effectiveness of the algorithm.

We propose a novel lightweight network for stereo estimation. Our network consists of a fully-convolutional densely connected neural network (FC-DCNN) that computes matching costs between rectified image pairs. Our FC-DCNN method learns expressive features and performs some simple but effective postprocessing steps. The densely connected layer structure connects the output of each layer to the input of each subsequent layer. This network structure and the fact that we do not use any fully-connected layers or 3D convolutions leads to a very lightweight network. The output of this network is used in order to calculate matching costs and create a cost-volume. Instead of using time and memory-inefficient cost-aggregation methods such as semiglobal matching or conditional random fields in order to improve the result, we rely on filtering techniques, namely median filter and guided filter. By computing a left-right consistency check we get rid of inconsistent values. Afterwards we use a watershed foreground-background segmentation on the disparity image with removed inconsistencies. This mask is then used to refine the final prediction. We show that our method works well for both challenging indoor and outdoor scenes by evaluating it on the Middlebury, KITTI and ETH3D benchmarks respectively. Our full framework is available at https://github.com/thedodo/FC-DCNN

Initializing and accelerating Stereo-DIC computation using semi-global matching with geometric constraints

To reduce inflow and motion artifacts in free-breathing, free-running, steady-state spoiled gradient echo T1 -weighted (SPGR) myocardial perfusion imaging. Unsaturated spins from inflowing blood or out-of-plane motion cause flashing artifacts in free-running SPGR myocardial perfusion. During free-running SPGR, 1 non-selective RF excitation was added after every 3 slice-selective RF excitations to suppress inflow artifacts by forcing magnetization in neighboring regions to steady-state. Bloch simulations and phantom experiments were performed to evaluate the impact of the flip angle and non-selective RF frequency on inflowing spins and tissue contrast. Free-running perfusion with (n = 11) interleaved non-selective RF or without (n = 11) were studied in 22 subjects (age = 60.2 ± 14.3 years, 11 male). Perfusion images were graded on a 5-point Likert scale for conspicuity of wall enhancement, inflow/motion artifact, and streaking artifact and compared using Wilcoxon sum-rank testing. Numeric simulation showed that 1 non-selective RF excitation applied after every 3 slice-selective RF excitations produced superior out-of-plane signal suppression compared to 1 non-selective RF excitation applied after every 6 or 9 slice-selective RF excitations. In vitro experiments showed that a 30° flip angle produced near-optimal myocardial contrast. In vivo experiments demonstrated that the addition of interleaved non-selective RF significantly (P < .01) improved conspicuity of wall enhancement (mean score = 4.4 vs. 3.2) and reduced inflow/motion (mean score = 4.5 vs. 2.5) and streaking (mean score = 3.9 vs. 2.4) artifacts. Non-selective RF excitations interleaved between slice-selective excitations can reduce image artifacts in free-breathing, ungated perfusion images. Further studies are warranted to assess the diagnostic accuracy of the proposed solution for evaluating myocardial ischemia.

To suppress the streak artifacts in images reconstructed from sparse-view projections in computed tomography (CT), a residual, attention-based, dense UNet (RAD-UNet) deep network is proposed to achieve accurate sparse reconstruction. The filtered back projection (FBP) algorithm is used to reconstruct the CT image with streak artifacts from sparse-view projections. Then, the image is processed by the RAD-UNet to suppress streak artifacts and obtain high-quality CT image. Those images with streak artifacts are used as the input of the RAD-UNet, and the output-label images are the corresponding high-quality images. Through training via the large-scale training data, the RAD-UNet can obtain the capability of suppressing streak artifacts. This network combines residual connection, attention mechanism, dense connection and perceptual loss. This network can improve the nonlinear fitting capability and the performance of suppressing streak artifacts. The experimental results show that the RAD-UNet can improve the reconstruction accuracy compared with three existing representative deep networks. It may not only suppress streak artifacts but also better preserve image details. The proposed networks may be readily applied to other image processing tasks including image denoising, image deblurring, and image super-resolution.

The stereo matching algorithm is a technique that analyses two or more images captured at diverse view points in order to find positions in real 3D space for the pixels of 2D image. The stereo matching methods have been used in various fields such as drawing the topographical map from aerial photograph and finding the depth information of objects in machine vision system. Nowdays, optical motion capture techniques using stereo matching algorithms are being developed for visual applications such as virtual reality or 3D graphics. Stereo matching algorithms can be generally classified into two methods: feature-based and area-based ones. The feature-based method matches the feature elements between two images, and uses the interpolation to obtain the disparity information for the pixels other than the feature elements, while the area-based method performs matching between pixels in two images by calculating the correlations of the pixels residing in the search window. Area-based method cannot match feature element with more accuracy than feature-based method even though it can make more dense disparity map. Moreover, it has more possibility of error in the area of insufficient texture information or depth discontinuities. In this paper we present a novel technique for area-based stereo matching algorithm which provides more accurate and error-prone matching capabilities by using adaptive search range and window size. We propose two new strategies (1) for determining search range adaptively from the disparity map and multiresolutional images of region segments obtained by applying feature-based algorithm, and (2) for changing the window size adaptively according to the edge information derived from the wavelet transform such that the combination of two adaptive methods in search range and window size greatly enhance accuracy while reducing errors. We test our matching algorithms for various types of images, and shall show the outperformance of our stereo matching algorithm.

LiDAR-guided stereo matching with a spatial consistency constraint

Mobile stereo matching systems are becoming more important in many applications such as auto-driving and autonomous robots. However, to maintain its low power consumption, mobile platforms have only limited hardware resources. Accurate stereo matching methods require a high computational complexity, and it is difficult to maintain both acceptable accuracy and processing speed on the mobile platforms. To solve this trade-off, in this paper, we propose a novel acceleration approach for a well-known matching algorithm Zero-means Normalized Cross Correlation (ZNCC), and show its effectiveness on a Jetson TX2 embedded GPU. By combining our new approach, Z2- ZNCC, with the Semi-Global Matching (SGM) algorithm, our system achieves a low error rate of 7.76% while keeping 28 fps for 1242×375 pixels images with the maximum disparity of 128 on the KITTI 2015 dataset. This performance is higher than previous state-of-the-art system on the same hardware platform.

Stacking learning with coalesced cost filtering for accurate stereo matching