Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching.

With the emergence of the Smart City concept, the rapid advancement of urban three-dimensional (3D) reconstruction becomes imperative. While current developments in the field of 3D reconstruction have enabled the generation of 3D products such as Digital Surface Models (DSM), challenges persist in accurately reconstructing shadows, handling occlusions, and addressing low-texture areas in very-high-resolution remote sensing images. These challenges often lead to difficulties in calculating satisfactory disparity maps using existing stereo matching methods, thereby reducing the accuracy of 3D reconstruction. This issue is particularly pronounced in urban scenes, which contain numerous super high-rise and densely distributed buildings, resulting in large disparity values and occluded regions in stereo image pairs, and further leading to a large number of mismatched points in the obtained disparity map. In response to these challenges, this paper proposes a method to refine the disparity in urban scenes based on open-source GIS data. First, we register the GIS data with the epipolar-rectified images since there always exists unignorable geolocation errors between them. Specifically, buildings with different heights present different offsets in GIS data registering; thus, we perform multi-modal matching for each building and merge them into the final building mask. Subsequently, a two-layer optimization process is applied to the initial disparity map based on the building mask, encompassing both global and local optimization. Finally, we perform a post-correction on the building facades to obtain the final refined disparity map that can be employed for high-precision 3D reconstruction. Experimental results on SuperView-1, GaoFen-7, and GeoEye satellite images show that the proposed method has the ability to correct the occluded and mismatched areas in the initial disparity map generated by both hand-crafted and deep-learning stereo matching methods. The DSM generated by the refined disparity reduces the average height error from 2.2 m to 1.6 m, which demonstrates superior performance compared with other disparity refinement methods. Furthermore, the proposed method is able to improve the integrity of the target structure and present steeper building facades and complete roofs, which are conducive to subsequent 3D model generation.

Depth information can be obtained using stereo matching algorithms, which compute the horizontal displacement (disparity) of the corresponding points and convert to depth information using the triangular relation. However, the matching process is challenging with presence of textureless regions. This paper proposes a novel disparity refinement method for stereo matching based on Semi-global Matching (SGM) algorithm for texture less images. To be brief, the SGM algorithm is a high-performance matching algorithm of a stereo image pair and it reaches a trade off between matching quality and computing complexity. The main contribution comes from the improvement of matching quality on texture less regions. At the end of SGM, a right-to-left consistency check is performed to remove the invalid pixels in the disparity map. The proposed method is added after the right-to-left consistent check. We assume that the texture less regions in the original stereo pairs are planar. We employ edge detection to extract the textureless regions. The fitting method is employed in horizontal and vertical directions respectively. If the distance of two adjacent edge pixels is sufficiently large, the corresponding line segment will be considered as texturelss. For every texturelss line segment, the RANSAC algorithm is performed on the corresponding line segment in the disparity map. We use the well-known Middlebury dataset to compare our method with the normal SGM and other matching algorithms. It shows that our method performs well for most texture less stereo pairs.

Stereo matching is the process of finding disparity map between a pair of stereo images. Markov Random Field (MRF) model has been used in stereo matching algorithms in which the goal is to minimize total data, smoothness and occlusion cost. Recent MRF based stereo matching algorithms assign the same smoothness cost to adjacent pixels which is not effective at objects boundaries. In this paper, we propose a method to constrain the smoothness cost such that the cost applied to disparity discontinuity within an object is higher than that of the object's boundary. Data cost function is also normalized before computing its probability distribution function (PDF) to enforce an uniform PDF for the entire image. Experimental results have demonstrated that the proposed method is competitive to the state-of-the-art stereo matching algorithms.

Stereo matching is a difficult and challenging task due to many uncontrollable factors that affect the results. These factors include the non-ideal radiometric conditions and the presence of weak-textured regions. A stereo matching algorithm including cost computation, cost aggregation, disparity computation, and disparity refinement is proposed to overcome these limitations. First, census, gradient, and HSL measures are combined as the pixel-wise matching cost to reduce the radiometric distortions. Second, a second-order smoothness constraint based on angle direction priors is utilized to improve the matching accuracy in the weak-textured regions. Then, the matching cost and smoothness constraint are applied to semi-global matching. Third, a winner-takes-all strategy is adopted to calculate the initial disparity maps. Finally, a coarse-to-fine simple linear iterative clustering superpixel segmentation is proposed to split the image into some regions with similar colors, adjacent pixels, and similar disparity planes. Through the segmentation result, boundary lines are classified into occluded, hinge, and coplanar lines. Slanted plane iterative optimization is performed to obtain the optimized disparity maps according to the segmentation result and boundary lines classification. The algorithm is evaluated in the Middlebury and KITTI datasets. The experimental results show that the method has good performance under non-ideal radiometric conditions and in the weak-textured regions. The disparity maps obtained by the method have a lower mismatching rate compared with other algorithms.

The stereo matching <span>process is one of the key areas that impact the stereo vision technologies which are commonly used in the application of three-dimensional reconstructions. The accuracy of the depth information used in three-dimensional reconstruction is directly proportional to the accuracy of the disparity obtained from stereo matching. The challenging issue in the stereo matching process is to determine the accurate corresponding point between the left image and right image, especially for image pairs that have different exposure such as different illumination and image pair with less texture region. In order to increase the accuracy of disparity value, a new stereo matching algorithm is proposed based on the combination of Sum of absolute different and census transform at matching cost computation. guided filter was used in the matching cost aggregation in order to remove noise and preserve the edge of the image. In the optimization step, the winner take all strategy is used to select the minimum matching cost. Finally, a median filter is applied to the initial disparity map for refinement purposes. The experimental results show that the algorithm is effective in reducing the error and improving the accuracy of the disparity map in different illumination regions, less textured regions and different environmental exposure.</span>

Stereo vision is used in many application areas, such as robot-assisted manufacturing processes. Recently, many different efficient stereo matching algorithms based on deep learning have been developed to solve the limitations of traditional correspondence point analysis, among others. The challenges include texture-poor objects or non-cooperative objects. One of these end-to-end learning algorithms is the Adaptive Aggregation Network (AANet/AANet+), which is divided into five steps: feature extraction, cost volume construction, cost aggregation, disparity computation and disparity refinement. By combining different components, it is easy to create an individual stereo matching model. Our goal is to develop efficient learning methods for robot-assisted manufacturing processes for cross-domain data streams. The aim is to improve recognition tasks and process optimisation. To achieve this, we have investigated the AANet+ in terms of usability and efficiency on our own test-dataset with different measurement setups (passive stereo system). Input of the AANet+ are rectified stereo pairs of the test-dataset and a pre-trained model. Instead of generating our own training dataset, we used two pre-trained models based on the KITTI-2015 and SceneFlow datasets. Our research has shown that the pretrained model based on the Scene Flow dataset predicts disparities with better object delimination. Due to the Out-of-Distribution inputs, only reliable disparity predictions of the AANet are possible for test data sets with parallel measurement setup. We compared the results with two traditional stereo matching algorithms (SemiGlobal block matching and DAISY). Compared to the traditionally computed disparity maps, the AANet+ is able to robustly detect texture-poor objects and optically non-cooperative objects.

This paper proposes a new hybrid method between the learning-based and handcrafted methods for a stereo matching algorithm. The main purpose of the stereo matching algorithm is to produce a disparity map. This map is essential for many applications, including three-dimensional (3D) reconstruction. The raw disparity map computed by a convolutional neural network (CNN) is still prone to errors in the low texture region. The algorithm is set to improve the matching cost computation stage with hybrid CNN-based combined with truncated directional intensity computation. The difference in truncated directional intensity value is employed to decrease radiometric errors. The proposed method’s raw matching cost went through the cost aggregation step using the bilateral filter (BF) to improve accuracy. The winner-take-all (WTA) optimization uses the aggregated cost volume to produce an initial disparity map. Finally, a series of refinement processes enhance the initial disparity map for a more accurate final disparity map. This paper verified the performance of the algorithm using the Middlebury online stereo benchmarking system. The proposed algorithm achieves the objective of generating a more accurate and smooth disparity map with different depths at low texture regions through better matching cost quality.

Outlier detection and disparity refinement in stereo matching

In this paper, we extend the standard belief propagation (BP) sequential technique proposed in the tree-reweighted sequential method [15] to the fully connected CRF models with the geodesic distance affinity. The proposed method has been applied to the stereo matching problem. Also a new approach to the BP marginal solution is proposed that we call one-view occlusion detection (OVOD). In contrast to the standard winner takes all (WTA) estimation, the proposed OVOD solution allows to find occluded regions in the disparity map and simultaneously improve the matching result. As a result we can perform only one energy minimization process and avoid the cost calculation for the second view and the left-right check procedure. We show that the OVOD approach considerably improves results for cost augmentation and energy minimization techniques in comparison with the standard one-view affinity space implementation. We apply our method to the Middlebury data set and reach state-of-the-art especially for median, average and mean squared error metrics.

Aim. Determination of the elements of external spatial orientation of the surveying systems at the moment of image acquisition is the fundamental task in photogrammetry. Principally, this problem is solving in two ways. The first way is direct positioning and measuring of directions of camera optical axis in the geodetic space with the help of GNSS/INS equipment. The second way is the analytical solution of the problem using a set of reference information (often such information is a set of ground control points whose geodetic positions are known with sufficient accuracy and which are reliably recognised on aerial images of the photogrammetric block). The authors consider the task of providing reference and control information using the second approach, which has a number of advantages in terms of reliability and accuracy of determining the unknown image exterior orientation parameters. It is proposed to obtain additional images of ground control points by the method of their auxiliary aerial photography using an unmanned aerial vehicle (UAV) on a larger scale compared to the scale of the images of the photogrammetric block. The aim of the presented work is the implementation of the method of creating reference points and experimental confirmation of its effectiveness for photogrammetric processing. Methods and results. For the entire realization of the potential of the analytical way to determine the elements of external orientation of images, it is necessary to have a certain number of ground control points (GCP) and to keep the defined scheme of their location on the photogrammetric block. As the main source of input data authors use UAV aerial images of the terrain, which are obtained separately from the block of aerial survey, and have a better geometric resolution and which clearly depict the control reference points. Application of such auxiliary images gives the possibility of automated transferring of the position of ground control point into images of the main photogrammetric block. In our interpretation, these images of ground control points and their surroundings on the ground are called "control reference images". The basis of the work is to develop a method for obtaining the auxiliary control reference images and transferring of position of GCP depicted on them into aerial or space images of terrain by means of computer stereo matching. To achieve this goal, we have developed a processing method for the creation of control reference images of aerial image or a series of auxiliary multi-scale aerial images obtained by a drone from different heights above the reference point. The operator identifies and measures the GCP once on the auxiliary aerial image of the highest resolution. Then there is an automatic stereo matching of the control reference image in the whole series of auxiliary images in succession with a decrease in the resolution and, ultimately, directly with the aerial images of photogrammetric block. On this stage there are no recognition/cursor targeting by the human operator, and therefore there are no discrepancies, errors or mistakes related to it. In addition, if to apply fairly large size of control reference images, the proposed method can be used on a low-texture terrain, and therefore deal in many cases without the physical marking of points measured by GNSS method. And this is a way to simplify and reduce the cost of photogrammetric technology. The action of the developed method has been verified experimentally to provide the control reference information of the block of archival aerial images of the low-texture terrain. The results of the experimental approbation of the proposed method give grounds to assert that the method makes it possible to perform geodetic reference of photogrammetric projects more efficiently due to the refusal of the physical marking of the area before aerial survey. The proposed method can also be used to obtain the information for checking the quality of photogrammetric survey for provision of check points. The authors argue that the use of additional equipment - UAV of semi-professional class to obtain control reference images is economically feasible. Scientific novelty and practical relevance. The results of approbation of the "control reference image" method with obtaining stereo pairs of aerial images with vertical placement of the base are presented for the first time. There was implemented the study of the properties of such stereo pairs of aerial images to obtain images of reference points. The effectiveness of including reference images in the main block of the digital aerial triangulation network created on UAV’s images is shown.

Efficient local stereo matching algorithm based on fast gradient domain guided image filtering

In this paper, we present a robust technique of stereo calibration using homography constraints. Our method is novel as stereo calibration is performed by solving a polynomial equation system including two radial distortion parameters, using a minimal number of five image point correspondences. This enables us to calibrate from only a pair of stereo images of a planar scene, and to provide the exact algebraic solution to the stereo calibration problem. The minimal case solution is useful to reduce the computation time and increase the calibration robustness when using random sample consensus (RANSAC) from the correspondences of the stereo image pair. Further, a non-linear parameter optimization for the intrinsic and extrinsic parameters of stereo cameras is performed using the inliers, which are determined after RANSAC. In addition, our method can achieve more robust calibration results with multiple stereo image pairs by performing joint optimization. In contrast to the previous stereo calibration methods, our method works without requiring any special hardware and has no problems with one stereo image pair, even corrupted by severe radial distortions. Finally, by evaluating our method on both synthetic and real scene data, we demonstrate that our method is both efficient and accurate for stereo calibration.

In this paper, we propose a stereo matching method based on adaptive plane-fitting. In the proposed method, we use the disparity map adjustment with adaptive weight filter to preserve the boundary region. To estimate the initial disparity map, we use two cost aggregation functions, and in addition to that, we combined color and gradient for cost computation. In addition, the error value of the initial disparity map is calibrated by applying plane-fitting using mean-shift segmentation. The disparity map is adjusted by using adaptive weights to improve the accuracy of the disparity map. Experimental results show that the proposed method generates a high-quality disparity map which has the distinct boundary between objects compared to other methods.

In this paper, a bi-directional stereo matching algorithm based-on adaptive matching window is proposed. That is, by adaptively predicting the mutual correlation between stereo images pair using the proposed algorithm, the bandwidth of stereo input images pair can be compressed to the level of a conventional 2D image and a predicted image also can be effectively reconstructed using a reference image and disparity vectors. Especially, in the proposed algorithm, first feature values are extracted from input stereo images pair. Then, a matching window for stereo matching is adaptively selected depending on the magnitude of these feature values. That is, for the region having larger feature values, a smaller matching window is selected while, for the opposite case, a larger matching window is selected by comparing predetermined threshold values. This approach is not only able to reduce a mismatching of disparity vectors which occurs in the conventional dense disparity estimation with a small matching window, but is also able to reduce blocking effects which occur in the coarse disparity estimation with a large matching window. In addition, from some experiments using stereo sequences of 'Man' and 'Fichier', it is shown that the proposed algorithm improves the PSNRs of a reconstructed image to about 6.78 dB on average at ± 30 search ranges by comparing with that of conventional algorithms. And also, it is found that there is almost no difference between an original image and a reconstructed image through the proposed algorithm by comparison to that of conventional algorithms.

Although convolutional neural network (CNN)-based stereo matching methods have become increasingly popular thanks to their robustness, they primarily have been focused on the matching cost computation. By leveraging CNNs, we present a novel method for matching cost aggregation to boost the stereo matching performance. Our insight is to learn the convolution kernel within CNN architecture for cost aggregation in a fully convolutional manner. Tailored to cost aggregation problem, our method differs from handcrafted methods in terms of its convolutional aggregation through optimally learned CNNs. First, the matching cost is aggregated with cost volume unary network, and then optimized with explicit disparity boundary, estimated through disparity boundary pairwise network, within a global energy minimization. Experiments demonstrate that our method outperforms conventional hand-crafted aggregation methods.