Dense multiview stereo based on image texture enhancement

While learning-based multi-view stereo (MVS) methods have recently shown successful performances in quality and efficiency, limited MVS data hampers generalization to unseen environments. A simple solution is to generate various large-scale MVS datasets, but generating dense ground truth for 3D structure requires a huge amount of time and resources. On the other hand, if the reliance on dense ground truth is relaxed, MVS systems will generalize more smoothly to new environments. To this end, we first introduce a novel semi-supervised multi-view stereo framework called a Sparse Ground truth-based MVS Network (SGT-MVSNet) that can reliably reconstruct the 3D structures even with a few ground truth 3D points. Our strategy is to divide the accurate and erroneous regions and individually conquer them based on our observation that a probability map can separate these regions. We propose a self-supervision loss called the 3D Point Consistency Loss to enhance the 3D reconstruction performance, which forces the 3D points back-projected from the corresponding pixels by the predicted depth values to meet at the same 3D co-ordinates. Finally, we propagate these improved depth pre-dictions toward edges and occlusions by the Coarse-to-fine Reliable Depth Propagation module. We generate the spare ground truth of the DTU dataset for evaluation and extensive experiments verify that our SGT-MVSNet outperforms the state-of-the-art MVS methods on the sparse ground truth setting. Moreover, our method shows comparable reconstruction results to the supervised MVS methods though we only used tens and hundreds of ground truth 3D points.

The present Multi-view stereo (MVS) methods with supervised learning-based networks have an impressive performance comparing with traditional MVS methods. However, the ground-truth depth maps for training are hard to be obtained and are within limited kinds of scenarios. In this paper, we propose a novel unsupervised multi-metric MVS network, named M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> VSNet, for dense point cloud reconstruction without any supervision. To improve the robustness and completeness of point cloud reconstruction, we propose a novel multi-metric loss function that combines pixel-wise and feature-wise loss function to learn the inherent constraints from different perspectives of matching correspondences. Besides, we also incorporate the normal-depth consistency in the 3D point cloud format to improve the accuracy and continuity of the estimated depth maps. Experimental results show that M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> VSNet establishes the state-of-the-arts unsupervised method and achieves better performance than previous supervised MVSNet on the DTU dataset and demonstrates the powerful generalization ability on the Tanks & Temples benchmark with effective improvement.

Digital Surface Models (DSM) play an important role in topographic mapping, disaster mitigation, and True Digital Ortho Model (TDOM) generation. Traditionally, DSM have been produced using Multi-View Stereo (MVS) method, which remains widely used due to its efficiency and ability to capture large-scale surface geometry but often struggle in low texture and complex surface conditions. Neural Radiance Fields (NeRF) a recent machine learning-based innovation has shown promise in capturing fine geometric details and continuous surface structures. This study presents a comparative evaluation of DSMs generated using MVS (using Colmap) and NeRF (using Nerfstudio). We conducted the study in a complex topographic area with turbid water bodies, dense vegetation, irregular and compact residential areas, bare open land, small patterned plantations, and paved roads. Results demonstrate that NeRF achieved superior vertical accuracy (RMSE-Z = 2.404 m, LE90 = 3.103 m) compared to MVS (RMSE-Z = 5.092 m, LE90 = 8.359 m) and produced more complete surfaces in visually ambiguous areas. These findings suggest that NeRF can complement photogrammetric workflows by improving DSM quality in challenging environments. Nevertheless, limitations remain in terms of computational cost, scalability, and hardware requirements. The outcomes of this study highlight both the opportunities and constraints of applying NeRF to DSM generation, with implications for future geospatial practices such as urban monitoring, environmental assessment, and disaster management.

Albeit remarkable progress has been made to improve the accuracy and completeness of multi-view stereo (MVS), existing methods still suffer from either sparse reconstructions of low-textured surfaces or heavy computational burden. In this paper, we propose a Confidence-based Large-scale Dense Multi-view Stereo (CLD-MVS) method for high resolution imagery. Firstly, we formulate MVS as a multi-view depth estimation problem, and employ a normal-aware efficient PatchMatch stereo to estimate the initial depth and normal map for each reference view. A self-supervised deep learning method is then developed to predict the spatial confidence for multi-view depth maps, which is combined with cross-view consistency to generate the ground control points. Subsequently, a confidence-driven and boundary-aware interpolation scheme using static and dynamic guidance is adopted to synthesize dense depth and normal maps. Finally, a refinement procedure which leverages synthesized depth and normal as prior is conducted to estimate cross-view consistent surface. Experiments show that the proposed CLD-MVS method achieves high geometric completeness while preserving fine-scale details. In particular, it has ranked No. 1 on the ETH3D high-resolution MVS benchmark in terms of F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -score.

Estimating economic benefit of sugar beet based on three-dimensional computer vision: a case study in Inner Mongolia, China

In this work, we develop a new surface reconstruction pipeline that combines monocular camera images and LiDAR measurements from a moving sensor rig to reconstruct dense 3D mesh models of indoor scenes. For surface reconstruction, the 3D LiDAR and camera are widely deployed for gathering geometric information from environments. Current state-of-the-art multi-view stereo or LiDAR-only reconstruction methods cannot reconstruct indoor environments accurately due to shortcomings of each sensor type. In our approach, LiDAR measurements are integrated into a multi-view stereo pipeline for point cloud densification and tetrahedralization. In addition to that, a graph cut algorithm is utilized to generate a watertight surface mesh. Because our proposed method leverages the complementary nature of these two sensors, the accuracy and completeness of the output model are improved. The experimental results on real world data show that our method significantly outperforms both the state-of-the-art camera-only and LiDAR-only reconstruction methods in accuracy and completeness.

Estimation of plant canopy using low-altitude imagery can help monitor the normal growth status of crops and is highly beneficial for various digital farming applications such as precision crop protection. However, extracting 3D canopy information from raw images requires studying the effect of sensor viewing angle by taking into accounts the limitations of the mobile platform routes inside the field. The main objective of this research was to estimate wheat (Triticum aestivum L.) leaf parameters, including leaf length and width, from the 3D model representation of the plants. For this purpose, experiments with different camera viewing angles were conducted to find the optimum setup of a mono-camera system that would result in the best 3D point clouds. The angle-control analytical study was conducted on a four-row wheat plot with a row spacing of 0.17 m and with two seeding densities and growth stages as factors. Nadir and six oblique view image datasets were acquired from the plot with 88% overlapping and were then reconstructed to point clouds using Structure from Motion (SfM) and Multi-View Stereo (MVS) methods. Point clouds were first categorized into three classes as wheat canopy, soil background, and experimental plot. The wheat canopy class was then used to extract leaf parameters, which were then compared with those values from manual measurements. The comparison between results showed that (i) multiple-view dataset provided the best estimation for leaf length and leaf width, (ii) among the single-view dataset, canopy, and leaf parameters were best modeled with angles vertically at −45° and horizontally at 0° (VA −45, HA 0), while (iii) in nadir view, fewer underlying 3D points were obtained with a missing leaf rate of 70%. It was concluded that oblique imagery is a promising approach to effectively estimate wheat canopy 3D representation with SfM-MVS using a single camera platform for crop monitoring. This study contributes to the improvement of the proximal sensing platform for crop health assessment.

Learning-based multi-view stereo (MVS) methods have been widely studied recently. However, current works are limited to using fixed-size convolution kernels, leading to suboptimal features that lack anisotropy in low-textured regions and tend to produce invalid depth blending at the edge of the foreground and background. In this paper, we propose N2MVSNet, which learns adaptive non-local neighbors matching (ANNM) and their spatial impact to overcome these deficiencies. Furthermore, we explore the ability of spatial perception to depth dimension and propose 3D ANNM. Besides, following the coarse-to-fine scheme, severe mismatchings in coarse stages will result in error accumulation and propagation in finer stages. To this end, we adopt the pretrained RGB guided depth refinement for depth hypothesis repolish. The robustness of the training process is further elevated by the energy aggregation loss. Extensive experiments on the DTU and Tanks and Temples datasets demonstrate that the proposed network achieves state-of-the-art results.

This paper proposes a novel method for estimating disparity accurately. To achieve the ideal result, an optimal adjusting framework is proposed to address the noise, occlusions, and outliners. Different from the typical multi-view stereo (MVS) methods, the proposed approach not only use the color constraint, but also use the geometric constraint associating multiple frame from the image sequence. The result shows the disparity with a good visual quality that most of the noise is eliminated, the errors in occlusion area are suppressed and the details of scene objects are preserved.

Multi-view photometric stereo (MVPS) is a preferred method for detailed and precise 3D acquisition of an object from images. Although popular methods for MVPS can provide outstanding results, they are often complex to execute and limited to isotropic material objects. To address such limitations, we present a simple, practical approach to MVPS, which works well for isotropic as well as other object material types such as anisotropic and glossy. The proposed approach in this paper exploits the benefit of uncertainty modeling in a deep neural network for a reliable fusion of photometric stereo (PS) and multi-view stereo (MVS) network predictions. Yet, contrary to the recently proposed state-of-the-art, we introduce neural volume rendering methodology for a trustworthy fusion of MVS and PS measurements. The advantage of introducing neural volume rendering is that it helps in the reliable modeling of objects with diverse material types, where existing MVS methods, PS methods, or both may fail. Furthermore, it allows us to work on neural 3D shape representation, which has recently shown outstanding results for many geometric processing tasks. Our suggested new loss function aims to fit the zero level set of the implicit neural function using the most certain MVS and PS network predictions coupled with weighted neural volume rendering cost. The proposed approach shows state-of-the-art results when tested extensively on several benchmark datasets.

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.

Multi-view stereo plays an important role in 3D reconstruction but suffers from low reconstruction efficiency and has difficulties reconstructing areas with low or repeated textures. To address this, we propose MVP-Stereo, a novel multi-view parallel patchmatch stereo method. MVP-Stereo employs two key techniques. First, MVP-Stereo utilizes multi-view dilated ZNCC to handle low texture and repeated texture by dynamically adjusting the matching window size based on image variance and using a portion of pixels to calculate matching costs without increasing computational complexity. Second, MVP-Stereo leverages multi-scale parallel patchmatch to reconstruct the depth map for each image in a highly efficient manner, which is implemented by CUDA with random initialization, multi-scale parallel spatial propagation, random refinement, and the coarse-to-fine strategy. Experiments on the Strecha dataset, the ETH3D benchmark, and the UAV dataset demonstrate that MVP-Stereo can achieve competitive reconstruction quality compared to state-of-the-art methods with the highest reconstruction efficiency. For example, MVP-Stereo outperforms COLMAP in reconstruction quality by around 30% of reconstruction time, and achieves around 90% of the quality of ACMMP and SD-MVS in only around 20% of the time. In summary, MVP-Stereo can efficiently reconstruct high-quality point clouds and meet the requirements of several photogrammetric applications, such as emergency relief, infrastructure inspection, and environmental monitoring.

Abstract. Structure from motion (SfM) has been widely used to achieve automatic 3D reconstructions. However, as the 3D point clouds obtained via SfM are sparse, multi-view stereo (MVS) was developed to compensate for this sparseness. The accuracy of the 3D surface depends on the accuracy of the orientation elements based on the SfM. Additionally, in the case of an unmanned aerial vehicle (UAV), SfM exhibits a decrease in the accuracy of the orientation elements during complex camera movements. This paper proposes a patch-based MVS (PMVS) method considering the accuracy of the orientation elements. The proposed method involves applying the global SfM, estimating accuracy of exterior orientation (EO) elements, and introducing the accuracy of EO elements to PMVS. The PMVS approximates an object surface by using small rectangular patches, namely local tangent plane approximation. The patches are optimized by minimizing the sum of the photometric discrepancy scores. The accuracy of the EO elements is introduced to the patch optimization as weighting function. This accuracy is defined using the variances of the estimated parameters in the bundle adjustment. We also investigate the types of weighting functions. The results indicate that the proposed method is capable of considering geometric conditions during patch estimation. The proposed method was applied to the three types of image datasets, i.e., images captured using an SLR camera at ground level, images captured using a UAV equipped with a SLR camera, and images captured using an airplane equipped with an oblique camera. Through the experimental results, the improved accuracy and the effectiveness of the proposed method were confirmed.

Structure tensor analysis on epipolar plane images (EPIs) is a successful approach to estimate disparity from a light field, i.e. a dense set of multi-view images. However, the disparity range allowable for the light field is limited, because the estimation becomes less accurate as the range of disparities become larger. To overcome this limitation, we propose a new method called sheared EPI analysis, where EPIs are sheared before the structure tensor analysis. The results of analysis obtained with different shear values are integrated into a final disparity map. As verified by extensive evaluations on 12 datasets with large disparity ranges, our method is comparably accurate to and much faster than a multi-view stereo method.

We investigated the feasibility of using a 3D point cloud estimated by a neural radiance field (NeRF). We evaluated the quality of point clouds reconstructed with NeRF and multi-view stereo. The point cloud reconstructed by NeRF contains substantial noise and shows a wavy surface even on a flat surface. Therefore, a point cloud generated by NeRF is not appropriate as a highly accurate 3D model with the size and shape of an actual object. However, it is effective for applications that do not require highly accurate point clouds, such as the detection and 3D modelling of objects with few feature-point surfaces or optically transparent objects, which have been difficult to achieve using conventional methods.