结构化场景重建中一种融合直线信息的平面拟合方法

The paper describes a robust method to extract 3D lines from stereo point clouds. This method combines 2D image information with 3D point clouds from a stereo camera. 2D lines are first extracted from the image in the stereo pair, followed by 3D line regression from the back-projected 3D point set of the images points in the detected 2D lines. In this paper, random sample consensus (RANSAC) is used to estimate 3D line from the 3D point set, the Mahalanobis distance from each 3D point to the 3D line is derived, and the statistically motivated distance measure is used to compute the support for the detected 3D line. Experimental results on real environment with high level of clutter, occlusion, and noise demonstrate the robustness of the algorithm.

The automatic reconstruction of photogrammetric mesh models often suffers from issues such as erroneous adhesion of different planes, local distortions, and the loss of edge details. This study aims to improve mesh quality by incorporating three-dimensional (3D) line segments and plane regions as geometric constraints during mesh refinement. Using previously obtained 3D line segments, depth, and normal information, accurate 3D plane regions are extracted to complete missing sections of the dense point cloud, with depth-based outlier removal applied. Correspondences between mesh model vertices and the 3D line segments and plane regions are established, followed by mesh refinement through an energy function that integrates photo-consistency, line and plane constraints, and regularization. Experiments conducted on four open-source datasets demonstrate that the proposed method substantially improves 3D plane region extraction and point cloud completion. The refined mesh model achieves a 24.30% reduction in the average distance related to 3D lines and a 15.75% reduction for 3D planes compared with the original model, confirming the effectiveness of incorporating 3D line and plane constraints in enhancing reconstruction accuracy.

3D reconstruction of building facade with fused data of terrestrial LiDAR data and optical image

To quantify the architecture and select the ideal ideotype, it is vital to accurately measure the dimension of each part of the mantis shrimp. Point clouds have become increasingly popular in recent years as an efficient solution. However, the current manual measurement is labor intensive and costly and has high uncertainty. Automatic organ point cloud segmentation is a prerequisite and core step for phenotypic measurements of mantis shrimps. Nevertheless, little work focuses on mantis shrimp point cloud segmentation. To fill this gap, this paper develops a framework for automated organ segmentation of mantis shrimps from multiview stereo (MVS) point clouds. First, a Transformer-based MVS architecture is applied to generate dense point clouds from a set of calibrated phone images and estimated camera parameters. Next, an improved point cloud segmentation (named ShrimpSeg) that exploits both local and global features based on contextual information is proposed for organ segmentation of mantis shrimps. According to the evaluation results, the per-class intersection over union of organ-level segmentation is 82.4%. Comprehensive experiments demonstrate the effectiveness of ShrimpSeg, outperforming other commonly used segmentation methods. This work may be helpful for improving shrimp phenotyping and intelligent aquaculture at the level of production-ready.

This work concerns the assessment of mold parts with multi-view scanning point cloud data. With the soaring demand for complex geometry parts, multi-axis electrical discharge machining (EDM) manufacturing also faces the challenge of high-cost and the requirement of high-accuracy. As a part of EDM, well-judged life-time and wear condition of mold parts are critical to ensuring product quality. In this work, we present a novel non-contact surface inspection solution for mold part 3D reconstruction and inspection based on light-section scanning. After calibration and registration, multi-view stereo (MVS) point cloud data for the parts can be obtained, which can then be used to generate 2D section profiles. Comparison method is designed to examine if the design tolerance is met by comparing the section profiles against those based on the CAD model of the parts. Our method is verified by assessing worn electrode parts, which are commonly used in machining centers. The method proposed in this paper ensures the detection accuracy and improves the detection speed. It overcomes the problem of slow detection speed of mold part wear using 3D point cloud.

Detecting three-dimensional (3D) windows is vital for creating semantic building models with high level of detail, furnishing smart city and digital twin programs. Existing studies on window extraction using street imagery or laser scanning data often rely on limited types of features, resulting in compromised accuracy and completeness due to shadows and geometric decorations caused by curtains, balconies plants, and other objects. To enhance the effectiveness and robustness of building window extraction in 3D, this paper proposes an automatic method that leverages synergistic information from multi-view stereo (MVS) point clouds, through an adaptive divide-and combine pipeline. Colour information inherited from the imagery serves as a main clue to acquire the point clouds of individual building façades that may be coplanar and connected. The geometric information associated with normal vectors is then combined with colour, to adaptively divide individual building façade into an irregular grid that conforms to the window edges. Subsequently, HSV colour and depth distances within each grid cell are computed, and the grid cells are encoded to quantify the global arrangement features of windows. Finally, the multi-type features are fused in an integer programming model, by solving which the optimal combination of grid cells corresponding to windows is obtained. Benefitting from the informative MVS point clouds and the fusion of multi-type features, our method is able to directly produce 3D models with high regularity for buildings with different appearances. Experimental results demonstrate that the proposed method is effective in 3D window extraction while overcoming variations in façade appearances caused by foreign objects and missing data, with a high point-wise precision of 92.7%, recall of 77.09%, IoU of 71.95%, and F1-score of 83.42%. The results also exhibit a high level of integrity, with the accuracy of correctly extracted windows reaching 89.81%. In the future, we will focus on the development of a more universal façade dividing method to deal with even more complicated windows.

The fully automatic registration of 3D terrestrial laser scanning (TLS) point clouds, which is the first step in the usage of point clouds, is a highly challenging task in Light Detection and Ranging (LiDAR) remote sensing applications. Here, an automatic algorithm for pairwise coarse registration of TLS point clouds using 3D line features is proposed. First, the 3D line sets were extracted from the original pair of point clouds respectively, and two arbitrary lines from a point cloud were used to construct a 2-line base; Then, a pair of conjugate 2-line bases were identified from the source and target 3D line sets at a time, based on which a 3D rotation matrix together with its corresponding overlap between the pairwise 3D line sets was calculated; Third, a series of 3D rotation matrixes together with their overlaps were obtained using the traversing strategy to identify conjugate 2-line base pairs, and the 3D rotation matrix with the highest overlap was outputted as the final 3D rotation matrix, which was further used to compute the corresponding lines set from the pairwise 3D line sets; Next, based on the set of common perpendicular midpoints between the line correspondences, the 3D translation vector was computed; Finally, the 3D transformation matrix between pairwise point clouds was obtained by combining the 3D rotation matrix and the 3D translation vector. The proposed method was tested on three different TLS datasets, with experimental results demonstrating that the proposed algorithm could perform well on registering pairwise TLS point clouds. The rotation and translation errors of aligning the nine experimental pairwise point clouds were all less than 0.50° and 0.55m, respectively. This registration framework was also shown to be superior to state-of-the-art methods in terms of registration accuracy.

The traditional point-based scene reconstruction scheme always leads to inaccurate 3D reconstruction results due to its lack of scene texture and object occlusions. With the help of geometric characteristics and line features of 3D point cloud scenes, a novel extraction, matching, and registration method of 3D line features is presented by fusing RGB gradients and the spatial distribution of neighboring point clouds. Firstly, the supporting region of 3D linear segments with similar gradient directions can be determined by the gradients of an RGB scene image. Secondly, owing to the neighboring geometric features contained in the depth map, the point set of boundary lines and folded edge lines can also be extracted according to the spatial distribution of the current point and its neighboring points. The 3D line features can thus be fitted by these extracted point sets respectively. Thirdly, taking each extracted 3D line feature as a central axis, the supporting region can be constructed as the co-axial cylinder with the same thickness of each cylindrical layer. According to the 3D gradient statistics of point cloud within each cylindrical layer, we can calculate an intrinsic descriptor for each 3D line, which can be adopted to match 3D line features between scene frames. Finally, a novel line-to-line iterative closest line registration algorithm is introduced to compute the rotation and translation transformations, and the inter-frame point cloud scene data can be registered efficiently. The experimental results illustrate that the proposed line feature-based scheme has high registration efficiency, less computational cost, accurate registration, and strong robustness when compared with the existing line feature extraction and point-based registration methods.

Abstract. This study developed a simple but effective strategy to combine 3D volume and mesh models for representing complicated heritage buildings and structures. The idea is to seamlessly integrate 3D parametric or polyhedral models and mesh-based digital surfaces to generate a hybrid 3D model that can take advantages of both modeling methods. The proposed hybrid model generation framework is separated into three phases. Firstly, after acquiring or generating 3D point clouds of the target, these 3D points are partitioned into different groups. Secondly, a parametric or polyhedral model of each group is generated based on plane and surface fitting algorithms to represent the basic structure of that region. A “bare-bones” model of the target can subsequently be constructed by connecting all 3D volume element models. In the third phase, the constructed bare-bones model is used as a mask to remove points enclosed by the bare-bones model from the original point clouds. The remaining points are then connected to form 3D surface mesh patches. The boundary points of each surface patch are identified and these boundary points are projected onto the surfaces of the bare-bones model. Finally, new meshes are created to connect the projected points and original mesh boundaries to integrate the mesh surfaces with the 3D volume model. The proposed method was applied to an open-source point cloud data set and point clouds of a local historical structure. Preliminary results indicated that the reconstructed hybrid models using the proposed method can retain both fundamental 3D volume characteristics and accurate geometric appearance with fine details. The reconstructed hybrid models can also be used to represent targets in different levels of detail according to user and system requirements in different applications.

Robust detection of non-overlapping ellipses from points with applications to circular target extraction in images and cylinder detection in point clouds

In this study, we present a high temporal resolution (30-minute) multi-view stereo (MVS) photogrammetric dataset capturing the front of the lake-terminating Perito Moreno Glacier during daylight over more than one year. We aim to provide novel data to investigate glacier dynamics, which are strongly affected by climate change and pose increasing risks to ecosystems and human infrastructure. The images for the photogrammetric reconstruction are automatically inspected and prepared to create high-precision 4D (3D + time) point clouds, which are then compared per epoch to detect surface changes along an approx. 300 m wide study reach.The MVS system comprises eight DSLR cameras and uses 4G connectivity for daily data transfer, aiming for near-real-time monitoring. This has resulted in more than 75,000 images stored on a central server, which allows the calving analysis based on more than 5,000 models. To isolate image regions relevant for the generation of dense clouds with maximum precision across as many epochs as possible, the images are segmented using the artificial intelligence (AI)-based image segmentation model SAM2. The relevant regions are assessed using blur metrics, which identify low contrast caused by harsh glacier conditions, such as moisture or water droplets, thereby reducing the risk of including images of low quality that interfere with image correlation success.The results of our image pre-processing demonstrate that lighting conditions have the greatest impact on image segmentation performance. In contrast, the final model quality of the 4D point cloud reconstructions, which are based on a multi-epoch multi-imagery (MEMI) strategy, is mostly affected by the presence of adjacent dynamically changing regions, such as floating ice on the lake, highlighting the need for masking these regions. Applying masking further seems to improve the robustness of detecting subtle glacier surface changes, which is essential for pre-failure deformation analysis, providing valuable input for future calving prediction efforts.The point cloud sequences are analyzed using the Multiscale Model to Model Cloud Comparison (M3C2) algorithm to quantify surface changes. Although calibration parameters of focal length and principal point exhibit temporal variability, a constant calibration strategy is examined to ensure consistent alignment across all epochs, which furthermore enables the observation of glacier flow velocities in both horizontal and vertical directions. Initial results for a test period during a week in summer indicate that a constant calibration does not adversely affect model generation, suggesting that calibration stability may be sufficient under favorable summer conditions, while ongoing analyses will assess the robustness during winter months. Current efforts focus on refining dense cloud quality by separating glacier surfaces from points reconstructed on stable rock areas, which were intentionally retained during point cloud generation to provide stable reference regions for the MEMI workflow.Our introduced workflow enables the creation of a reliable calving inventory, exceeding the spatio-temporal resolution of conventional glacier monitoring techniques. In a next step, we aim to combine the created calving inventory with associated dynamic parameters, such as glacier velocity, and with environmental variables to support the development of AI-based calving prediction models.

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

Estimation of plant canopy using low-altitude imagery can help monitor the normal growth status of crops and is highly beneficial for various digital farming applications such as precision crop protection. However, extracting 3D canopy information from raw images requires studying the effect of sensor viewing angle by taking into accounts the limitations of the mobile platform routes inside the field. The main objective of this research was to estimate wheat (Triticum aestivum L.) leaf parameters, including leaf length and width, from the 3D model representation of the plants. For this purpose, experiments with different camera viewing angles were conducted to find the optimum setup of a mono-camera system that would result in the best 3D point clouds. The angle-control analytical study was conducted on a four-row wheat plot with a row spacing of 0.17 m and with two seeding densities and growth stages as factors. Nadir and six oblique view image datasets were acquired from the plot with 88% overlapping and were then reconstructed to point clouds using Structure from Motion (SfM) and Multi-View Stereo (MVS) methods. Point clouds were first categorized into three classes as wheat canopy, soil background, and experimental plot. The wheat canopy class was then used to extract leaf parameters, which were then compared with those values from manual measurements. The comparison between results showed that (i) multiple-view dataset provided the best estimation for leaf length and leaf width, (ii) among the single-view dataset, canopy, and leaf parameters were best modeled with angles vertically at −45° and horizontally at 0° (VA −45, HA 0), while (iii) in nadir view, fewer underlying 3D points were obtained with a missing leaf rate of 70%. It was concluded that oblique imagery is a promising approach to effectively estimate wheat canopy 3D representation with SfM-MVS using a single camera platform for crop monitoring. This study contributes to the improvement of the proximal sensing platform for crop health assessment.

This work presents a new method of automatic window detection in 3D LiDAR point clouds obtained from mobile terrestrial data acquisition systems in the urban environment. The proposed method first segments out 3D points belonging to the building facade from the 3D urban point cloud and then projects them onto a 2D plane parallel to the building facade. After point inversion within a watertight boundary, windows are segmented out based on geometrical information. The window features are then estimated exploiting both symmetrically corresponding windows in the facade as well as temporally corresponding windows in successive passages, based on ANOVA measurements. This unique fusion of information not only accommodates for lack of symmetry but also helps complete missing features due to occlusions. The estimated windows are then used to refine the 3D point cloud of the building facade. The results, evaluated on real data using different standard evaluation metrics, demonstrate the efficacy of the method.

While learning-based multi-view stereo (MVS) methods have recently shown successful performances in quality and efficiency, limited MVS data hampers generalization to unseen environments. A simple solution is to generate various large-scale MVS datasets, but generating dense ground truth for 3D structure requires a huge amount of time and resources. On the other hand, if the reliance on dense ground truth is relaxed, MVS systems will generalize more smoothly to new environments. To this end, we first introduce a novel semi-supervised multi-view stereo framework called a Sparse Ground truth-based MVS Network (SGT-MVSNet) that can reliably reconstruct the 3D structures even with a few ground truth 3D points. Our strategy is to divide the accurate and erroneous regions and individually conquer them based on our observation that a probability map can separate these regions. We propose a self-supervision loss called the 3D Point Consistency Loss to enhance the 3D reconstruction performance, which forces the 3D points back-projected from the corresponding pixels by the predicted depth values to meet at the same 3D co-ordinates. Finally, we propagate these improved depth pre-dictions toward edges and occlusions by the Coarse-to-fine Reliable Depth Propagation module. We generate the spare ground truth of the DTU dataset for evaluation and extensive experiments verify that our SGT-MVSNet outperforms the state-of-the-art MVS methods on the sparse ground truth setting. Moreover, our method shows comparable reconstruction results to the supervised MVS methods though we only used tens and hundreds of ground truth 3D points.