Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement

In the agricultural field, concrete headworks is the most important structure for the irrigation system. In recent years, several agricultural concrete infrastructures aging for a long-term period has been increasing. For maintenance and management, conventional inspection methods are time-consuming and costly, such as the electromagnetic wave method and elastic wave method. The detection of surface damage is more effective, safe and reliable than before since the laser scanning method provides detailed geometric information about the structure. The fundamental studies on point cloud data have been conducted in the civil engineering fields, nevertheless, the characteristics of point cloud in agricultural infrastructures, such as dam, headworks and canal, have not been discussed. In this study, 3D point clouds are generated for a concrete irrigation structure using the laser scanning method. We focused on the characteristics of surface damage, which are quantitatively evaluated using point cloud information, geometric information and intensity parameter. The types of detected damage are efflorescence and cracks. Which aligned point clouds or point clouds from a single scan is effective for high-precision detection is confirmed. The characteristics of surface damage are evaluated by geometric features. The distance between the fitted plane and points is calculated by RANSAC algorithm and roughness parameter. The amount of efflorescence is detected by the distance between the fitted plane from RANSAC algorithm and points. The crack is detected by the local plane fitting method. The types of damage are characterized by the intensity parameter which is related to the color, roughness and moisture of the object. The surface damage and condition are evaluated by both geometric information and intensity parameter. These results show the unique parameters of point clouds from laser scanning methods, such as geometric features and intensity parameter, are useful to evaluate the characteristics of surface damage.

Point cloud completion recovers the complete point clouds from partial ones, providing numerous point cloud information for downstream tasks such as 3-D reconstruction and target detection. However, previous methods usually suffer from unstructured prediction of points in local regions and the discrete nature of the point cloud. To resolve these problems, we propose a point cloud completion network called TPDC. Representing the point cloud as a set of unordered features of points with local geometric information, we devise a Triangular Pyramid Extractor (TPE), using the simplest 3-D structure-a triangular pyramid-to convert the point cloud to a sequence of local geometric information. Our insight of revealing local geometric information in a complex environment is to design a Divide-and-Conquer Splitting Module in a Divide-and-Conquer Splitting Decoder (DCSD) to learn point-splitting patterns that can fit local regions the best. This module employs the Divide-and-Conquer approach to parallelly handle tasks related to fitting ground-truth values to base points and predicting the displacement of split points. This approach aims to make the base points align more closely with the ground-truth values while also forecasting the displacement of split points relative to the base points. Furthermore, we propose a more realistic and challenging benchmark, ShapeNetMask, with more random point cloud input, more complex random item occlusion, and more realistic random environmental perturbations. The results show that our method outperforms both widely used benchmarks as well as the new benchmark.

Theoretically, the appearance of shadows in aerial imagery is not desirable for researchers because it leads to errors in object classification and bias in the calculation of indices. In contrast, shadows contain useful geometrical information about the objects blocking the light. Several studies have focused on estimation of building heights in urban areas using the length of shadows. This type of information can be used to predict the population of a region, water demand, etc., in urban areas. With the emergence of unmanned aerial vehicles (UAVs) and the availability of high- to super-high-resolution imagery, the important questions relating to shadows have received more attention. Three-dimensional imagery generated using UAV-based photogrammetric techniques can be very useful, particularly in agricultural applications such as in the development of an empirical equation between biomass or yield and the geometrical information of canopies or crops. However, evaluating the accuracy of the canopy or crop height requires labor-intensive efforts. In contrast, the geometrical relationship between the length of the shadows and the crop or canopy height can be inversely solved using the shadow length measured. In this study, object heights retrieved from UAV point clouds are validated using the geometrical shadow information retrieved from three sets of high-resolution imagery captured by Utah State University's AggieAir UAV system. These flights were conducted in 2014 and 2015 over a commercial vineyard located in California for the USDA Agricultural Research Service Grape Remote sensing Atmospheric Profile and Evapotranspiration Experiment (GRAPEX) Program. The results showed that, although this approach could be computationally expensive, it is faster than fieldwork and does not require an expensive and accurate instrument such as a real-time kinematic (RTK) GPS.

In the current methods of point cloud processing, there are still several limitations, particularly in achieving high precision and accuracy for large objects in complex environments. Existing techniques often struggle with incomplete or noisy data, leading to inaccurate contour extraction. In view of the challenges associated with the sparse and discrete nature of point clouds in complex environments, which lead to poor accuracy and stability in object contour extraction, this paper proposes a novel method for accurately extracting the contours of three-dimensional target point clouds. The method integrates high-resolution images with sparse point cloud information to address these issues. Firstly, the local characteristics of the point cloud are calculated, allowing for the selection of a contour point cloud. Next, depth information from two-dimensional images is obtained through a fuzzy mapping relationship. Finally, constraint conditions are established to derive a more accurate predicted value of the contour point cloud. Experiments demonstrate that the proposed method effectively improves the precision and accuracy of contour extraction for large objects, reducing measurement deviation by approximately 64.9% compared to using the original point cloud alone. Additionally, the method shows a more accurate completion effect on parts of the contour that are missing, underscoring its robustness and effectiveness in challenging scenarios.

Road geometric information and a digital model based on light detection and ranging (LiDAR) can perform accurate geometric inventories and three-dimensional (3D) descriptions for as-built roads and infrastructures. However, unorganized point clouds and complex road scenarios would reduce the accuracy of geometric information extraction and digital modeling. There is a standardization need for information extraction and 3D model construction that integrates point cloud processing and digital modeling. This paper develops a framework from semantic segmentation to geometric information extraction and digital modeling based on LiDAR data. A semantic segmentation network is improved for the purpose of dividing the road surface and infrastructure. The road boundary and centerline are extracted by the alpha-shape and Voronoi diagram methods based on the semantic segmentation results. The road geometric information is obtained by a coordinate transformation matrix and the least square method. Subsequently, adaptive road components are constructed using Revit software. Thereafter, the road route, road entity model, and various infrastructure components are generated by the extracted geometric information through Dynamo and Revit software. Finally, a detailed digital model of the road scenario is developed. The Toronto-3D and Semantic3D datasets are utilized for analysis through training and testing. The overall accuracy (OA) of the proposed net for the two datasets is 95.3 and 95.0%, whereas the IoU of segmented road surfaces is 95.7 and 97.9%. This indicates that the proposed net could accomplish superior performance for semantic segmentation of point clouds. The mean absolute errors between the extracted and manually measured geometric information are marginal. This demonstrates the effectiveness and accuracy of the proposed extraction methods. Thus, the proposed framework could provide a reference for accurate extraction and modeling from LiDAR data.

Digital modeling is an essential means for preserving and passing down historical culture within cultural heritage. Point cloud registration technology, by aligning point cloud data captured from multiple perspectives, enhances the accuracy of reconstructing the complex structures of artifacts and buildings and provides a reliable digital foundation for their protection, exhibition, and research. Due to the challenges posed by complex morphology, noise, and missing data when processing cultural heritage data, this paper proposes a point cloud registration method based on the Diffusion Transformer (PointDT). Compared to traditional methods, the Diffusion Transformer can better capture both the global features and local structures of point cloud data, more accurately capturing the geometric and semantic information of the target point cloud, thereby achieving precise digital reconstruction. In this study, we trained our method using indoor datasets such as 3DMatch and large-scale outdoor datasets like KITTI, and validated it on various cultural heritage datasets, including those of the Terracotta Warriors and heritage buildings. The results demonstrate that this method not only significantly improves accuracy but also shows advantages in computational efficiency.

In open‐world robotic manipulation tasks, language‐guided model‐free grasping has garnered increasing attention. However, existing approaches often overlook the geometric structure of target objects, which limits the effectiveness of subsequent tasks such as manipulation and placement. To address this limitation, a novel method called Language‐Guided Grasping via Primitive Fitting is proposed. This approach integrates language instructions with multimodal perception to enhance the semantic interpretability and downstream usability of the grasp through structured geometric modeling. Specifically, the user‐specified object using 2D images and depth data via multimodal understanding is first localized. Then, primitive fitting on the object's point cloud using basic geometric shapes (e.g., cuboids, ellipsoids, truncated cones) to extract approximate size and structural features is performed. Based on the geometric information, a grasp pose generation strategy guided by semantic geometry is defined, and modules for grasp feasibility filtering and task‐oriented optimization to select the optimal grasp pose are introduced. This method is validated in real‐world complex environments and achieved grasp success rates of 95% in structured and 90% in cluttered scenes. Geometric fitting enhances post‐grasp predictability and semantic consistency, enabling better generalization and planning.

Point cloud data can accurately and intuitively reflect the spatial relationship between the coal wall and underground fully mechanized mining equipment. However, the indirect method of point cloud feature extraction based on deep neural networks will lose some of the spatial information of the point cloud, while the direct method will lose some of the local information of the point cloud. Therefore, we propose the use of dynamic graph convolution neural network (DGCNN) to extract the geometric features of the sphere in the point cloud of the fully mechanized mining face (FMMF) in order to obtain the position of the sphere (marker) in the point cloud of the FMMF, thus providing a direct basis for the subsequent transformation of the FMMF coordinates to the national geodetic coordinates with the sphere as the intermediate medium. Firstly, we completed the production of a diversity sphere point cloud (training set) and an FMMF point cloud (test set). Secondly, we further improved the DGCNN to enhance the effect of extracting the geometric features of the sphere in the FMMF. Finally, we compared the effect of the improved DGCNN with that of PointNet and PointNet++. The results show the correctness and feasibility of using DGCNN to extract the geometric features of point clouds in the FMMF and provide a new method for the feature extraction of point clouds in the FMMF. At the same time, the results provide a direct early guarantee for analyzing the point cloud data of the FMMF under the national geodetic coordinate system in the future. This can provide an effective basis for the straightening and inclining adjustment of scraper conveyors, and it is of great significance for the transparent, unmanned, and intelligent mining of the FMMF.

Nowadays, 3D city models are used by the increasing number of applications. Most applications require not only geometric information but also semantic information. As a standard and tool for 3D city model, CityGML, provides a method for storing and managing both geometric and semantic information. Moreover, it also provides the multi-scale representation of 3D building model for efficient visualization. In CityGML, building models are represented in five LODs (Level of Detail), start from LOD0, LOD1, LOD2, LOD3, and LOD4. Each level has different accuracy and detail requirement for visualization. Usually, for obtaining multi-LOD of 3D building model, several data sources are integrated. For example, LiDAR data is used for generating LOD0, LOD1, and LOD2 as close-range photogrammetry data is used for generating more detailed model in LOD3 and LOD4. However, using additional data sources is increasing cost and time consuming. Since the development of TLS (Terrestrial Laser Scanner), data collection for detailed model can be conducted in a relative short time compared to photogrammetry. Point cloud generated from TLS can be used for generating the multi-LOD of building model. This paper gives an overview about the representation of 3D building model in CityGML and also method for generating multi-LOD of building from TLS data. An experiment was conducted using TLS. Following the standard in CityGML, point clouds from TLS were processed resulting 3D model of building in different level of details. Afterward, models in different LOD were converted into XML schema to be used in CityGML. From the experiment, final result shows that TLS can be used for generating 3D models of building in LOD1, LOD2, and LOD3.

We propose to overcome a significant limitation of the iterative closest point (ICP) algorithm used by KinectFusion, namely, its sole reliance upon geometric information. Our approach uses both geometric and color information in a direct manner that uses all the data in order to accurately estimate camera pose. Data association is performed by Lucas–Kanade to compute an affine warp between the color images associated with two RGBD point clouds. A subsequent step then estimates the Euclidean transformation between the point clouds using either a point-to-point or point-to-plane error metric, with a novel method based on a normal covariance test for automatically selecting between them. Together, Lucas–Kanade data association with covariance testing enables robust camera tracking through areas of low geometric features, without sacrificing accuracy in environments in which the existing ICP technique succeeds. Experimental results on several publicly available datasets demonstrate the improved performance both qualitatively and quantitatively.

In industrial plants, piping and equipment are intricately interconnected, and there are many components with a variety of shapes. To use point clouds of industrial plants for simulation of maintenance work, it is necessary to extract the components. Deep learning is effective for recognizing components in point clouds of industrial plants. However, training classifiers is challenging due to the difficulty in acquiring diverse point cloud datasets and the labor-intensive process of annotating large-scale point clouds. A promising approach to address these issues is to train the classifier on virtual point clouds generated from CAD models. However, classifiers trained on these virtual point clouds often fail to achieve sufficient segmentation accuracy due to discrepancies between virtual point clouds and actual point clouds captured by terrestrial laser scanners. This paper proposes methods to improve segmentation accuracy by reducing these discrepancies. First, we introduce a method to incorporate features such as missing points, noise, and outliers observed in actual point clouds. Furthermore, we propose a data augmentation approach that applies up-sampling using a deep learning model trained on paired virtual and real point clouds to reduce the discrepancy between them. Our evaluation demonstrates that the proposed methods effectively improve the segmentation accuracy of point clouds of industrial plants.

As 3D acquisition technology develops and 3D sensors become increasingly affordable, large quantities of 3D point cloud data are emerging. How to effectively learn and extract the geometric features from these point clouds has become an urgent problem to be solved. The point cloud geometric information is hidden in disordered, unstructured points, making point cloud analysis a very challenging problem. To address this problem, we propose a novel network framework, called Tree Graph Network (TGNet), which can sample, group, and aggregate local geometric features. Specifically, we construct a Tree Graph by explicit rules, which consists of curves extending in all directions in point cloud feature space, and then aggregate the features of the graph through a cross-attention mechanism. In this way, we incorporate more point cloud geometric structure information into the representation of local geometric features, which makes our network perform better. Our model performs well on several basic point clouds processing tasks such as classification, segmentation, and normal estimation, demonstrating the effectiveness and superiority of our network. Furthermore, we provide ablation experiments and visualizations to better understand our network.

Airborne LIDAR technology which has an increasing importance in recent years, has entered into the field of application of many disciplines by obtaining fast and highly accurate 3D data. It provides precise topography information with dense point cloud data as well as all details on the surface. Thus, it has become useful in all disciplines associated with space such as cartography, construction, city planning, forest, energy, hydrology, geology, transportation, telecommunications, security, disaster, aviation, and infrastructure. By mounting LIDAR measurement units on aircraft large areas can be measured relatively quickly and cost-effectively. In this study, Riegl Q680i scanner and CCNS5 flight management system were mounted to the aircraft. The digital elevation models; DEM (Digital Elevation Model) and DSM (Digital Surface Model) of the Melen basin, which is located within the boundaries of Düzce and Sakarya was generated using LIDAR point cloud data (.las format) with a point density of 16 points/m2 and also 1/1000 base maps of the basin were produced. In addition, many details such as road, slope, culvert, electricity poles were drawn in accordance with the principles of large-scale map construction regulations and transferred to GIS environment. The Melen basin with an important water storage area, boundaries, basin model, water collection lines, determination of flow directions and connections, the topographic surface of the basin sub-areas, morphology were created using 3D laser point cloud data. So, the digital terrain model of the basin in GIS environment is visualized with linear maps. LIDAR data provides 3D geometric and morphological information that cannot be obtained according to classical methods in this kind of engineering studies. Results suggest that the higher spatial resolution LIDAR-derived data are preferable and can introduce more detailed information about basin hydro geomorphic behaviours.

LSGRNet: Local Spatial Latent Geometric Relation Learning Network for 3D point cloud semantic segmentation

Human mesh recovery (HMR) holds significant utility in many applications. Studying HMR involving various types of sensors is necessary, as it enables the acquisition of human meshes in diverse scenes. Unlike HMR based on RGB images, HMR based on LiDAR has received considerably less attention in previous works. The major challenge in estimating human poses and meshes from sparse point clouds lies in the sparsity, noise, and incompletion of LiDAR point clouds. To address these challenges, we propose a LiDAR-based 3D human mesh recovery algorithm, called LiDAR-HMR. This algorithm involves estimating a sparse representation of a human (3D human pose) and gradually reconstructing the body mesh. To better leverage the 3D structural information of point clouds, we propose a point-cloud-to-SMPL pipeline that uses the original point cloud features to guide the reconstruction. The experimental results on four publicly available datasets demonstrate the effectiveness of LiDAR-HMR. The codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/soullessrobot/LiDAR-HMR</uri>.