Dual attention-based deep learning network for multi-class object semantic segmentation of tunnel point clouds

At present, 3D vision technology has been more and more applied in the field of livestock breading industry. Three depth cameras are installed in the data acquisition walkway from different views, as the animals pass through the walkway, the depth cameras will get the local point cloud of livestock at the same time. By this method the livestock point cloud exists inevitable noise, as well as the background point cloud including railing and ground. Therefore, the segmentation of livestock point cloud from the original point cloud mixed with noise and background is the important first step of the application of three-dimensional vision in livestock breading industry. This paper proposed using the PointNet++ model to separate the target point cloud from the background point cloud. The improved DBSCAN clustering algorithm is proposed to denoise and segment the data preliminarily, which is helpful to label the data manually. The labeled data are used to train the PointNet++ neural network model to extract target point cloud efficiently and accurately. The experimental results show that the mean Intersection over Union (MIOU) of pig target point cloud segmentation based on single-scale grouping and multi-scale grouping Pointnet++ are 0.932 and 0.940, respectively, which both obtain good segmentation results.

In recent years, point cloud segmentation technology has increasingly played a pivotal role in tunnel construction and maintenance. Currently, traditional methods for segmenting point clouds in tunnel scenes often rely on a multitude of attribute information, including spatial distribution, color, normal vectors, intensity, and density. However, the underground tunnel scenes show greater complexity than road tunnel scenes, such as dim light, indistinct boundaries of tunnel walls, and disordered pipelines. Furthermore, issues pertaining to data quality, such as the lack of color information and insufficient annotated data, contribute to the subpar performance of conventional point cloud segmentation algorithms. To address this issue, a 3D point cloud segmentation framework specifically for underground tunnels is proposed based on the Segment Anything Model (SAM). This framework effectively leverages the generalization capability of the visual foundation model to automatically adapt to various scenes and perform efficient segmentation of tunnel point clouds. Specifically, the tunnel is first sliced along its direction on the tunnel line. Then, each sliced point cloud is projected onto a two-dimensional plane. Various projection methods and point cloud coloring techniques are employed to enhance SAM’s segmentation performance in images. Finally, the semantic segmentation of the entire underground tunnel is achieved by a small set of manually annotated semantic labels used as prompts in a progressive and recursive manner. The key feature of this method lies in its independence from model training, as it directly and efficiently addresses tunnel point cloud segmentation challenges by capitalizing on the generalization capability of foundation model. Comparative experiments against classical region growing algorithms and PointNet++ deep learning algorithms demonstrate the superior performance of our proposed algorithm.

The classification and part segmentation of point clouds have gained significant attention in the field of artificial intelligence (AI), especially in the construction industry. However, addressing the dataset directly in AI models remains a challenge, as most existing methods are not well-suited for processing point cloud data. PointNet has emerged as an AI architecture algorithm. It transforms individual points independently to learn local and global features. This research aims to develop a comprehensive framework for classification and part segmentation for point clouds of steel structure elements. The framework enhances the accuracy of the PointNet algorithm, and it consists of three stages: (1) dataset creation; (2) model classification; and (3) part segmentation. First, the dataset creation procedure encompasses modeling steel columns, beams, and braces using Autodesk Revit software. For the classification dataset, a dataset comprising 580 columns and 920 beams is obtained. In the part segmentation dataset, five categories of steel braced frame elements are generated, yielding a total of 21,870 elements for braced frame structures. Several point cloud experiments have been applied, including adjusting the number of points in the point cloud, altering the batch size, and fine-tuning the number of epochs. These experimental settings were systematically investigated to identify the optimal combination that yields the highest (AI) model accuracy. PointNet model achieved 100% accuracy across all classification experiments, while part segmentation experiments reached up to 97.10% accuracy, with a mean intersection over union (MIOU) of 93.70%. The comprehensive analysis of the point cloud dataset is applied on an actual case study to demonstrate the practical features of the proposed research.

Reconstructing as‐built building information modeling from point clouds in underground garages holds great potential for enhancing facility management. However, this task is particularly challenging due to the complex and diverse architectural and mechanical, electrical, and plumbing (MEP) components present in such environments. Moreover, there is no publicly available dataset and no dedicated semantic segmentation algorithm for underground garages environments. To address this gap, this paper first builds an expert‐labeled underground garage dataset, which includes 10 categories of components. Subsequently, this paper introduces GarageNet, a deep learning (DL) network designed for semantic segmentation of point clouds for scenes including both building and MEP components. GarageNet integrates a residual squeeze‐and‐excitation module to enhance layer and channel connection and a multi‐head attention feature aggregation module to capture local geometric structures. The results indicate that the GarageNet achieved a high overall accuracy of 95.09% and a mean intersection over union of 87.15%, surpassing other DL algorithms such as PointNext. Ablation studies and cross‐validation further demonstrate the model's reliability and generalization capability.

The expansion of tunnel scale has led to a massive demand for inspections. Light Detection And Ranging (LiDAR) has been widely applied in tunnel structure inspections due to its fast data acquisition speed and strong environmental adaptability. However, raw tunnel point-cloud data contain noise point clouds, such as non-structural facilities, which affect the detection of tunnel lining structures. Methods such as point-cloud filtering and machine learning have been applied to tunnel point-cloud denoising, but these methods usually require a lot of manual data preprocessing. In order to directly denoise the tunnel point cloud without preprocessing, this study proposes a comprehensive processing method for cross-section fitting and point-cloud denoising in subway shield tunnels based on the Huber loss function. The proposed method is compared with classical fitting denoising methods such as the least-squares method and random sample consensus (RANSAC). This study is experimentally verified with 40 m long shield-tunnel point-cloud data. Experimental results show that the method proposed in this study can more accurately fit the geometric parameters of the tunnel lining structure and denoise the point-cloud data, achieving a better denoising effect. Meanwhile, since coordinate system transformations are required during the point-cloud denoising process to handle the data, manual rotations of the coordinate system can introduce errors. This study simultaneously combines the Huber loss function with principal component analysis (PCA) and proposes a three-dimensional spatial coordinate system transformation method for tunnel point-cloud data based on the characteristics of data distribution.

Plant phenotypic analysis is of great importance to the development of agricultural engineering, and is one of the core issues in crop science and plant breeding. Since plant growth is spatio-temporal and synchronous, understanding the growth and development of individual plants can help to reveal the growth potential of the whole plot and thus improve planting methods. In recent years, the technical means to analyze the growth situation using 3D point cloud data has received extensive attention. The plant point cloud obtained by scanning plants with LiDAR has the characteristics of high resolution, high precision, etc. Periodic scanning of the same plant for spatio-temporal point cloud data sets allows monitoring of growth through subtle changes of plant organs. Organ tracking of growing plants remains challenging due to the lightward nature of growth, the potential for topological changes and the unpredictability of plant growth over time, with the possibility of new leaf growth and leaf death. This paper designs a plant organ growth tracking method based on point cloud. First of all, for growing plants, we have established a crop point cloud spatio-temporal dataset based on two publicly available point cloud datasets. The data set includes four species, tomato, tobacco, sorghum and maize, each species contains complete organ instance labels, and each organ of the same plant has a unique label, which means that the labels of the same organ of an individual at different scan dates correspond one-to-one. Second, this paper proposes a point cloud data-based plant organ growth tracking method, which uses a cost correlation matrix to automatically track growing plant organs. Finally, based on the set of quantitative evaluation metrics, our algorithm achieves a matching accuracy of 82.89% on the plant spatio-temporal dataset and good growth tracking results in the qualitative analysis.

At present, three-dimensional laser scanners are used to scan subway shield tunnels and generate point cloud data as the basis for extracting a variety of information about tunnel defects. However, there are obstacles in the tunnel such as pipelines, tracks, and signaling systems that cause noise in the point cloud. Usually, the data of the tunnel point cloud are huge, and the efficiency of artificial denoising is low. Faced with this problem, based on the respective characteristics of the geometric shape and reflection intensity of the tunnel point cloud and their correlation, this paper proposes a tunnel point cloud denoising method. The method includes the following three parts: reflection intensity threshold denoising, joint shape and reflection intensity denoising, and shape denoising. Through the experiment on the single-ring segment point cloud of a shield tunnel, the method proposed in this paper takes 2 min to remove 99.77% of the noise in the point cloud. Compared with manual denoising, the method proposed in this paper takes two fifteenths of the time to achieve the same denoising effect. The method proposed in this paper meets the requirements of a tunnel point cloud data survey. Thus, it provides support for the efficient, accurate, and automatic daily maintenance and surveys of tunnels.

Point cloud segmentation is essential for scene understanding, which provides advanced information for many applications, such as autonomous driving, robots, and virtual reality. To improve the accuracy and robustness of point cloud segmentation, many researchers have attempted to fuze camera images to complement the color and texture information. The common fusion strategy is the combination of convolutional operations with concatenation, element-wise addition or element-wise multiplication. However, conventional convolutional operators tend to confine the fusion of modal features within their receptive fields, which can be incomplete and limited. In addition, the inability of encoder–decoder segmentation networks to explicitly perceive segmentation boundary information results in semantic ambiguity and classification errors at object edges. These errors are further amplified in point cloud segmentation tasks, significantly affecting the accuracy of point cloud segmentation. To address the above issues, we propose a novel self-attention multi-modal fusion semantic segmentation network for point cloud semantic segmentation. Firstly, to effectively fuze different modal features, we propose a self-cross fusion module (SCF), which models long-range modality dependencies and transfers complementary image information to the point cloud to fully leverage the modality-specific advantages. Secondly, we design the salience refinement module (SR), which calculates the importance of channels in the feature maps and global descriptors to enhance the representation capability of salient modal features. Finally, we propose the local-aware anisotropy loss measure the element-level importance in the data and explicitly provide boundary information for the model, which alleviates the inherent semantic ambiguity problem in segmentation networks. Extensive experiments on two benchmark datasets demonstrate that our proposed method surpasses current state-of-the-art methods.

In interpreting a scene for numerous applications, including autonomous driving and robotic navigation, semantic segmentation is crucial. Compared to single-modal data, multi-modal data allow us to extract a richer set of features, which is the benefit of improving segmentation accuracy and effect. We propose a point cloud semantic segmentation method, and a fusion graph convolutional network (FGCN) which extracts the semantic information of each point involved in the two-modal data of images and point clouds. The two-channel k-nearest neighbors (KNN) module of the FGCN was created to address the issue of the feature extraction's poor efficiency by utilizing picture data. Notably, the FGCN utilizes the spatial attention mechanism to better distinguish more important features and fuses multi-scale features to enhance the generalization capability of the network and increase the accuracy of the semantic segmentation. In the experiment, a self-made semantic segmentation KITTI (SSKIT) dataset was made for the fusion effect. The mean intersection over union (MIoU) of the SSKIT can reach 88.06%. As well as the public datasets, the S3DIS showed that our method can enhance data features and outperform other methods: the MIoU of the S3DIS can reach up to 78.55%. The segmentation accuracy is significantly improved compared with the existing methods, which verifies the effectiveness of the improved algorithms.

Structural health monitoring of underground tunnels increasingly uses advanced sensing and data-driven methods. Laser-scanned 3D point clouds capture spatially rich measurements of segmental tunnel linings and require segmentation as a prerequisite for downstream analysis. Deep learning (DL) is effective for point-cloud segmentation, but scarce datasets and costly annotation limit practical use. This paper presents Tunnel Scanner , a high-fidelity simulator that synthesises realistic tunnel point clouds with automatic annotation. The plug-and-play module Hybrid Position–Normal Local Spatial Encoding embeds geometric priors into DL backbones and combines with transfer learning (TL) to exploit synthetic data for domain adaptation. Models trained only on synthetic data achieved 71.9% mean Intersection-over-Union (mIoU) and 86.6% Overall Accuracy (OA), and TL increased performance to at least 78.8% mIoU and 90.9% OA with limited real data. This paper highlights geometry-informed data synthesis as a viable augmentation approach for digital inspection and asset management of large-scale tunnels. • Develop a high-fidelity simulator to address the scarcity of tunnel point clouds. • Propose a plug-and-play module encoding geometric features into DL backbones. • Investigate transfer learning to enhance 3D Sim-to-Real domain adaptation. • The end-to-end framework yields ＋24% segmentation accuracy with limited real data.

This paper presents a novel approach to recognize and estimate pose of the 3D objects in cluttered range images. The key technical breakthrough of the developed approach can enable robust object recognition and localization under undesirable condition such as environmental illumination variation as well as optical occlusion to viewing the object partially. First, the acquired point clouds are segmented into individual object point clouds based on the developed 3D object segmentation for randomly stacked objects. Second, an efficient shape-matching algorithm called Sub-OBB based object recognition by using the proposed oriented bounding box (OBB) regional area-based descriptor is performed to reliably recognize the object. Then, the 3D position and orientation of the object can be roughly estimated by aligning the OBB of segmented object point cloud with OBB of matched point cloud in a database generated from CAD model and 3D virtual camera. To detect accurate pose of the object, the iterative closest point (ICP) algorithm is used to match the object model with the segmented point clouds. From the feasibility test of several scenarios, the developed approach is verified to be feasible for object pose recognition and localization.

<p>The size distribution of sediments together with their shape inform on their transport history, are important factors controlling the efficiency of erosion and transport, and control the quality of aquatic ecosystems. However, the size distribution of sediments is generally assessed using poorly representative field measurements and determining the grain-scale shape of sediments remains a real challenge in geomorphology. To tackle this issue, we develop a new methodological approach based on the segmentation and geomorphological fitting of 3D point clouds. Point cloud segmentation into individual grains is performed using a watershed algorithm applied here to 3D point clouds. Once the grains are individualized into several sub-clouds, the morphology of each grain is determined by fitting a 3D ellipsoid to each sub-cloud. These 3D models are then used to extract the size distribution and the grain-scale shape of the sediment population. The algorithm is validated against field data acquired by Wolman counts in coastal and fluvial environments. The main benefits of this automatic and non-destructive method are that it provides, with a fast and efficient approach, access to 1) an un-biased estimate of surface grain-size distribution on a large range of scales, from centimeters to tens of meters; 2) a very large number of data, only limited by the number of grains in the point-cloud dataset; 3) the 3D morphology of grains, in turn allowing to develop new metrics characterizing the size and shape of grains; and 4) the in-situ orientation and organization of grains and grain clusters. The main limit of this method is that it is only able to detect grains with a characteristic size significantly greater than the resolution of the point cloud.</p>

With the development of airborne light detection and ranging (LiDAR) technology, it has become a common and efficient way to collect large-scale 3D spatial information. However, efficient and automatic semantic segmentation of LiDAR data, in the form of 3D point clouds, remains a persistent challenge. To address this, a dual attention neural network (DA-Net) is proposed, consisting of two different blocks, namely augmented edge representation (AER) and elevation attentive pooling (EAP). First, the AER can adaptively represent local orientation and position, thereby effectively enhancing geometric information. Second, the captured local features of centroid points are utilized to further encode discriminative features using the EAP with the learned attention scores. Finally, a location homogeneity (LH) module is devised to explore the long-range relationship in an encoder-decoder network. Benefiting from the dual attention module, geometric information hidden in unorganized point clouds can be effectively propagated. Besides, the LH forces the network to pay attention to the semantic consistency of elevated objects, which facilitates both point- and object-level point cloud semantic segmentation for scene understanding. A benchmark dataset is used to assess the proposed method, which achieves an overall accuracy of 85.98% and an average F1 score of 72.31%. In addition, comparisons with other latest deep learning methods on the 2019 Data Fusion Contest dataset further demonstrate the robustness and generalization ability of the proposed method.

Learning discriminative shape representation directly on point clouds is still challenging in 3D shape analysis and understanding. Recent studies usually involve three steps: first splitting a point cloud into some local regions, then extracting the corresponding feature of each local region, and finally aggregating all individual local region features into a global feature as shape representation using simple max-pooling. However, such pooling-based feature aggregation methods do not adequately take the spatial relationships (e.g. the relative locations to other regions) between local regions into account, which greatly limits the ability to learn discriminative shape representation. To address this issue, we propose a novel deep learning network, named Point2SpatialCapsule, for aggregating features and spatial relationships of local regions on point clouds, which aims to learn more discriminative shape representation. Compared with the traditional max-pooling based feature aggregation networks, Point2SpatialCapsule can explicitly learn not only geometric features of local regions but also the spatial relationships among them. Point2SpatialCapsule consists of two main modules. To resolve the disorder problem of local regions, the first module, named geometric feature aggregation, is designed to aggregate the local region features into the learnable cluster centers, which explicitly encodes the spatial locations from the original 3D space. The second module, named spatial relationship aggregation, is proposed for further aggregating the clustered features and the spatial relationships among them in the feature space using the spatial-aware capsules developed in this paper. Compared to the previous capsule network based methods, the feature routing on the spatial-aware capsules can learn more discriminative spatial relationships among local regions for point clouds, which establishes a direct mapping between log priors and the spatial locations through feature clusters. Experimental results demonstrate that Point2SpatialCapsule outperforms the state-of-the-art methods in the 3D shape classification, retrieval and segmentation tasks under the well-known ModelNet and ShapeNet datasets.

To investigate the feasibility of tracking targets in 2D fluor images using a novel deep learning network. Our model design aims to capture the consistent motion of tumors in fluoroscopic images by neural network. Specifically, the model is trained by generative adversarial methods. The network is a coarse-to-fine architecture design. Convolutional LSTM (Long Short-term Memory) modules are introduced to account for the time correlation between different frames of the fluoroscopic images. The model was trained and tested on a digital X-CAT phantom in two studies. Series of coherent 2D fluoroscopic images representing the full respiration cycle were fed into the model to predict the localized tumor regions. In first study to test on massive scenarios, phantoms of different scales, tumor positions, sizes, and respiration amplitudes were generated to evaluate the accuracy of the model comprehensively. In second study to test on specific sample, phantoms were generated with fixed body and tumor sizes but different respiration amplitudes to investigate the effects of motion amplitude on the tracking accuracy. The tracking accuracy was quantitatively evaluated using intersection over union (IOU), tumor area difference, and centroid of mass difference (COMD). In the first comprehensive study, the mean IOU and dice coefficient achieved 0.93±0.04 and 0.96±0.02. The mean tumor area difference was 4.34%±4.04%. And the COMD was 0.16 cm and 0.07 cm on average in SI (superior-interior) and LR (left-right) directions, respectively. In the second amplitude study, the mean IOU and dice coefficient achieved 0.98 and 0.99. The mean tumor difference was 0.17%. And the COMD was 0.03cm and 0.01 cm on average in SI and LR directions, respectively. Results demonstrated the robustness of our model against breathing variations. Our study showed the feasibility of using deep learning to track targets in x-ray fluoroscopic projection images without the aid of markers. The technique can be valuable for both pre- and during-treatment real-time target verification using fluoroscopic imaging in lung SBRT treatments.