A point cloud dataset and deep learning method for semantic segmentation of underground garages

Point cloud has emerged as the most popular three-dimensional (3D) data format in recent years for several scientific and industrial applications. Point cloud semantic segmentation has piqued the researcher’s interest, which is a crucial stage in 3D analysis and scene comprehension. Deep learning-based processing is more feasible to increase the availability of point cloud acquisition tools that is LiDAR systems at the user end. The point cloud learning achieves tremendous success in object detection, object categorization, and semantic segmentation. To summarize the recent works with chronological development, comprehensive review of projection-, voxel-, and direct point-based point cloud semantic segmentation methods is performed from various perspectives. The commonly used point cloud benchmark datasets with their characteristics are discussed, and they are used for the performance analysis and comparison of several state-of-the-art segmentation methods. The quantitative performance analysis of these deep learning models summarizes the trend of semantic segmentation of point clouds. In the context of point cloud semantic segmentation, the various methods have specific roles. Based on the review of methods working and their performance analysis, it is concluded that the projection-based methods prioritize efficiency, which is ideal in unavailability of high-performance computing system. Voxel-based methods capture overall context, serving well in 3D object classification. Point-based approaches excel in fine details and efficiency, suited for tasks like 3D semantic segmentation. Choosing the suitable method depends on the task, data, and resources. KPConv and DGCNN are popular choices, especially for precision and adaptability to point density. However, method performance varies, underlining the need for tailored selection. Hybrid approaches, combining method strengths, promise superior results.

Point cloud data provide three-dimensional (3D) information about objects in the real world, containing rich semantic features. Therefore, the task of semantic segmentation of point clouds has been widely applied in fields such as robotics and autonomous driving. Although existing research has made unprecedented progress, achieving real-time semantic segmentation of point clouds on airborne devices still faces challenges due to excessive computational and memory requirements. To address this issue, we propose a novel sequence convolution semantic segmentation architecture that integrates Convolutional Neural Networks (CNN) with a sequence-to-sequence (seq2seq) structure, termed SeqConv-Net. This architecture views point cloud semantic segmentation as a sequence generation task. Based on our unique perspective of spatially ordered sequences, we use Recurrent Neural Networks (RNN) to encode elevation information, then input the structured hidden states into a CNN for planar feature extraction. The results are combined with the RNN’s encoded outputs via residual connections and are fed into a decoder for sequence prediction in a seq2seq manner. Experiments show that the SeqConv-Net architecture achieves 75.5% mean Intersection Over Union (mIOU) accuracy on the DALES dataset, with the total processing speed from data preprocessing to prediction being several to tens of times faster than existing methods. Additionally, SeqConv-Net can balance accuracy and speed by adjusting the hyperparameters and using different RNNs and CNNs, providing a new solution for real-time point cloud semantic segmentation in airborne environments.

Semantic segmentation in the context of 3D point clouds for the railway environment holds a significant economic value, but its development is severely hindered by the lack of suitable and specific datasets. Additionally, the models trained on existing urban road point cloud datasets demonstrate poor generalisation on railway data due to a large domain gap caused by non‐overlapping special/rare categories, for example, rail track, track bed etc. To harness the potential of supervised learning methods in the domain of 3D railway semantic segmentation, we introduce RailPC, a new point cloud benchmark. RailPC provides a large‐scale dataset with rich annotations for semantic segmentation in the railway environment. Notably, RailPC contains twice the number of annotated points compared to the largest available mobile laser scanning (MLS) point cloud dataset and is the first railway‐specific 3D dataset for semantic segmentation. It covers a total of nearly 25 km railway in two different scenes (urban and mountain), with 3 billion points that are finely labelled as 16 most typical classes with respect to railway, and the data acquisition process is completed in China by MLS systems. Through extensive experimentation, we evaluate the performance of advanced scene understanding methods on the annotated dataset and present a synthetic analysis of semantic segmentation results. Based on our findings, we establish some critical challenges towards railway‐scale point cloud semantic segmentation. The dataset is available at https://github.com/NNU‐GISA/GISA‐RailPC, and we will continuously update it based on community feedback.

Three-dimensional (3D) semantic segmentation of point clouds is important in many scenarios, such as automatic driving, robotic navigation, while edge computing is indispensable in the devices. Deep learning methods based on point sampling prove to be computation and memory efficient to tackle large-scale point clouds (e.g. millions of points). However, some local features may be abandoned while sampling. In this paper, We present one end-to-end 3D semantic segmentation framework based on dilated nearest neighbor encoding. Instead of down-sampling point cloud directly, we propose a dilated nearest neighbor encoding module to broaden the network’s receptive field to learn more 3D geometric information. Without increase of network parameters, our method is computation and memory efficient for large-scale point clouds. We have evaluated the dilated nearest neighbor encoding in two different networks. The first is the random sampling with local feature aggregation. The second is the Point Transformer. We have evaluated the quality of the semantic segmentation on the benchmark 3D dataset S3DIS, and demonstrate that the proposed dilated nearest neighbor encoding exhibited stable advantages over baseline and competing methods.

Semantic segmentation of large-scale point clouds by integrating attention mechanisms and transformer models

The semantic segmentation of point clouds is a crucial undertaking in 3D reconstruction and holds great importance. However, achieving precise semantic segmentation represents a significant hurdle. In this paper, we present BEMF-Net, an efficient method for large-scale environments. It starts with an effective feature extraction method. Unlike images, 3D data comprise not only geometric relations but also texture information. To accurately depict the scene, it is crucial to take into account the impacts of texture and geometry on the task, and incorporate modifications to improve feature description. Additionally, we present a multi-scale feature fusion technique that effectively promotes the interaction between features at different resolutions. The approach mitigates the problem of the smoothing of detailed information caused by downsampling mechanisms, while ensuring the integrity of features across different layers, allowing a more comprehensive representation of the point cloud. We confirmed the effectiveness of this method by testing it on benchmark datasets such as S3DIS, SensatUrban, and Toronto3D.

In interpreting a scene for numerous applications, including autonomous driving and robotic navigation, semantic segmentation is crucial. Compared to single-modal data, multi-modal data allow us to extract a richer set of features, which is the benefit of improving segmentation accuracy and effect. We propose a point cloud semantic segmentation method, and a fusion graph convolutional network (FGCN) which extracts the semantic information of each point involved in the two-modal data of images and point clouds. The two-channel k-nearest neighbors (KNN) module of the FGCN was created to address the issue of the feature extraction's poor efficiency by utilizing picture data. Notably, the FGCN utilizes the spatial attention mechanism to better distinguish more important features and fuses multi-scale features to enhance the generalization capability of the network and increase the accuracy of the semantic segmentation. In the experiment, a self-made semantic segmentation KITTI (SSKIT) dataset was made for the fusion effect. The mean intersection over union (MIoU) of the SSKIT can reach 88.06%. As well as the public datasets, the S3DIS showed that our method can enhance data features and outperform other methods: the MIoU of the S3DIS can reach up to 78.55%. The segmentation accuracy is significantly improved compared with the existing methods, which verifies the effectiveness of the improved algorithms.

Point cloud semantic segmentation in urban scenes plays a vital role in intelligent city modeling, autonomous driving, and urban planning. Point cloud semantic segmentation based on deep learning methods has achieved significant improvement. However, it is also challenging for accurate semantic segmentation in large scenes due to complex elements, variety of scene classes, occlusions, and noise. Besides, most methods need to split the original point cloud into multiple blocks before processing and cannot directly deal with the point clouds on a large scale. We propose a novel context-aware network (CAN) that can directly deal with large-scale point clouds. In the proposed network, a Local Feature Aggregation Module (LFAM) is designed to preserve rich geometric details in the raw point cloud and reduce the information loss during feature extraction. Then, in combination with a Global Context Aggregation Module (GCAM), capture long-range dependencies to enhance the network feature representation and suppress the noise. Finally, a Context-Aware Upsampling Module (CAUM) is embedded into the proposed network to capture the global perception from a broad perspective. The ensemble of low-level and high-level features facilitates the effectiveness and efficiency of 3D point cloud feature refinement. Comprehensive experiments were carried out on three large-scale point cloud datasets in both outdoor and indoor environments to evaluate the performance of the proposed network. The results show that the proposed method outperformed the state-of-the-art representative semantic segmentation networks, and the overall accuracy (OA) of Tongji-3D, Semantic3D, and S3DIS is 96.01%, 95.0%, and 88.55%, respectively.

Performing semantic segmentation on point clouds is the primary method by which machines perceive 3D scenes in a fine-grained manner. Deep learning algorithms usually require many pointwise annotations obtained with specialized tools, which is a laborious and inefficient process. To this end, we develop two frameworks for training point cloud semantic segmentation networks, one that utilizes fewer projected image annotations and another that employs sparse scribble image annotations, making the process more flexible and user friendly. However, back-projecting 2D-pixel labels to 3D points during loss calculations always introduces errors. To increase the back-projection accuracy of our approach, we first identify and record potential pixel-point correspondence errors and then develop strategies for constructing an accurate back-projection mapping matrix. Specifically, we filter out occluded and noisy points to avoid incorrect label allocations and permit multiclass assignments to adjust the ambiguity of boundary points. By incorporating an accurate back-projection mechanism into the loss functions of the proposed training frameworks, our networks can perform well with only four projected image annotations or even sparse scribble image annotations for each scene. This results in state-of-the-art performance compared with that of other weakly supervised point cloud semantic segmentation approaches, and the outcomes are even comparable to those produced by fully supervised methods on the S3DIS and ScanNet-v2 datasets.

LESA-Net: Semantic segmentation of multi-type road point clouds in complex agroforestry environment

As buildings have almost come to a saturation point in most developed countries, the management and maintenance of existing buildings have become the major problem of the field. Building Information Modeling (BIM) is the underlying technology to solve this problem. It is a 3D semantic representation of building construction and facilities that contributes to not only the design phase but also the construction and maintenance phases, such as life-cycle management and building energy performance measurement. This study aims at the processes of creating as-built BIM models, which are constructed after the design phase. Point cloud, a set of points in 3D space, is an intermediate product of as-built BIM models that is often acquired by 3D laser scanning and photogrammetry. A raw point cloud typically requires further procedures, e.g. registration, segmentation, classification, etc. In terms of segmentation and classification, machine learning methodologies are trending due to the enhanced speed of computation. However, supervised machine learning methodologies require labelling the training point clouds in advance, which is time-consuming and often leads to inevitable errors. And due to the complexity and uncertainty of real-world environments, the attributes of one point vary from the attributes of others. These situations make it difficult to analyze how one single attribute contributes to the result of segmentation and classification. This study developed a method of producing point clouds from a fast-generating 3D virtual indoor environment using procedural modeling. This research focused on two attributes of simulated point clouds, point density and the level of random errors. According to Silverman (1986), point density is associated with the point features around each output raster cell. The number of points within a neighborhood divided the area of the neighborhood is the point density. However, in this study, there was a little different. The point density was defined as the number of points on a surface divided by the surface area. And the unit is points per square meters (pts/m2). This research compared the performances of a machine learning segmentation and classification algorithm on ten different point cloud datasets. The mean loss and accuracy of segmentation and classification were analyzed and evaluated to show how the point density and level of random errors affect the performance of the segmentation and classification models. Moreover, the real-world point cloud data were used as additional data to evaluate the applicability of produced models.

A training dataset for semantic segmentation of urban point cloud map for intelligent vehicles

Deep learning network for indoor point cloud semantic segmentation with transferability

Attention-based relation and context modeling for point cloud semantic segmentation

Road geometric information and a digital model based on light detection and ranging (LiDAR) can perform accurate geometric inventories and three-dimensional (3D) descriptions for as-built roads and infrastructures. However, unorganized point clouds and complex road scenarios would reduce the accuracy of geometric information extraction and digital modeling. There is a standardization need for information extraction and 3D model construction that integrates point cloud processing and digital modeling. This paper develops a framework from semantic segmentation to geometric information extraction and digital modeling based on LiDAR data. A semantic segmentation network is improved for the purpose of dividing the road surface and infrastructure. The road boundary and centerline are extracted by the alpha-shape and Voronoi diagram methods based on the semantic segmentation results. The road geometric information is obtained by a coordinate transformation matrix and the least square method. Subsequently, adaptive road components are constructed using Revit software. Thereafter, the road route, road entity model, and various infrastructure components are generated by the extracted geometric information through Dynamo and Revit software. Finally, a detailed digital model of the road scenario is developed. The Toronto-3D and Semantic3D datasets are utilized for analysis through training and testing. The overall accuracy (OA) of the proposed net for the two datasets is 95.3 and 95.0%, whereas the IoU of segmented road surfaces is 95.7 and 97.9%. This indicates that the proposed net could accomplish superior performance for semantic segmentation of point clouds. The mean absolute errors between the extracted and manually measured geometric information are marginal. This demonstrates the effectiveness and accuracy of the proposed extraction methods. Thus, the proposed framework could provide a reference for accurate extraction and modeling from LiDAR data.