AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation

RGAM: A novel network architecture for 3D point cloud semantic segmentation in indoor scenes

Aiming at the problem that the relationship between geometric features and semantic features is ignored in the point cloud data downsampling process, which leads to inaccurate segmentation of object boundaries and structural details, this paper proposes a 3D point cloud semantic segmentation network based on multi-scale dense nested type. Firstly, a dense nested network architecture is constructed by nesting multiple multi-scale feature fusion modules to fuse multi-scale features of different directions between encoder-decoder paths, so as to effectively propagate local ge-ometric context information and enhance the ability of cross-scale information interaction. Secondly, a local feature aggregation unit is constructed in the multi-scale feature fusion module, which strengthens the structural awareness within the local point set based on graph convolution and attention mechanism, and promotes the complementarity of local geometric features and abstract semantic information. Then, the cross-layer multi-loss supervision module is combined to further optimize the multi-scale feature propagation, which makes the network training more stable and improves the point cloud segmentation accuracy. Finally, this paper verified the proposed network on the S3DIS dataset. The experimental results show that the proposed network has the mean intersection over Union of 71.2% and the overall accuracy of 88.7%, which is 1.2 and 0.7 percentage points higher than RandLA-Net, respectively, which proves that the proposed network can effectively improve the accuracy of 3D point cloud semantic segmentation.

3D point cloud semantic segmentation is a fundamental task for 3D scene understanding. However, most existing pipelines usually use k-NN or ball query operation to form hard neighborhoods, which may cross different semantic objects, resulting low-quality local features. To address this issue, we propose a multi-scale superpoint network that gradually generates multi-scale soft neighborhoods to extract geometric local features, thereby boosting the 3D semantic segmentation performance. Specifically, we present a simple yet efficient superpoint merging module that merge small-scale superpoints to obtain large-scale superpoint by considering the feature similarity of superpoints, so that we can obtain multi-scale geometric features of point clouds. We also develop a superpoint upsampling module that adopt inverse mapping function to propagate multi-scale features from low-resolution point cloud to high-resolution point cloud. By integrating our multi-scale superpoint network into a simple point based semantic segmentation network, our method can obtain SOTA results on S3DIS Area 5 and 6-fold, and competitive results on ScanNet v2.

Improving performance of deep learning models for 3D point cloud semantic segmentation via attention mechanisms

In this paper, a Backward Attentive Fusing Network with Local Aggregation Classifier (BAF-LAC) is proposed to improve the performance of 3D point cloud semantic segmentation. It consists of a Backward Attentive Fusing Encoder-Decoder (BAF-ED) to learn semantic features and a Local Aggregation Classifier (LAC) to maintain the context-awareness of points. BAF-ED narrows the semantic gap between the encoder and the decoder via fusing multi-layer encoder features with the decoder features. High-level encoder features are transformed into an attention map to modulate low-level encoder features backward. LAC adaptively enhances the intermediate features in point-wise MLPs via aggregating the features of neighboring points into the center point. It takes the place of commonly used post-processing techniques and retains context consistency into the classifier. Equipped with these modules, BAF-LAC can extract discriminative semantic features and predict smoother results. Extensive experiments on Semantic3D, SemanticKITTI, and S3DIS demonstrate that the proposed method can achieve competitive results against the state-of-the-art methods.

Unsupervised domain adaptation (UDA) is increasingly used for 3D point cloud semantic segmentation tasks due to its ability to address the issue of missing labels for new domains. However, most existing unsupervised domain adaptation methods focus only on uni‐modal data and are rarely applied to multi‐modal data. Therefore, we propose a cross‐modal UDA on multi‐modal datasets that contain 3D point clouds and 2D images for 3D Semantic Segmentation. Specifically, we first propose a Dual discriminator‐based Domain Adaptation (Dd‐bDA) module to enhance the adaptability of different domains. Second, given that the robustness of depth information to domain shifts can provide more details for semantic segmentation, we further employ a Dense depth Feature Fusion (DdFF) module to extract image features with rich depth cues. We evaluate our model in four unsupervised domain adaptation scenarios, i.e., dataset‐to‐dataset (A2D2 → SemanticKITTI), Day‐to‐Night, country‐to‐country (USA → Singapore), and synthetic‐to‐real (VirtualKITTI → SemanticKITTI). In all settings, the experimental results achieve significant improvements and surpass state‐of‐the‐art models.

As a pivotal task in 3D scene understanding, 3D point cloud semantic segmentation is often challenged by the presence of mislabeled data in real-world annotations. Such noisy labels can mislead the segmentation model into learning erroneous features during training, thereby degrading its performance. To address this issue, this paper proposes a robust training framework for 3D semantic segmentation based on noisy label correction. The proposed method does not rely on prior knowledge of point cloud density and is capable of learning robust segmentation models under various noise rates. Our approach consists of two core components: a Density-Adaptive Neighborhood Clustering algorithm that dynamically adjusts the neighborhood radius and minimum point count according to local point density variations, and a Dynamically-Weighted Label Refurbishing mechanism, which rectifies noisy labels at the cluster level based on historical prediction confidence. Extensive experiments on the S3DIS dataset demonstrate that our method outperforms existing baselines and state-of-the-art (SOTA) approaches under symmetric, asymmetric, and mixed noise settings. Notably, it maintains strong robustness even under high noise ratios. Furthermore, our method integrates label structures and stores them in a vectorized format, significantly reducing data transfer overhead between CPU and GPU and mitigating resource wastage. This design makes the approach particularly efficient in resource-constrained environments such as cloud servers, where CPU performance is often a bottleneck. As a result, our method achieves a notable improvement in training efficiency, with each training epoch requiring only approximately one-tenth of the time compared to current SOTA methods.

Semantic segmentation of 3D point cloud data is essential for enhanced high-level perception in autonomous platforms. Furthermore, given the increasing deployment of LiDAR sensors onboard of cars and drones, a special emphasis is also placed on non-computationally intensive algorithms that operate on mobile GPUs. Previous efficient state-of-the-art methods relied on 2D spherical projection of point clouds as input for 2D fully convolutional neural networks to balance the accuracy-speed trade-off. This paper introduces a novel approach for 3D point cloud semantic segmentation that exploits multiple projections of the point cloud to mitigate the loss of information inherent in single projection methods. Our Multi-Projection Fusion (MPF) framework analyzes spherical and bird's-eye view projections using two separate highly-efficient 2D fully convolutional models then combines the segmentation results of both views. The proposed framework is validated on the SemanticKITTI dataset where it achieved a mIoU of 55.5 which is higher than state-of-the-art projection-based methods RangeNet++ [23] and PolarNet [44] while being 1.6x faster than the former and 3.1x faster than the latter.

LiDAR has been widely used in autonomous driving systems to provide high-precision 3D geometric information about the vehicle’s surroundings for perception, localization, and path planning. LiDAR-based point cloud semantic segmentation is an important task with a critical real-time requirement. However, most of the existing convolutional neural network (CNN) models for 3D point cloud semantic segmentation are very complex and can hardly be processed at real-time on an embedded platform. In this study, a lightweight CNN structure was proposed for projection-based LiDAR point cloud semantic segmentation with only 1.9 M parameters that gave an 87% reduction comparing to the state-of-the-art networks. When evaluated on a GPU, the processing time was 38.5 ms per frame, and it achieved a 47.9% mIoU score on Semantic-KITTI dataset. In addition, the proposed CNN is targeted on an FPGA using an NVDLA architecture, which results in a 2.74x speedup over the GPU implementation with a 46 times improvement in terms of power efficiency.

In recent years, the numerous advantages introduced by Building Information modelling (BIM) have led in its application on the heritage environment and giving birth to the concept of H-BIM (Heritage BIM). The resulting demand in heritage survey data processing has focused this research on the development of strategies and methods to improve the construction of three-dimensional and informative models starting from 3D point clouds. The implementation of an automated procedure is fundamental for easing and speeding up the survey data processing and one of the most challenging tasks to achieve this purpose is the problem of semantic segmentation. The research presented in this paper aims at testing already existing methods and exploring new strategies for 3D point cloud semantic segmentation on heritage scenarios focusing on deep learning and neural network techniques.

Semantic segmentation is an important component in understanding the 3D point cloud scene. Whether we can effectively obtain local and global contextual information from points is of great significance in improving the performance of 3D point cloud semantic segmentation. In this paper, we propose a self-attention feature extraction module: the local transformer structure. By stacking the encoder layer composed of this structure, we can extract local features while preserving global connectivity. The structure can automatically learn each point feature from its neighborhoods and is invariant to different point orders. We designed two unique key matrices, each of which focuses on the feature similarities and geometric structure relationships between the points to generate attention weight matrices. Additionally, the cross-skip selection of neighbors is used to obtain larger receptive fields for each point without increasing the number of calculations required, and can therefore better deal with the junction between multiple objects. When the new network was verified on the S3DIS, the mean intersection over union was 69.1%, and the segmentation accuracies on the complex outdoor scene datasets Semantic3D and SemanticKITTI were 94.3% and 87.8%, respectively, which demonstrate the effectiveness of the proposed methods.

Neuronal soma segmentation plays a crucial role in neuroscience applications. However, the fine structure, such as boundaries, small-volume neuronal somata and fibers, are commonly present in cell images, which pose a challenge for accurate segmentation. In this paper, we propose a 3D semantic segmentation network for neuronal soma segmentation to address this issue. Using an encoding-decoding structure, we introduce a Multi-Scale feature extraction and Adaptive Weighting fusion module (MSAW) after each encoding block. The MSAW module can not only emphasize the fine structures via an upsampling strategy, but also provide pixel-wise weights to measure the importance of the multi-scale features. Additionally, a dynamic convolution instead of normal convolution is employed to better adapt the network to input data with different distributions. The proposed MSAW-based semantic segmentation network (MSAW-Net) was evaluated on three neuronal soma images from mouse brain and one neuronal soma image from macaque brain, demonstrating the efficiency of the proposed method. It achieved an F1 score of 91.8% on Fezf2-2A-CreER dataset, 97.1% on LSL-H2B-GFP dataset, 82.8% on Thy1-EGFP-Mline dataset, and 86.9% on macaque dataset, achieving improvements over the 3D U-Net model by 3.1%, 3.3%, 3.9%, and 2.3%, respectively.

When 3D reconstruction of plant seedlings is performed to obtain plant point clouds, there are many noisy points between the leaves and stems of plant point clouds due to the influence of ambient light and the limitation of the camera vision, which affects the automatic measurement of plant phenotypes.In order to achieve plant point cloud stem and leaf segmentation with a large amount of noise, We propose a network for semantic segmentation of noisy plant point clouds based on self-attentive feature fusion (abbreviated as SAFF-Net).The network first extracts shallow spatial features and higher-level semantic features between points by using correlations between pairs of points in the neighborhood through the local feature fusion module, and enhances the ability to extract plant shape features;The dual-branch attention pooling module is used to aggregate features, and one branch uses the channel attention mechanism to adaptively filter low-correlation features, avoiding redundancy of features and mitigating the bias effect of noise points.Another branch performs max pooling for the highest level feature map to obtain global contextual features, and finally merges local and global contextual features to learn more discriminative features.The point cloud dataset for the experiment is derived from multi-angle photographs of the plant taken by a high-definition camera and reconstructed in 3D by Structure from Motion(SFM) algorithm.The experimental results indicate that SAFF-Net performs better than the mainstream semantic segmentation networks and extracts more fine-grained local features. OA and (mIoU) of this method on the plant dataset are 93.7% and 83.4%, respectively. It outperforms existing methods in the segmentation of noise points and leaves, and the segmentation results have higher precision and recall.

Adversarial unsupervised domain adaptation for 3D semantic segmentation with multi-modal learning

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories – less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.