3D point cloud classification and segmentation based on dual attention and weighted dynamic graph convolution

In recent years, there has been significant growth in the ubiquity and popularity of three-dimensional (3D) point clouds, with an increasing focus on the classification of 3D point clouds. To extract richer features from point clouds, many researchers have turned their attention to various point set regions and channels within irregular point clouds. However, this approach has limited capability in attending to crucial regions of interest in 3D point clouds and may overlook valuable information from neighboring features during feature aggregation. Therefore, this paper proposes a novel 3D point cloud classification method based on global attention and adaptive graph convolution (Att-AdaptNet). The method consists of two main branches: the first branch computes attention masks for each point, while the second branch employs adaptive graph convolution to extract global features from the point set. It dynamically learns features based on point interactions, generating adaptive kernels to effectively and precisely capture diverse relationships among points from different semantic parts. Experimental results demonstrate that the proposed model achieves 93.8% in overall accuracy and 90.8% in average accuracy on the ModeNet40 dataset.

We propose a novel deep architecture for semantic labeling of 3D point clouds referred to as Global and Local Streams Network (GLSNet) which is designed to capture both global and local structures and contextual information for large scale 3D point cloud classification. Our GLSNet tackles a hard problem - large differences of object sizes in large-scale point cloud segmentation including extremely large objects like water, and small objects like buildings and trees, and we design a two-branch deep network architecture to decompose the complex problem to separate processing problems at global and local scales and then fuse their predictions. GLSNet combines the strength of Submanifold Sparse Convolutional Network [1] for learning global structure with the strength of PointNet++ [2] for incorporating local information. The first branch of GLSNet processes a full point cloud in the global stream, and it captures long range information about the geometric structure by using a U-Net structured Submanifold Sparse Convolutional Network (SSCN-U) architecture. The second branch of GLSNet processes a point cloud in the local stream, and it partitions 3D points into slices and processes one slice of the cloud at a time by using the PointNet ++ architecture. The two streams of information are fused by max pooling over their classification prediction vectors. Our results on the IEEE GRSS Data Fusion Contest Urban Semantic 3D, Track 4 (DFT4) [3] [4] [5] point cloud classification dataset have shown that GLSNet achieved performance gains of almost 4% in mIOU and 1% in overall accuracy over the individual streams on the held-back testing dataset.

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications [1,2].METHODS & RESULTS: While new approaches for deep neural networks are rapidly developing [1], the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ [3], can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments [1], purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo [5], enables to perform ML for change analysis in multitemporal point clouds [6]. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo). REFERENCES:[1] Esmorís Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.[2] Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.112772 [3] HELIOS++: https://github.com/3dgeo-heidelberg/helios[4] VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d[5] py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo[6] Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034

Recently, the researches on 3D model analysis begin to come into focus. With the development of 3D data capturing devices, people can obtain 3D data much easier than before. At present, many existing models for 3D classification are constructed with convolution operator which can benefit from improvements on convolution. Meanwhile, attention mechanisms have demonstrated their utility across quite a few tasks including sequence learning, understanding in images. Some lightweight and general modules based on the attention mechanism are designed to improve the performance of network based on CNN architectures. Inspired by the research on image representations based on the attention mechanism, this paper applies the mechanism to address the problem of 3D shape classification. In this work, a channel attention module named 3DSE block is used to improve the performance of some representative models for 3D point cloud classification. The dependency relationship between the different channels is explicitly modeled to obtain the channel descriptor of the feature map, and it is used to refine the original feature map, so as to sift through the information from the feature and enhance the ability to feature extraction. This paper uses 3DSE module to improve the performance of the existing typical 3D classification networks. Then some experiments are performed on popular shape classification benchmark to test our ideas. The result demonstrates the effectiveness of our method.

Abstract. Point-wise classification of 3D point clouds is a challenging task in point cloud processing, whereas, in particular, its application to high-density point clouds needs special attention because a large number of point clouds affect computational efficiency negatively. Although deep learning based models have been gaining popularity in recent years and have reached state-of-the-art results in accuracy for point-wise classification, their requirements of the high number of training samples and computational resources make those models inefficient for high-density 3D point clouds. However, traditional machine learning classifiers require less training samples, so they are capable of reducing computational requirements, even considering the latest machine learning classifiers, particularly in ensemble learning of gradient boosting machines, the results can compete with deep learning models. In this study, we are studying the point-wise classification of high-density UAV LiDAR data and focusing on efficient feature extraction and a recent state-of-the-art gradient boosting machine learning classifier, LightGBM. Our proposed framework includes the following steps: at first, we are using point cloud sampling for creating sub-sampled point clouds, then we are calculating the features based on those scales implemented on GPU. Finally, we are using the LightGBM classifier for training and testing. For the evaluation of our framework, we used a publicly available benchmark dataset, Hessigheim 3D. According to the results, we achieved an overall accuracy of 87.59% and an average F1 score of 75.92%. Our framework has promising results and scores closer to deep learning models. However, more distinctive features are required to obtain more accurate results.

We proposed using Conditional Random Fields with adaptive data reduction for the classification of 3D point clouds acquired from a Riegl Terrestrial laser scanner. The training and inference of the acquired large outdoor urban data can be time consuming. We approach the problem by computing an adaptive support region for each data point using 3D scale theory. For training and inference of the discriminative Conditional Random Fields, smaller set of data samples that contains relevant information within the support region is selected instead of using all point cloud data. We tested the algorithm on synthetically generated data and urban point clouds data acquired from the laser scanner. The computed support region is also used in feature extraction for urban point clouds data. The results showed improvement in the training and inference rate while maintaining comparable classification accuracy.

3D point cloud analysis and processing find numerous applications in computer-aided design, 3D printing, autonomous driving, etc. Most state-of-the-art point cloud processing methods are based on convolutional neural networks (CNNs). Although they outperform traditional methods in terms of accuracy, they demand heavy supervision and higher training complexity. Besides, they lack mathematical transparency. In this talk, I will present three interpretable and effective machine learning methods for 3D point cloud registration, classification and segmentation, respectively. First, an unsupervised registration method that extracts salient points for matching is presented. Second, an unambiguous way to order points sequentially in a point cloud set is developed. Then, their spatial coordinates can be treated as geometric attributes of 1 D data array. This idea facilitates the classification task. Third, for the segmentation task, we show how to leverage prior knowledge on point clouds to derive an intuitive and effective segmentation method. Extensive experiments are conducted to demonstrate the performance of the three new methods. I will also provide performance benchmarking between these interpretable methods and deep learning methods.

Under increased urban settlement density, access to a high resolution (land-parcel scale) bare-earth Digital Elevation Model (DEM) is a pre-requisite for much decision support for planning: stormwater assessment, flood control, 3D visualisation, automatic delineation of flow paths, sub watersheds and flow networks for hydrological modelling. In these terms, a range of options face the DEM-building team. Apart from using necessarily expensive field survey, or use of out-of-date terrain information (usually in the form of digital contours of less-than-satisfactory interval) the model will be built from point-clouds. These will have been assembled via digital photogrammetry or acquisition of LiDAR data. In the first instance, both these data types soon yield a model that is known as a digital surface model (DSM). It includes any buildings, vehicles, vegetation (canopy and understory), as well as the bare ground". To generate the required bare-earth' DEM, ground and non-ground features/data points must be distinguished from each other so that the latter can be eliminated before DEM building. Existing methods for doing this are based on data filtering routines, and are known to produce errors of omission and commission. Moreover, their implementation is complex and time consuming.I report here, the results of deploying spatial data integration instead of the previously favoured filtering routines. The challenge was to identify a process flow for separating the ground and non-ground points. It is shown that this alternative approach can be implemented if the client can supply a range of ancillary height data, this being most economically forthcoming if archivally available. The relative significance of these archival datasets emerges from exploring the various process flow paths and devising relevant quality tests designed to distinguish input suitable to support modelling at a land-parcel scale of analysis. Then the ArcGIS topological overlay technique was used to collect zonal statistics for each 3D point. Thus each output 3D point acquires z (elevation) values derived from the digital photogrammetry and z-statistics (minimum, maximum, mean) of its assigned zone. Clearly there is value in spatial data integration for a city with spatial data archives of adequately supportive scope and quality.

Under increased urban settlement density, access to a high resolution (land-parcel scale) bare-earth Digital Elevation Model (DEM) is a pre-requisite for much decision support for planning: stormwater assessment, flood control, 3D visualisation, automatic delineation of flow paths, sub watersheds and flow networks for hydrological modelling. In these terms, a range of options face the DEM-building team. Apart from using necessarily expensive field survey, or use of out-of-date terrain information (usually in the form of digital contours of less-than-satisfactory interval) the model will be built from point-clouds. These will have been assembled via digital photogrammetry or acquisition of LiDAR data. In the first instance, both these data types soon yield a model that is known as a digital surface model (DSM). It includes any buildings, vehicles, vegetation (canopy and understory), as well as the bare ground". To generate the required bare-earth' DEM, ground and non-ground features/data points must be distinguished from each other so that the latter can be eliminated before DEM building. Existing methods for doing this are based on data filtering routines, and are known to produce errors of omission and commission. Moreover, their implementation is complex and time consuming.I report here, the results of deploying spatial data integration instead of the previously favoured filtering routines. The challenge was to identify a process flow for separating the ground and non-ground points. It is shown that this alternative approach can be implemented if the client can supply a range of ancillary height data, this being most economically forthcoming if archivally available. The relative significance of these archival datasets emerges from exploring the various process flow paths and devising relevant quality tests designed to distinguish input suitable to support modelling at a land-parcel scale of analysis. Then the ArcGIS topological overlay technique was used to collect zonal statistics for each 3D point. Thus each output 3D point acquires z (elevation) values derived from the digital photogrammetry and z-statistics (minimum, maximum, mean) of its assigned zone. Clearly there is value in spatial data integration for a city with spatial data archives of adequately supportive scope and quality.

A fast graph-based data classification method with applications to 3D sensory data in the form of point clouds

Recent years, the research on object classification based on three‐dimensional (3D) point cloud pays more attention to extract the features from point sets directly. PointNet++ is the latest network structure for 3D classification which has achieved acceptable results. Although it has achieved acceptable results, there are still two problems: (i) The farthest point sampling (FPS) algorithm in PointNet++ ignores the fact that the feature of each point contributes differently to the classification and segmentation tasks. Therefore, FPS cannot guarantee that the selected point sets can correctly represent the main features of the object. (ii) The multi‐scale grouping and multi‐resolution grouping in PointNet++ do not consider the features between different levels of the same point. In order to solve these problems, the authors propose the point‐selection structure which can calculate the importance of each point's feature. In addition, multi‐level‐point‐feature fusion module is proposed to extract the features of the point at all levels and fuse them as a new feature of that point. In this Letter, they make some experiments on ModelNet40 and ScanNet datasets, which achieves better results compared to the state‐of‐the‐art methods.

Nearly every structured prediction problem in computer vision requires approximate inference due to large and complex dependencies among output labels. While graphical models provide a clean separation between modeling and inference, learning these models with approximate inference is not well understood. Furthermore, even if a good model is learned, predictions are often inaccurate due to approximations. In this work, instead of performing inference over a graphical model, we instead consider the inference procedure as a composition of predictors. Specifically, we focus on message-passing algorithms, such as Belief Propagation, and show how they can be viewed as procedures that sequentially predict label distributions at each node over a graph. Given labeled graphs, we can then train the sequence of predictors to output the correct labeling s. The result no longer corresponds to a graphical model but simply defines an inference procedure, with strong theoretical properties, that can be used to classify new graphs. We demonstrate the scalability and efficacy of our approach on 3D point cloud classification and 3D surface estimation from single images.

Compared to traditional learning from scratch, knowledge distillation sometimes makes the DNN achieve superior performance. In this paper, we provide a new perspective to explain the success of knowledge distillation based on the information theory, i.e., quantifying knowledge points encoded in intermediate layers of a DNN for classification. To this end, we consider the signal processing in a DNN as a layer-wise process of discarding information. A knowledge point is referred to as an input unit, the information of which is discarded much less than that of other input units. Thus, we propose three hypotheses for knowledge distillation based on the quantification of knowledge points. 1. The DNN learning from knowledge distillation encodes more knowledge points than the DNN learning from scratch. 2. Knowledge distillation makes the DNN more likely to learn different knowledge points simultaneously. In comparison, the DNN learning from scratch tends to encode various knowledge points sequentially. 3. The DNN learning from knowledge distillation is often more stably optimized than the DNN learning from scratch. To verify the above hypotheses, we design three types of metrics with annotations of foreground objects to analyze feature representations of the DNN, i.e., the quantity and the quality of knowledge points, the learning speed of different knowledge points, and the stability of optimization directions. In experiments, we diagnosed various DNNs on different classification tasks, including image classification, 3D point cloud classification, binary sentiment classification, and question answering, which verified the above hypotheses.

We propose MaxUp, a simple and effective technique for improving the generalization performance of machine learning models, especially deep neural networks. The idea is to generate a set of augmented data with some random perturbations or transforms, and minimize the maximum, or worst case loss over the augmented data. By doing so, we implicitly introduce a smoothness or robustness regu-larization against the random perturbations, and hence improve the generation performance. For example, in the case of Gaussian perturbation, MaxUp is asymptotically equivalent to using the gradient norm of the loss as a penalty to encourage smoothness. We test MaxUp on a range of tasks, including image classification, 3D point cloud classification, and adversarial certification, on which MaxUp consistently outperforms the baseline methods, without introducing substantial computational overhead. In particular, we improve ImageNet classification from the top-1 accuracy 85.5% without extra data to 85.8%.

Today, agricultural vehicles are available that can drive autonomously and follow exact route plans more precisely than human operators. Combined with advancements in precision agriculture, autonomous agricultural robots can reduce manual labor, improve workflow, and optimize yield. However, as of today, human operators are still required for monitoring the environment and acting upon potential obstacles in front of the vehicle. To eliminate this need, safety must be ensured by accurate and reliable obstacle detection and avoidance systems.In this thesis, lidar-based obstacle detection and recognition in agricultural environments has been investigated. A rotating multi-beam lidar generating 3D point clouds was used for point-wise classification of agricultural scenes, while multi-modal fusion with cameras and radar was used to increase performance and robustness. Two research perception platforms were presented and used for data acquisition. The proposed methods were all evaluated on recorded datasets that represented a wide range of realistic agricultural environments and included both static and dynamic obstacles.For 3D point cloud classification, two methods were proposed for handling density variations during feature extraction. One method outperformed a frequently used generic 3D feature descriptor, whereas the other method showed promising preliminary results using deep learning on 2D range images. For multi-modal fusion, four methods were proposed for combining lidar with color camera, thermal camera, and radar. Gradual improvements in classification accuracy were seen, as spatial, temporal, and multi-modal relationships were introduced in the models. Finally, occupancy grid mapping was used to fuse and map detections globally, and runtime obstacle detection was applied on mapped detections along the vehicle path, thus simulating an actual traversal.The proposed methods serve as a first step towards full autonomy for agricultural vehicles. The study has thus shown that recent advancements in autonomous driving can be transferred to the agricultural domain, when accurate distinctions are made between obstacles and processable vegetation. Future research in the domain has further been facilitated with the release of the multi-modal obstacle dataset, FieldSAFE.