A simulation‐assisted point cloud segmentation neural network for human–robot interaction applications

For the problem that it is difficult to effectively cluster lidar point clouds with irregular shapes and uneven densities, a Neighborhood Effective Line Density (NELD)-based Euclidean Clustering (NELD-EC) algorithm is proposed in this paper. The NELD-EC algorithm first eliminates the interfering points within the neighborhood of the data point by utilizing the distance relationship and calculates the NELD of the data point using the effective neighborhood set without interfering points of the data point. The NELD of a data point is taken as the local density of that data point. Then, the NELD-EC algorithm conducts clustering processing using the NELD of all data points and uses the reciprocal of the harmonic average of the local densities of all data points within each cluster after clustering as the distance threshold for the data points within the cluster. Finally, the NELD-EC algorithm completes the clustering of the point cloud based on the adjusted adaptive distance threshold. The clustering experimental results on simulated point clouds, fixed point clouds, and sequential point clouds indicate that, compared with several other typical Euclidean clustering algorithms, the NELD-EC algorithm requires simpler parameters to be set, is less sensitive to the initial distance threshold, can effectively reduce the occurrence probabilities of over-segmentation and under-segmentation, and has strong stability in clustering performance. The NELD-EC algorithm is more suitable for processing sequential point clouds in actual dynamic and complex scenarios.

Point cloud segmentation is the key to the research of robotic grasping in multi-object stacked scenes. Aiming to improve the point cloud segmentation methods, which have low accuracy and poor robustness in complex stacked scenes, this paper proposes a point cloud segmentation method based on a dynamic graph convolutional neural network (DGCNN). A channel-spatial attention module combining the channel attention mechanism and the spatial attention mechanism is added to the DGCNN feature extraction. This improved method can effectively use the features of point clouds. To train the improved DGCNN network, we build a small rectangular object segmentation dataset containing 800 real-world stacked scenes. Experimental results show that our segmentation accuracy can reach 88.61%. After sufficient experiments, it is shown that the robotic grasping with our point cloud segmentation method can get 96% success rate, which can meet the actual production needs.

Lidar has received a lot of attention due to its precise ranging accuracy. Ground points filtering is an important task in point clouds processing. It’s a challenge to model the ground surface and filter the point clouds accurately in the case of complex ground undulations, occlusions, and sparse point clouds. A novel ground surface modeling method based on a hybrid regression technique is proposed in this paper. The method integrates Gaussian process regression (GPR) and robust locally weighted regression (RLWR) by dividing the point clouds that are projected on the polar grid map into radial and circumferential filtering processes to form a hybrid regression model, which has the ability to eliminate the influence of outliers and model the ground surface robustly. First, the RLWR combined with gradient filter is applied to fit the sampled points in the radial direction, which will exclude outliers and get the fitting ground line. All radial fitting lines constitute the seed skeleton of the whole plane. Then, based on the seeds in the same circumferential of the skeleton, the GPR is applied to construct the ground surface model. The comparative experiments are implemented quantitatively and qualitatively on the simulated point clouds and measured data. The results show that the proposed method performs well in most real scenarios, even in the cases of ground undulation, occlusion, and sparse point clouds.

Point cloud segmentation is essential for scene understanding, which provides advanced information for many applications, such as autonomous driving, robots, and virtual reality. To improve the accuracy and robustness of point cloud segmentation, many researchers have attempted to fuze camera images to complement the color and texture information. The common fusion strategy is the combination of convolutional operations with concatenation, element-wise addition or element-wise multiplication. However, conventional convolutional operators tend to confine the fusion of modal features within their receptive fields, which can be incomplete and limited. In addition, the inability of encoder–decoder segmentation networks to explicitly perceive segmentation boundary information results in semantic ambiguity and classification errors at object edges. These errors are further amplified in point cloud segmentation tasks, significantly affecting the accuracy of point cloud segmentation. To address the above issues, we propose a novel self-attention multi-modal fusion semantic segmentation network for point cloud semantic segmentation. Firstly, to effectively fuze different modal features, we propose a self-cross fusion module (SCF), which models long-range modality dependencies and transfers complementary image information to the point cloud to fully leverage the modality-specific advantages. Secondly, we design the salience refinement module (SR), which calculates the importance of channels in the feature maps and global descriptors to enhance the representation capability of salient modal features. Finally, we propose the local-aware anisotropy loss measure the element-level importance in the data and explicitly provide boundary information for the model, which alleviates the inherent semantic ambiguity problem in segmentation networks. Extensive experiments on two benchmark datasets demonstrate that our proposed method surpasses current state-of-the-art methods.

This paper presents a novel approach to recognize and estimate pose of the 3D objects in cluttered range images. The key technical breakthrough of the developed approach can enable robust object recognition and localization under undesirable condition such as environmental illumination variation as well as optical occlusion to viewing the object partially. First, the acquired point clouds are segmented into individual object point clouds based on the developed 3D object segmentation for randomly stacked objects. Second, an efficient shape-matching algorithm called Sub-OBB based object recognition by using the proposed oriented bounding box (OBB) regional area-based descriptor is performed to reliably recognize the object. Then, the 3D position and orientation of the object can be roughly estimated by aligning the OBB of segmented object point cloud with OBB of matched point cloud in a database generated from CAD model and 3D virtual camera. To detect accurate pose of the object, the iterative closest point (ICP) algorithm is used to match the object model with the segmented point clouds. From the feasibility test of several scenarios, the developed approach is verified to be feasible for object pose recognition and localization.

In recent years, point cloud segmentation technology has increasingly played a pivotal role in tunnel construction and maintenance. Currently, traditional methods for segmenting point clouds in tunnel scenes often rely on a multitude of attribute information, including spatial distribution, color, normal vectors, intensity, and density. However, the underground tunnel scenes show greater complexity than road tunnel scenes, such as dim light, indistinct boundaries of tunnel walls, and disordered pipelines. Furthermore, issues pertaining to data quality, such as the lack of color information and insufficient annotated data, contribute to the subpar performance of conventional point cloud segmentation algorithms. To address this issue, a 3D point cloud segmentation framework specifically for underground tunnels is proposed based on the Segment Anything Model (SAM). This framework effectively leverages the generalization capability of the visual foundation model to automatically adapt to various scenes and perform efficient segmentation of tunnel point clouds. Specifically, the tunnel is first sliced along its direction on the tunnel line. Then, each sliced point cloud is projected onto a two-dimensional plane. Various projection methods and point cloud coloring techniques are employed to enhance SAM’s segmentation performance in images. Finally, the semantic segmentation of the entire underground tunnel is achieved by a small set of manually annotated semantic labels used as prompts in a progressive and recursive manner. The key feature of this method lies in its independence from model training, as it directly and efficiently addresses tunnel point cloud segmentation challenges by capitalizing on the generalization capability of foundation model. Comparative experiments against classical region growing algorithms and PointNet++ deep learning algorithms demonstrate the superior performance of our proposed algorithm.

Point cloud segmentation is a key part of geometric processing. In this paper, a point cloud segmentation method based on the Fast Point Feature Histograms (FPFH) is proposed. Fast Point Feature Histograms (FPFH) is used to extract features of point clouds, and then Gaussian Mixture Model (GMM) is employed to cluster the point clouds. Experimental results on SHREC2014 dataset show the effectiveness of the proposed method. It can avoid both under- segmentation and over-segmentation.

Three dimensional point cloud data sets are easy to acquire and manipulate, but are often too large to process directly for embedded real-time applications. The spatial information in a point cloud can be represented in a variety of reduced forms, such as voxel grids, Gaussian mixture models, or spatial semantic structures. In this article, we show how a segmented point cloud can be represented as a spatial relationship graph using bounding boxes and triangular fuzzy numbers. This model is a lightweight encoding of the relative distance and direction between objects, and can be used to describe and query for particular spatial configurations using linguistic terms in a multicriteria framework. We show how this approach can be applied on a hand-segmented subset of the NPM3D data set with several illustrative examples. The work herein has useful applications in many applied domains, such as human-robot interaction with unmanned aerial systems.

Fitting and recognition of geometric primitives in segmented 3D point clouds using a localized voting procedure

XLPE cable joint defects measurement method based on point cloud remapping

The accurate segmentation of roof point cloud will directly affect the quality of building model reconstruction. Because of the complexity of building roof structure, the uneven distribution of point cloud density obtained by lidar, and the interference of external environment, it is very difficult to accurately segment the point cloud on the roof. In order to improve the quality of point cloud segmentation, this paper proposes a joint optimization method of region growth and RANSAC for Airborne LIDAR point cloud segmentation. Firstly, the point cloud is voxeled, and the normal vector and curvature of the point cloud are calculated based on principal component analysis (PCA). After setting the correlation threshold, the region growing algorithm is used to extract the initial reliable patch. Then, RANSAC algorithm is used to regularize the irregular or deviation areas in the extracted patch, optimize and set new roof surface parameters. Finally, for the unclassified points, the Euclidean distance to each plane is calculated. According to the minimum distance principle and the set distance threshold, they are classified to the corresponding roof surface, and the segmentation results are refined. Tests on several representative building point clouds show that this method significantly improves the accuracy, recall and overall accuracy of roof point cloud segmentation, and provides high-quality basic data for building 3D model reconstruction

Global localization in a prior map is an important field in virtual and augmented reality systems, but it is always a challenge to conduct point cloud based localization in the large-scale scene prior map. A large-scale prior map usually means huge amount of calculation for point cloud processing, which leads to the long time required for global localization. To deal with this problem, we propose a fast point cloud global localization method based on point clouds segmentation and learning-based descriptor. On the one hand, cylindrical filtering, ground-point removal and point cloud segmentation are adopted to eliminate a large number of useless points and retain points with rich structures, which improves the efficiency of point cloud registration. On the other hand, reliable 3D point cloud descriptor, two-phase search strategy for place recognition and geometric consistency verification are used to ensure the localization accuracy. Experiments prove that the proposed method achieves good localization effect on both KITTI and MVSEC datasets. Under the condition of ensuring the high localization accuracy, the time for point clouds to complete the global localization is greatly reduced.

Point cloud segmentation neural network with same-type point cloud assistance

In Autonomous Driving (AD), detection and tracking of obstacles on the roads is a critical task. Deep-learning based methods using annotated LiDAR data have been the most widely adopted approach for this. Unfortunately, annotating 3D point cloud is a very challenging, time- and money-consuming task. In this paper, we propose a novel LiDAR simulator that augments real point cloud with synthetic obstacles (e.g., cars, pedestrians, and other movable objects). Unlike previous simulators that entirely rely on CG models and game engines, our augmented simulator bypasses the requirement to create high-fidelity background CAD models. Instead, we can simply deploy a vehicle with a LiDAR scanner to sweep the street of interests to obtain the background point cloud, based on which annotated point cloud can be automatically generated. This unique "scan-and-simulate" capability makes our approach scalable and practical, ready for large-scale industrial applications. In this paper, we describe our simulator in detail, in particular the placement of obstacles that is critical for performance enhancement. We show that detectors with our simulated LiDAR point cloud alone can perform comparably (within two percentage points) with these trained with real data. Mixing real and simulated data can achieve over 95% accuracy.

Early stopping techniques can be utilized to decrease the time cost, however currently the ultimate goal of early stopping techniques is closely related to the accuracy upgrade or the ability of the neural network to generalize better on unseen data without being large or complex in structure and not directly with its efficiency. Time efficiency is a critical factor in neural networks, especially when dealing with the segmentation of 3D point cloud data, not only because a neural network itself is computationally expensive, but also because point clouds are large and noisy data, making learning processes even more costly. In this paper, we propose a new early stopping technique based on fundamental mathematics aiming to upgrade the trade-off between the learning efficiency and accuracy of neural networks dealing with 3D point clouds. Our results show that by employing our early stopping technique in four distinct and highly utilized neural networks in segmenting 3D point clouds, the training time efficiency of the models is greatly improved, with efficiency gain values reaching up to 94\%, while the models achieving in just a few epochs approximately similar segmentation accuracy metric values like the ones that are obtained in the training of the neural networks in 200 epochs. Also, our proposal outperforms four conventional early stopping approaches in segmentation accuracy, implying a promising innovative early stopping technique in point cloud segmentation.