SSC: Semantic Scan Context for Large-Scale Place Recognition

As a fundamental yet challenging problem in intelligent transportation systems, point cloud registration attracts vast attention and has been attained with various deep learning-based algorithms. The unsupervised registration algorithms take advantage of deep neural network-enabled novel representation learning while requiring no human annotations, making them applicable to industrial applications. However, unsupervised methods mainly depend on global descriptors, which ignore the high-level representations of local geometries. In this paper, we propose to jointly use both global and local descriptors to register point clouds in a self-supervised manner, which is motivated by a critical observation that all local geometries of point clouds are transformed consistently under the same transformation. Therefore, local geometries can be employed to enhance the representation ability of the feature extraction module. Moreover, the proposed local descriptor is flexible and can be integrated into most existing registration methods and improve their performance. Besides, we also utilize point cloud reconstruction and normal estimation to enhance the transformation awareness of global and local descriptors. Lastly, extensive experimental results on one synthetic and three real-world datasets demonstrate that our method outperforms existing state-of-art unsupervised registration methods and even surpasses supervised ones in some cases. Robustness and computational efficiency evaluations also indicate that the proposed method applies to intelligent vehicles.

3D point cloud-based place recognition has gotten more attention since 3D LiDAR sensors are widely used for robotic applications and autonomous driving. Most of the existing deep point cloud-based methods take a few regular number points or image-like formats as inputs which are inability to make full use of point clouds' geometric information. This paper proposes a novel place recognition approach that is flexible and effective to handle diverse numbers of 3D LiDAR point clouds in large-scale environments. The approach is composed of feature extraction and a global descriptor encoding. The feature extraction consumes the 3D LiDAR point cloud with KPConv that can extract features efficiently and flexibly. Before the global descriptor encoding, a transformer module is employed to aggregate the contextual information that exploits the relationship of all features. The NetVLAD layer encodes the features into a global descriptor for recognizing a similar place rapidly. The proposed approach is evaluated on the KITTI odometry dataset, which demonstrates the validity of the proposed approach.

LiDAR point cloud-based place recognition (LPR) in unstructured natural environments remains an open challenge with limited existing research. To address the limitations of unstructured environments, such as sparse structural features, uneven point cloud density, and significant viewpoint variations, we present BSPR-Net, a dual-branch point cloud feature extraction approach for point cloud place recognition, which consists of a BEV - projection rotation - invariant convolution branch and a point cloud sparse convolution branch. This design enhances the representation capability of geometric structural features while aggregating rotation-invariant characteristics of point clouds, thereby better addressing the challenge of large viewpoint disparities in reverse-revisited unstructured environments. The proposed network was tested on multiple reverse-revisited sequences of the Wild-Places data set, a benchmark for unstructured natural environment place recognition. It achieved a maximum F1 score of 85.46 %, exceeding other classical methods by more than 4 %. The ablation experiments further confirmed the effectiveness of each module in improving place recognition performance.

Remote sensing technologies, such as LiDAR, produce billions of points that commonly exceed the storage capacity of the GPU, restricting their processing and rendering. Level of detail (LoD) techniques have been widely investigated, but building the LoD structures is also time-consuming. This study proposes a GPU-driven culling system focused on determining the number of points visible in every frame. It can manipulate point clouds of any arbitrary size while maintaining a low memory footprint in both the CPU and GPU. Instead of organizing point clouds into hierarchical data structures, these are split into groups of points sorted using the Hilbert encoding. This alternative alleviates the occurrence of anomalous groups found in Morton curves. Instead of keeping the entire point cloud in the GPU, points are transferred on demand to ensure real-time capability. Accordingly, our solution can manipulate huge point clouds even in commodity hardware with low memory capacities. Moreover, hole filling is implemented to cover the gaps derived from insufficient density and our LoD system. Our proposal was evaluated with point clouds of up to 18 billion points, achieving an average of 80 frames per second (FPS) without perceptible quality loss. Relaxing memory constraints further enhances visual quality while maintaining an interactive frame rate. We assessed our method on real-world data, comparing it against three state-of-the-art methods, demonstrating its ability to handle significantly larger point clouds.

For 3D object recognition, a discriminative point cloud descriptor is required to represent the object. The existing global descriptors encode the whole object into a vector but they are sensitive to occlusion. On the contrary, the local descriptors encode only a small neighbor of a key point and are more robust to occlusion, but many objects have the same local surface. This paper presents a novel mixture method which segments a point cloud into multiple subparts to overcome the above shortcomings. In offline training stage, we propose to build up the model library that integrates both global and local surface of partial point clouds. In online recognition stage, the scene objects are represented by its subparts, and a voting scheme is performed for the recognition of scene objects. Experimental results on public datasets show that the proposed method promotes the recognition performance significantly compared to the conventional global and local descriptors.

At present, three-dimensional laser scanners are used to scan subway shield tunnels and generate point cloud data as the basis for extracting a variety of information about tunnel defects. However, there are obstacles in the tunnel such as pipelines, tracks, and signaling systems that cause noise in the point cloud. Usually, the data of the tunnel point cloud are huge, and the efficiency of artificial denoising is low. Faced with this problem, based on the respective characteristics of the geometric shape and reflection intensity of the tunnel point cloud and their correlation, this paper proposes a tunnel point cloud denoising method. The method includes the following three parts: reflection intensity threshold denoising, joint shape and reflection intensity denoising, and shape denoising. Through the experiment on the single-ring segment point cloud of a shield tunnel, the method proposed in this paper takes 2 min to remove 99.77% of the noise in the point cloud. Compared with manual denoising, the method proposed in this paper takes two fifteenths of the time to achieve the same denoising effect. The method proposed in this paper meets the requirements of a tunnel point cloud data survey. Thus, it provides support for the efficient, accurate, and automatic daily maintenance and surveys of tunnels.

Point cloud based retrieval for place recognition is still a challenging problem since the drastic appearance changes of scenes due to seasonal or artificial changes in the environments. Existing deep learning based global descriptors for the retrieval task usually consume a large amount of computational resources ( e.g ., memory), which may not be suitable for the cases of limited hardware resources. In this paper, we develop an efficient point cloud learning network (EPC-Net) to generate global descriptors of point clouds for place recognition. While obtaining good performance, it can greatly reduce computational memory and inference time. First, we propose a lightweight but effective neural network module, called ProxyConv, to aggregate the local geometric features of point clouds. We leverage the adjacency matrix and proxy points to simplify the original edge convolution for lower memory consumption. Then, we design a lightweight grouped VLAD network to form global descriptors for retrieval. Compared with the original VLAD network, we propose a grouped fully connected layer to decompose the high-dimensional vectors into a group of low-dimensional vectors, which can reduce the number of parameters of the network and maintain the discrimination of the feature vector. Finally, we further develop a simple version of EPC-Net, called EPC-Net-L, which consists of two ProxyConv modules and one max pooling layer to aggregate global descriptors. By distilling the knowledge from EPC-Net, EPC-Net-L can obtain discriminative global descriptors for retrieval. Extensive experiments on the Oxford dataset and three in-house datasets demonstrate that our method achieves good results with lower parameters, FLOPs, GPU memory, and shorter inference time. Our code is available at https://github.com/fpthink/EPC-Net.

Multisource forest point cloud registration with semantic-guided keypoints and robust RANSAC mechanisms

The affinities of 177 nonameric peptides binding to the HLA-A*0201 molecule were measured using a FACS-based MHC stabilisation assay and analysed using chemometrics. Their structures were described by global and local descriptors, QSAR models were derived by genetic algorithm, stepwise regression and PLS. The global molecular descriptors included molecular connectivity chi indices, kappa shape indices, E-state indices, molecular properties like molecular weight and log P, and three-dimensional descriptors like polarizability, surface area and volume. The local descriptors were of two types. The first used a binary string to indicate the presence of each amino acid type at each position of the peptide. The second was also position-dependent but used five z-scales to describe the main physicochemical properties of the amino acids forming the peptides. The models were developed using a representative training set of 131 peptides and validated using an independent test set of 46 peptides. It was found that the global descriptors could not explain the variance in the training set nor predict the affinities of the test set accurately. Both types of local descriptors gave QSAR models with better explained variance and predictive ability. The results suggest that, in their interactions with the MHC molecule, the peptide acts as a complicated ensemble of multiple amino acids mutually potentiating each other.

Semantic context plays a significant role in image segmentation. However, few prior works have explored semantic contexts for 3D point cloud segmentation. In this paper, we propose a simple yet effective Point Context Encoding (PointCE) module to capture semantic contexts of a point cloud and adaptively highlight intermediate feature maps. We also introduce a Semantic Context Encoding loss (SCE-loss) to supervise the network to learn rich semantic context features. To avoid hyperparameter tuning and achieve better convergence performance, we further propose a geometric mean loss to integrate both SCE-loss and segmentation loss. Our PointCE module is general and lightweight, and can be integrated into any point cloud segmentation architecture to improve its segmentation performance with only marginal extra overheads. Experimental results on the ScanNet, S3DIS and Semantic3D datasets show that consistent and significant improvement can be achieved for several different networks by integrating our PointCE module.

The paper presents a learning-based method for computing a discriminative 3D point cloud descriptor for place recognition purposes. Existing methods, such as Point-NetVLAD, are based on unordered point cloud representation. They use PointNet as the first processing step to extract local features, which are later aggregated into a global descriptor. The PointNet architecture is not well suited to capture local geometric structures. Thus, state-of-the-art methods enhance vanilla PointNet architecture by adding different mechanism to capture local contextual information, such as graph convolutional networks or using hand-crafted features. We present an alternative approach, dubbed Min-kLoc3D, to compute a discriminative 3D point cloud descriptor, based on a sparse voxelized point cloud representation and sparse 3D convolutions. The proposed method has a simple and efficient architecture. Evaluation on standard benchmarks proves that MinkLoc3D outperforms current state-of-the-art. Our code is publicly available on the project website. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Recognition in open-world scenarios is an important and challenging field, where Vision-Language Pre-training paradigms have greatly impacted the 2D domain. This inspires a growing interest in introducing 2D pre-trained models, such as CLIP, into the 3D domain to enhance the ability of point cloud understanding. Considering the difference between discrete 3D point clouds and real-world 2D images, reducing the domain gap is crucial. Some recent works project point clouds onto a 2D plane to enable 3D zero-shot capabilities without training. However, this simplistic approach leads to an unclear or even distorted geometric structure, limiting the potential of 2D pre-trained models in 3D. To address the domain gap, we propose Point2Real, a training-free framework based on the realistic rendering technique to automate the transformation of the 3D point cloud domain into the Vision-Language domain. Specifically, Point2Real leverages a shape recovery module that devises an iterative ball-pivoting algorithm to convert point clouds into meshes, narrowing the gap in shape at first. To simulate photo-realistic images, a set of refined textures as candidates is applied for rendering, where the CLIP confidence is utilized to select the suitable one. Moreover, to tackle the viewpoint challenge, a heuristic multi-view adapter is implemented for feature aggregation, which exploits the depth surface as an effective indicator of view-specific discriminability for recognition. We conduct experiments on ModelNet10, ModelNet40, and ScanObjectNN datasets, and the results demonstrate that Point2Real outperforms other approaches in zero-shot and few-shot tasks by a large margin.

Global localization in a prior map is an important field in virtual and augmented reality systems, but it is always a challenge to conduct point cloud based localization in the large-scale scene prior map. A large-scale prior map usually means huge amount of calculation for point cloud processing, which leads to the long time required for global localization. To deal with this problem, we propose a fast point cloud global localization method based on point clouds segmentation and learning-based descriptor. On the one hand, cylindrical filtering, ground-point removal and point cloud segmentation are adopted to eliminate a large number of useless points and retain points with rich structures, which improves the efficiency of point cloud registration. On the other hand, reliable 3D point cloud descriptor, two-phase search strategy for place recognition and geometric consistency verification are used to ensure the localization accuracy. Experiments prove that the proposed method achieves good localization effect on both KITTI and MVSEC datasets. Under the condition of ensuring the high localization accuracy, the time for point clouds to complete the global localization is greatly reduced.

Point clouds represent one of the prevalent formats for 3D content. Distortions introduced at various stages in the point cloud processing pipeline affect the visual quality, altering their geometric composition, texture information, or both. Understanding and quantifying the impact of the distortion domain on visual quality is vital to driving rate optimization and guiding post-processing steps to improve the quality of experience. In this paper, we propose a multi-task guided multi-modality no reference metric (M3-Unity), which utilizes 4 types of modalities across attributes and dimensionalities to represent point clouds. An attention mechanism establishes inter/intra associations among 3D/2D patches, which can complement each other, yielding local and global features, to fit the highly nonlinear property of the human vision system. A multi-task decoder involving distortion type classification selects the best association among 4 modalities, aiding the regression task and enabling the in-depth analysis of the interplay between geometrical and textural distortions. Furthermore, our framework design and attention strategy enable us to measure the impact of individual attributes and their combinations, providing insights into how these associations contribute particularly in relation to distortion type. Extensive experimental results on 4 datasets consistently outperform the state-of-the-art metrics by a large margin. The code is available at https://github.com/cwi-dis/ACMMM2024-Oral.

Nowadays, pre-training big models on large-scale datasets has achieved great success and dominated many downstream tasks in natural language processing and 2D vision, while pre-training in 3D vision is still under development. In this paper, we provide a new perspective of transferring the pre-trained knowledge from 2D domain to 3D domain with Point-to-Pixel Prompting in data space and Pixel-to-Point distillation in feature space, exploiting shared knowledge in images and point clouds that display the same visual world. Following the principle of prompting engineering, Point-to-Pixel Prompting transforms point clouds into colorful images with geometry-preserved projection and geometry-aware coloring. Then the pre-trained image models can be directly implemented for point cloud tasks without structural changes or weight modifications. With projection correspondence in feature space, Pixel-to-Point distillation further regards pre-trained image models as the teacher model and distills pre-trained 2D knowledge to student point cloud models, remarkably enhancing inference efficiency and model capacity for point cloud analysis. We conduct extensive experiments in both object classification and scene segmentation under various settings to demonstrate the superiority of our method. In object classification, we reveal the important scale-up trend of Point-to-Pixel Prompting and attain 90.3% accuracy on ScanObjectNN dataset, surpassing previous literature by a large margin. In scene-level semantic segmentation, our method outperforms traditional 3D analysis approaches and shows competitive capacity in dense prediction tasks.