S 2 PU-Net: Sparse Semantic-Guided Progressive Point Cloud Upsampling for Indoor Scenes

Global Navigation Satellite System (GNSS) can provide accurate absolute position but requires fine observational conditions. In urban canyons and indoor scenes, GNSS suffers from severe performance degradation and is even unavailable. Thereby, to provide ubiquitous positioning solutions, it is very necessary to incorporate other complementary sensors. A popular option is the 3D LiDAR sensor, which emits its own light and thus is robust to illumination variation. 3D LiDAR sensors can efficiently capture plentiful point patterns of the ambient environment. Besides perception tasks, the obtained point clouds are also used for relative localization in a registration fashion, known as the LiDAR odometry (LO). Generally, LO is based on structural information, where edge and planar features are extensively exploited.In this work, we propose a novel method aiming to efficiently extract planes from the sparse and noisy 3D LiDAR point clouds. To fully exploit the scanning pattern of this sensor, our method follows a framework of point-to-line-to-plane. The point cloud is firstly projected onto a range image by investigating the azimuth and elevation of each point. In the point-to-line stage, consecutive flat points in a column are grouped into line segments, where a new flat point detector is introduced. In the line-to-plane stage, we extend the classical line extraction method, i.e., Douglas-Peucker algorithm, to find planes in line segments. Considering the over-segmentation caused by occlusion and deformation, we finally conduct region growing and merging to acquire more complete results. Most importantly, we bridge the measurement noise model and the parameter uncertainty via error propagation to determine reasonable thresholds throughout our method.We test the proposed method on datasets collected by various LiDAR sensors. The experiments are conducted in indoor scenes and urban scenes, which contains abundant planar objects such as walls and building facades. Three point-level metrics, namely positive predictive value (PPV), true positive rate (TPR) and F1 score, are employed for quantitative evaluation. The average PPV, TPR, F1 of the proposed method are 89.92%, 86.38% and 88.11%, respectively. The results show that the proposed method is able to recover the dominant planar structure, which is valuable to LO. Moreover, compared to the rotation rate of the LiDAR sensor, which is generally set to 10 Hz, the average runtime of the proposed method is 15.6 ms/frame, so it is qualified for online operation.

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

Although 3D Gaussian Splatting (3DGS) has greatly advanced the development of novel view synthesis (NVS) and surface reconstruction tasks, it still faces serious challenges in indoor scenes with only sparse views due to the lack of initialization point cloud. Existing GS-like methods are geometrically initialized using point clouds from Structure from Motion (SfM), and their performance is heavily limited by the point cloud quality. However, in indoor scenes containing large areas of textureless or weakly textured regions, the SfM algorithm struggles to generate point clouds in these areas, especially when only a limited number of input views are available. This situation can cause the GS optimization process to overfit the photometric error, resulting in severe geometric degradation in regions lacking proper initialization. To address the above problem, we propose a novel sparse-view 3DGS method using global scale-aligned monocular depth to initialize Gaussian primitives and optimize them in view space, named VGA-GS. First, we design a global scale-aligned monocular depth initialization strategy, which provides geometric priors in textureless or weakly textured regions while avoiding the local geometric distortions in vanilla alignments. Second, we propose a novel Gaussian primitive representation based on view space and a refinement strategy based on pixel errors, which can constrain each primitive to always be within the field of view where it is observed to be fully optimized. Finally, we extract the surface prior from monocular depth to regularize the rendering depth and normal, and further propagate the constraints of the training view to the neighboring pseudo-views via image warping. Our method achieves state-of-the-art reconstruction performance in both textureless regions and local details for indoor scenes with sparse views. Moreover, it is highly scalable, supporting integration with various 3DGS rasterizers and depth estimation methods. We release the code at https://github.com/XT5un/VGA-GS.

Indoor scene understanding is an active research topic that has recently gained increasing attention in both computer graphics and computer vision community. This paper presents a novel automatic point cloud instance abstraction and segmentation of indoor scenes using learned pairwise planar patch relations. Planar patches have sufficient representative power to abstract the semantics in mostly man-made 3D indoor scenes. To exploit the planar patch relationships, a new indoor point clouds dataset is automatically derived from the public dataset ScanNet, which can be adopted for training a pairwise patch relation classifier. Given the pre-processed indoor scene point clouds, a set of planar patches are generated to provide robust and compact representation of indoor objects and parts. For each extracted planar patch, a feature descriptor in view of its geometric properties and appearance is calculated. Owing to the proposed feature descriptors, the relations of pairwise planar patch are measured and can be fed into the pre-trained patch relation classifier. To yield the instance segmentation of indoor scenes, a robust graph-based algorithm is adopted to group these planar patches which incorporate the learned patch relations. Experimental results demonstrate that our proposed approach can produce instance semantic segmentation of various cluttered indoor scenes robustly and efficiently.

The fusion of laser point cloud and visual image depends on the point cloud density and the target framing effect, the traditional laser point cloud processing for sparse point cloud clustering effect is poor, it is difficult to frame small objects as well as medium and long distance objects. Then the subsequent sensor fusion is easy to miss the recognition of obstacles. In this paper, we improve the frame selection method for sparse point clouds, firstly build a deep learning framework pointpillar, use pointpillar to frame the sparse laser point clouds, then spatially calibrate the lidar coordinate system and camera coordinate system, project the lidar point clouds to the camera image, improve the late fusion method, effectively use the detection results of single sensor, and finally The late-fusion is performed with the target detection results of the camera image to output the exact distance as well as the category of the target. Experiments show that compared with the recognition effect of the traditional fusion algorithm, the number of frames is increased by 6 and the missed recognition rate is reduced from 31.41% to 12.31%.

Since the development of 3D applications, the point cloud, as a spatial description easily acquired by sensors, has been widely used in multiple areas such as SLAM and 3D reconstruction. Point Cloud Compression (PCC) has also attracted more attention as a primary step before point cloud transferring and saving, where the geometry compression is an important component of PCC to compress the points geometrical structures. However, existing non-learning-based geometry compression methods are often limited by manually pre-defined compression rules. Though learning-based compression methods can significantly improve the algorithm performances by learning compression rules from data, they still have some defects. Voxel-based compression networks introduce precision errors due to the voxelized operations, while point-based methods may have relatively weak robustness and are mainly designed for sparse point clouds. In this work, we propose a novel learning-based point cloud compression framework named 3D Point Cloud Geometry Quantiation Compression Network (3QNet), which overcomes the robustness limitation of existing point-based methods and can handle dense points. By learning a codebook including common structural features from simple and sparse shapes, 3QNet can efficiently deal with multiple kinds of point clouds. According to experiments on object models, indoor scenes, and outdoor scans, 3QNet can achieve better compression performances than many representative methods.

It is an important task to reconstruct surfaces from 3D point clouds. Current methods are able to reconstruct surfaces by learning Signed Distance Functions (SDFs) from single point clouds without ground truth signed distances or point normals. However, they require the point clouds to be dense, which dramatically limits their performance in real applications. To resolve this issue, we propose to reconstruct highly accurate surfaces from sparse point clouds with an on-surface prior. We train a neural network to learn SDFs via projecting queries onto the surface represented by the sparse point cloud. Our key idea is to infer signed distances by pushing both the query projections to be on the surface and the projection distance to be the minimum. To achieve this, we train a neural network to capture the on-surface prior to determine whether a point is on a sparse point cloud or not, and then leverage it as a differentiable function to learn SDFs from unseen sparse point cloud. Our method can learn SDFs from a single s parse point cloud without ground truth signed distances or point normals. Our numerical evaluation under widely used benchmarks demonstrates that our method achieves state-of-the-art reconstruction accuracy, especially for sparse point clouds. Code and data are available at https://github.com/mabaorui/OnSurfacePrior.

K-nearest neighbor (kNN) search is a fundamental operation in various point cloud applications, such as autonomous driving. However, the heavy computational intensity and memory demands of kNN search pose significant challenges for efficient implementation, especially in resource-constrained scenarios. To address these challenges, we propose PICK, a processing-in-memory (PIM) architecture designed to accelerate kNN search in point cloud applications. PICK leverages bit-serial-based PIM (BS-PIM) and customized circuits to efficiently handle key operations of kNN search: distance calculation and top-k selection. The run-time off-chip access is eliminated thanks to the large on-chip memory. For distance calculation, we introduce a bit-width clipping technique to reduce the latency of bit-serial execution with negligible accuracy degradation, providing flexible trade-offs between performance and precision. Besides, we propose a filtering-and-selection strategy that realizes approximately constant time complexity for arbitrary values of k. Furthermore, a two-stage pipeline is implemented to parallelize distance calculation and top-k search, effectively hiding latency and improving throughput. According to our experiments, PICK achieves $4.17 \times$ speedup and a $4.42 \times$ energy saving over the state-of-the-art design.

Accurate human activity recognition (HAR) is the key to enable emerging context-aware applications that require an understanding and identification of human behavior, e.g., monitoring disabled or elderly people who live alone. Traditionally, HAR has been implemented either through ambient sensors, e.g., cameras, or through wearable devices, e.g., a smartwatch, with an inertial measurement unit (IMU). The ambient sensing approach is typically more generalizable for different environments as this does not require every user to have a wearable device. However, utilizing a camera in privacy-sensitive areas such as a home may capture superfluous ambient information that a user may not feel comfortable sharing. Radars have been proposed as an alternative modality for coarse-grained activity recognition that captures a minimal subset of the ambient information using micro-Doppler spectrograms. However, training fine-grained, accurate activity classifiers is a challenge as low-cost millimeter-wave (mmWave) radar systems produce sparse and non-uniform point clouds. In this paper, we propose RadHAR, a framework that performs accurate HAR using sparse and non-uniform point clouds. RadHAR utilizes a sliding time window to accumulate point clouds from a mmWave radar and generate a voxelized representation that acts as input to our classifiers. We evaluate RadHAR using a low-cost, commercial, off-the-shelf radar to get sparse point clouds which are less visually compromising. We evaluate and demonstrate our system on a collected human activity dataset with 5 different activities. We compare the accuracy of various classifiers on the dataset and find that the best performing deep learning classifier achieves an accuracy of 90.47%. Our evaluation shows the efficacy of using mmWave radar for accurate HAR detection and we enumerate future research directions in this space.

With the increasing popularity of RGB-depth (RGB-D) sensor, research on the use of RGB-D sensors to reconstruct three-dimensional (3D) indoor scenes has gained more and more attention. In this paper, an automatic point cloud registration algorithm is proposed to efficiently handle the task of 3D indoor scene reconstruction using pan-tilt platforms on a fixed position. The proposed algorithm aims to align multiple point clouds using extrinsic parameters of the RGB-D camera obtained from every preset pan-tilt control point. A computationally efficient global registration method is proposed based on transformation matrices formed by the offline calibrated extrinsic parameters. Then, a local registration method, which is an optional operation in the proposed algorithm, is employed to refine the preliminary alignment result. Experimental results validate the quality and computational efficiency of the proposed point cloud alignment algorithm by comparing it with two state-of-the-art methods.

Lidar has received a lot of attention due to its precise ranging accuracy. Ground points filtering is an important task in point clouds processing. It’s a challenge to model the ground surface and filter the point clouds accurately in the case of complex ground undulations, occlusions, and sparse point clouds. A novel ground surface modeling method based on a hybrid regression technique is proposed in this paper. The method integrates Gaussian process regression (GPR) and robust locally weighted regression (RLWR) by dividing the point clouds that are projected on the polar grid map into radial and circumferential filtering processes to form a hybrid regression model, which has the ability to eliminate the influence of outliers and model the ground surface robustly. First, the RLWR combined with gradient filter is applied to fit the sampled points in the radial direction, which will exclude outliers and get the fitting ground line. All radial fitting lines constitute the seed skeleton of the whole plane. Then, based on the seeds in the same circumferential of the skeleton, the GPR is applied to construct the ground surface model. The comparative experiments are implemented quantitatively and qualitatively on the simulated point clouds and measured data. The results show that the proposed method performs well in most real scenarios, even in the cases of ground undulation, occlusion, and sparse point clouds.

Road intersection is a kind of important navigation landmark, while existing detection methods exhibit clear limitations in terms of their robustness and efficiency. A real-time algorithm for road intersection detection and location in large-scale sparse point clouds is proposed in this paper. Different from traditional approaches, our method establishes the augmented viewpoints beam model to perceive the road bifurcation structure. Explicitly, the spatial features from point clouds are jointly extracted in various viewpoints in front of the robot. In addition, the evaluation metrics are designed to self-assess the quality of detection results, enabling our method to optimize the detection process in real time. Considering the scarcity of datasets for intersection detection, we also collect and annotate a VLP-16 point cloud dataset specifically for intersections, called NCP-Intersection. Quantitative and qualitative experiments demonstrate that the proposed method performs favorably against the other parallel methods. Specifically, our method performs an average precision exceeding 90% and an average processing time of approximately 88 ms/frame.

Stockpile volume estimation plays a critical role in several industrial/commercial bulk material management applications. LiDAR systems are commonly used for this task. Thanks to Global Navigation Satellite System (GNSS) signal availability in outdoor environments, Uncrewed Aerial Vehicles (UAV) equipped with LiDAR are frequently adopted for the derivation of dense point clouds, which can be used for stockpile volume estimation. For indoor facilities, static LiDAR scanners are usually used for the acquisition of point clouds from multiple locations. Acquired point clouds are then registered to a common reference frame. Registration of such point clouds can be established through the deployment of registration targets, which is not practical for scalable implementation. For scans in facilities bounded by planar walls/roofs, features can be automatically extracted/matched and used for the registration process. However, monitoring stockpiles stored in dome facilities remains to be a challenging task. This study introduces an image-aided fine registration strategy of acquired sparse point clouds in dome facilities, where roof and roof stringers are extracted, matched, and modeled as quadratic surfaces and curves. These features are then used in a Least Squares Adjustment (LSA) procedure to derive well-aligned LiDAR point clouds. Planar features, if available, can also be used in the registration process. Registered point clouds can then be used for accurate volume estimation of stockpiles. The proposed approach is evaluated using datasets acquired by a recently developed camera-assisted LiDAR mapping platform—Stockpile Monitoring and Reporting Technology (SMART). Experimental results from three datasets indicate the capability of the proposed approach in producing well-aligned point clouds acquired inside dome facilities, with a feature fitting error in the 0.03–0.08 m range.

Purpose Sensors are vital for autonomous driving, but they are susceptible to weather noise. To address the performance degradation of point cloud object detection in foggy conditions, this paper aims to propose an efficient dynamic pillar-based point cloud detection network tailored for foggy traffic scenarios. Design/methodology/approach The point cloud data are first pillarized, and then the model adaptively adjusts the feature extraction process through DynamicPillarVFE to significantly enhance the processing capability for sparse and irregular point cloud data. Second, a bird's eye view (BEV) backbone network based on residual connections is used to effectively mitigate the gradient vanishing problem in deep networks. In addition, this paper introduces Triplet Attention (TA) in the feature enhancement part, which can focus on important features more comprehensively and suppress noise interference. Findings In this paper, we mixed a small portion of the Multifog KITTI dataset on top of the KITTI dataset. Experimental results demonstrate that under moderate difficulty 3D evaluation metrics, our method achieves accuracy improvements of 5.03%, 6.66% and 7.78% for cars, pedestrians and cyclists, respectively, compared to the PointPillars baseline, significantly enhancing point cloud object detection performance in foggy conditions. Furthermore, the DPSF-Net architecture achieves an inference speed of 32.36 ms per frame, fully meeting the real-time processing requirements for autonomous driving applications. Originality/value This method is more economical to simulate the influence of sensors under foggy conditions, and through the training of mixed data sets, the network model can better cope with foggy interference.

Current 3D object detection methods achieve accurate and efficient results on the standard point cloud dataset. However, in real-world applications, the point cloud samples obtained in the real-time running may be much sparser due to various reasons (occlusion, low reflectivity of objects and fewer laser beams) and existing methods do not consider the limitations of their models on sparse point clouds. To improve the robustness of an object detector to sparser point clouds, we propose PointDrop, which learns to drop the features of some key points in the point clouds to generate challenging sparse samples for data augmentation. Moreover, PointDrop is able to adjust the difficulty of the generated samples based on the capacity of the detector and thus progressively improve the performance of the detector. We create two sparse point clouds datasets from the KITTI dataset to evaluate our method, and the experimental results show that PointDrop significantly improves the robustness of the detector to sparse point clouds.