Point Cloud Inpainting on Graphs from Non-Local Self-Similarity

Point cloud is an important and commonly used signal representation for volume objects or scenes in the real world. Due to the imperfect acquisition of the point cloud, there is nonnegligible noise in the point cloud. Most literatures that use graph signal processing (GSP) for point cloud denoising (PCD) generally construct k-NN graph to represent the point cloud. However, the graph constructed based on this scheme can not compactly represent the underlying structure of a noisy point cloud. In this paper, we propose a feature graph that can effectively and naturally represent the structure of the point cloud. To construct the feature graph, a feature sampling method is exploited to obtain the feature points. Then, patches are built based on the feature points. After that, the feature graph is constructed by connecting all the points in the patches. Finally, we apply the feature graph to the PCD problem and exploit graph Laplacian regularization (GLR) as smoothing prior information for denoising. Experimental results show that our proposed PCD method not only outperforms the existing PCD methods in objective evaluation metrics, but also performs better in processing the inner and edge structure of the point cloud.

Point cloud is a collection of 3D coordinates that are discrete geometric samples of an object's 2D surfaces. Imperfection in the acquisition process means that point clouds are often corrupted with noise. Building on recent advances in graph signal processing, we design local algorithms for 3D point cloud denoising. Specifically, we design a signal-dependent feature graph Laplacian regularizer (SDFGLR) that assumes surface normals computed from point coordinates are piecewise smooth with respect to a signal-dependent graph Laplacian matrix. Using SDFGLR as a signal prior, we formulate an optimization problem with a general 'p-norm fidelity term that can explicitly remove only two types of additive noise: small but non-sparse noise like Gaussian (using '2 fidelity term) and large but sparser noise like Laplacian (using '1 fidelity term). To establish a linear relationship between normals and 3D point coordinates, we first perform bipartite graph approximation to divide the point cloud into two disjoint node sets (red and blue). We then optimize the red and blue nodes' coordinates alternately. For '2-norm fidelity term, we iteratively solve an unconstrained quadratic programming (QP) problem, efficiently computed using conjugate gradient with a bounded condition number to ensure numerical stability. For '1-norm fidelity term, we iteratively minimize an '1-'2 cost function using accelerated proximal gradient (APG), where a good step size is chosen via Lipschitz continuity analysis. Finally, we propose simple mean and median filters for flat patches of a given point cloud to estimate the noise variance given the noise type, which in turn is used to compute a weight parameter trading off the fidelity term and signal prior in the problem formulation. Extensive experiments show state-of-the-art denoising performance among local methods using our proposed algorithms.

For point cloud data obtained from 3D scanning devices, excessively large storage and long post-processing time are required. Due to this, it is very important to simplify the point cloud to reduce calculation cost. In this paper, we propose a new point cloud simplification method that can maintain the characteristics of surface shape for unstructured point clouds. In our method, a segmentation range based on mean curvature of point cloud can be controlled. The simplification process is completed by maintaining the position of the representative point and removing the represented points using the range. Our method can simplify results with highly simplified rate with preserving the form feature. Applying the proposed method to 3D stone tool models, the method is evaluated precisely and effectively.

In the acquisition process of 3D cultural relics, it is common to encounter noise. To facilitate the generation of high-quality 3D models, we propose an approach based on graph signal processing that combines color and geometric features to denoise the point cloud. We divide the 3D point cloud into patches based on self-similarity theory and create an appropriate underlying graph with a Markov property. The features of the vertices in the graph are represented using 3D coordinates, normal vectors, and color. We formulate the point cloud denoising problem as a maximum a posteriori (MAP) estimation problem and use a graph Laplacian regularization (GLR) prior to identifying the most probable noise-free point cloud. In the denoising process, we moderately simplify the 3D point to reduce the running time of the denoising algorithm. The experimental results demonstrate that our proposed approach outperforms five competing methods in both subjective and objective assessments. It requires fewer iterations and exhibits strong robustness, effectively removing noise from the surface of cultural relic point clouds while preserving fine-scale 3D features such as texture and ornamentation. This results in more realistic 3D representations of cultural relics.

Sharpness fields in point clouds using deep learning

Point cloud is emerged as a promising media format to represent realistic 3D objects or scenes in applications, such as virtual reality, teleportation, etc. How to accurately quantify the subjective point cloud quality for application-driven optimization, however, is still a challenging and open problem. In this paper, we attempt to tackle this problem in a systematic means. First, we produce a fairly large point cloud dataset where ten popular point clouds are augmented with seven types of impairments (e.g., compression, photometry/color noise, geometry noise, scaling) at six different distortion levels, and organize a formal subjective assessment with tens of subjects to collect mean opinion scores (MOS) for all 420 processed point cloud samples (PPCS). We then try to develop an objective metric that can accurately estimate the subjective quality. Towards this goal, we choose to project the 3D point cloud onto six perpendicular image planes of a cube for the color texture image and corresponding depth image, and aggregate image-based global (e.g., Jensen-Shannon (JS) divergence) and local features (e.g., edge, depth, pixel-wise similarity, complexity) among all projected planes for a final objective index. Model parameters are fixed constants after performing the regression using a small and independent dataset previously published. The proposed metric has demonstrated the state-of-the-art performance for predicting the subjective point cloud quality compared with multiple full-reference and no-reference models, e.g., the weighted peak signal-to-noise ratio (PSNR), structural similarity (SSIM), feature similarity (FSIM) and natural image quality evaluator (NIQE). The dataset is made publicly accessible at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://smt.sjtu.edu.cn</uri> or <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://vision.nju.edu.cn</uri> for all interested audiences.

With the increasing attention in various 3D safety-critical applications, point cloud learning models have been shown to be vulnerable to adversarial attacks. Although existing 3D attack methods achieve high success rates, they delve into the data space with point-wise perturbation, which may neglect the geometric characteristics. Instead, we propose point cloud attacks from a new perspective-the graph spectral domain attack, aiming to perturb graph transform coefficients in the spectral domain that correspond to varying certain geometric structures. Specifically, leveraging on graph signal processing, we first adaptively transform the coordinates of points onto the spectral domain via graph Fourier transform (GFT) for compact representation. Then, we analyze the influence of different spectral bands on the geometric structure, based on which we propose to perturb the GFT coefficients via a learnable graph spectral filter. Considering the low-frequency components mainly contribute to the rough shape of the 3D object, we further introduce a low-frequency constraint to limit perturbations within imperceptible high-frequency components. Finally, the adversarial point cloud is generated by transforming the perturbed spectral representation back to the data domain via the inverse GFT. Experimental results demonstrate the effectiveness of the proposed attack in terms of both the imperceptibility and attack success rates.

We present a fast point cloud clustering technique which is suitable for outlier detection, object segmentation and region labeling for large multi-dimensional data sets. The basis is a minimal data structure similar to a kd-tree which enables us to detect connected subsets very fast. The proposed algorithms utilizing this tree structure are parallelizable which further increases the computation speed for very large data sets. The procedures given are a vital part of the data preprocessing. They improve the input data properties for a more reliable computation of surface measures, polygonal meshes and other visualization techniques. In order to show the effectiveness of our techniques we evaluate sets of point clouds from different 3D scanning devices.

With the growth of 3D sensing technology, the deep learning system for 3D point clouds has become increasingly important, especially in applications such as autonomous vehicles where safety is a primary concern. However, there are growing concerns about the reliability of these systems when they encounter noisy point clouds, either occurring naturally or introduced with malicious intent. This paper highlights the challenges of point cloud classification posed by various forms of noise, from simple background noise to malicious adversarial/backdoor attacks that can intentionally skew model predictions. While there's an urgent need for optimized point cloud denoising, current point outlier removal approaches, an essential step for denoising, rely heavily on handcrafted strategies and are not adapted for higher-level tasks, such as classification. To address this issue, we introduce an innovative point outlier cleansing method that harnesses the power of downstream classification models. Using gradient-based attribution analysis, we define a novel concept: point risk. Drawing inspiration from tail risk minimization in finance, we recast the outlier removal process as an optimization problem, named PointCVaR. Extensive experiments show that our proposed technique not only robustly filters diverse point cloud outliers but also consistently and significantly enhances existing robust methods for point cloud classification. A notable feature of our approach is its effectiveness in defending against the latest threat of backdoor attacks in point clouds.

Most UAVs depend on realtime 3D point clouds to navigate through unknown environments. However, both point clouds processing and trajectory planning are computationally expensive and will deplete UAV's battery quickly. There are also inevitable uncertainties in point clouds, which further makes collision-free trajectory planning a very challenging problem. To address these issues, we propose an <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>e</u></b> nergy-efficient and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>c</u></b> loud-assisted aut <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>onom</u></b> ous navigation s <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><u>y</u></b> stem, called ECONOMY, which allows a UAV to transmit the point clouds through a cellular network to a computing cloud that plans the trajectories for the UAV in realtime. To maximize the UAV's energy-efficiency, we jointly optimize its velocity, trajectory, and transmission power. Since the formulated problem is non-convex, we decompose it into a UAV trajectory and velocity optimization problem (UTVO), and a UAV communication optimization problem (UCO), which are resolved by an intelligent solver built upon the Suggest-and-Improve framework, and gradient ascent, respectively. To address point clouds uncertainties, we devise probabilistic collision-free constraints which can be handled in a deterministic manner by exploiting the tight <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$(n,2,\mathcal {R}^n)$</tex-math></inline-formula> -upper bound for convex sets. Simulation results for a UAV exploring a simulated urban environment demonstrate the efficacy and efficiency of ECONOMY.

An increased interest in immersive applications has drawn attention to\nemerging 3D imaging representation formats, notably light fields and point\nclouds (PCs). Nowadays, PCs are one of the most popular 3D media formats, due\nto recent developments in PC acquisition, namely with new depth sensors and\nsignal processing algorithms. To obtain high fidelity 3D representations of\nvisual scenes a huge amount of PC data is typically acquired, which demands\nefficient compression solutions. As in 2D media formats, the final perceived PC\nquality plays an important role in the overall user experience and, thus,\nobjective metrics capable to measure the PC quality in a reliable way are\nessential. In this context, this paper proposes and evaluates a set of\nobjective quality metrics for the geometry component of PC data, which plays a\nvery important role in the final perceived quality. Based on the popular PSNR\nPC geometry quality metric, the novel improved PSNR-based metrics are proposed\nby exploiting the intrinsic PC characteristics and the rendering process that\nmust occur before visualization. The experimental results show the superiority\nof the best-proposed metrics over the state-of-the-art, obtaining an\nimprovement of up to 32% in the Pearson correlation coefficient.\n

Point clouds have evolved into one of the most important data formats for 3D representation. It is becoming more popular as a result of the increasing affordability of acquisition equipment and growing usage in a variety of fields. Volumetric grid-based approaches are among the most successful models for processing point clouds because they fully preserve data granularity while additionally making use of point dependency. However, using lower order local estimate functions to close 3D objects, such as the piece-wise constant function, necessitated the use of a high-resolution grid in order to capture detailed features that demanded vast computational resources. This study proposes an improved fused feature network as well as a comprehensive framework for solving shape classification and segmentation tasks using a two-branch technique and feature learning. We begin by designing a feature encoding network with two distinct building blocks: layer skips within, batch normalization (BN), and rectified linear units (ReLU) in between. The purpose of using layer skips is to have fewer layers to propagate across, which will speed up the learning process and lower the effect of gradients vanishing. Furthermore, we develop a robust grid feature extraction module that consists of multiple convolution blocks accompanied by max-pooling to represent a hierarchical representation and extract features from an input grid. We overcome the grid size constraints by sampling a constant number of points in each grid using a simple K-points nearest neighbor (KNN) search, which aids in learning approximation functions in higher order. The proposed method outperforms or is comparable to state-of-the-art approaches in point cloud segmentation and classification tasks. In addition, a study of ablation is presented to show the effectiveness of the proposed method.

In this paper, we propose an algorithm for the registration of the GPS sensor and the stereo camera for vehicle localization within 3D dense point clouds. We adopt the particle swarm optimization algorithm to perform the sensor registration and the vehicle localization. The registration of the GPS sensor and the stereo camera is performed to increase the robustness of the vehicle localization algorithm. In the standard GPS-based vehicle localization, the algorithm is affected by noisy GPS signals in certain environmental conditions. We can address this problem through the sensor fusion or registration of the GPS and stereo camera. The sensors are registered by estimating the coordinate transformation matrix. Given the registration of the two sensors, we perform the point cloud-based vehicle localization. The vision-based localization is formulated as an optimization problem, where the “optimal” transformation matrix and corresponding virtual point cloud depth image is estimated. The transformation matrix, which is optimized, corresponds to the coordinate transformation between the stereo and point cloud coordinate systems. We validate the proposed algorithm with acquired datasets, and show that the algorithm robustly localizes the vehicle.

3D point cloud super-resolution (PCSR) plays an important role in many applications, which can infer a dense geometric shape from a sparse one. However, existing PCSR methods only leverage the geometric properties to predict dense geometric coordinates without considering the importance of correlated attributes in the prediction of complex geometric structures. In this paper, we propose a novel PCSR network by leveraging color attributes to improve the reconstruction quality of dense geometric shape. In the proposed network, we utilize graph convolutions to obtain cross-domain structure representation for point cloud from both geometric coordinates and color attributes, which is constructed by aggregating local points based on the similarity of cross-domain features. Furthermore, we propose a shape-aware loss function to cooperate with network training, which constrains the point cloud generation from both overall and detailed aspects. Extensive experimental results show that our proposed method outperforms the state-of-the-art methods from both objective and subjective quality.

Point clouds have been gaining importance as a solution to the problem of efficient representation of 3D geometric and visual information. They are commonly represented by large amounts of data, and compression schemes are important for their manipulation transmission and storing. However, the selection of appropriate compression schemes requires effective quality evaluation. In this work a subjective quality evaluation of point clouds using a surface representation is analyzed. Using a set of point cloud data objects encoded with the popular octree pruning method with different qualities, a subjective evaluation was designed. The point cloud geometry was presented to observers in the form of a movie showing the 3D Poisson reconstructed surface without textural information with the point of view changing in time. Subjective evaluations were performed in three different laboratories. Scores obtained from each test were correlated and no statistical differences were observed. Scores were also correlated with previous subjective tests and a good correlation was obtained when compared with mesh rendering in 2D monitors. Moreover, the results were correlated with state of the art point cloud objective metrics revealing poor correlation. Likewise, the correlation with a subjective test using a different representation of the point cloud data also showed poor correlation. These results suggest the need for more reliable objective quality metrics and further studies on adequate point cloud data representations.