Compression and Rendering of Textured Point Clouds via Sparse Coding

3D point cloud presentation has been widely used in computer vision, automatic driving, augmented reality, smart cities and virtual reality. 3D point cloud compression method with higher compression ratio and tiny loss is the key to improve data transportation efficiency. In this paper, we propose a new 3D point cloud geometry compression method based on deep learning, also an auto-encoder performing better than other networks in detail reconstruction. It can reach much higher compression ratio than the state-of-art while keeping tolerable loss. It also supports parallel compressing multiple models by GPU, which can improve processing efficiency greatly. The compression process is composed of two parts. Firstly, Raw data is compressed into codeword by extracting feature of raw model with encoder. Then, the codeword is further compressed with sparse coding. Decompression process is implemented in reverse order. Codeword is recovered and fed into decoder to reconstruct point cloud. Detail reconstruction ability is improved by a hierarchical structure in our decoder. Latter outputs are grown from former fuzzier outputs. In this way, details are added to former output by latter layers step by step to make a more precise prediction. We compare our method with PCL compression and Draco compression on ShapeNet40 part dataset. Our method may be the first deep learning-based point cloud compression algorithm. The experiments demonstrate it is superior to former common compression algorithms with large compression ratio, which can also reserve original shapes with tiny loss.

With the promptly growing population of the low-cost Microsoft Kinect sensor, action recognition, which is a hard yet important problem in computer vision, has been received substantial attention. However, most existing approaches in action recognition spend much time on feature detection even though these methods can achieve high recognition rates. In this paper, we propose a local mean spatio-temporal feature (LMSF) to speed up depth image based action recognition. In particular, we solve the problem from three aspects: (1) associate the 4D normals by a local mean spatio-temporal neighborhood; (2) extract motion frames by detecting the differences between consecutive frames; (3) reduce redundant normals extracted from depth cloud points by sparse coding. The proposed approach is tested on two public benchmark datasets, i.e., MSRAction3D and MSRGesture3D. Experimental results demonstrate the advantages of our improvement method and the state-of-the-art performance on processing speed.

Deep neural networks for 3D point cloud classification, such as PointNet, have been demonstrated to be vulnerable to adversarial attacks. Current adversarial defenders often learn to denoise the (attacked) point clouds by reconstruction, and then feed them to the classifiers as input. In contrast to the literature, we propose a novel bilevel optimization framework for robust point cloud classification, where the internal optimization can effectively defend the adversarial attacks as denoising and the external optimization can learn the classifiers accordingly. As a demonstration, we further propose an effective and efficient instantiation of our approach, namely, Lattice Point Classifier (LPC), based on structured sparse coding in the permutohedral lattice and 2D convolutional neural networks (CNNs) that integrates both internal and external optimization problems into end-to-end trainable network architectures. We demonstrate state-of-the-art robust point cloud classification performance on ModelNet40 and ScanNet under seven different attackers. Our demo code is available at: https://github.com/Zhang-VISLab .

Abstract. The aim of this article is to demonstrate the possibility of using photogrammetry as a method for documenting and digitizing data reconstructed from a display hologram. We discuss the importance of the task of digitizing information from holograms and the search for its solution, which will allow display holography to gain new distribution. Textured point clouds were obtained for the object and for its image reconstructed from the display reflective type hologram using monochromatic radiation. The requirements for the experimental setup for the digitization of the reconstructed from hologram object wavefront, as well as for the number of photographs for the hologram capturing are formulated and explained. The calculated three-dimensional models of the object and the reconstructed object wavefront were compared according to the Pearson correlation criterion. The achieved correlation value is 0.903. The results of comparison of two 3D models in the estimation of the standard deviation are given.

In this paper, a method of acquiring intraoperative data using a laser range scanner (LRS) is presented within the context of model-updated image-guided surgery. Registering textured point clouds generated by the LRS to tomographic data is explored using established point-based and surface techniques as well as a novel method that incorporates geometry and intensity information via mutual information (SurfaceMI). Phantom registration studies were performed to examine accuracy and robustness for each framework. In addition, an in vivo registration is performed to demonstrate feasibility of the data acquisition system in the operating room. Results indicate that SurfaceMI performed better in many cases than point-based (PBR) and iterative closest point (ICP) methods for registration of textured point clouds. Mean target registration error (TRE) for simulated deep tissue targets in a phantom were 1.0 +/- 0.2, 2.0 +/- 0.3, and 1.2 +/- 0.3 mm for PBR, ICP, and SurfaceMI, respectively. With regard to in vivo registration, the mean TRE of vessel contour points for each framework was 1.9 +/- 1.0, 0.9 +/- 0.6, and 1.3 +/- 0.5 for PBR, ICP, and SurfaceMI, respectively. The methods discussed in this paper in conjunction with the quantitative data provide impetus for using LRS technology within the model-updated image-guided surgery framework.

3D vehicle shape completion is a key task in urban environment reconstruction, since the available sensors and 3D scanning procedures (such as Mobile Laser Scanning) in an outdoor city scene cannot usually extract the entire car shapes, which can be necessary for specifying their geometric properties for further analysis or realistic visualization. In this paper, we propose a novel multi-view based 3D object point cloud completion technique. In contrast to existing approaches, our method operates on 2D images formed by projecting the point cloud from several virtual camera positions around the object of interest. Both color and geometrical information is considered during the process, generating dense textured point clouds, displaying realistic patterns in the missing regions from the partial inputs. We present both quantitative and qualitative tests on various synthetic and real laser scanned vehicle point clouds, which demonstrate that our method surpasses existing state-of-the-art approaches. By applying it to vehicles from the Shapenet dataset, our approach outperforms recent techniques in terms of Earth Mover’s Distance (EMD) and Chamfer Distance (CD) by 43.8% and 12.17%, respectively.

This paper reports a novel mutual information (MI) based algorithm for automatic registration of unstructured 3D point clouds comprised of co-registered 3D lidar and camera imagery. The proposed method provides a robust and principled framework for fusing the complementary information obtained from these two different sensing modalities. High-dimensional features are extracted from a training set of textured point clouds (scans) and hierarchical k-means clustering is used to quantize these features into a set of codewords. Using this codebook, any new scan can be represented as a collection of codewords. Under the correct rigid-body transformation aligning two overlapping scans, the MI between the codewords present in the scans is maximized. We apply a James-Stein-type shrinkage estimator to estimate the true MI from the marginal and joint histograms of the codewords extracted from the scans. Experimental results using scans obtained by a vehicle equipped with a 3D laser scanner and an omnidirectional camera are used to validate the robustness of the proposed algorithm over a wide range of initial conditions. We also show that the proposed method works well with 3D data alone.