Differential Transform for Video-Based Plenoptic Point Cloud Coding.

Efficient point cloud compression is essential for applications like virtual and mixed reality, autonomous driving, and cultural heritage. This paper proposes a deep learning-based inter-frame encoding scheme for dynamic point cloud geometry compression. We propose a lossy geometry compression scheme that predicts the latent representation of the current frame using the previous frame by employing a novel feature space inter-prediction network. The proposed network utilizes sparse convolutions with hierarchical multiscale 3D feature learning to encode the current frame using the previous frame. The proposed method introduces a novel predictor network for motion compensation in the feature domain to map the latent representation of the previous frame to the coordinates of the current frame to predict the current frame's feature embedding. The framework transmits the residual of the predicted features and the actual features by compressing them using a learned probabilistic factorized entropy model. At the receiver, the decoder hierarchically reconstructs the current frame by progressively rescaling the feature embedding. The proposed framework is compared to the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG). The proposed method achieves more than 88% BD-Rate (Bjøntegaard Delta Rate) reduction against G-PCCv20 Octree, more than 56% BD-Rate savings against G-PCCv20 Trisoup, more than 62% BD-Rate reduction against V-PCC intra-frame encoding mode, and more than 52% BD-Rate savings against V-PCC P-frame-based inter-frame encoding mode using HEVC. These significant performance gains are cross-checked and verified in the MPEG working group.

In recent years, 3D point cloud content has gained attention due to its application possibilities, such as multimedia systems, virtual, augmented, and mixed reality, through the mapping and visualization of environments and/or 3D objects, real-time immersive communications, and autonomous driving systems. However, raw point clouds demand a large amount of data for their representation, and compression is mandatory to allow efficient transmission and storage. The MPEG group proposed the Video-based Point Cloud Compression (V-PCC) standard, which is a dynamic point cloud encoder based on the use of video encoders through projections into 2D space. However, V-PCC demands a high computational cost, demanding fast implementations for real-time processing and, especially, for mobile device applications. In this paper, a machine-learning-based fast implementation of V-PCC is proposed, where the main approach is the use of trained decision trees to speed up the block partitioning process during the point cloud compression. The results show that the proposed fast V-PCC solution is able to achieve an encoding time reduction of 42.73% for the geometry video sub-stream and 55.3% for the attribute video sub-stream, with a minimal impact on bitrate and objective quality.

Video-based point cloud compression (V-PCC) has achieved remarkable performance in dynamic point cloud compression. However, due to the 3D-to-2D projection in V-PCC, the temporal continuity across projected video frames is broken, limiting inter-frame coding efficiency. In this paper, we propose a virtual reference-based predictive coding (VRPC) scheme for V-PCC attribute compression to improve coding efficiency. This coding scheme uses the temporal correspondence between point clouds to guide the alignment of attribute video frames for virtual reference frame generation. Specifically, we first construct a neural deformation field-based geometry alignment module to align the reference and target point clouds to reduce the effect of motion on temporal correspondence estimation. Then, temporal correspondence is obtained in 3D space and used to align the reference and target video frames to generate the occupied region of the virtual reference frame. Finally, after padding the unoccupied region, the resulting reference frame is added to the reference picture list and used by the target frame for inter-frame coding. Experimental results demonstrate that the proposed method effectively improves the attribute coding efficiency of V-PCC.

The dynamic point cloud is widely needed in 3D vision related applications such as virtual reality and telepresence. Due to the huge amount of data, a key technology before the effective application is the dynamic point cloud compression. The state-of-the-art dynamic point cloud compression scheme, video-based point cloud compression (V-PCC), generates 2D videos with some uncorrelation due to the patch segmentation and packing process, which will affect the compression efficiency. In this paper, we propose a Packing with Patch Correlation Improvement (PPCI) algorithm to adaptively remove the uncorrelated parts between matched patches in packing for the sake of inter-prediction performance. We first propose a basic unidirectional patch re-segmentation operator to remove the uncorrelated parts of the patches in the current point cloud relative to the patches in its reference point cloud. The removed parts will be formed as new patches and added to the patch collection of the current point cloud. Then we propose a back-and-forth structure, which is a combination of several basic patch re-segmentation operators, to bilaterally remove the uncorrelated parts of matched patches in a back-and-forth (BF) unit. Furthermore, we propose a framework to adaptively decide the best length of each BF unit in a point cloud sequence. Experimental results show that our method achieves noticeable bitrate savings compared with the existing V-PCC packing methods, particularly for sequences with small motion.

Point clouds are a versatile 3D data representation format that captures spatial and attribute information about objects or environments. It has been widely used in applications like autonomous driving, augmented and virtual reality, and 3D scanning. However, the large volume of data involved in point cloud processing poses challenges in terms of storage, transmission, and real-time processing. The Video-based Point Cloud Compression (V-PCC) standard addresses these challenges by employing 2D video compression techniques to encode dynamic 3D point clouds. Despite its effectiveness, V-PCC demands a high computational cost, particularly in encoding geometry and attribute video sub-streams. This paper presents a machine learning-based approach to accelerate the V-PCC encoding, focusing on the Coding Unit partitioning process in the geometry and attribute sub-streams deployed in the reference software (TMC2). The proposed method significantly reduces the encoding complexity by using decision tree models with negligible impact on coding efficiency. Experimental results demonstrate an average encoding time reduction of 36.12% for geometry and attribute sub-streams, with a minimal impact on coding efficiency between -0.43% and 0.83% in terms of BD-Rate.

Video-based Point Cloud Compression (V-PCC) standard offers state-of-the-art tools and efficiency for volumetric video compression. The V-PCC reference software, TMC2, is able to demonstrate the compression efficiency of V-PCC, but with computational complexity that is impractical for real-life applications. This paper introduces a novel open-source V-PCC encoder, named uvgVPCCenc, that brings V-PCC compression speed to a practical level. To reduce computational bottlenecks, uvgVPCCenc is implemented from the ground up in C++ with advanced multi-threading and streamlined algorithms. It also supports a flexible parameterization and different presets that facilitate its adaptation to various content and target bitrates. Our comparative evaluations show that uvgVPCCenc achieves substantial speedups over TMC2 while maintaining competitive rate-distortion-complexity tradeoff. To the best of our knowledge, uvgVPCCenc is the first open-source encoder explicitly designed for practical V-PCC compression, paving the way from theoretical research to practical volumetric video applications.

Immersive media representation format based on point clouds has underpinned significant opportunities for extended reality applications. Point cloud in its uncompressed format require very high data rate for storage and transmission. The video based point cloud compression (V-PCC) technique projects a dynamic point cloud into geometry and texture video sequences. The projected texture video is then coded using modern video coding standard like HEVC. Since the properties of projected texture video frames are different from traditional video frames, HEVC-based commonality modeling can be inefficient. An improved commonality modeling technique is proposed that employs edge position difference oriented motion model. Experimental results show that the proposed commonality modeling technique can yield savings in bit rate of up to 3.15% over the V-PCC HEVC reference encoder.

This paper presents a novel method to determine rate-distortion optimized\ntransform coefficients for efficient compression of videos generated from point\nclouds. The method exploits a generalized frequency selective extrapolation\napproach that iteratively determines rate-distortion-optimized coefficients for\nall basis functions of two-dimensional discrete cosine and sine transforms. The\nmethod is applied to blocks containing both occupied and unoccupied pixels in\nvideo based point cloud compression for HEVC encoding. In the proposed\nalgorithm, only the values of the transform coefficients are changed such that\nresulting bit streams are compliant to the V-PCC standard. For all-intra coded\npoint clouds, bitrate savings of more than 4% for geometry and more than 6% for\ntexture error metrics with respect to standard encoding can be observed. These\nsavings are more than twice as high as savings obtained using competing methods\nfrom literature. In the randomaccess case, our proposed method outperforms\ncompeting V-PCC methods by more than 0.5%.\n

We present a study on the impact of packet loss on dynamic 3D point cloud streaming, encoded with MPEG Video-based Point Cloud Compression (V-PCC) standard. We show the distortion when different channels of V-PCC bitstream are lost, with the loss of occupancy and geometry data impacting the quality most significantly. Our results point to the need for better error concealment techniques. We end the paper by presenting preliminary thoughts and experimental results of two naive error concealment techniques in the point cloud domain, for attributes and geometry data, respectively, and highlight the limitations of each.

With the maturity of 3D capture technology, the explosive growth of point cloud data has burdened the storage and transmission process. Traditional hybrid point cloud compression (PCC) tools relying on handcrafted priors have limited compression performance and are increasingly weak in addressing the burden induced by data growth. Recently, deep learning-based PCC methods have been introduced to continue to push the PCC performance boundary. With the thriving of deep PCC, the community urgently demands a systematic overview to conclude the past progress and present future research directions. In this paper, we have a detailed review that covers popular point cloud datasets, algorithm evolution, benchmarking analysis, and future trends. Concretely, we first introduce several widely-used PCC datasets according to their major properties. Then the algorithm evolution of existing studies on deep PCC, including lossy ones and lossless ones proposed for various point cloud types, is reviewed. Apart from academic studies, we also investigate the development of relevant international standards (i.e., MPEG standards and JPEG standards). To help have an in-depth understanding of the advance of deep PCC, we select a representative set of methods and conduct extensive experiments on multiple datasets. Comprehensive benchmarking comparisons and analysis reveal the pros and cons of previous methods. Finally, based on the profound analysis, we highlight the challenges and future trends of deep learning-based PCC, paving the way for further study.

In video-based point cloud compression (V-PCC), occupancy map video is utilized to indicate whether a 2-D pixel corresponds to a valid 3-D point or not. In the current design of V-PCC, the occupancy map video is directly compressed losslessly with High Efficiency Video Coding (HEVC). However, the coding tools in HEVC are specifically designed for natural images, thus unsuitable for the occupancy map. In this paper, we present a novel quadtree-based scheme for lossless occupancy map coding. In this scheme, the occupancy map is firstly divided into several coding tree units (CTUs). Then, the CTU is divided into coding units (CUs) recursively using a quadtree. The quadtree partition is terminated when one of the three conditions is satisfied. Firstly, all the pixels have the same value. Secondly, the pixels in the CU only have two kinds of values and they can be separated by a continuous edge whose endpoints lie on the side of the CU. The continuous edge is then coded using chain code. Thirdly, the CU reaches the minimum size. This scheme simplifies the design of block partitioning in HEVC and designs simpler yet more effective coding tools. Experimental results show significant reduction of bit-rate and complexity compared with the occupancy map coding scheme in V-PCC. In addition, this scheme is also very efficient to compress the semantic map.

Since the development of 3D applications, the point cloud, as a spatial description easily acquired by sensors, has been widely used in multiple areas such as SLAM and 3D reconstruction. Point Cloud Compression (PCC) has also attracted more attention as a primary step before point cloud transferring and saving, where the geometry compression is an important component of PCC to compress the points geometrical structures. However, existing non-learning-based geometry compression methods are often limited by manually pre-defined compression rules. Though learning-based compression methods can significantly improve the algorithm performances by learning compression rules from data, they still have some defects. Voxel-based compression networks introduce precision errors due to the voxelized operations, while point-based methods may have relatively weak robustness and are mainly designed for sparse point clouds. In this work, we propose a novel learning-based point cloud compression framework named 3D Point Cloud Geometry Quantiation Compression Network (3QNet), which overcomes the robustness limitation of existing point-based methods and can handle dense points. By learning a codebook including common structural features from simple and sparse shapes, 3QNet can efficiently deal with multiple kinds of point clouds. According to experiments on object models, indoor scenes, and outdoor scans, 3QNet can achieve better compression performances than many representative methods.

Existing methods of unstructured point cloud compression usually exploit the spatial sparseness of point clouds using hierarchical tree data structures for spatial encoding. However, such methods can be inefficient when very deep octrees are applied to sparse point cloud data to maintain low level of geometric error during compression. This paper proposes a novel octree structure called truncated octree that improves the compression ratio by representing the deep octree with a set of shallow sub-octrees which can save storage without losing the original structure. We also propose a variable length addressing scheme, to adaptively choose the length of an octree’s node address based on the truncation level—shorter (resp. longer) address when octree is truncated near the leaf (resp. root) which leads to further compression. The method is able to achieve 40% to 90% compression ratio on our tested models for point clouds of different spatial distributions. For extremely sparse point clouds, the method achieves approximately 7 times higher compression ratio than previous methods. Moreover, the method is designed to run in parallel for octree construction, encoding and decoding.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

3D point cloud presentation has been widely used in computer vision, automatic driving, augmented reality, smart cities and virtual reality. 3D point cloud compression method with higher compression ratio and tiny loss is the key to improve data transportation efficiency. In this paper, we propose a new 3D point cloud geometry compression method based on deep learning, also an auto-encoder performing better than other networks in detail reconstruction. It can reach much higher compression ratio than the state-of-art while keeping tolerable loss. It also supports parallel compressing multiple models by GPU, which can improve processing efficiency greatly. The compression process is composed of two parts. Firstly, Raw data is compressed into codeword by extracting feature of raw model with encoder. Then, the codeword is further compressed with sparse coding. Decompression process is implemented in reverse order. Codeword is recovered and fed into decoder to reconstruct point cloud. Detail reconstruction ability is improved by a hierarchical structure in our decoder. Latter outputs are grown from former fuzzier outputs. In this way, details are added to former output by latter layers step by step to make a more precise prediction. We compare our method with PCL compression and Draco compression on ShapeNet40 part dataset. Our method may be the first deep learning-based point cloud compression algorithm. The experiments demonstrate it is superior to former common compression algorithms with large compression ratio, which can also reserve original shapes with tiny loss.