Virtual Reference-Based Predictive Coding for V-PCC Attribute Compression

The dynamic point cloud is widely needed in 3D vision related applications such as virtual reality and telepresence. Due to the huge amount of data, a key technology before the effective application is the dynamic point cloud compression. The state-of-the-art dynamic point cloud compression scheme, video-based point cloud compression (V-PCC), generates 2D videos with some uncorrelation due to the patch segmentation and packing process, which will affect the compression efficiency. In this paper, we propose a Packing with Patch Correlation Improvement (PPCI) algorithm to adaptively remove the uncorrelated parts between matched patches in packing for the sake of inter-prediction performance. We first propose a basic unidirectional patch re-segmentation operator to remove the uncorrelated parts of the patches in the current point cloud relative to the patches in its reference point cloud. The removed parts will be formed as new patches and added to the patch collection of the current point cloud. Then we propose a back-and-forth structure, which is a combination of several basic patch re-segmentation operators, to bilaterally remove the uncorrelated parts of matched patches in a back-and-forth (BF) unit. Furthermore, we propose a framework to adaptively decide the best length of each BF unit in a point cloud sequence. Experimental results show that our method achieves noticeable bitrate savings compared with the existing V-PCC packing methods, particularly for sequences with small motion.

In recent years, 3D point cloud content has gained attention due to its application possibilities, such as multimedia systems, virtual, augmented, and mixed reality, through the mapping and visualization of environments and/or 3D objects, real-time immersive communications, and autonomous driving systems. However, raw point clouds demand a large amount of data for their representation, and compression is mandatory to allow efficient transmission and storage. The MPEG group proposed the Video-based Point Cloud Compression (V-PCC) standard, which is a dynamic point cloud encoder based on the use of video encoders through projections into 2D space. However, V-PCC demands a high computational cost, demanding fast implementations for real-time processing and, especially, for mobile device applications. In this paper, a machine-learning-based fast implementation of V-PCC is proposed, where the main approach is the use of trained decision trees to speed up the block partitioning process during the point cloud compression. The results show that the proposed fast V-PCC solution is able to achieve an encoding time reduction of 42.73% for the geometry video sub-stream and 55.3% for the attribute video sub-stream, with a minimal impact on bitrate and objective quality.

Efficient point cloud compression is essential for applications like virtual and mixed reality, autonomous driving, and cultural heritage. This paper proposes a deep learning-based inter-frame encoding scheme for dynamic point cloud geometry compression. We propose a lossy geometry compression scheme that predicts the latent representation of the current frame using the previous frame by employing a novel feature space inter-prediction network. The proposed network utilizes sparse convolutions with hierarchical multiscale 3D feature learning to encode the current frame using the previous frame. The proposed method introduces a novel predictor network for motion compensation in the feature domain to map the latent representation of the previous frame to the coordinates of the current frame to predict the current frame's feature embedding. The framework transmits the residual of the predicted features and the actual features by compressing them using a learned probabilistic factorized entropy model. At the receiver, the decoder hierarchically reconstructs the current frame by progressively rescaling the feature embedding. The proposed framework is compared to the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG). The proposed method achieves more than 88% BD-Rate (Bjøntegaard Delta Rate) reduction against G-PCCv20 Octree, more than 56% BD-Rate savings against G-PCCv20 Trisoup, more than 62% BD-Rate reduction against V-PCC intra-frame encoding mode, and more than 52% BD-Rate savings against V-PCC P-frame-based inter-frame encoding mode using HEVC. These significant performance gains are cross-checked and verified in the MPEG working group.

Point cloud compression has been studied in standard bodies and we are here concerned with the Moving Picture Experts Group video-based point cloud compression (V-PCC) solution. Plenoptic point clouds (PPC) is a novel volumetric data representation wherein points are associated with colors in all viewing directions to improve realism. It is sampled as a number ( Nc ) of attribute colors per point. We propose a new method for the efficient video-based compression of PPC that is backwards compatible with the existing single-color V-PCC decoder. V-PCC generates three image atlases which are encoded using an image/video encoder. We assume there may be a reference color which is to be encoded as the main payload. We generate Nc+3 atlases and we produce Nc differential images against the reference color image. Those difference images are pixel-wise transformed using an Nc -point discrete cosine transform, generating Nc transformed atlases which are encoded, forming the secondary payload. Such secondary information is the plenoptic enhancement to the point cloud. If there is no reference attribute, we skip the differences and use the lowest frequency of the transformed atlases as the main payload. Results are presented that show an unrivaled performance of the proposed method.

Immersive media representation format based on point clouds has underpinned significant opportunities for extended reality applications. Point cloud in its uncompressed format require very high data rate for storage and transmission. The video based point cloud compression (V-PCC) technique projects a dynamic point cloud into geometry and texture video sequences. The projected texture video is then coded using modern video coding standard like HEVC. Since the properties of projected texture video frames are different from traditional video frames, HEVC-based commonality modeling can be inefficient. An improved commonality modeling technique is proposed that employs edge position difference oriented motion model. Experimental results show that the proposed commonality modeling technique can yield savings in bit rate of up to 3.15% over the V-PCC HEVC reference encoder.

Point clouds are a versatile 3D data representation format that captures spatial and attribute information about objects or environments. It has been widely used in applications like autonomous driving, augmented and virtual reality, and 3D scanning. However, the large volume of data involved in point cloud processing poses challenges in terms of storage, transmission, and real-time processing. The Video-based Point Cloud Compression (V-PCC) standard addresses these challenges by employing 2D video compression techniques to encode dynamic 3D point clouds. Despite its effectiveness, V-PCC demands a high computational cost, particularly in encoding geometry and attribute video sub-streams. This paper presents a machine learning-based approach to accelerate the V-PCC encoding, focusing on the Coding Unit partitioning process in the geometry and attribute sub-streams deployed in the reference software (TMC2). The proposed method significantly reduces the encoding complexity by using decision tree models with negligible impact on coding efficiency. Experimental results demonstrate an average encoding time reduction of 36.12% for geometry and attribute sub-streams, with a minimal impact on coding efficiency between -0.43% and 0.83% in terms of BD-Rate.

In video-based point cloud compression (V-PCC), occupancy map video is utilized to indicate whether a 2-D pixel corresponds to a valid 3-D point or not. In the current design of V-PCC, the occupancy map video is directly compressed losslessly with High Efficiency Video Coding (HEVC). However, the coding tools in HEVC are specifically designed for natural images, thus unsuitable for the occupancy map. In this paper, we present a novel quadtree-based scheme for lossless occupancy map coding. In this scheme, the occupancy map is firstly divided into several coding tree units (CTUs). Then, the CTU is divided into coding units (CUs) recursively using a quadtree. The quadtree partition is terminated when one of the three conditions is satisfied. Firstly, all the pixels have the same value. Secondly, the pixels in the CU only have two kinds of values and they can be separated by a continuous edge whose endpoints lie on the side of the CU. The continuous edge is then coded using chain code. Thirdly, the CU reaches the minimum size. This scheme simplifies the design of block partitioning in HEVC and designs simpler yet more effective coding tools. Experimental results show significant reduction of bit-rate and complexity compared with the occupancy map coding scheme in V-PCC. In addition, this scheme is also very efficient to compress the semantic map.

Video-based Point Cloud Compression (V-PCC) standard offers state-of-the-art tools and efficiency for volumetric video compression. The V-PCC reference software, TMC2, is able to demonstrate the compression efficiency of V-PCC, but with computational complexity that is impractical for real-life applications. This paper introduces a novel open-source V-PCC encoder, named uvgVPCCenc, that brings V-PCC compression speed to a practical level. To reduce computational bottlenecks, uvgVPCCenc is implemented from the ground up in C++ with advanced multi-threading and streamlined algorithms. It also supports a flexible parameterization and different presets that facilitate its adaptation to various content and target bitrates. Our comparative evaluations show that uvgVPCCenc achieves substantial speedups over TMC2 while maintaining competitive rate-distortion-complexity tradeoff. To the best of our knowledge, uvgVPCCenc is the first open-source encoder explicitly designed for practical V-PCC compression, paving the way from theoretical research to practical volumetric video applications.

This paper presents a novel method to determine rate-distortion optimized\ntransform coefficients for efficient compression of videos generated from point\nclouds. The method exploits a generalized frequency selective extrapolation\napproach that iteratively determines rate-distortion-optimized coefficients for\nall basis functions of two-dimensional discrete cosine and sine transforms. The\nmethod is applied to blocks containing both occupied and unoccupied pixels in\nvideo based point cloud compression for HEVC encoding. In the proposed\nalgorithm, only the values of the transform coefficients are changed such that\nresulting bit streams are compliant to the V-PCC standard. For all-intra coded\npoint clouds, bitrate savings of more than 4% for geometry and more than 6% for\ntexture error metrics with respect to standard encoding can be observed. These\nsavings are more than twice as high as savings obtained using competing methods\nfrom literature. In the randomaccess case, our proposed method outperforms\ncompeting V-PCC methods by more than 0.5%.\n

We present a study on the impact of packet loss on dynamic 3D point cloud streaming, encoded with MPEG Video-based Point Cloud Compression (V-PCC) standard. We show the distortion when different channels of V-PCC bitstream are lost, with the loss of occupancy and geometry data impacting the quality most significantly. Our results point to the need for better error concealment techniques. We end the paper by presenting preliminary thoughts and experimental results of two naive error concealment techniques in the point cloud domain, for attributes and geometry data, respectively, and highlight the limitations of each.

With the maturity of 3D capture technology, the explosive growth of point cloud data has burdened the storage and transmission process. Traditional hybrid point cloud compression (PCC) tools relying on handcrafted priors have limited compression performance and are increasingly weak in addressing the burden induced by data growth. Recently, deep learning-based PCC methods have been introduced to continue to push the PCC performance boundary. With the thriving of deep PCC, the community urgently demands a systematic overview to conclude the past progress and present future research directions. In this paper, we have a detailed review that covers popular point cloud datasets, algorithm evolution, benchmarking analysis, and future trends. Concretely, we first introduce several widely-used PCC datasets according to their major properties. Then the algorithm evolution of existing studies on deep PCC, including lossy ones and lossless ones proposed for various point cloud types, is reviewed. Apart from academic studies, we also investigate the development of relevant international standards (i.e., MPEG standards and JPEG standards). To help have an in-depth understanding of the advance of deep PCC, we select a representative set of methods and conduct extensive experiments on multiple datasets. Comprehensive benchmarking comparisons and analysis reveal the pros and cons of previous methods. Finally, based on the profound analysis, we highlight the challenges and future trends of deep learning-based PCC, paving the way for further study.

Since the development of 3D applications, the point cloud, as a spatial description easily acquired by sensors, has been widely used in multiple areas such as SLAM and 3D reconstruction. Point Cloud Compression (PCC) has also attracted more attention as a primary step before point cloud transferring and saving, where the geometry compression is an important component of PCC to compress the points geometrical structures. However, existing non-learning-based geometry compression methods are often limited by manually pre-defined compression rules. Though learning-based compression methods can significantly improve the algorithm performances by learning compression rules from data, they still have some defects. Voxel-based compression networks introduce precision errors due to the voxelized operations, while point-based methods may have relatively weak robustness and are mainly designed for sparse point clouds. In this work, we propose a novel learning-based point cloud compression framework named 3D Point Cloud Geometry Quantiation Compression Network (3QNet), which overcomes the robustness limitation of existing point-based methods and can handle dense points. By learning a codebook including common structural features from simple and sparse shapes, 3QNet can efficiently deal with multiple kinds of point clouds. According to experiments on object models, indoor scenes, and outdoor scans, 3QNet can achieve better compression performances than many representative methods.

Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding.

The growing demand for visual telecommunication services over the last decade has greatly increased the need for efficient image coding and compression schemes. The work presented in this thesis examines several aspects of the problem of coding and compressing the various image and video-based services which are likely to utilize Broadband Integrated Services Digital Networks (BISDNs) in the future. This research has two major thrusts. The first is the development of a general-purpose, high-performance and high-quality image coding and compression scheme for these broadband-based visual services. The second is the development of an accurate traffic source modelling scheme for the Variable BitRate (VBR) packet video traffic produced by the proposed coding scheme. The proposed high-performance visual coding and compression scheme combines both statistical and psychovisual coding techniques to produce an optimum scheme which removes both statistical and psychovisual redundancies from images in the coding process. When used to encode four standard 512x512x8-bit test images, this scheme results in sub-distortion compression ratios of up to 27:1 without the use of any form of interframe coding. An efficient and flexible Asynchronous Transfer Mode (ATM) cell packaging scheme is also developed which allows the production of either 'priority' or 'non-priority' cell streams suitable for transmission over broadband networks. Three suitable VBR packet video traffic source models are developed during this study. Each of these sources reproduces the 'low-level' cell generation process and switches between different 'modes' of cell generation in order to capture the inherent `burstiness' and 'variability' of typical VBR packet video traffic streams. However, the way in which this cell production process is modelled, as well as the traffic levels which can be reproduced, differs for each of these sources. A relatively complex, general-purpose traffic source is proposed which is based on the hidden Markov statistical model and is able to model all levels of VBR traffic (up to 20 Mbps). A simpler 'switched-fractal' source is also proposed as an accurate model for the low (below 5 Mbps) VBR traffic only. The third proposed source is a 'switched-Markov' model which specifically caters for the high (5 to 20 Mbps) levels of VBR traffic. Among these three artificial traffic sources, the characteristics of all levels of traffic produced by the proposed video and image coding scheme are able to be reproduced accurately.

Point cloud data from LiDAR sensors is the currently the basis of most L4 autonomous driving systems. Sharing and storing point clouds will also be important for future applications, such as accident investigation or V2V/V2X networks. Due to the huge volume of data involved, storing point clouds collected over long periods of time and transmitting point clouds in real-time are difficult tasks, making compression an indispensable step before storing or transmitting. Previous streaming point cloud compression methods, such as octree compression or video compression-based approaches, have difficulty compressing this data in real-time into very small volumes with low information loss. To reduce temporal redundancy efficiently and rapidly, in this paper we propose a real-time streaming point cloud data compression method using U-net. By utilizing raw packet data from LiDAR sensors, we can store 3D point cloud information losslessly in a 2D matrix, and convert streaming point cloud data into a video-like format. By designating some frames as reference frames and then using U-net to interpolate the remaining LiDAR frames, we can greatly reduce temporal redundancy. Our use of U-net was inspired by a video interpolation approach employed in another study. Noise in LiDAR data is a big issue which significantly affects network training and compression results. In this paper, we propose a padding strategy to alleviate the negative impact of this noise. As a result of these improvements, our proposed method can outperform octree compression, MPEG-based compression and our previously proposed SLAM-based compression method.