Depth Coding using Downsampling and View based Upsampling

In this paper, we propose boundary-preserving depth upsampling without texture copying artifacts and holes. Since high-resolution (HR) color images contain many edges and textures, they inevitably cause artifacts such as texture copying when they are used in depth upsampling. Thus, we perform rolling guidance filtering on HR color images before depth upsampling to preserve object boundaries in depth maps while successfully removing texture copying artifacts. Moreover, upsampled depth maps are often corrupted by noise and contain holes along object boundaries. Thus, we utilize fast weighted median filtering after depth upsampling to remove noise and fill holes. Experimental results demonstrate that the proposed method achieves boundary-preserving depth upsampling results without texture copying artifacts and holes through visual and quantitative evaluations.

Chapter 10 - A study on the relationship between depth map quality and stereoscopic image quality using upsampled depth maps

Due to the limitation of the bandwidth resource and capture resolution of depth cameras, low resolution depth maps should be up-sampled to high resolution so that they can correspond to their texture images. In this paper, a novel depth map up-sampling algorithm is proposed by exploiting the fractal internal self-referential feature. Fractal parameters which are extracted from a depth map, describe the internal self-referential feature of the depth map, do not introduce inherent scale and just retain the relational information of the depth map, i.e., fractal transforms provide a resolution-independent description for depth maps and could up-sample depth maps to an arbitrary high resolution. Then, an enhancement method is also proposed to further improve the performance of the up-sampled depth map. The experimental results demonstrate that better quality of synthesized views is achieved both on objective and subjective performance. Most important of all, arbitrary resolution depth maps can be obtained with the aid of the proposed scheme.

In this paper, we propose a simple and effective depth upsampling technique using self-guided residual interpolation. The original residual interpolation requires guidance information such as high-resolution RGB color image. However, self-guided residual interpolation requires only a single depth map. In the proposed algorithm, a tentative estimation of a high-resolution depth map is first generated from an input low-resolution depth map. Then, re-interpolation is applied to the residual domain, which is defined by differences between the input depth map and the tentative estimate. A precise high-resolution depth map is obtainable by interpolating in the residual domain. Experimental results demonstrate that our algorithm can outperform state-of-the-art depth map upsampling algorithms.

A new multiview just-noticeable-depth-difference(MJNDD) Model is presented and applied to compress the joint multiview video plus depth. Many video coding algorithms remove spatial and temporal redundancies and statistical redundancies but they are not capable of removing the perceptual redundancies. Since the final receptor of video is the human eyes, we can remove the perception redundancy to gain higher compression efficiency according to the properties of human visual system (HVS). Traditional just-noticeable-distortion (JND) model in pixel domain contains luminance contrast and spatial-temporal masking effects, which describes the perception redundancy quantitatively. Whereas HVS is very sensitive to depth information, a new multiview-just-noticeable-depth-difference(MJNDD) model is proposed by combining traditional JND model with just-noticeable-depth-difference (JNDD) model. The texture video is divided into background and foreground areas using depth information. Then different JND threshold values are assigned to these two parts. Later the MJNDD model is utilized to encode the texture video on JMVC. When encoding the depth video, JNDD model is applied to remove the block artifacts and protect the edges. Then we use VSRS3.5 (View Synthesis Reference Software) to generate the intermediate views. Experimental results show that our model can endure more noise and the compression efficiency is improved by 25.29 percent at average and by 54.06 percent at most compared to JMVC while maintaining the subject quality. Hence it can gain high compress ratio and low bit rate.

The multiview video-plus-depth format provides a promising solution for the transmission of three-dimensional video. This format can be transmitted using the multiview high efficiency video coding (MV-HEVC) standard, where the texture and the depth videos are encoded and transmitted separately. The depth maps have large homogeneous areas making their downsampling before transmission and their upsampling at the receiver an attractive option to reduce bandwidth requirements. However, this process results in distortions of the sharp edges also characteristic of depth maps, causing artefacts in the synthesized views. This work exploits the edges found in the corresponding texture view and uses a novel edge detection and depth map painting algorithm to improve the reconstruction of the edges in the upsampled depth maps. Test results show that this solution can improve the quality of depth maps by up to 2.35dB compared to other methods found in literature.

A new feedforward control method to control encod-ing parameters is proposed for low bit rate video encoding. At low transmission bit rates, all input video frames cannot be transmitted and an encoding frame interval varies with the information amount in each encoded frame. Subjective video quality is evaluated from both picture degradation and encoding frame intervals. Because picture degradation and encoding frame intervals are traded off, encoding parameter control is very important. The proposed method is based on a hybrid transform predictive coding scheme with adaptive coefficient selection. The method estimates the amount of information for some encoding parameters and selects parameters so that given control characteristics can be achieved. Computer simulations for both the proposed method and a conventional feedback method show the effectiveness of the new method. This method will be applied to an NETEC series low bit rate video codec.

High-quality 3D content generation requires high-quality depth maps. In practice, depth maps generated by stereo-matching, depth sensing cameras, or decoders, have low resolution and suffer from unreliable estimates and noise. Therefore, depth enhancement is necessary. Depth enhancement comprises two stages: depth upsampling and temporal post-processing. In this paper, we extend our previous work on depth upsampling in two ways. First we propose PWAS-MCM, a new depth upsampling method, and we show that it achieves on average the highest depth accuracy compared to other efficient state-of-the-art depth upsampling methods. Then, we benchmark all relevant state-of-the-art filter-based temporal post-processing methods on depth accuracy by conducting a parameter space search to find the optimum set of parameters for various upscale factors and noise levels. Then we analyze the temporal post-processing methods qualitatively. Finally, we analyze the computational complexity of each depth upsampling and temporal post-processing method by measuring the throughput and hardware utilization of the GPU implementation that we built for each method.

In free-viewpoint television, high-quality depth maps are substantial for a virtual view synthesis for free navigation purposes. In this paper, we propose a new method of depth map quality improvement and resolution increase. Our method is intended for low resolution depth maps acquired through estimation using multiview video. The proposed depth map quality improvement is based on segmentation of an acquired high-resolution image. In the estimated depth map, searching for outliers is performed in the neighborhood created from similar segments in the acquired view. Experimental results show high effectiveness of the presented method of depth maps refinement, especially for object edges.

This paper presents a method for increasing spatial resolution of a depth map using its corresponding high-resolution (HR) color image as a guide. Most of the previous methods rely on the assumption that depth discontinuities are highly correlated with color boundaries, leading to artifacts in the regions where the assumption is broken. To prevent scene texture from being erroneously transferred to reconstructed scene surfaces, we propose a framework for dividing the color image into different regions and applying different methods tailored to each region type. For the region classification, we first segment the low-resolution (LR) depth map into regions of smooth surfaces, and then use them to guide the segmentation of the color image. Using the consensus of multiple image segmentations obtained by different super-pixel generation methods, the color image is divided into continuous and discontinuous regions: in the continuous regions, their HR depth values are interpolated from LR depth samples without exploiting the color information. In the discontinuous regions, their HR depth values are estimated by sequentially applying more complicated depth-histogram-based methods. Through experiments, we show that each step of our method improves depth map upsampling both quantitatively and qualitatively. We also show that our method can be extended to handle real data with occluded regions caused by the displacement between color and depth sensors.

H.26L is an emerging video encoding standard proposed by ITU, which goal is for video communication and multicast in low bit rate conditions. We give detailed analysis and performance tests for the parallelism of the H.26L video encoder test model long-term (TML8.0). Based on these tests, several parallel schedule strategies are discussed in different aspects, and the optimal one is selected and implemented by using MPI. At last, experiment results on Dawning-2000 MPP machines with different processors are given. The analysis and simulation results show that the H.26L has a good parallel performance and scalability on slice layer, and parallel computing can greatly speed up the processing of H.26L encoder. The experiment methods and results can be applied to many parallel systems for H.26L video processing.

A wavelet Application Programmers' Interface (API) for the TriMedia TM1000 multimedia processor will be described and demonstrated. The challenge undertaken for this API was to provide a single scalable interface to a subband encoder with acceptable performance at bit rates from 9600 bps to 2 Mbps. Further, the API as designed supports the utilization of multiple TriMedia resources. Specifically, the encoder to be demonstrated, is a fully motion compensated wavelet transform which does not utilize the Discrete Cosine Transform (DCT) for the encoding of the prediction errors. The video encoder has numerous novel features including: an integer valued limited precision wavelet transform; a fully motion compensated wavelet API; integral support for multiple processors; and simple mapping to hardware. The integer valued wavelet transform being demonstrated fits into a general class of processing methods. This facilitates the calculation of wavelet transforms in limited precision arithmetic. Specifically, the transform to be described permits the calculation of 25 subbands from 8 bit data with a 16 bit word length. Details of the loading on the Philips TriMedia will be provided including the utilization of the instruction pipeline for the inner loops of the wavelet transform. The API includes a fully motion compensated wavelet transform which greatly improves the visual quality and achievable frame rate over low bit rate channels. One unique feature of the motion compensation (again, patent pending) is that the interframe coding does not require the use of the DCT, which greatly simplifies the implementation of the motion prediction. We will demonstrate the gains achievable with and without motion compensation in the real time encoding and encoding of video sequences. How the API maps across multiple processors will also be demonstrated and a sample application for high field rate with broadcast quality will be presented. The demonstration program control flow will be provided and system performance presented. Finally, the above wavelet encoder is being mapped to both low cost Application Specific Integrated Circuits (ASICs) and Field Programmable Gate Arrays (FPGAs) and the resulting implementations and performance, including gate count and required pixel rates, outlined.

A new video encoder proposal, H.26L, is compared against H.263 and H.263+. In the comparison, both computational complexity and compression performance are analyzed. Moreover, the trade-off possibilities between the complexity and compression performance within H.26L are presented. Experimental comparisons with H.263 and H.263+ show that H.26L reduces the output bit rate about 30% with the same quality. The computation time increases about three times compared to H.263 and leads into the encoding speed of 3-6 fps for QCIF sequences on a 400 MHz Pentium III processor. Realtime operation can be achieved by applying additional, algorithmic and platform-specific optimizations.

Wyner-Ziv coding refers to lossy source coding with side information at the decoder. Recently some practical applications of Wyner-Ziv coding to video compression have been studied due to its advantage of error robustness over standard video coding standards. Based on recent theoretical result on successive Wyner-Ziv coding, we propose in this paper a practical layered Wyner-Ziv video codec using the DCT, nested scalar quantizer (NSQ), and irregular LDPC code based Slepian-Wolf coding (or lossless source coding with side information). The DCT is applied as an approximation to the conditional KLT, which makes the components of the transformed block conditionally independent given the side information. NSQ is a binning scheme that facilitates layered bit-plane coding of the bin indices while reducing the bit rate. LDPC code based Slepian-Wolf coding exploits the correlation between the quantized version of the source and the side information to achieve further compression. Different from previous works, an attractive feature of our proposed system is that video encoding is done only once but decoding allowed at many lower bit rates without quality loss.

In multi-view video plus depth (MVD) format based 3D video (3DV) systems, the quality of the rendered virtual views depends highly on the quality of the compressed depth maps. Therefore, depth coding technology is vital for 3DV applications. This paper proposed a superpixel based non-regular homogenous block (NRHB) representation for low bit rate depth map coding. At the encoder side, with the aid of the superpixel segmentation of the associated texture images and the synthesis distortion estimation of the virtual views, the original depth maps are partitioned into several NRHBs and represented with only a small amount of pixels. Then, the NRHB represented depth maps are compressed by using the standard video coding approach afterward. At the decoder side, the depth maps are reconstructed through position mapping of the superpixel segmentation of the decoded texture images, and enhanced by a false edge filter. The experimental results show that the proposed method could impressively increase both the coding efficiency of the depth maps and the rendering quality of the virtual views. It also can be seen that the proposed method significantly reduced the encoding computational complexity for depth maps, which makes it particularly useful for the low bandwidth mobile consumer devices with limited computational resources.