New Network Bandwidth-limited Multi-view Video plus Depth Coding Method for 3D Video

Recent developments of three-dimensional (3D) video coding greatly rely on the use of Multiview Video plus Depth (MVD) data format for representing a 3D scene. This type of data can be coded with conventional video compression schemes and can enable advanced 3D video functionality, such as rendering of virtual views at the decoder side. The MVD represents a 3D scene from different viewing angles as video and pixel-wise associated depth data. In this paper we consider the redundancy of depth data in the MVD representation and propose a novel scheme, called Unpaired MVD (UP-MVD) format to be used in 3D video applications. Being a subset of the MVD this new format assumes that a reduced number of depth views compared to the number of texture views can reduce bitrate as well as the encoding/decoding complexity while still providing the 3DV functionality in many scenarios. The simulation results show that the proposed unpaired MVD format outperforms the MVD format on average from 0.9% to 6.95% of Bjontegaard delta bitrate (dBR) for the baseline disparity adjustments from 50% to 100% of the coded baseline, respectively. Moreover, UP-MVD provides equal or better rate-distortion results for all test sequences for up to 20% view separation adjustment, and in five out of seven sequences a better rate-distortion performance is observed when 50% view separation adjustment is applied.

Multi-view video plus depth (MVD) is a 3D video representation. In MVD, the depth map provides the scene distance information and is used to render the virtual view through Depth Image Based Rendering (DIBR) technique. The depth map coding error will induce distortion in the rendered virtual views. This paper proposes a mathematic model that can estimate the synthesized virtual view distortion induced by depth map compression, and the model is employed to the rate distortion optimization (RDO) in the depth map coding. Based on the rendered virtual view quality, a Lagrangian optimization adjustment scheme at Coding Unit (CU) level is proposed to improve the depth map encoding efficiency. Experimental results demonstrate that the proposed method can improve the BD-PSNR of virtual view for 0.62 dB, and the encoding complexity reduces compared with the view synthesis optimization (VSO) technique in the 3D-HEVC Test Model (HTM).

In multi-view video plus depth (MVD) format based 3D video (3DV) systems, the quality of the rendered virtual views depends highly on the quality of the compressed depth maps. Therefore, depth coding technology is vital for 3DV applications. This paper proposed a superpixel based non-regular homogenous block (NRHB) representation for low bit rate depth map coding. At the encoder side, with the aid of the superpixel segmentation of the associated texture images and the synthesis distortion estimation of the virtual views, the original depth maps are partitioned into several NRHBs and represented with only a small amount of pixels. Then, the NRHB represented depth maps are compressed by using the standard video coding approach afterward. At the decoder side, the depth maps are reconstructed through position mapping of the superpixel segmentation of the decoded texture images, and enhanced by a false edge filter. The experimental results show that the proposed method could impressively increase both the coding efficiency of the depth maps and the rendering quality of the virtual views. It also can be seen that the proposed method significantly reduced the encoding computational complexity for depth maps, which makes it particularly useful for the low bandwidth mobile consumer devices with limited computational resources.

Compression of depth maps is important for “texture plus depth” format of multiview images, which enables synthesis of novel intermediate views via depth-image-based rendering (DIBR) at decoder. Previous depth map coding schemes exploit unique depth data characteristics to compactly and faithfully reproduce the original signal. In contrast, since depth map is only a means to the end of view synthesis and not itself viewed, in this paper we explicitly manipulate depth values, without causing severe synthesized view distortion, in order to maximize representation sparsity in the transform domain for compression gain - we call this process transform domain spar-sification (TDS). Specifically, for each pixel in the depth map, we first define a quadratic penalty function, with minimum at ground truth depth value, based on synthesized view's distortion sensitivity to the pixel's depth value during DIBR. We then define an objective for a depth signal in a block as a weighted sum of: i) signal's sparsity in the transform domain, and ii) per-pixel synthesized view distortion penalties for the chosen signal. Given that sparsity (l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> -norm) is non-convex and difficult to optimize, we replace the l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> -norm in the objective with a computationally inexpensive weighted l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -norm; the optimization is then an unconstrained quadratic program, solvable via a set of linear equations. For the weighted l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -norm to promote sparsity, we solve the optimization iteratively, where at each iteration weights are readjusted to mimic sparsity-promoting l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">τ</sub> -norm, 0 ≤ τ ≤ 1. Using JPEG as an example transform codec, we show that our TDS approach gained up to 1.7dB in rate-distortion performance for the interpolated view over compression of unaltered depth maps.

Multiview video plus depth (MVD) is the most popular 3D video format due to its efficient compression and provision for novel view generation enabling the free-viewpoint applications. In addition to color images, MVD format provides depth maps which are exploited to generate intermediate virtual views using the depth image-based rendering (DIBR) techniques. Compression affects the quality of the depth maps which in turn may introduce various structural and textural distortions in the DIBR-synthesized images. Estimation of the compression-related distortion in depth maps is very important for a high-quality 3D experience. The task becomes challenging when the corresponding reference depth maps are unavailable, e.g., when evaluating the quality on the decoder side. In this paper, we present a no-reference quality assessment algorithm to estimate the distortion in the depth maps induced by compression. The proposed algorithm exploits the depth saliency and local statistical characteristics of the depth maps to predict the compression distortion. The proposed ‘depth distortion evaluator’ (DDE) is evaluated on depth videos from standard MVD database compressed with the state-of-the-art high-efficiency video coding at various quality levels. The results demonstrate that DDE can be used to effectively estimate the compression distortion in depth videos.

This paper presents an efficient error concealment method for multiview video plus depth (MVD) delivery over error-prone channels based on the view synthesis information. Specifically, we propose an enhanced temporal error concealment (TEC) method by estimating the probable partition and missing MVs of a corrupted macroblock (MB) using the synthesized information. Furthermore, the view synthesis error concealment (VSEC) approach that directly uses the synthesized pixels to conceal the corrupted MBs is also adaptively utilized together with the TEC approach. Experimental results have shown the effectiveness of the proposed method compared with the existing methods in terms of both objective and subjective measures.

View synthesis prediction (VSP) is an important tool for improving the coding efficiency in the next generation three-dimensional (3-D) video systems. However, VSP will result in a new type of inter-view error propagation when the multi-view video plus depth (MVD) data are transmitted over the lossy networks. In this paper, this new type of error propagation is characterized and modeled. Firstly, a new analytic model is formulated to estimate the expected transmission distortion caused by error propagation from the synthesized reference view. Then, the compound impact of the transmission distortions of both the texture video and the depth map on the quality of the synthetic reference view is mathematically analysed. Our extensive simulation results demonstrate that the proposed transmission distortion model is very accurate.

There are several data formats available for 3D video, among which is the Multiview Video plus Depth (MVD) representation that enables depth-image-based rendering (DIBR). In addition to DIBR, the MVD data format can enable more efficient compression for texture video coding. In this paper, we propose advanced motion vector prediction (MVP) techniques which utilize the availability of depth information in the MVD format for more efficient coding of the corresponding texture pictures. The proposed MVP was implemented on top of the Multiview Video Coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard and its reference software and tested under the simulation conditions of the Call for Proposals of 3D video coding technology issued by the Moving Picture Experts Group (MPEG). In the provided simulation results the proposed scheme significantly outperformed the conventional MVP (by up to 9.9% and on average by 7.5% in Bjontegaard delta bitrate) when it was applied for coding of 3-view MVD data.

In this paper, we propose a new Light Field Image (LFI) compression scheme based on Multiview Video plus Depth (MVD) coding architecture. Through LF function analysis, we preliminarily estimate depth map according to the concept of Epipolar Plane Image (EPI). Such a rough estimation causes some error pixels within initial depth map, so we design a weighting mean filter to smooth the inaccurate region. The final estimated depth maps can be encoded by MVD coding standard jointly with a small number of viewpoint images in LFI, so as to improve compression efficiency of LFI. Ultimately, massive experiments are conducted on 4 LFIs to verify the effectiveness of the proposed compression scheme. The simulated results demonstrate that our LFI compression scheme can achieve a high LFI compression performance and outperform the-state-of-art coding solution.

We propose a novel 3D space representation for multi-view video, using epipolar plane depth images (EPDI). Multi-view video plus depth (MVD) is used as common data format for FTV (Free-viewpoint TV), which enables synthesizing virtual view images. Due to the large amount of data and complexity of the multi-view video coding (MVC), compression of MVD is a challenging issue. We address this problem and propose a new representation that is constructed from MVD using ray-space. MVD is converted into image and depth ray-spaces. The proposed representation is obtained by converting each of ray-spaces into a global depth map and a global view using EPDI. Experiments demonstrate the analysis of this representation.

Automated precision weed control requires visual methods to discriminate between crops and weeds. State-of-the-art plant detection methods fail to reliably detect weeds, especially in dense and occluded scenes. In the past, using hand-crafted detection models, both color (RGB) and depth (D) data were used for plant detection in dense scenes. Remarkably, the combination of color and depth data is not widely used in current deep learning-based vision systems in agriculture. Therefore, we collected an RGB-D dataset using a stereo vision camera. The dataset contains sugar beet crops in multiple growth stages with a varying weed densities. This dataset was made publicly available and was used to evaluate two novel plant detection models, the D-model, using the depth data as the input, and the CD-model, using both the color and depth data as inputs. For ease of use, for existing 2D deep learning architectures, the depth data were transformed into a 2D image using color encoding. As a reference model, the C-model, which uses only color data as the input, was included. The limited availability of suitable training data for depth images demands the use of data augmentation and transfer learning. Using our three detection models, we studied the effectiveness of data augmentation and transfer learning for depth data transformed to 2D images. It was found that geometric data augmentation and transfer learning were equally effective for both the reference model and the novel models using the depth data. This demonstrates that combining color-encoded depth data with geometric data augmentation and transfer learning can improve the RGB-D detection model. However, when testing our detection models on the use case of volunteer potato detection in sugar beet farming, it was found that the addition of depth data did not improve plant detection at high vegetation densities.

The compression of multiview video in an end-to-end 3D system is required to reduce the amount of visual information. Since multiple cameras usually have a common field of view, high compression ratios can be achieved if both the temporal and inter-view redundancy are exploited. View synthesis prediction is a new coding tool for multiview video that essentially generates virtual views of a scene using images from neighboring cameras and estimated depth values. In this work, we consider depth estimation for view synthesis in multiview video encoding. We focus on generating smooth and accurate depth maps, which can be efficiently coded. We present several improvements to the reference block-based depth estimation approach and demonstrate that the proposed method of depth estimation is not only efficient for view synthesis prediction, but also produces depth maps that require much fewer bits to code.

In the multiview video plus depth (MVD) coding format, both texture and depth views are jointly compressed to represent the 3-D video content. The MVD format enables synthesis of virtual views through depth-image-based rendering; hence, distortion in the texture and depth views affects the quality of the synthesized virtual views. Bit allocation between texture and depth views has been studied with some promising results. However, to the best of our knowledge, most of the existing bit-allocation methods attempt to allocate a fixed amount of total bit rate between texture and depth views; that is, to select appropriate pair of quantization parameters for texture and depth views to maximize the synthesized view quality subject to a fixed total bit rate. In this paper we propose a scalable bit-allocation scheme, where a single ordering of texture and depth packets is derived and used to obtain optimal bit allocation between texture and depth views for any total target rates. In the proposed scheme, both texture and depth views are encoded using the quality scalable coding method; that is, medium grain scalable (MGS) coding of the Scalable Video Coding (SVC) extension of the Advanced Video Coding (H.264/AVC) standard. For varying target total bit rates, optimal bit truncation points for both texture and depth views can be obtained using the proposed scheme. Moreover, we propose to order the enhancement layer packets of the H.264/SVC MGS encoded depth view according to their contribution to the reduction of the synthesized view distortion. On one hand, this improves the depth view packet ordering when considered the rate-distortion performance of synthesized views, which is demonstrated by the experimental results. On the other hand, the information obtained in this step is used to facilitate optimal bit allocation between texture and depth views. Experimental results demonstrate the effectiveness of the proposed scalable bit-allocation scheme for texture and depth views.

In this paper, we propose a joint coding scheme for both multi-view video and its corresponding depth map. After we synthesize a virtual image for the target view using adjacent view images and their depth information, we apply a view interpolation prediction (VIP) method for both multi-view video coding and its depth data coding. In order to improve the synthesized virtual view, we also propose a hole filling method that can compensate for empty regions caused by the 3D warping operation. With the proposed algorithm, we have obtained approximately 0.65 dB of the PSNR gain on average for the multi-view depth data, and 0.17 dB of the PSNR gain for the multi-view video data, compared to JMVM 1.0.

This paper presents a new approach for the depth coding part of a 3D video coding extension based on the Multiview Video plus Depth (MVD) representation. Our approach targets a higher coding efficiency for the depth component and is motivated by the fact that depth signals have specific characteristics that differ from video. For this purpose we apply the method of wedgelet segmentation with residual adaptation for depth blocks by implementing a new set of coding and prediction modes and by optimizing the algorithms for efficient processing and signaling. The results show that a bit rate reduction of up to 6% is achieved for the depth component, using a 3D video codec based on the high-efficiency video coding (HEVC) technology. Apart from the depth coding gains, wedgelets lead to a considerably better rendered view quality.