Depth No-Synthesis-Error Time-Consistent Preprocessing for Depth Coding in FVV System

Quality assessment of stereoscopic video in free viewpoint video system

An overview of 3D and free viewpoint video is given in this paper with special focus on related standardization activities in MPEG. Free viewpoint video allows the user to freely navigate within real world visual scenes, as known from virtual worlds in computer graphics. Suitable 3D scene representation formats are classified and the processing chain is explained. Examples are shown for image-based and model-based free viewpoint video systems, highlighting standards conform realization using MPEG-4. Then the principles of 3D video are introduced providing the user with a 3D depth impression of the observed scene. Example systems are described again focusing on their realization based on MPEG-4. Finally multi-view video coding is described as a key component for 3D and free viewpoint video systems. MPEG is currently working on a new standard for multi-view video coding. The conclusion is that the necessary technology including standard media formats for 3D and free viewpoint is available or will be available in the near future, and that there is a clear demand from industry and user side for such applications. 3DTV at home and free viewpoint video on DVD will be available soon, and will create huge new markets.

We have been developing a new television system named Free Viewpoint Television (FTV) that can generate free views. We propose a new user interface of FTV using an automultiscopic display (multi-viewing autostereoscopic display) and a head tracking system. We made a head tracking system which used cameras and didn't need to attach any sensors to a person. We succeeded to extend the viewing zone of an automultiscopic display. It brought us to interface with FTV naturally.

This paper presents an H.264-based scheme for compressing 3D content captured by 3D depth range cameras. Existing MPEG-2 based schemes take advantage of the correlation between the 2D video sequence and its corresponding depth map sequence, and use the 2D motion vectors (MV) for the depth video sequence as well. This improves the speed of encoding the depth map sequence, but it results in an increase in the bitrate or a drop in the quality of the reconstructed 3D video. This is found to be due to the MVs of the 2D video sequence not being the best choice for encoding some parts of the depth map sequence containing sharp edges or corresponding to distant objects. To solve this problem, we propose an H.264-based method which re-estimates the MVs and re-selects the appropriate modes for these regions. Experimental results show that the proposed method enhances the quality of the encoded depth map sequence by an average of 1.77 dB. Finding the MVs of the sharp edge-included regions of the depth map sequence amounts to 30.64% of the computational effort needed to calculate MVs for the whole depth map sequence.

To improve the depth estimation precision at the object edges, a novel depth estimation method is proposed in this paper. In the proposed method, based on the depth characteristics of the object edges, the pixels in the image are firstly classified into three categories, and then the corresponding smoothness cost functions are derived. With the modified smoothness cost functions, the depth value at the object edges with high quality can be obtained using the proposed depth estimation method. Compared with the existing depth estimation methods for the free-viewpoint television (FTV) system, the experimental results have shown that the proposed method can greatly improve the depth estimation precision for the object edges and guarantee the correctness of the edges in the synthesized virtual view at the receiver of the FTV system.

In multi-view video plus depth (MVD) format based 3D video (3DV) systems, the quality of the rendered virtual views depends highly on the quality of the compressed depth maps. Therefore, depth coding technology is vital for 3DV applications. This paper proposed a superpixel based non-regular homogenous block (NRHB) representation for low bit rate depth map coding. At the encoder side, with the aid of the superpixel segmentation of the associated texture images and the synthesis distortion estimation of the virtual views, the original depth maps are partitioned into several NRHBs and represented with only a small amount of pixels. Then, the NRHB represented depth maps are compressed by using the standard video coding approach afterward. At the decoder side, the depth maps are reconstructed through position mapping of the superpixel segmentation of the decoded texture images, and enhanced by a false edge filter. The experimental results show that the proposed method could impressively increase both the coding efficiency of the depth maps and the rendering quality of the virtual views. It also can be seen that the proposed method significantly reduced the encoding computational complexity for depth maps, which makes it particularly useful for the low bandwidth mobile consumer devices with limited computational resources.

Recurrent neural network (RNN) has proven successful recently in action recognition. However, depth sequences are of high dimensionality and contain rich human dynamics, which makes traditional RNNs difficult to capture complex action information. This paper addresses the problem of human action recognition from sequences of depth maps using sequential deep learning. The proposed method first synthesizes multi-view depth sequences by rotating 3D point clouds from depth maps. Each depth sequence is then split into short-term temporal segments. For each segment, a multi-view depth motion template (MVDMT), which compresses the segment to a motion template, is constructed for short-term multi-view action representation. The MVDMT effectively characterizes the multi-view appearance and motion patterns within a short-term duration. Convolutional Neural Network (CNN) models are leveraged to extract features from MVDMT, and a CNN-RNN network is subsequently employed to learn an effective representation for sequential patterns of the multi-view depth sequence. The proposed multi-view sequential deep learning framework can simultaneously capture spatial-temporal appearance and motion features in the depth sequence. The proposed method has been evaluated on the MSR Action3D and MSR Action Pairs datasets, achieving promising results compared with the state-of-the-art methods based on depth data.

The depth image based rendering (DIBR) is a popular technology for 3D video and free viewpoint video (FVV) synthesis, by which numerous virtual views can be generated from a single reference view and its depth image. However, some artifacts are produced in the DIBR process and reduce the visual quality of virtual view. Due to the diversity of artifacts, effectively handling them becomes a challenging task. In this paper, an artifact handling method based on depth image is proposed. The reference image and its depth image are extended to fill the holes that belong to the out-of-field regions. A depth image preprocessing method is applied to project the ghosts to their correct place. The 3D warping process is optimized by an adaptive one-to-four method to deal with the cracks and pixel overlapping. For disocclusions, we calculate depth and background terms of the filling priority based on depth information. The search for the best matching patch is performed simultaneously in the reference image and the virtual image. Moreover, adaptive patch size is used in all hole-filling processes. Experimental results demonstrate the effectiveness of the proposed method, which has better performance compared with previous methods in subjective and objective evaluation.

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.32-2.43dB, 0.0036-0.0155, 0.0041-0.0198 and 0.0012-0.0057 respectively compared with competitive methods.

In free viewpoint video, a viewer can choose at will any camera angle or the so-called “virtual view” to observe a dynamic 3-D scene, enhancing his/her depth perception. The virtual view is synthesized using texture and depth videos of two anchor camera views via depth-image-based rendering (DIBR). We consider, for the first time, collaborative live streaming of a free viewpoint video, where a group of users may interactively pull and cooperatively share streams of different anchor views. There is a cost to access the anchor views from the live source, a cost to “reconfigure” the peer network due to a change in selected anchors during view switching, and a distortion cost due to the distance of the virtual views to the received anchor views at users. We optimize the anchor views allocated to users so as to minimize the overall streaming cost given by the access cost, reconfiguration cost, and view distortion cost. We first show that, if the reconfiguration cost due to view switching is negligible, the view allocation problem can be optimally and efficiently solved in polynomial time using dynamic programming. For the case of non-negligible reconfiguration cost, the problem becomes NP-hard. We thus present a locally optimal and centralized algorithm inspired by Lloyd's algorithm used in non-uniform scalar quantization. We further propose a distributed algorithm with convergence guarantee, where each peer group independently makes merge-and-split decisions with a well-defined fairness criteria. Simulation results show that our algorithms achieve low streaming cost due to its excellent anchor view allocation.

The DIBR technique generates a 3-D perception through a 2-D color and a depth map sequences. Compressing depth map sequence is necessary for storage and transmission considerations. However, the tradeoff between bit rate reduction and quality degradation from compression should be addressed. We propose a bit rate-saving intra coding scheme which modifies, prior to compression, each depth map to minimize the prediction errors. Empirical results show that the proposed intra coding scheme can substantially reduce the bit rate while preserving the quality of the rendered virtual view.

Improving Graph Cuts algorithm to transform sequence of stereo image to depth map

In free-viewpoint television, high-quality depth maps are substantial for a virtual view synthesis for free navigation purposes. In this paper, we propose a new method of depth map quality improvement and resolution increase. Our method is intended for low resolution depth maps acquired through estimation using multiview video. The proposed depth map quality improvement is based on segmentation of an acquired high-resolution image. In the estimated depth map, searching for outliers is performed in the neighborhood created from similar segments in the acquired view. Experimental results show high effectiveness of the presented method of depth maps refinement, especially for object edges.

Multi-view video can provide users a 3-D and virtual reality perception by its multiple viewing angles. In recent years, the depth image-based rendering (DIBR) is generally used to synthesize virtual view images in free viewpoint television (FTV) and three-dimensional (3-D) video. In order to conceal the zero-region more accurately and improve the quality of virtual view synthesized frame, an integrated Hole-filling Algorithm for View Synthesis is proposed in this paper. It contains five parts: the different regions distinguishing algorithm, foreground and background boundary detection, the texture image isophotes detection, the textural and structural isophotes prediction algorithm, the in-painting algorithm with gradient priority order. Based on the texture isophotes prediction with geometrical principle and the in-painting algorithm with gradient priority order, the boundary information of the foreground is much clearer and the texture information in the zero-region can be concealed much more accurately than the previous work. The vision quality mainly depends on the distortion of the structural information. Through the experimental results, the proposed algorithm not only improves the objective quality of the virtual image, but also improves the subjective quality of the virtual image a lot, and the human vision quality is also improved obviously based on the subjective results. Especially, it ensures the boundary contours of the foreground objects and the textural and the structural information.KeywordsDIBRVirtual viewHole-fillingIn-painting

Human action recognition based on the depth maps is an important yet challenging task. In this paper, a new framework based on the 3D motion trail model (3DMTM) and Pyramid Histograms of Oriented Gradient (PHOG) is proposed to recognize human actions from sequences of depth maps. Specifically, a discriminative descriptor called 3DMTM-PHOG is proposed for depth-based human action recognition. The 3DMTM is generated through the entire depth video sequence to encode additional motion information from three projected orthogonal planes. By adding pyramid representation, Histograms of Oriented Gradient (HOG) descriptor is extended to PHOG which can well characterize local shapes at different spatial grid sizes for action recognition. PHOG is then computed from the 3DMTM as the 3DMTM-PHOG descriptor for the representation of an action. The proposed approach based on 3DMTM-PHOG descriptor is evaluated on MSR Action3D dataset captured by depth cameras. Experimental results show that the proposed approach outperforms the state-of-the-art methods and demonstrate the effectiveness and robustness of the proposed 3DMTM-PHOG descriptor.