Light Field Multi-View Video Coding With Two-Directional Parallel Inter-View Prediction.

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, inter-view and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multi-view video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access.

In this article we present a novel dictionary learning framework designed for compression and sampling of light fields and light field videos. Unlike previous methods, where a single dictionary with one-dimensional atoms is learned, we propose to train a Multidimensional Dictionary Ensemble (MDE). It is shown that learning an ensemble in the native dimensionality of the data promotes sparsity, hence increasing the compression ratio and sampling efficiency. To make maximum use of correlations within the light field data sets, we also introduce a novel nonlocal pre-clustering approach that constructs an Aggregate MDE (AMDE). The pre-clustering not only improves the image quality but also reduces the training time by an order of magnitude in most cases. The decoding algorithm supports efficient local reconstruction of the compressed data, which enables efficient real-time playback of high-resolution light field videos. Moreover, we discuss the application of AMDE for compressed sensing. A theoretical analysis is presented that indicates the required conditions for exact recovery of point-sampled light fields that are sparse under AMDE. The analysis provides guidelines for designing efficient compressive light field cameras. We use various synthetic and natural light field and light field video data sets to demonstrate the utility of our approach in comparison with the state-of-the-art learning-based dictionaries, as well as established analytical dictionaries.

Light field has great applications in AR/VR. It is particu-larly usefulfor resolving the vergence-accommodation con-flict (VAC) and creating correct depth cues for AR/VR dis-plays. However, the source data, especially light field video, are not widely available yet. To resolve the scarcity is-sue, one may resort to data such as stereo image sequences that are commonly available. In this paper, we propose an end-to-end deep learning framework for synthesizing light field sequences from stereo image sequences. Our frame-work consists of a disparity estimation network, a guided synthesis network, and a refinement network and is able to resolve the flickering issue caused by temporal incon-sistency, an artifact that is commonly seen in synthesized light field videos. Our experimental results are quantitatively and qualitatively better than the results of existing light field synthesis algorithms that were originally developed for static light fields.

The drastic increasing of multimedia applications, such as IPTV, Virtual Reality (VR, 360°) and Light Field videos has led to a high computing complexity in video compression and content quality assessment. In the last five years, HEVC standard has been widely used in the industrial community due to its bit-rate gain compared to its predecessor H.264/AVC. Recently, a new coding tools have been developed under the Joint Exploration Model (JEM) software, with the main goal to provide high bit rate saving compared to the HEVC standard. In this paper we investigate the performance and the associated computational complexity of these emerging video coding tools, from both encoding and decoding sides. Two spatial resolutions (HD & 4K) and several video contents have been used in this study. Results have shown that despite the bit-rate saving, a considerable computational complexity can be noticed. A bit-rate saving up to 37% and quality enhancements up to 30% can be achieved by JEM. However, these emerging tools are time consuming between x5 and x12 times compared to the HM reference software, depending on video sequence and encoding/decoding processes.

Multidimensional Signal, Image, and Video Processing and Coding

In multiview video coding (MVC), disparity-compensated prediction (DCP) exploits the correlation among different views. A common approach is to use block-based motion-compensated prediction (MCP) tools to predict the disparity effect among different views. However, some regions in different views may have various deformations due to nonconstant depth. Thus, performance of DCP is not satisfactory with the simple translational model assumed in conventional block-based MCP tools. Previous attempts to achieve better disparity prediction were usually too complex for practical use. In this paper, horizontal scaling and shearing (HSS) effects are investigated to increase interview prediction accuracy for stereo video. HSS deformations are common among images of horizontally aligned views, due to horizontal and vertical flat surfaces that are not parallel with projection image planes. To achieve HSS-based DCP with minimal complexity, an efficient subsampled block-matching technique is adopted and integrated into MVC extension of H.264/AVC in stereo profile. Affine parameters estimation and additional frame buffers are not required and the overall increase of computational complexity and memory requirements are moderate. Experimental results show that the new technique can achieve up to 5.25% bitrate reduction in interview prediction using JM17.0 reference software implementation.

Light fields are becoming an increasingly popular method of digital content production for visual effects and virtual/augmented reality as they capture a view dependent representation enabling photo realistic rendering over a range of viewpoints. Light field video is generally captured using arrays of cameras resulting in tens to hundreds of images of a scene at each time instance. An open problem is how to efficiently represent the data preserving the view-dependent detail of the surface in such a way that is compact to store and efficient to render. In this paper we show that constructing an Eigen texture basis representation from the light field using an approximate 3D surface reconstruction as a geometric proxy provides a compact representation that maintains view-dependent realism. We demonstrate that the proposed method is able to reduce storage requirements by > 95% while maintaining the visual quality of the captured data. An efficient view-dependent rendering technique is also proposed which is performed in eigen space allowing smooth continuous viewpoint interpolation through the light field.

Multiview video has gained a wide interest recently. The huge amount of data needed to be processed by multiview applications is a heavy burden for both transmission and decoding. The joint video team has recently devoted part of its effort to extend the widely deployed H.264/AVC standard to handle multiview video coding (MVC). The MVC extension of H.264/AVC includes a number of new techniques for improved coding efficiency, reduced decoding complexity, and new functionalities for multiview operations. MVC takes advantage of some of the interfaces and transport mechanisms introduced for the scalable video coding (SVC) extension of H.264/AVC, but the system level integration of MVC is conceptually more challenging as the decoder output may contain more than one view and can consist of any combination of the views with any temporal level. The generation of all the output views also requires careful consideration and control of the available decoder resources. In this paper, multiview applications and solutions to support generic multiview as well as 3D services are introduced. The proposed solutions, which have been adopted to the draft MVC specification, cover a wide range of requirements for 3D video related to interface, transport of the MVC bitstreams, and MVC decoder resource management. The features that have been introduced in MVC to support these solutions include marking of reference pictures, supporting for efficient view switching, structuring of the bitstream, signalling of view scalability supplemental enhancement information (SEI) and parallel decoding SEI.

Multi-Intensity Illuminated Infrared (MIIR) video is a new type of infrared video which solves the problem of improper illumination conditions for ordinary infrared cameras for nighttime surveillance. However, its time-varying intensity makes it hard to be compressed with traditional video coding methods. In this paper we use MV-HEVC and 3D-HEVC video coding standards, which are originally designed for Multi-view and 3D video coding, to compress the MIIR videos. With the proposed encoding method, the coding structures of MV- and 3D-HEVC can be employed to compress the MIIR video in a more efficient way. Experimental results show that the coding structures of MV- and 3D-HEVC can compress such video in a more efficient way, i.e., up to 43.8%/50.1% bitrate reductions with MV/3D-HEVC compared with standalone HEVC compression and 74.3%/80.5% bitrate reductions compared with simulcast HEVC coding in the enhancement layers.

Multiview video coding has been attracting increased popularity for facilitating stereo-vision and free-viewpoint video applications. Compares to conventional single view video coding, multiview video coding utilizes inter and intra view motion compensations to yield an improved coding efficiency, and multiple description video coding helps improving video streaming over multiple lossy channels; on the other hand, multiview and multiple description coding techniques increase video frame dependencies, with trades off video streaming quality with an increased drifting error probability. This paper examines the frame dependencies of multi-view video coding against configurable parameters such as the length of group-of-pictures, and the number of viewpoints.

This study proposes a framework of video coding based on Laplacian eigenmaps (LEM) and its related embedding and reconstruction algorithm (ERA). Firstly, a one-dimensional (1D) representation of LEM is adopted to achieve an extremely low bit per pixel (BPP). Secondly, dual k-nearest neighbours, which keeps neighbour relationships both in high-dimensional data space and low-dimensional representation space and overcomes the disadvantage of classical non-linear dimensionality reduction methods which cannot preserve the neighbour properties in both of the spaces, based ERA of LEM is employed to gain extraordinarily high peak-signal-to-noise ratio (PSNR). Thirdly, a unified framework of video coding is fit for intra-frame, inter-frame and multi-view video coding. Finally, it is evaluated by simulation experiments that, in the situation of low bitrate transmission, the proposed method can attain better performance of BPP and PSNR than that of the state-of-the-art methods, such as highly efficient video coding.

High Efficiency Video Coding (HEVC) and VP9 are the latest state-of-the-art high-efficiency video coding standards. Both standards aim to decrease the bit-rate while improving the compression efficiency. HEVC/H.265 video standard was collaboratively developed by ITU-T and ISO/IEC with up to 50% bit-rate reduction as compared to its predecessor H.264/AVC. On the other hand, VP9 is an open source video standard developed by Google. While maintaining the same video quality, VP9 has achieved 50% bit-rate reduction as compared to its predecessor VP8. This research paper focuses on the comparison of HEVC and VP9 based on both subjective as well as objective assessments. First, detailed differences in design and specification of the two standards are discussed. Later experiment results are provided, which compare the coding efficiency of both standards on various high-definition 720p, 1090p, and 2160p sample test bench videos. According to our experimental results, it has been empirically proved that HEVC/H.265 outperforms VP9 in bit-rate savings.

Light field imaging is becoming a key technology, which provides users with a realistic visual experience through the capability of dynamic viewpoint shifting. This ability comes at the cost of capturing huge amounts of information, leaving the problem of compression and transmission a challenge. The encoder complexity is the key to achieve efficient coding in conventional light field coding schemes, where a complicated prediction process is essentially used at the encoder side to exploit the redundancy present in the light field image. We employ Distributed Source Coding (DSC) for light field images, which can extensively lift the computational requirement from the encoding side at the expense of increased computational complexity at the decoder side. The efficiency of DSC is heavily dependent on the quality of side information at the decoder. Therefore, we propose to leverage a learning-based view synthesis method, which takes into account the light field structure to generate high-quality side information. We compare our approach to Distributed Video Coding and Distributed Multi-view Video Coding schemes adapted to the light field framework and relevant standard-based approach, and demonstrate that the proposed view synthesis-based approach can achieve similar performance, while substantially reducing the number of key views to be transmitted.

Multiview video coding (MVC) systems have attracted much attention because of their applications in free view point television (FTV), three-dimensional television (3DTV), surveillance systems, etc. Compared with the traditional 2D media, multiview videos usually involve large data size. It is essential to develop high performance MVC systems with low computational complexity. This thesis aims to develop an efficient multiview video codec based on the Audio Video Coding Standard of China (AVS) and investigate its application to image-based rendering (IBR).
\n
\nFirstly, a new fast motion estimation (ME) algorithm based on the simplified unsymmetrical hexagonal method (UMHexagonS) used in the AVS standard is developed. Different from the conventional method, the proposed approach uses an extended searching area and a square-based search to reduce computation complexity considerably. To obtain real-time implementation, we proposed a macroblock-based parallel structure which utilizes multi-core processing. Such approach can improve the encoding frame rate by approximately 4 times compared to traditional implementations. The proposed method has been tested on an Intel Core i7 3960x six cores 3.3GHz CPU. The real-time encoding speed reaches 25 frames per second (fps) for a 1280×720 high-definition video.
\n
\nSecondly, a multiview video coding algorithm based on AVS which encode three views as a group is developed. By taking advantages of the reduced inter-view dependency among adjacent views, the proposed algorithm can simplify the selective transmission and the decoding of multiview videos. Moreover, a new disparity estimation (DE) algorithm based on an unsymmetrical search window is proposed to further reduce the computational complexity. The developed encoder can achieve real-time encoding for 640×480 3-view videos at more than 26fps on a computer equipped with an Intel Core i7 3960x six cores 3.3GHz CPU.
\n
\nFinally, an image-based rendering and object-based compression system based on the Kinect depth camera and algorithm are developed for indoor applications. The proposed system uses the depth information obtained by the Kinect depth camera together with the texture information to perform IBR. Experimental results illustrate that good performance and usefulness can be achieved by the proposed approach.

There are several data formats available for 3D video, among which is the Multiview Video plus Depth (MVD) representation that enables depth-image-based rendering (DIBR). In addition to DIBR, the MVD data format can enable more efficient compression for texture video coding. In this paper, we propose advanced motion vector prediction (MVP) techniques which utilize the availability of depth information in the MVD format for more efficient coding of the corresponding texture pictures. The proposed MVP was implemented on top of the Multiview Video Coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard and its reference software and tested under the simulation conditions of the Call for Proposals of 3D video coding technology issued by the Moving Picture Experts Group (MPEG). In the provided simulation results the proposed scheme significantly outperformed the conventional MVP (by up to 9.9% and on average by 7.5% in Bjontegaard delta bitrate) when it was applied for coding of 3-view MVD data.