Inter-frame dependency in multiview multi-description video streaming

The use of multiple description (MD) video coding and path diversity has been proposed to provide improved performance over lossy packet networks [1]. The goal of this work was to develop models to accurately and quickly predict and compare the distortion of MD video coding and path diversity against conventional single description (SD) video delivered over a single path. In the process, we developed (1) a model for the loss process of a two-path path diversity system, and (2) a distortion model that maps the loss model to MD distortion values. Given these models we present a number of comparisons between MD video coding and path diversity and conventional SD video over a single path. The proposed model for path diversity may also be useful in other applications not related to MD coding. Furthermore, other forms of MD coding may be analyzed using similar models for MD distortion.

Multiview video has gained a wide interest recently. The huge amount of data needed to be processed by multiview applications is a heavy burden for both transmission and decoding. The joint video team has recently devoted part of its effort to extend the widely deployed H.264/AVC standard to handle multiview video coding (MVC). The MVC extension of H.264/AVC includes a number of new techniques for improved coding efficiency, reduced decoding complexity, and new functionalities for multiview operations. MVC takes advantage of some of the interfaces and transport mechanisms introduced for the scalable video coding (SVC) extension of H.264/AVC, but the system level integration of MVC is conceptually more challenging as the decoder output may contain more than one view and can consist of any combination of the views with any temporal level. The generation of all the output views also requires careful consideration and control of the available decoder resources. In this paper, multiview applications and solutions to support generic multiview as well as 3D services are introduced. The proposed solutions, which have been adopted to the draft MVC specification, cover a wide range of requirements for 3D video related to interface, transport of the MVC bitstreams, and MVC decoder resource management. The features that have been introduced in MVC to support these solutions include marking of reference pictures, supporting for efficient view switching, structuring of the bitstream, signalling of view scalability supplemental enhancement information (SEI) and parallel decoding SEI.

Multidimensional Signal, Image, and Video Processing and Coding

Multiview video coding (MVC) systems have attracted much attention because of their applications in free view point television (FTV), three-dimensional television (3DTV), surveillance systems, etc. Compared with the traditional 2D media, multiview videos usually involve large data size. It is essential to develop high performance MVC systems with low computational complexity. This thesis aims to develop an efficient multiview video codec based on the Audio Video Coding Standard of China (AVS) and investigate its application to image-based rendering (IBR).
\n
\nFirstly, a new fast motion estimation (ME) algorithm based on the simplified unsymmetrical hexagonal method (UMHexagonS) used in the AVS standard is developed. Different from the conventional method, the proposed approach uses an extended searching area and a square-based search to reduce computation complexity considerably. To obtain real-time implementation, we proposed a macroblock-based parallel structure which utilizes multi-core processing. Such approach can improve the encoding frame rate by approximately 4 times compared to traditional implementations. The proposed method has been tested on an Intel Core i7 3960x six cores 3.3GHz CPU. The real-time encoding speed reaches 25 frames per second (fps) for a 1280×720 high-definition video.
\n
\nSecondly, a multiview video coding algorithm based on AVS which encode three views as a group is developed. By taking advantages of the reduced inter-view dependency among adjacent views, the proposed algorithm can simplify the selective transmission and the decoding of multiview videos. Moreover, a new disparity estimation (DE) algorithm based on an unsymmetrical search window is proposed to further reduce the computational complexity. The developed encoder can achieve real-time encoding for 640×480 3-view videos at more than 26fps on a computer equipped with an Intel Core i7 3960x six cores 3.3GHz CPU.
\n
\nFinally, an image-based rendering and object-based compression system based on the Kinect depth camera and algorithm are developed for indoor applications. The proposed system uses the depth information obtained by the Kinect depth camera together with the texture information to perform IBR. Experimental results illustrate that good performance and usefulness can be achieved by the proposed approach.

The use of interview prediction and hierarchical pictures in the Multiview Video Coding (MVC) extension of the H.264 Advance Video Coding (AVC) standard produces better compression efficiency in multi-view video. This however, results in a bitstream that is considerably more prone to transmission errors than its simulcast coding counterpart. In this paper, we present an experimental evaluation of the transmission of MVC coded multi-view video over multiple RTP sessions. Our results show that by separating the transmission of frames based on their level of importance and protecting the important streams, we can minimise the effect of transmission errors in the hierarchical B frames. We also discuss the use of multi-session transmission in delivering a more adaptable bitstream where the number of streams can be adjusted based on the network conditions. In extreme circumstances all but the base view stream can be dropped to avoid further congesting the network and to deliver at least one high quality view to the end user.

In single viewpoint video coding, there are many fast block matching motion estimation algorithms proposed, such as three-step search, four-step search, diamond search as well as hexagon-based search and so on. However, experimental analysis show that these algorithms are not suitable for using directly in Multiview View Coding (MVC). Since the increased search range, the larger format of multiview video as well as the correlations between inter-view frames are not considered by these algorithms, they may easy led the block matching search into local minimum, the Rate-Distortion (R-D) performance will degrade dramatically. In this paper, we propose a novel multiple hexagon search algorithm to address this problem. Firstly, according to the original initial search point, four sub-search windows are constructed. Then, the hexagon based search algorithm will be performed respectively in the four sub-search windows. The final result is the best search point with the minimum R-D cost among the best points in the four sub-search windows. In order to trade off the computational complexity and R-D performance, two adaptive early termination strategies are proposed. The experimental results show that the proposed algorithm yields a quite promising coding performance in terms of R-D performance and computational complexity. Especially, the proposed algorithm can work well in multiview video sequences with various motion and disparity activities.

Multi-view video consists of multiple views which are taken by multiple cameras from different positions and angles. It allows the users to freely change their viewpoints. Typical applications of multi-view video include Free Viewpoint TV, remote medical surgery, wireless multimedia sensor networks and so on. However, the traffic of multi-view video is also several times larger than traditional multimedia, which brings much more increment in the bandwidth requirement. To overcome this problem, Simulcast and MVC (Multi-view Video Coding) can be used to decrease the traffic for multi-view video. However, although Simulcast removes redundant information within each view, it still contains a large amount of inter-view redundant information. MVC further removes the correlation between the views, but all the views should be encoded and transmitted. So the bit-rates are still too high in these schemes. In this paper, in order to reduce the bit-rate for multi-view video transmission, we analyze the user's motion which is classified into random access model and successive motion model. We find that in the successive motion model only part of frames are possible to be displayed in a period of time. According to the analytical result, we propose a user dependent system to decrease the bit-rate of multi-view video transmission. The proposed system only encodes and transmits those frames that are possible to be displayed according to the user's motion. Further, in order to support the proposed system, we also propose a prediction structure as a substitute of the prediction structure in MVC. Performance evaluation proves that the proposed system reduces the average bit-rate for the transmission of multi-view video and is to be more effective when the number of views is large.

The traffic of multi-view video is several times larger than traditional multimedia, which brings much more increment in the bandwidth requirement. To overcome this problem, Simulcast and MVC (Multi-view Video Coding) can be used to decrease the traffic for multi-view video. However, although Simulcast removes redundant information within each view, it still contains a large amount of inter-view redundant information. MVC further removes the correlation between the views, but all the views should be encoded and transmitted. So the bit-rates are still too high in these schemes. In this paper, in order to reduce the bit-rate for multi-view video transmission, we analyze the user's motion which is classified into random access model and successive motion model. We find that in the successive motion model only part of frames are possible to be displayed in a period of time. According to the analytical result, we propose a user dependent scheme to decrease the bit-rate of multi-view video transmission. The proposed scheme only encodes and transmits those frames that are possible to be displayed according to the user's motion. Further, in order to support the proposed scheme, we also propose a prediction structure as a substitute of the prediction structure in MVC. Performance evaluation proves that this proposed scheme reduces the average bit-rate for the transmission of multi-view video.

There are several data formats available for 3D video, among which is the Multiview Video plus Depth (MVD) representation that enables depth-image-based rendering (DIBR). In addition to DIBR, the MVD data format can enable more efficient compression for texture video coding. In this paper, we propose advanced motion vector prediction (MVP) techniques which utilize the availability of depth information in the MVD format for more efficient coding of the corresponding texture pictures. The proposed MVP was implemented on top of the Multiview Video Coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard and its reference software and tested under the simulation conditions of the Call for Proposals of 3D video coding technology issued by the Moving Picture Experts Group (MPEG). In the provided simulation results the proposed scheme significantly outperformed the conventional MVP (by up to 9.9% and on average by 7.5% in Bjontegaard delta bitrate) when it was applied for coding of 3-view MVD data.

In this paper, we propose a joint coding scheme for both multi-view video and its corresponding depth map. After we synthesize a virtual image for the target view using adjacent view images and their depth information, we apply a view interpolation prediction (VIP) method for both multi-view video coding and its depth data coding. In order to improve the synthesized virtual view, we also propose a hole filling method that can compensate for empty regions caused by the 3D warping operation. With the proposed algorithm, we have obtained approximately 0.65 dB of the PSNR gain on average for the multi-view depth data, and 0.17 dB of the PSNR gain for the multi-view video data, compared to JMVM 1.0.

Fast synthesized and predicted just noticeable distortion maps for perceptual multiview video coding

Multiple Description Video Coding (MDC) and Layered Coding (LC) are both error-resilient source coding techniques used for transmission over error-prone channels. Both techniques generate multiple streams. The streams generated by MDC correspond to different descriptions of the same source whereas the streams produced by LC are differentiated as base and enhancement layer streams. Moreover whereas the MDC streams are independently decodable the decoding of the enhancement layer stream is dependent on the decoding of the base layer stream. In this work we concentrate on specific MDC and LC schemes, i.e. Multi-State Video Coding (MSVC) and Temporal Layered Coding (TLC). MSVC was introduced by John Apostolopoulos and it was showed that if each frame is transmitted in a separate packet and if motion information for each lost frame is also lost, MSVC outperforms Single Layer Coding (SC) in recovering from single as well as burst losses. Here we compared MSVC with TLC as an extension of SC based on transmission simulations over lossy channels under the assumption that the motion vectors are always available. Using different coding modes and specific reconstruction methods average reconstructed frame PSNR (peak signal to noise ratio) is measured and compared. Results show that when motion vectors are received TLC performs better than MSVC for every coding option tested. The performance difference is bigger for low motion sequences.

Multilevel region of interest guided bit allocation for multiview video coding

In several telepresence applications the transmitting end is resource constrained, hence multi-view video has to be source coded and transmitted to a location where stereo matching is performed. The effect of video source coding on stereo matching techniques is presently not well understood. In this paper, we present an analysis which quantitatively demonstrates the impact of standard video coding techniques on the quality of well-known stereo matching algorithms. We propose a quantitative evaluation methodology and framework. Evaluation is also presented for the state-of-the-art Annex H of H.264/AVC (multi-view coding). The results shed a light on the tradeoff between video coding parameters and the quality of dense stereo matching. This enables us to select suitable operating regions for both stereo matching and video coding, for a diverse range of applications. In addition, various system configurations with varying performance-complexity tradeoff are also compared, giving insight on selecting the configuration suitable for a variety of telepresence applications.

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, inter-view and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multi-view video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access.