A Multiple Hexagon Search Algorithm for Motion and Disparity Estimation in Multiview Video Coding

Multiview video coding (MVC) systems have attracted much attention because of their applications in free view point television (FTV), three-dimensional television (3DTV), surveillance systems, etc. Compared with the traditional 2D media, multiview videos usually involve large data size. It is essential to develop high performance MVC systems with low computational complexity. This thesis aims to develop an efficient multiview video codec based on the Audio Video Coding Standard of China (AVS) and investigate its application to image-based rendering (IBR).
\n
\nFirstly, a new fast motion estimation (ME) algorithm based on the simplified unsymmetrical hexagonal method (UMHexagonS) used in the AVS standard is developed. Different from the conventional method, the proposed approach uses an extended searching area and a square-based search to reduce computation complexity considerably. To obtain real-time implementation, we proposed a macroblock-based parallel structure which utilizes multi-core processing. Such approach can improve the encoding frame rate by approximately 4 times compared to traditional implementations. The proposed method has been tested on an Intel Core i7 3960x six cores 3.3GHz CPU. The real-time encoding speed reaches 25 frames per second (fps) for a 1280×720 high-definition video.
\n
\nSecondly, a multiview video coding algorithm based on AVS which encode three views as a group is developed. By taking advantages of the reduced inter-view dependency among adjacent views, the proposed algorithm can simplify the selective transmission and the decoding of multiview videos. Moreover, a new disparity estimation (DE) algorithm based on an unsymmetrical search window is proposed to further reduce the computational complexity. The developed encoder can achieve real-time encoding for 640×480 3-view videos at more than 26fps on a computer equipped with an Intel Core i7 3960x six cores 3.3GHz CPU.
\n
\nFinally, an image-based rendering and object-based compression system based on the Kinect depth camera and algorithm are developed for indoor applications. The proposed system uses the depth information obtained by the Kinect depth camera together with the texture information to perform IBR. Experimental results illustrate that good performance and usefulness can be achieved by the proposed approach.

Publication in the conference proceedings of EUSIPCO, Rhodes, Greece, 1998

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

Multilevel region of interest guided bit allocation for multiview video coding

Block matching motion estimation (BMME) plays a very important role in video coding. The performance of BMME greatly affects the quality of the encoded bit stream. In order to reduce the high computational demands of H.264 video encoder, a fast block matching motion estimation algorithm based on temporal and spatial motion correlation is proposed by the deep research of the characteristics of motion estimation. This algorithm well exploits the correlation between video sequence motion activity and macro-block coding pattern and motion vector statistic characteristics to employ different patterns. Experimental results show that the algorithm, without function losing of PNSR, can not only reduce the search complexity of motion estimation and the high computational demands but also raise the real-time performance of encoder.

Multi-view video coding (MVC) is an ongoing standard in which variable size disparity estimation (DE) and motion estimation (ME) are both employed to select the best coding mode for each macroblock (MB). This technique achieves the highest possible coding efficiency, but it results in extremely large encoding time which obstructs it from practical use. In this paper, a fast DE and ME algorithm based on motion homogeneity is proposed to reduce MVC computational complexity. The basic idea of the method is to utilize the spatial property of motion field in prediction where DE and variable size ME are needed, and only in these regions DE and variable size ME are enabled. The motion field is generated by the corresponding motion vectors (MVs) in spatial window. Simulation results show that the proposed algorithm can save 63% average computational complexity, with negligible loss of coding efficiency.

For video compression, motion estimation is popularly employed to exploit temporal correlation existing in video sequences. If we employ the full search block matching algorithm for estimating motion vectors, it requires very heavy computational complexity. Although several fast block matching algorithms have been proposed to solve this problem, they sacrifice their reconstructed image quality. In this paper, we derive optimal search patterns for fast block matching motion estimation. By analyzing the block matching algorithm as a function of the block size and the shape, we find optimal search patterns for initial motion estimation. The proposed idea can provide an analytical ground for the current MPEG-2 proposals. In addition, we propose a new fast motion estimation algorithm using adaptive search patterns, considering matching criteria and statistical properties of object displacement. In order to select an appropriate search pattern, we exploit the relationship between the motion vector and the frame difference of each block. By changing the search pattern adaptively, we can improve the motion prediction accuracy, while reducing the required computational complexity compared to other fast block matching algorithms.

Multi-view video consists of multiple views which are taken by multiple cameras from different positions and angles. It allows the users to freely change their viewpoints. Typical applications of multi-view video include Free Viewpoint TV, remote medical surgery, wireless multimedia sensor networks and so on. However, the traffic of multi-view video is also several times larger than traditional multimedia, which brings much more increment in the bandwidth requirement. To overcome this problem, Simulcast and MVC (Multi-view Video Coding) can be used to decrease the traffic for multi-view video. However, although Simulcast removes redundant information within each view, it still contains a large amount of inter-view redundant information. MVC further removes the correlation between the views, but all the views should be encoded and transmitted. So the bit-rates are still too high in these schemes. In this paper, in order to reduce the bit-rate for multi-view video transmission, we analyze the user's motion which is classified into random access model and successive motion model. We find that in the successive motion model only part of frames are possible to be displayed in a period of time. According to the analytical result, we propose a user dependent system to decrease the bit-rate of multi-view video transmission. The proposed system only encodes and transmits those frames that are possible to be displayed according to the user's motion. Further, in order to support the proposed system, we also propose a prediction structure as a substitute of the prediction structure in MVC. Performance evaluation proves that the proposed system reduces the average bit-rate for the transmission of multi-view video and is to be more effective when the number of views is large.

The multiview video coding (MVC) adopts hierarchical B picture prediction structure and offers many prediction modes to effectively remove the spatial, temporal, and inter-view redundancies inherited in multiview video (MVV), but at the price of extremely high computational complexity. To address this problem, a fast MVC method by jointly using adaptive prediction structure (APS) and hierarchical mode decision (HMD) is proposed in this paper. The complexity reduction is achieved by: 1) designing four APSs for different MVV contents based on the fact that the contribution of the inter-view prediction varies from sequence to sequence and 2) developing an HMD scheme based on the observation that the relationship between the rate distortion (RD) cost and size of prediction mode is a unimodal function. In particular, for the current group of picture of the input MVV, the prediction structure is adaptively selected based on its characteristic, which is measured by the ratio of the average RD cost of the base view frames to the sum of the average RD cost of the base view frames and that of anchor frames in nonbase views, and then an HMD scheme is further performed to skip the checking process of those unlikely modes. The experimental results have shown that compared with the exhaustive mode decision in the MVC, the proposed algorithm achieves a reduction of the computational complexity by 83.49% on average, whereas incurring only a 0.086 dB loss in Bjontegaard delta peak signal-to-noise ratio and 2.97% increment on the total Bjontegaard delta bit rate.

In recent years, many fast Block Matching Algorithm or known as BMA have been proposed and developed in order to reduce the computational cost while maintaining the quality of the video signal. In this paper, the algorithm proposed is called Orthogonal-Diamond Search (ODS) which employs an orthogonal-shaped search pattern in the first step and then is switched into diamond-shaped search pattern for the next step. The proposed algorithm and several established algorithms namely Three Step Search (TSS), New-Three-Step Search (NTSS), Four-Step Search (4SS), Diamond Search (DS) and Hexagon-Diamond Search (HDS) are implemented using MATLAB and their performance are being compared and analyzed in terms of number of search point, peak signal-to-noise ratio (PSNR) and computational complexity. Simulation result show that the proposed algorithm can find motion vector with fever number of search points while maintains close performance of video quality with other selected algorithms.

Early SKIP mode decision for MVC using inter-view correlation

The traffic of multi-view video is several times larger than traditional multimedia, which brings much more increment in the bandwidth requirement. To overcome this problem, Simulcast and MVC (Multi-view Video Coding) can be used to decrease the traffic for multi-view video. However, although Simulcast removes redundant information within each view, it still contains a large amount of inter-view redundant information. MVC further removes the correlation between the views, but all the views should be encoded and transmitted. So the bit-rates are still too high in these schemes. In this paper, in order to reduce the bit-rate for multi-view video transmission, we analyze the user's motion which is classified into random access model and successive motion model. We find that in the successive motion model only part of frames are possible to be displayed in a period of time. According to the analytical result, we propose a user dependent scheme to decrease the bit-rate of multi-view video transmission. The proposed scheme only encodes and transmits those frames that are possible to be displayed according to the user's motion. Further, in order to support the proposed scheme, we also propose a prediction structure as a substitute of the prediction structure in MVC. Performance evaluation proves that this proposed scheme reduces the average bit-rate for the transmission of multi-view video.

Multiview video has gained a wide interest recently. The huge amount of data needed to be processed by multiview applications is a heavy burden for both transmission and decoding. The joint video team has recently devoted part of its effort to extend the widely deployed H.264/AVC standard to handle multiview video coding (MVC). The MVC extension of H.264/AVC includes a number of new techniques for improved coding efficiency, reduced decoding complexity, and new functionalities for multiview operations. MVC takes advantage of some of the interfaces and transport mechanisms introduced for the scalable video coding (SVC) extension of H.264/AVC, but the system level integration of MVC is conceptually more challenging as the decoder output may contain more than one view and can consist of any combination of the views with any temporal level. The generation of all the output views also requires careful consideration and control of the available decoder resources. In this paper, multiview applications and solutions to support generic multiview as well as 3D services are introduced. The proposed solutions, which have been adopted to the draft MVC specification, cover a wide range of requirements for 3D video related to interface, transport of the MVC bitstreams, and MVC decoder resource management. The features that have been introduced in MVC to support these solutions include marking of reference pictures, supporting for efficient view switching, structuring of the bitstream, signalling of view scalability supplemental enhancement information (SEI) and parallel decoding SEI.

In this paper, a new rate---distortion (R---D) optimized error-resilient scheme is proposed to improve error resilience for multi-view video transmission over lossy networks. Based on the study on the characteristics of multi-view video coding and the propagating behavior of channel errors, a recursive model to estimate the end-to-end distortion is first developed, in which the channel-induced distortion takes into consideration both motion and disparity compensation prediction. Then, based on the estimated distortion and the derived total bit rate, R---D optimized motion and disparity estimation are employed to find the reliable motion vector and disparity vector to achieve the optimum trade-off between prediction efficiency and error robustness. Finally, R---D optimization is applied again to perform coding mode decision, which explicitly considers intra, inter and inter-view prediction modes. Extensive experimental results demonstrate significant performance gains can be achieved for multi-view video communications against transmission errors.

Multiview video coding (MVC) plays an important role in a 3-D video system. In addition, the resolution of HDTV is increasing to present more vivid perception for users. To realize real-time processing of dozens of TOPS, VLSI solution is necessary. However, ultra high computational complexity, a large amount of external memory bandwidth and on-chip SRAM size, and complex MVC prediction structures are three main design challenges of implementation of MVC hardware architecture. In this paper, an MVC single-chip encoder is proposed for H.264/AVC Multiview High Profile and High Profile for 3-D and quad full high definition (QFHD) TV applications, respectively. The 4096 × 2160 p multiview video encoder chip is implemented on a 11.46 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> die with 90 nm CMOS technology. An eight-stage macroblock pipelined architecture with proposed system scheduling and cache-based prediction core supports real-time processing from one-view 4096 × 2160 p to seven-view 720 p videos. The 212 Mpixels/s throughput is 3.4 to 7.7 times higher than previous work. The 407 Mpixels/W power efficiency is achieved, and 94% on-chip SRAM size and 79% external memory bandwidth are saved by the proposed techniques.