An integrated approach to block based motion estimation for video coding

Motion estimation (ME) is one of the most critical tools in video coding and consumes the majority of the encoding complexity. Three types of ME are utilized in the latest video coding standards, namely integer, fractional, and affine MEs. They are implemented as three searches for the integer motion vector (IMV), fractional motion vector (FMV), and control point motion vectors (CPMVs). Many algorithms were proposed to reduce the complexity for them individually, but the overall overhead of three searches is still challenging for hardware implementations. Therefore, we propose a hardware-efficient Unified Motion Estimation (UME) to derive three types of MVs with only one search. An IME with sub-block refinement is performed to collect extra motion information while searching for the IMV. The FMV and CPMVs are then derived from the collected information using a mixed error surface and an overdetermined system. Compared to the default ME algorithms in VVC, the time cost for ME is reduced by 41.63% with a coding loss of only 0.87% under LDB configuration. For hardware implementations, the minimum required resources and corresponding latency are significantly reduced by 75.35% and 69.17%, respectively.

In this paper, a new optimised method of coding stereoscopic image sequences is presented and compared with already known methods. Two basic methods of coding a stereoscopic image sequence are the compatible and joint. The first one uses MPEC for coding the left channel and takes advantage of the spatial disparity redundancy between the two sequences for coding the right channel. The second one employs MPEG for coding the left channel but takes advantage of both temporal redundancy among the right channel frames and spatial redundancy between the corresponding frames of the two channels The proposed method, which is called IMDE, estimates the P and B type of frames of the right channel by an interpolative scheme that takes in to account both the temporal and disparity characteristics. Investigating the effectiveness of the joint motion and disparity vectors estimation as well as the choice of the weighting factors that participate in the proposed interpolative scheme optimises the whole framework.

This thesis presents the rate distortion analysis of conditional motion estimation, a process in which motion computation is restricted to only active pixels in the video. We model active pixels as independent and identically distributed Gaussian process and inactive pixels as Gaussian-Markov process and derive the rate distortion function based on conditional motion estimation. Rate-Distortion curves for the conditional motion estimation scheme are also presented. In addition this thesis also presents the hardware implementation of a block based motion estimation algorithm. Block matching algorithms are difficult to implement on FPGA chip due to its complexity. We implement 2D-Logarithmic search algorithm to estimate the motion vectors for the image. The matching criterion used in the algorithm is Sum of Absolute Differences (SAD). VHDL code for the motion estimation algorithm is verified using ISim and is implemented using Xilinx ISE Design tool. Synthesis results for the algorithm are also presented.

동영상 부호화에서 움직임 추정은 구현 복잡도에 가장 크게 영향을 미친다. 본 논문에서는 깊이 카메라와 색상 카메라를 동시에 이용하여 움직임 추정의 복잡도를 줄이는 방법을 제안한다. 깊이 카메라로부터 얻어진 거리 정보로 동영상내 객체 정보를 얻고, 비슷한 거리에 있는 화소들을 그룹화하여 동일한 객체로 라벨링을 수행한다. 라벨링된 현재 및 참조 화면내에서 움직임 추정 블록을 세부분(배경, 객체내부, 경계)의 탐색영역으로 구분하여 적응적으로 판단한다. 현재 블록이 객체내부영역이면 참조 화면에서 객체내부영역에만 움직임 추정을 탐색하고, 배경영역이면 블록은 참조 화면에서 배경영역에만 탐색한다. 모의실험을 바탕으로 전역탐색방법에 비하여 제안된 방법은 움직임 추정 차신호가 동일하게 유지되면서 탐색 복잡도가 크게 줄어듬을 확인할 수 있다. Motion estimation in video coding greatly affects implementation complexity. In this paper, a reducing method of the complexity in motion estimation is proposed by using both the depth and color cameras. We obtain object information with video sequence from distance information calculated by depth camera, then perform labeling for grouping pixels within similar distances as the same object. Three search regions (background, inside-object, boundary) are determined adaptively for each of motion estimation blocks within current and reference pictures. If a current block is the inside-object region, then motion is searched within the inside-object region of reference picture. Also if a current block is the background region, then motion is searched within the background region of reference picture. From simulation results, we can see that the proposed method compared to the full search method remains the almost same as the motion estimated difference signal and significantly reduces the searching complexity.

In video coding system, block matching algorithms are used to identify the matching blocks in digital video sequence for motion estimation. Motion estimation is used to discover temporal redundancy in the digital video sequence. This paper shows the analysis of popular existing algorithms for motion estimation. The analysis is based on the number of search points/macroblocks and speed-up ratio. The endeavor of this revision is to present the successive researchers with some productive information in design of block matching algorithms.

To exploit the temporal and spatial redundancy and to expedite the process of video encoding it is required to find the motion Vector (MV) by a process called Motion Estimation (ME). In this paper, optimal motion estimation algorithms based on correlation that exists between consecutive macro blocks in a frame of a video with stopping criteria are proposed. The different neighborhood macro blocks (MB) form the Locality of Reference (LOR) of the candidate macro blocks in the reference frame and these LOR defines the types of algorithms which are named as Correlation Based Rood Pattern Search (CBRPS) in four variations which effectively reduce the search time, computational complexity without compromising the PSNR values i.e. quality of the compensated image. The results obtained are compared with the results with relevant existing techniques to highlight the superior performance of the proposed algorithms. All the proposed algorithms outperform the existing techniques both in terms of computational complexity and PSNR values. Amongst the four proposed algorithms the fourth type of CBRPS i.e. CBRPS_T4 is found to be the best for linear and transverse motion of objects in the videos in terms of computational complexity.

The progress of VLSI technology towards deep sub-micron feature sizes, e.g., sub-100 nanometer technology, has created a growing impact of hardware defects and fabrication process variability, which lead to reductions in yield rate. To address these problems, a new approach, system-level error tolerance (ET), has been recently introduced. Considering that a significant percentage of the entire chip production is discarded due to minor imperfections, this approach is based on accepting imperfect chips that introduce imperceptible/acceptable system-level degradation; this leads to increases in overall effective yield. In this paper, we investigate the impact of hardware faults on the video compression performance, with a focus on the motion estimation (ME) process. More specifically, we provide an analytical formulation of the impact of single and multiple stuck-at-faults within ME computation. We further present a model for estimating the system-level performance degradation due to such faults, which can be used for the error tolerance based decision strategy of accepting a given faulty chip. We also show how different faults and ME search algorithms compare in terms of error tolerance and define the characteristics of search algorithm that lead to increased error tolerance. Finally, we show that different hardware architectures performing the same metric computation have different error tolerance characteristics and we present the optimal ME hardware architecture in terms of error tolerance. While we focus on ME hardware, our work could also applied to systems (e.g., classifiers, matching pursuits, vector quantization) where a selection is made among several alternatives (e.g., class label, basis function, quantization codeword) based on which choice minimizes an additive metric of interest.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

Based on the temporal and interview redundancy of stereoscopic video coding, a fast motion and disparity estimation algorithm is proposed in this paper. Firstly, a credible start point is designed taking use of the view-temporal correlation of neighboring motion and disparity vectors. Secondly, an improved searching strategy is proposed to reduce the computational complexity, in which only 2-pixel refinement is needed around the start point. The proposed algorithm can get both motion vector and disparity vector during one estimation process. The experimental results demonstrate that the proposed algorithm can reduce up to 96% coding time compared with inter-view full search with a negligible quality drop.

Color vector quantization (VQ) has been an efficient still image compression scheme as well as a popular bitmap graphics format for display devices with limited color capability. In this paper we are proposing a color vector quantization-based video coder, exploiting the temporal stationary nature of color distribution among a group of pictures (GOP) over a short period. Color VQ is applied first to reduce the RGB image sequence into a single channel color index image. Motion estimation and compression is then performed in the index space, instead of the separate YCbCr channels. Initial results demonstrated that the proposed coder can provide good compression rates. By eliminating the need for an inverse DCT and color conversion, typical requirements in a JPEG/MPEG type of coders, the decoding is computationally very simple. This makes it suitable for certain applications like media playback, and visual communications with low-end mobile devices.

The scalable extension of the HEVC Video Coding Standard (H.265) offers elaborate mechanisms for motion vector prediction and estimation. S-HEVC builds on the standard by extending predictor lists for Coding Unit blocks, utilizing base-layer information in the inference of enhancement-layer Coding Units. The complex, exhaustive search schemes in use can be aided by hierarchical optimizations in subpixel motion estimation, which we propose for slow-moving CUs per frame. In this paper we implement and test an adaptive optimization of motion estimation in the standard (SHM 6.1 software release), based on a statistical analysis of the behavior of subpixel motion vector differentials in each spatial mode per Coding Unit. We propose that the least granular mode (64×64 PEL macro-block in current release) contains sufficient information at subpixel levels to decide best-mode selection, i.e., whether a complete recursion through the inner partitions (higher granularity) is required in the estimation of a CU motion vector. We further propose that subpixel motion estimation overheads can be avoided below a set threshold, given conditions set in base and enhancement layer motion estimation for priorly computed modes in the same CU. Both optimization methods are tested across a diverse set of video sequences, producing negligible quality penalties at for a sizable reduction in encoding time.

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

Classical block-based motion-compensated video coders need to find and code a motion field with one motion vector per image block. All motion vectors are computed and encoded with the same fixed accuracy, typically 1 or one-half pixel accuracy. Higher motion vector accuracies have been shown to significantly reduce the total bit rate in some video sequences, but motion estimation at such subpixel accuracies is computationally expensive and is usually not performed in practice. In this paper we show that computing and encoding different motion vectors with different accuracies in the same frame can significantly reduce the total bit rate, and that the complexity of the block adaptive motion estimation procedure can be as low as that of the classical motion estimation at 1 pixel accuracy. Our new block adaptive motion estimator uses a simple technique to decide how accurately to compute the motion vector for each block. This technique results from an analysis on the effect of the motion vector accuracy on the total bit rate, and indicates that motion vectors of higher texture blocks must be computed more accurately and that at higher levels of compression lower motion vector accuracies are needed. We implement two video coders based on our technique, present results on real video frames, and describe the rate/complexity benefits of our procedure.

In this paper, we propose a temporal dynamic index reordering vector quantization for wavelet-based embedded coding. Da Silva et al. introduced a vector quantization concept that is similar to EZW called a successive approximation vector quantizer (SAVQ). The successive refinement process is defined as a temporal process in our proposed algorithm. The temporal updates are performed in every refinement pass, and the updates of codevectors reflect the updates of angles for vector approximation. Because the approximation trajectory of SAVQ is similar to that of the least-mean- square algorithm, temporal redundancy does not seem to be obvious. However, the redundancy becomes more clear in the improved SAVQ. By carefully analyzing the angle transitions, we are successfully able to apply dynamic index reordering vector quantization (DIRVQ) in the temporal domain and improve the coding performance. Efficient encoding and decoding algorithms for D4 lattice vector quantization are also proposed, and the algorithms can be applicable to DIRVQ to reduce the computational complexity of the reordering process. Experiments are performed with the 2x2 vector size. For almost all tested bpp's the PSNR performance of our proposed algorithm for a lena image outperforms that of SAVQ up to 0.18 dB. Improvement is measured for the highly detailed baboon image as well.