<title>Spatial correlation-based approach to motion vector estimation in video coding</title>

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

In this paper, we propose a new fast algorithm for motion vector (MV) estimation based on the correlation of MVs existing in spatially adjacent and hierarchically related blocks. We select a set of MV candidates from the MV information obtained at coarser resolution level and its spatially neighboring blocks in the same level, and then perform further local search to refine the MV result. Experiments are performed for several test video sequences to demonstrate its excellent performance. The proposed algorithm achieves a speed-up factor ranging from 140 to 200 with only 2-8% MSE increase in comparison with the full search block matching algorithm.© (1995) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

In this paper, we propose a new fast algorithm for motion vector (MV) estimation based on correlations of MVs existing in spatially and temporally adjacent as well as hierarchically related blocks. The main idea is to select a good set of MV candidates with information obtained at the coarser resolution level and spatio-temporal blocks at the same level, and then perform further local search to refine the MV result. The experimental results show that the proposed fast algorithm achieves a speed-up factor ranging from 150 to 310 with only 2 - 7% MSE increase in comparison with the full search block matching algorithm on typical test video sequences.

We present a novel multi-resolution block matching algorithm (BMA) for fast motion estimation. At the coarsest level, a full search BMA (FSBMA) is performed for searching complex or random motion. Concurrently, spatial correlation of motion vector (MV) field is used for searching continuous motion. Here we present an efficient method for searching full resolution MVs without MV decimation even at the coarsest level. After the coarsest level search, two or three initial MV candidates are chosen for the next level. At the further levels, the MV candidates are refined within much smaller search areas. Simulation results show that in comparison with FSBMA, the proposed BMA achieves a speed-up factor over 710 with minor PSNR degradation of 0.2 dB at most, under a normal MPEG-2 coding environment. Furthermore, our scheme is also suitable for hardware implementation due to regular data-flow.

For video compression, motion estimation is popularly employed to exploit temporal correlation existing in video sequences. If we employ the full search block matching algorithm for estimating motion vectors, it requires very heavy computational complexity. Although several fast block matching algorithms have been proposed to solve this problem, they sacrifice their reconstructed image quality. In this paper, we derive optimal search patterns for fast block matching motion estimation. By analyzing the block matching algorithm as a function of the block size and the shape, we find optimal search patterns for initial motion estimation. The proposed idea can provide an analytical ground for the current MPEG-2 proposals. In addition, we propose a new fast motion estimation algorithm using adaptive search patterns, considering matching criteria and statistical properties of object displacement. In order to select an appropriate search pattern, we exploit the relationship between the motion vector and the frame difference of each block. By changing the search pattern adaptively, we can improve the motion prediction accuracy, while reducing the required computational complexity compared to other fast block matching algorithms.

A modified block matching algorithm (BMA) with motion correlation constraint (MCCBMA) is proposed at first. Then a novel motion compensation algorithm (INTPMC) which computes motion vector for each pixel by interpolating motion vectors is presented. In order to increase interframe prediction performance and decrease the computational complexity, an optimizing algorithm for partial motion vectors is described at last. Experimental results show that the proposed algorithm can improve the prediction performance obviously with a moderately increased complexity compared with the conventional full search block matching algorithm (FSBMA).

We present a new method for converting monoscopic video to stereoscopic video. The key characteristic of our proposed method is that it can process non-horizontal camera/object motion existing in most of image scenes. It is well known that the non-horizontal motion causes the vertical parallax to human eyes and accordingly visual discomfort. The proposed methodology is composed of four major steps. First, given a current video frame, we estimate a motion vector for each block by a conventional block matching motion estimation algorithm. The motion vector is composed of horizontal and vertical disparities. Second, the norm of the motion vector is computed for each block. Here, the vertical disparity is eliminated due to the usage of the norm of the motion vector. Due to the unreliability of estimated motion vectors, a low-pass filter is performed on the norm of the motion vector in order to enhance the reliability. Third, each block is shifted to the horizontal direction by the norm of the motion vector, which is transformed to binocular parallax. The shift of blocks in the horizontal direction eliminates the effects of the vertical disparity. Finally, all the shifted blocks are synthesized and a synthesized image is then generated. A stereoscopic image pair being composed of the original image and its associated synthesized image is produced. With proper 3D viewing devices, users can feel 3D depth from seeing the stereoscopic image. Preliminary experiments have demonstrated that stable stereoscopic image pairs can be produced by applying our proposed method to a variety of monoscopic video with non-horizontal camera panning and/or object motion.

We will be able to use highly parallel processing environments. This paper proposes a method for estimating translations, rotations and scaling simultaneously from the estimated motion vectors, and represents its performance with motion estimation experiments. A sector region luminosity correlation is used for estimating motion vectors. The sector region luminosity correlation is robust about the rotation and withstands large motion environments. The proposed method makes the assumption about the scaling and estimates the motion vectors based on the assumptions. Then it randomly creates the pair of the estimated motion vectors. Next, it selects the proper pair using the scaling factor of the pair. The selected pairs are included in the set of reliable motion vector pairs. The reliable motion vector pairs decide the translation, rotation and scaling. With large scaling, it is difficult to estimate the motion using the sector region luminosity correlation. But with the assumptions about the scaling, they can work. Experiments show that the proposed method makes much better correlations between images than SIFT does.

In contrast to the conventional block matching algorithm (BMA) in video coding, a backward predictive BMA which does not need the transmission of the motion vector is investigated. The motion vector is estimated using the previously estimated motion vectors and reconstructed pixels adjacent to a block to be coded. For application to low bit rate video coding, backward/forward adaptive BMA is devised. At 59kbit/s the proposed scheme provides 0.2–0.5dB improvement in peak signal to noise ratio (PSNR) over the conventional BMA.

Motion-vector-based error concealment algorithms recover lost blocks by motion compensation with the estimated motion vectors. Traditionally, the estimated motion vectors for the lost block are derived by the direct usage of the neighboring available motion vectors. In this work, the residual signal in a block encoded in the video bitstream is used to measure the reliability of the corresponding motion vector. The residual information is applied to weight the neighboring available motion vectors differently, to further improve a newly proposed mathematical formulation between encoded motion vectors and extrapolated motion vectors. The proposed error concealment method using the residual information outperforms a state-of-the-art error concealment algorithm by up to 2.12 dB in sequence PSNR, and outperforms a classic DMVE method by up to 8.27 dB.

We will be able to use highly parallel processing environments. This paper proposes a method for estimating translations, rotations and scaling reaching 10 times simultaneously based on the multiple scaling assumptions, and represents its performance with motion estimation experiments. A sector region luminosity correlation is used for estimating motion vectors. The sector region luminosity correlation is robust about the rotation and withstands large motion environments. The proposed method makes the assumptions about the scaling and estimates the motion vectors based on the assumptions. Then it randomly creates the pair of the estimated motion vectors. Next, it selects the proper pair using the pre-assumed scaling factor. The selected pairs are included in the set of reliable motion vector pairs. The reliable motion vector pairs decide the translation, rotation and scaling. With large scaling, it is difficult to estimate the motion using the sector region luminosity correlation. But with the assumptions about the scaling, they can work. Experiments show that the proposed method makes much better correlations between images than SIFT does in 10 times scaling changes.

A low-complexity iterative algorithm is introduced for joint estimation of forward and backward motion vectors in interpolative prediction of video. Starting from the initial values obtained by a commonly-used block matching independent search method, the motion vectors are iteratively refined until a locally optimal solution to the motion estimation problem for interpolative prediction is achieved. Each iteration consists of a series of two similar procedures. First, the backward motion vector is fixed and a new forward motion vector is searched to minimize the interpolation error. Then the forward motion vector is fixed and the backward motion vector is similarly refined by minimizing the interpolation error. This process is repeated until the interpolation error stops decreasing. Computer simulation results demonstrate that with this technique the prediction error in some scenes is significantly reduced.

Block matching motion estimation is a significant issue in image and video super resolution reconstruction. In order to decrease the number of search locations, the amount of computations, and enhance the prediction accuracy, an improved block matching motion estimation algorithm based on adaptive rood pattern search (ARPS) is proposed. Firstly, we use spatial correlation characteristic to predict motion vector (MV) of the current macro-block in the algorithm. Secondly, we consider to eliminate the influence of neighborhood macro-blocks having low relevance to the current macro-block in the prediction process. Finally, we utilize the weighted average MV of two macro-blocks with the smaller matching error in the neighborhood macro-blocks to calculate the predictive MV of current macro-block, instead of using the fixed single macro-block forecast pattern in ARPS. Experimental results show that our algorithm provides the better performance in terms of the computational complexity and the prediction precision in comparison with APRS and other block matching algorithms.

The scalable extension of the HEVC Video Coding Standard (H.265) offers elaborate mechanisms for motion vector prediction and estimation. S-HEVC builds on the standard by extending predictor lists for Coding Unit blocks, utilizing base-layer information in the inference of enhancement-layer Coding Units. The complex, exhaustive search schemes in use can be aided by hierarchical optimizations in subpixel motion estimation, which we propose for slow-moving CUs per frame. In this paper we implement and test an adaptive optimization of motion estimation in the standard (SHM 6.1 software release), based on a statistical analysis of the behavior of subpixel motion vector differentials in each spatial mode per Coding Unit. We propose that the least granular mode (64×64 PEL macro-block in current release) contains sufficient information at subpixel levels to decide best-mode selection, i.e., whether a complete recursion through the inner partitions (higher granularity) is required in the estimation of a CU motion vector. We further propose that subpixel motion estimation overheads can be avoided below a set threshold, given conditions set in base and enhancement layer motion estimation for priorly computed modes in the same CU. Both optimization methods are tested across a diverse set of video sequences, producing negligible quality penalties at for a sizable reduction in encoding time.

In this correspondence, a fast approach to motion estimation is presented. The algorithm uses the block sum pyramid to eliminate unnecessary search positions. It first constructs the sum pyramid structure of a block. Successive elimination is then performed hierarchically from the top level to the bottom level of the pyramid. Many search positions can be skipped from being considered as the best motion vector and, thus, the search complexity can be reduced. The algorithm can achieve the same estimation accuracy as the full search block matching algorithm with much less computation time.