An efficient low-bit rate adaptive mesh-based motion compensation technique

Motion estimation and compensation is widely used for exploiting temporal correlation within an image sequence. To find motion vectors that lead to high compression, most motion estimation approaches use a source distortion measure, such as mean-square error (MSE) or mean-absolute error (MAE), as a search criterion. When incorporated into a closed-loop motion compensated (MC) transform video coder, these schemes produce noisy motion fields which significantly increase the bit-rates required to represent motion vectors. In view of this problem, this paper presents a rate-distortion optimal motion estimation algorithm. The proposed scheme improves rate performance of the estimated motion field while maintaining the peak signal-to-noise ratio (PSNR) prediction quality of the distortion-based methods, thereby enabling an efficient bit allocation between motion information and transform-coded prediction residuals. For coders in which motion vectors are differentially encoded, the rate-distortion optimization process is formulated as a shortest-path-finding problem. Adopting this framework, we show that the optimal solution for the conventional block-based motion estimation, followed by one-dimensional (1-D) differential coding and Huffman coding, can be obtained by using dynamic programming or the Viterbi algorithm. We propose an effective fast algorithm that closely approximates the optimal performance while requiring considerably less complexity. Our experimental results demonstrate overall gains in the range of 0.3-1.5 dB.

The quad-tree structured motion-compensation technique effectively utilizes the motion content of a frame as opposed to the fixed size block motion compensation technique. The authors propose a novel quad-tree structured region-wise motion compensation technique that divides a frame into equilateral triangle blocks using the quad-tree structure. Arbitrary partition shapes are achieved by allowing 4-to-1, 3-to-1 and 2-1 merge/combine of sibling blocks having the same motion vector. We propose an optimal code scheme and a temporal predictive coding for the quad-tree. Simulation results show that our techniques reduce the bit rate by 40% as compared to other methods.

This paper introduces a novel scheme of motion estimation (ME) and motion compensation (MC) in wavelet transform (WT) video coding. The scheme is unique in that half-pixel resolution ME and MC are applied on filtered coefficients ahead of decimation to preserve motion consistency and yet achieve minimal prediction residue. An efficient motion-compensated WT video coding system is built based on the proposed ME and MC. In the coding scheme, according to the property of embedded zerotree wavelet (EZW) coding, the mean square error (MSE) criterion is used in ME instead of normally adopted sum of absolute difference, and an overlapped MC algorithm is developed to combat the block effect introduced by block-based ME and improve peak signal-to-noise ratio (PSNR) performance. The superior performance of the proposed scheme is demonstrated with experimental results.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

This paper proposes a motion estimation scheme to reduce the computational complexity of multilayer motion estimation for scalable video coding. Based on the result of the motion estimation of the lower resolution layer referred to as base layer, we developed a new approach for exploring the search range of the enhancement layer with high coding efficiency. This approach is based on the activity defined as the absolute difference between the motion vector predictor and the final motion vector. Based on the correlation of the activities between neighboring layers, an inter-layer activity model was developed using a curve-fitted linear equation to exploit the activity in the base layer for deciding the search center and the search range of the enhancement layer. Each activity pair in the neighboring layers is used to associate the relevant macroblock to one of two groups; boundary region and interior region. The base-layer motion vector predictor is basically selected over all the activity regions; for each activity region, the proposed motion estimation algorithm decides whether to include the median motion vector predictor or not. Minimal sufficient search range is also decided from the inter-layer activity prediction factor that is adjusted to the given sequence. The proposed scheme reduced the execution time of motion estimation by 99.26% at the cost of 1.56% bit-rate increase and 0.048 dB peak signal-to-noise ratio (PSNR) decrease on average compared with the conventional full-search algorithm. The fast full-search block matching algorithm can also be incorporated to obtain the extra CPU time reduction in the motion estimation process. By adopting the fast full-search block matching algorithm (FFSBMA) in JSVM reference software, the CPU time was reduced by up to 91.84% and the memory bandwidth was reduced by 90% at the sacrifice of 1.27% bit-rate increase and 0.041 dB PSNR decrease on average compared with the FFSBMA only.

In this work, a new block-based motion estimation method using the adaptive Kalman filtering (KF) based on one- dimensional (1-D) and two-dimensional (2-D) autoregressive (AR) models is proposed to improve the motion estimates. Conventional block-matching algorithms (BMAs) are utilized to obtain the measured motion vectors (MVs). Autoregressive models are employed to characterize the motion correlation between neighbouring blocks and predict motion information. The measured and predicted motions are utilized to generate an optimal estimate of motion vector. To improve the performance of the adaptive Kalman filtering, in our method, a new error function is defined and the statistics of MV measurements are utilized to dynamically adjust the state parameters during each iteration. The experimental results indicate that the proposed approach performs better than the existing methods in comparison, in terms of the peak-signal- to-noise-ratio (PNSR) of the motion compensated images. The developed adaptive KF method is independent of motion estimation, thus it can be applied to any block-matching motion estimation scheme for performance improvement. Additionally, after the Kalman filtering, the fractional-pixel accuracy of motion vector is achieved with no additional bits for MV.

1비트 변환 (one-bit transform) 및 2비트 변환 (two-bit transform)을 이용하는 이진 블록 매칭 움직임 예측 (motion estimation) 방법은 전역 탐색 (full search) 움직임 예측 방법에 비해 블록 매칭 연산의 복잡도를 감소시키지만 PSNR (Peak Signal-to-Noise Ratio)성능 저하를 야기한다. 이러한 이진 블록 매칭 움직임 예측 방법의 정확도를 개선하기 위해 조건부 국부 탐색 (conditional localsearch)이 더해져 보완된 1비트 변환 (modified one-bit transform) 및 보완된 2비트 변환 (modified two-bit transform) 방법이 제안되었다. 그러나 이와 같이 추가적인 국부 탐색은 움직임이 빠른 영상에 대한 <TEX>$16{\times}16$</TEX> 블록 크기의 움직임 예측에 있어서 많은 수의 추가적인 탐색을 필요로 한다. 본 논문은 기존의 조건부 국부 탐색 방법 대신 탐색 영역내의 각 후보 블록들의 (candidate blocks) NNMP(Number of Non-Matching Points)를 기반으로 한 효율적인 탐색 방법을 제안한다. NNMP 기반 탐색 방법을 통하여 작은 NNMP 값을 가지는 후보 블록들을 쉽게 탐색하여 최종 움직임 벡터(motion vector)를 효율적으로 찾을 수 있다. 실험을 통하여 제안하는 알고리듬이 기존의 방법들보다 복잡도 및 정확도 측면에서 좋은 성능을 보여주는 것을 확인하였다. Motion estimation using one-bit transform and two-bit transform reduces the complexity for computation of matching error; however, the peak signal-to-noise ratio (PSNR) is degraded. Modified 1BT (M1BT) and modified 2BT (M2BT) have been proposed to compensate degraded PSNR by adding conditional local search. However, these algorithms require many additional search points in fast moving sequences with a block size of <TEX>$16{\times}16$</TEX>. This paper provides more efficient search method by preparing candidate blocks using the number of non-matching points (NNMP) than the conditional local search. With this NNMP-based search, we can easily obtain candidate blocks with small NNMP and efficiently search final motion vector. Experimental results show that the proposed algorithm not only reduces computational complexity, but also improves PSNR on average compared with conventional search algorithm used in M1BT, M2BT and AM2BT.

The present paper introduces an original way to perform motion estimation and compensation. The scheme is asymmetric in the sense that the compensation stage is not merely the reverse operation of the estimation process. Motion estimation is computed using a classical block matching algorithm and the set of vectors that compose the motion information are classically transmitted to the decoder. The motion compensation uses them to deform a wireframe structure whose vertices are displaced according to the motion vectors. Conditions must be set up on the number of wireframe vertices and on their location. In comparison with a block-based compensation, the main advantage is the possibility to take into account the object information, which results in perceptually more pleasant predicted pictures.

Delogne et al. (see IEEE Transactions on Image Processing, vol.3, no.5, p.482-91, 1994) have shown how the motion estimation and compensation have to be carried out when interlaced images are processed under the assumption of a translational motion. Due to the finite accuracy of the motion vectors, the prediction error is nonzero after motion compensation. We therefore investigate the influence of quantizing the motion vectors on the statistical properties of the prediction error signal. The power spectral density and the variance of the prediction error are also computed.

H.264/AVC, the latest video coding standard, adopts rate-distortion optimization (RDO) technique to obtain the best intra- and inter-prediction, while maximizing visual quality and minimizing the required bitrate. However, the full RD cost calculation for all intra-prediction modes, the exhaustive searches for finding optimal motion vectors for all block sizes, and the multiple references frame procedure considerably increase its computational complexity with the allowed number of prediction modes. The authors have previously proposed an approach for both inter- and intra-mode decisions that takes into account the several effective parameters, image content type, quantization parameter, correlation of motion vectors and motion details of video objects. In order to reduce the complexity, here we propose a new two-step adaptive search range decision by developing a combination of previous algorithm. The first step of our adaptive search range decision determines the search range for current MB according to the motion vectors of the previous MBs. In the second step, the motion vector obtained in the current frame is used to adaptively searching the window size of the previous reference frame for multi-frame motion estimation. As such, only a subset of inter- and intra-modes is chosen for RDO calculation. Experimental results show that the proposed algorithm reduces the total encoding cost with negligible loss in PSNR and a slightly increase in the required bitrate, when compared to our previous algorithm and other algorithms reported in the literature.

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.

This dissertation reports on rate-distortion optimized and computation-constrained video coding algorithms which yield effective and efficient block-based video encoders. We first propose a new cost function for motion estimation which can achieve good tradeoffs among motion vector bit rate, compensation distortion, and number of computations. Based on the new cost function, we propose a predictive motion estimation algorithm which can effectively determine the size of the search region according to the statistics of the motion field. The proposed motion estimation algorithm allows control of the motion vector bit rate and the computational cost, simultaneously. We also develop two efficient rate-distortion optimized coding mode selection algorithms, based on thresholding and finite state machine (FSM) methods, respectively. The thresholding method can eliminate consideration of the computationally expensive modes. The FSM method determines the test orders of the modes and associated parameters by exploiting the local statistics of the input video sequence. The proposed two methods can reduce substantially the computation requirements as compared to the full search mode selection method. Based on the proposed motion estimation and mode selection algorithms, we implement an H.263-based video coder and an MPEG-2, compliant video coder for very low bit rate video applications, and high bit rate interlaced video applications, respectively. The resulting video coders achieve excellent tradeoffs among bit rate, quality, and computational complexity. In fact, experimental results show that our video coders outperform the best known video coders in terms of compression performance and encoding speed.

Convolutional neural networks (CNNs)-based video quality enhancement generally employs optical flow for pixelwise motion estimation and compensation, followed by utilizing motion-compensated frames and jointly exploring the spatiotemporal correlation across frames to facilitate the enhancement. This method, called the optical-flow-based method (OPT), usually achieves high accuracy at the expense of high computational complexity. In this article, we develop a new framework, referred to as biprediction-based multiframe video enhancement (PMVE), to achieve a one-pass enhancement procedure. PMVE designs two networks, that is, the prediction network (Pred-net) and the frame-fusion network (FF-net), to implement the two steps of synthesization and fusion, respectively. Specifically, the Pred-net leverages frame pairs to synthesize the so-called virtual frames (VFs) for those low-quality frames (LFs) through biprediction. Afterward, the slowly fused FF-net takes the VFs as the input to extract the correlation across the VFs and the related LFs, to obtain an enhanced version of those LFs. Such a framework allows PMVE to leverage the cross-correlation between successive frames for enhancement, hence capable of achieving high accuracy performance. Meanwhile, PMVE effectively avoids the explicit operations of motion estimation and compensation, hence greatly reducing the complexity compared to OPT. The experimental results demonstrate that the peak signal-to-noise ratio (PSNR) performance of PMVE is fully on par with that of OPT while its computational complexity is only 1% of OPT. Compared with other state-of-the-art methods in the literature, PMVE is also confirmed to achieve superior performance in both objective quality and visual quality at a reasonable complexity level. For instance, PMVE can surpass its best counterpart method by up to 0.42 dB in PSNR.

In this paper, a novel motion estimation and compensation algorithm for frame rate up conversion is presented. Compared to conventional motion estimation methods, motion vectors are processed by different approaches for static region or internal object region and moving boundary of video stream in this proposed algorithm. In the static or internal object region, motion vectors are decided according to the interaction principle between adjacent blocks and easily obtained by the adjacent motion vectors; while for the moving boundary, motion vectors are firstly calculated according to the adjacent blocks, and then the algorithm detects the movement of block in the region by SAD values, finally, the motion vectors of moving blocks are refined and corrected.