Chapter 4 - Basic Motion Estimation Techniques

Block matching motion estimation (BMME) plays a very important role in video coding. The performance of BMME greatly affects the quality of the encoded bit stream. In order to reduce the high computational demands of H.264 video encoder, a fast block matching motion estimation algorithm based on temporal and spatial motion correlation is proposed by the deep research of the characteristics of motion estimation. This algorithm well exploits the correlation between video sequence motion activity and macro-block coding pattern and motion vector statistic characteristics to employ different patterns. Experimental results show that the algorithm, without function losing of PNSR, can not only reduce the search complexity of motion estimation and the high computational demands but also raise the real-time performance of encoder.

Motion estimation (ME) is one of the most time-consuming parts in video coding. The use of multiple partition sizes in H.264/AVC makes it even more complicated when compared to ME in conventional video coding standards. It is important to develop fast and effective sub-pixel ME algorithms since: 1) the computation overhead by sub-pixel ME has become relatively significant while the complexity of integer-pixel search has been greatly reduced by fast algorithms, and 2) reducing sub-pixel search points can greatly save the computation for sub-pixel interpolation. In this letter, a novel fast sub-pixel ME algorithm is proposed which performs a “rough” sub-pixel search before the partition selection, and performs a “precise” sub-pixel search for the best partition. By reducing the searching load for the large number of non-best partitions, the computation complexity for sub-pixel search can be greatly decreased. Experimental results show that our method can reduce the sub-pixel search points by more than 50% compared to existing fast sub-pixel ME methods with negligible quality degradation.

The Block-Matching (BM) method for motion estimation in most video coding is largely discussed in the case of perspective images.The omnidirectional cameras provide images with large field of view.These images contain global information about motion and permit to remove the ambiguity present with little camera motion in perspective case.Nevertheless, these images contain significant radial distortions.The Block-Matching in these catadioptric images is not a resolved problem, and still a challenging research field.A rectangular block representing the neighborhood in BM of a point and used in the perspective images is not appropriate for catadioptric cameras.The work presented in this article concerns the local motion estimation in catadioptric videos with the Adapted Block-Matching (ABM).The ABM based on an adapted neighborhood, the local motion estimation allows successful compensation prediction in catadioptric images.The Adapted Block-Matching is obtained from the equivalence between the omnidirectional image and the projection of scene points on a unit sphere.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

As an essential component in applications such as video coding, autonomous navigation, and surveillance cameras, efficient and robust motion estimation is always required. This paper proposes a robust block-matching algorithm consisting of a rough matching step and a fine matching step for motion estimation. In the coarse matching step, an improved adaptive rood pattern search strategy combined with an anti-interference similarity criterion is developed to improve the computational efficiency and robustness. In the fine matching step, after performing a subpixel estimation procedure, a bilateral verification scheme is demonstrated to decrease the motion estimation errors caused by environmental disturbances. Experiments are carried out over popular video and image sequences, and several measurement indexes are used to quantify the performance of the proposed method and other motion estimation methods. Comparative analysis and quantitative evaluation demonstrate that the proposed method exhibits strong robustness and can achieve a good balance between computational efficiency and complexity.

The increasing processing speed of computers has made software-based video coding possible. Several reports have been published on increasing the processing speed of such video coding through improved software development techniques, and on increasing video coding speed by efficiently omitting the video coding process itself. However, the actual speed of software-based video coding depends on the implementation environments. Accordingly, under some environments of lower processing speed, the processor can not encode video data in real time, even if real-time coding is necessary. This has created a need to control a video coding process with depending on the processing speed. To address this need, we propose a new scheme for controlling motion estimation. This scheme, which we call “time adaptive motion estimation” (TAME), is based on the Successive Elimination Algorithm (SEA) method. It enables motion estimation to be completed within a pre-determined target time. Computer simulation results demonstrate that the proposed method achieves nearly the same quality as in the case where coding time is unlimited.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

Stereo image sequence coding based on three-dimensional motion estimation and compensation

We present a method for estimating large camera motions, including combinations of large rotations and displacements. We know that translation estimates are improved as the displacements between two camera positions increase. However, there is little work on motion estimation methods targeted at large motions since it is difficult to find correspondences between two planar perspective images under large motions. We can overcome the correspondence problem with omnidirectional cameras, which obtain a hemisphere or cylinder projection of the environment. We first develop a new motion estimation method for smooth omnidirectional camera motions. Our method incrementally improves estimates from incomplete information provided by a single feature correspondence, using an Implicit Extended Kalman Filter (IEKF). This general motion estimation method produces stable estimates from smooth camera motions, but it is not directly applicable to estimating large motions. We adapt the motion estimation method to large motions by using a novel Recursive Rotation Factorization (RRF) that removes the image motions due to rotation. Simulation results show that the RRF large motion estimates are accurate and robust.

Motion estimation (ME) is one of the most critical tools in video coding and consumes the majority of the encoding complexity. Three types of ME are utilized in the latest video coding standards, namely integer, fractional, and affine MEs. They are implemented as three searches for the integer motion vector (IMV), fractional motion vector (FMV), and control point motion vectors (CPMVs). Many algorithms were proposed to reduce the complexity for them individually, but the overall overhead of three searches is still challenging for hardware implementations. Therefore, we propose a hardware-efficient Unified Motion Estimation (UME) to derive three types of MVs with only one search. An IME with sub-block refinement is performed to collect extra motion information while searching for the IMV. The FMV and CPMVs are then derived from the collected information using a mixed error surface and an overdetermined system. Compared to the default ME algorithms in VVC, the time cost for ME is reduced by 41.63% with a coding loss of only 0.87% under LDB configuration. For hardware implementations, the minimum required resources and corresponding latency are significantly reduced by 75.35% and 69.17%, respectively.

This paper presents a new method for frame rate up‐conversion using sub‐sampled bilateral motion estimation, motion vector smoothing, simplified bidirectional motion compensated interpolation, and complex motion detection. The proposed method produces improved image quality and requires less computational complexity compared to conventional motion estimation. The average PSNRs of the proposed method for frame rate up‐conversion outperform those of the benchmark frame rate up‐conversion technique by up to 4.39 dB for the test sequences. Furthermore, the motion estimation of the proposed FRUC method reduces the required computational complexity to 12.5% of that required for full‐search block matching motion estimation.

The H.264 provides various and useful features for improved coding efficiency and error robustness, but its encoder complexity is extremely high due to the computational time of ME (motion estimation) and mode decision of inter coding. In this paper, we propose a fast and simple class-based scheme to reduce the computational complexity of ME. The image in block base is classified to k classifications with SQCM (spatial-domain quickly classified method). Referring to the classified indices, we discriminate two methods of ME: one is pattern search for stationary macroblock, synchronous motion and unchanged object; other is fast inter prediction for instable or variant content of object. We can save the dispensable motion detection for different contents of video, named CQME (class-based quickly ME). Experimental results shows that the proposed method can achieve a reduction of 67% computational time on average, with an average PSNR loss of only 0.30 dB compared with the original H.264 reference software jm12.1.

Motion estimation is the most computationally intensive component of a video coding algorithm. It could consume as much as 75% of the total processing power of a video codec. Although various international video coding standards have adopted the block-matching approach for motion estimation, a designer can still have the freedom to choose a specific technique to find the motion vectors. Consequently, numerous techniques ranging from full search to simple fast search algorithms have been proposed and some of them have been implemented in VLSI chips. It should be emphasized that one of the most important requirements for an effective motion estimation algorithm is its ability to perform in real-time. The choice of a specific algorithm for implementation, however, depends very much on the intended application as well as the trade-offs between the desired performance and affordable complexity. In this paper, we first discuss what are the essential ingredients for an effective block-matching motion estimation, then briefly, describe what is the status of current technology, and finally, present some new activities in this vital area of research.

An enhanced constrained one-bit transform (C1BT)-based fast motion estimation (ME) method is proposed. Binary transform-based ME identifies the proper motion vector by transforming the 8-bit pixels in the original image into a low bit depth of the bit plane, substantially reducing the computational complexity and facilitating hardware implementation by using the number of non-matching points (NNMPs) instead of the sum of absolute difference (SAD). However, a motion block size is N × N , therefore the dynamic range of NNMP (0 ≤ NNMP ≤ N × N ) is decreased to 1/256 compared to that of the SAD with 8-bit images (0 ≤ SAD ≤ 256 × N × N ). The higher the NNMP between the current bit plane and the previous bit plane, the more probable that it will be the block most similar to each other. Therefore, the matching error criterion of NNMP is extended to improve ME performance. Experimental results show that the proposed algorithm improves the performance of ME accuracy by 0.27, 0.38 and 0.67 dB compared to the C1BT-based ME, two-bit transform (2BT)-based ME and 1BT-based ME, respectively.

Fractal compression is the lossy compression technique in the field of gray/color image and video compression. It gives high compression ratio, better image quality with fast decoding time but improvement in encoding time is a challenge. This review paper/article presents the analysis of most significant existing approaches in the field of fractal based gray/color images and video compression, different block matching motion estimation approaches for finding out the motion vectors in a frame based on inter-frame coding and intra-frame coding i.e. individual frame coding and automata theory based coding approaches to represent an image/sequence of images. Though different review papers exist related to fractal coding, this paper is different in many sense. One can develop the new shape pattern for motion estimation and modify the existing block matching motion estimation with automata coding to explore the fractal compression technique with specific focus on reducing the encoding time and achieving better image/video reconstruction quality. This paper is useful for the beginners in the domain of video compression.