Bilateral frame rate up‐conversion algorithm based on the comparison of texture complexity

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.

The most recent video coding standards are usually based on a rate-distortion optimization (RDO) process that has been formulated in terms of an unconstrained Lagrangian optimization. The RDO provides outstanding results in exchange for a high computational cost, especially for the Inter frames, which require a computationally heavy motion estimation (ME) process. In particular, for H.264/AVC, this RDO process allows selecting both the MB partition size and the motion vector. However, as the optimum procedure is not feasible for computational reasons, the ME process uses a simplified rate-distortion (RD) cost function. Therefore, two RDO processes are involved, one for selecting the MB partition size and one for ME. Both RDO processes rely on an Lagrangian formulation and, for practical purposes, the corresponding Lagrangian parameters are related by a simple, experimentally obtained relationship. In this paper, some evidences of the weaknesses of such a relationship between the two Lagrangian parameters are given and a simple effective procedure to improve the R-D encoding performance is proposed according to such weaknesses. The proposed method has been comparatively evaluated with respect to one recently published method, showing significant average performance improvements, above 0.4 dB in terms of PSNR.

A 90-nm CMOS motion estimation (ME) processor was developed by employing dynamic voltage and frequency scaling (DVFS) to greatly reduce the dynamic power. To make full use of the advantages of DVTS, a fast ME algorithm and a small on-chip DC/DC converter were also developed. The fast ME algorithm can adaptively predict the optimum supply voltage (V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sub> ) and the optimum clock frequency (f <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</sub> .) before each block matching process stalls. Power dissipation of the MK processor, which contained an absolute difference accumulator as well as the on-chip DC/DC converter and DVFS controller, was reduced to 29.1 muW, which was only 3% that of a conventional ME processor.

Fast motion estimation for H.264

Efficiency of motion estimation is affected by serious cache miss and cache split load. Firstly, the data to be used in the motion estimation process have poor spatial locality due to the layout format of video data. Secondly, the motion estimation process accesses memory irregularly and sometimes it accesses the data that spread across two cache lines. Irregular memory access pattern brings difficulties for a processor to pre-fetch data into the cache. The cache split load, on the other hand, forces the processor to load more data into the crowded cache. In this paper, a new layout format of video data, named overlapped tiled format, is proposed. The new format addressed the aforementioned two issues. By organizing the video data in the proposed way, the issue of cache split load is completely removed from the motion estimation process and the data cache miss is also greatly reduced. The full search motion estimation is speed up by 28% when organizing the video data in an overlapped tiled way.

A novel Cooperative Motion Estimation Algorithm based on Particle Swarm Optimization and its multicore implementation

This paper proposes a motion estimation scheme to reduce the computational complexity of multilayer motion estimation for scalable video coding. Based on the result of the motion estimation of the lower resolution layer referred to as base layer, we developed a new approach for exploring the search range of the enhancement layer with high coding efficiency. This approach is based on the activity defined as the absolute difference between the motion vector predictor and the final motion vector. Based on the correlation of the activities between neighboring layers, an inter-layer activity model was developed using a curve-fitted linear equation to exploit the activity in the base layer for deciding the search center and the search range of the enhancement layer. Each activity pair in the neighboring layers is used to associate the relevant macroblock to one of two groups; boundary region and interior region. The base-layer motion vector predictor is basically selected over all the activity regions; for each activity region, the proposed motion estimation algorithm decides whether to include the median motion vector predictor or not. Minimal sufficient search range is also decided from the inter-layer activity prediction factor that is adjusted to the given sequence. The proposed scheme reduced the execution time of motion estimation by 99.26% at the cost of 1.56% bit-rate increase and 0.048 dB peak signal-to-noise ratio (PSNR) decrease on average compared with the conventional full-search algorithm. The fast full-search block matching algorithm can also be incorporated to obtain the extra CPU time reduction in the motion estimation process. By adopting the fast full-search block matching algorithm (FFSBMA) in JSVM reference software, the CPU time was reduced by up to 91.84% and the memory bandwidth was reduced by 90% at the sacrifice of 1.27% bit-rate increase and 0.041 dB PSNR decrease on average compared with the FFSBMA only.

Adaptive fractional motion and disparity estimation skipping in MV-HEVC

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

In an H.264 video encoder, motion estimation (ME) is the most time-consuming component. The ME process consists of two stages, integer pixel search and fractional pixel search. Since the complexity of integer pixel search has been greatly reduced by numerous fast ME algorithms, the computation overhead required by fractional pixel ME has become relatively significant. To reduce the complexity of fractional pixel ME, we propose a prediction-based directional fractional pixel ME algorithm. We utilize more accurate motion vector predictions and directional search to achieve better computation reduction. We further propose an early termination method to decrease the amount of search. Experimental results show that, compared to the full search sub-pel ME and the fast sub-pel ME proposed in H.264, the proposed method can reduce up to 84% and 74% of fractional pixel search points respectively, with a negligible degradation in quality.

Motion estimation is a vital function of video coders that use motion compensated prediction to exploit the temporal redundancy of video signals. The reference images used in the motion estimation process can be either original images (open‐loop architecture) or reconstructed images (closed‐loop architecture). While the closed‐loop architecture is intuitively superior to the open‐loop architecture, closed‐loop motion estimation is more difficult to implement in a real‐time video encoder. A compromise is to perform open‐loop integer‐pel accurate motion estimation followed by closed‐loop half‐pel accurate estimation. In this article we present a comparative study of the performance of half‐pel closed‐loop versus open‐loop motion estimation for coding of high definition television with the MPEG algorithm. Simulation results indicate that on average, closed‐loop half‐pel accurate motion estimation provides slightly better performance than open‐loop half‐pel accurate estimation. This improvement depends on the bit rate of the coded video biststream. For relatively high bit rates, the closed‐loop implementation often provides no improvement, and sometimes an actual degradation of performance. An analysis of why a closed‐loop implementation does not guarantee a better performance is presented.©1994 John Wiley &amp; Sons Inc

Because of its high computational complexity, the process of motion estimation has become a bottleneck problem in many video applications, such as mobile video terminals and software-based video codecs. This has motivated the development of a number of fast motion estimation techniques. This paper briefly reviews reduced complexity motion estimation techniques and classifies them into five categories. It then presents the results of a comparative study across all categories. The aim of this study is to provide the reader with a feel of the relative performance of the categories, with particular attention to the important trade-off between computational complexity and prediction quality.

In this paper we propose a new method of using motion estimation for the purpose of offline recognition of machine-print Hindi digits. The recognition process can be summarized as follows: an image of each of the ten numerals (numbers 0 to 9) is stored, these are called reference images. The differences between the image of the numeral to be recognized and the reference image are considered motions. The motions are estimated and then compensated. The resultant image after motion compensation is compared to the reference image and the difference between the two (treated as error) is calculated. The process of motion estimation and compensation is done for each of the ten numerals. The numeral with the minimum error is chosen as the recognized digit. While this proposed recognition system is applied to Hindi digits in this paper, it can be generalized to the recognition of any numerals in any language and can be extended to text and voice recognition. The proposed method is simple, fast, accurate, and reliable.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.