Frame Rate Up-Conversion Based on Improved Dual Motion Estimation

A new bilateral frame rate up-conversion (FRUC) algorithm based on the bi-directional comparison of the texture complexity in the reference frame is proposed. In the proposed algorithm, the accuracy of the motion estimation (ME) process is determined by the comparison of the prediction size in the reference region of the bi-directional ME process. Base on this comparison, the adaptive refinement process is adjusted. In the experimental results, the proposed algorithm outperforms the conventional bilateral FRUC algorithm by 0.27 dB.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

This paper presents a novel motion-compensated frame rate up-conversion algorithm that uses dual criterion motion estimation to obtain accurate motion vectors in the circumstance of scene luminance changing. In the proposed algorithm, we also present a selective vector median filtering method and a motion vector refinement method to address block mismatching problem. Finally, a motion-compensated interpolation algorithm based on adaptive overlapped block motion compensation is proposed to reduce block artifact. Experimental results demonstrate that the proposed algorithm improves visual qualities compared with the existing algorithms.

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and preserve the spatial correlation of MVs. Further, the proposed method can detect occlusion regions, where a continuously tracked feature does not have a corresponding point in the current frame. In the detected occlusion region, the intermediate frame is generated from either the previous frame or the current frame. Outside the occlusion region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results show that the proposed temporal frame interpolation (TFI) method can improve visual quality over conventional TFI methods, both objectively and subjectively.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

Traditionally, the re-use of motion vectors extracted from incoming video bit-stream during transcoding has been widely accepted. However, this simple re-use scheme introduces significant quality degradation in many applications including the situation when the frame-rate conversion is needed. In this paper, we analyzed the quantization errors that cause the extracted motion vectors to be non-optimal and we performed simulations to show the quality degradation due to the inaccurate motion vectors during transcoding. To improve the video quality, we proposed an adaptive motion vector refinement. With a highly reduced computational complexity, the proposed adaptive motion vector refinement achieves significant quality improvement in comparison to the conventional motion vector re-use scheme. In addition, the adaptive motion vector refinement is almost as good as performing a new full-scale motion estimation.

Frame rate up-conversion is a useful post processing technique to enhance the visual quality of low frame rate video materials. In this paper, we propose a novel frame rate up-conversion algorithm utilizing true motion estimation and adaptive motion vector refinement. First, block-matching algorithm (BMA) based on an efficient motion similarity constrained criterion is employed to track the true motion between adjacent frames and produce a preliminary intermediate motion vector field (MVF).We then classify the intermediate motion vectors into two categories, isolated group and connected group adaptively by motion clustering. A smooth resampling method is later applied to connected group to obtain consistent motion, while isolated motion vectors are re-estimated with bilateral edge-weighted block decomposition (BEWBD). Finally, overlapped block motion compensation (OBMC) is adopted to interpolate the intermediate frame. Extensive experimental results show that the proposed algorithm outperforms conventional methods both subjectively and objectively.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

A novel motion estimation method is presented which significantly reduces the computational complexity of overlapped block motion estimation (OBME). To eliminate the computationally intensive multiplications of OBME, the proposed method uses a predetermined pattern in selecting matched pixels within a window support. Simulation results show that the proposed method achieves about the same prediction performance as the conventional approach.

Frame rate up conversion has been discussed for several years. Especially in the emerge of large-size LCD display devices, frame rate up conversion erase the motion blur caused by liquid crystal rotation. Among existing techniques, frame rate up conversion by motion compensation is more and more popular. In this paper, a novel frame rate up conversion method, overlapped block-based adaptive bilateral motion estimation, is proposed combining adaptive sum of bilateral absolute difference and overlapped block motion estimation. Based on the analysis of block size transition, smaller block size compensation is applied while the search block size is large. According to the experiment results, the proposed work outperforms existing work by 6.25 dB on average. This concludes a high video quality frame rate up converter design.

In this letter, we present a new motion estimation algorithm for frame rate up-conversion. The proposed dual motion estimation algorithm enhances the estimation accuracy of motion vectors by using the unidirectional and bidirectional matching ratios of blocks in the previous and current frames. In addition, the proposed motion estimation approach uses motion vector validity to evaluate the accuracy of motion vectors thereby avoiding false motion vectors. In experiments using benchmark image sequences, the proposed motion estimation algorithm improved the average peak signal-to-noise ratio of interpolated frames by up to 2.272 dB, when compared to conventional motion estimation algorithms. For the comparison of the perceptual image quality using the structural similarity, the average value of the proposed dual motion estimation was by up to 0.062 higher than those of the conventional algorithms.

In this paper, we propose a frame rate up-conversion (FRUC) algorithm to increase the temporal resolution of video sequences at the decoder side. First, the proposed algorithm segments a frame into several objects. Then, perspective transforms are used to motion-compensate each object, while the translational block matching algorithm (BMA) is applied to the background. Furthermore, the overlapped block motion compensation (OBMC) technique is used to reduce blocking artifacts in boundary blocks. Experimental results show that the proposed algorithm provides better performance than the conventional approach.

Motion estimation and motion compensation (MEMC) has been applied inside the wavelet-transformed domain for video compression at low bit rates. However, pixel-based motion estimation in the wavelet domain is not as straightforward as in the spatial domain due to the shift variance property caused by decimation. The paper finds a new wavelet-based motion estimation and compensation method, named as the sub-decimation decomposition method, to perform shift invariant motion estimation inside the wavelet transformed domain, so that better matching blocks are compensated to achieve more efficient video compression.

Liquid crystal displays (LCDs), which serve as receivers of high visual quality video, suffer from motion blur issues. One of the methods in terminating motion blur is motion compensated frame rate upconversion, which is widely adopted in LCDs. In the previous work, i.e., particle-based frame rate upconversion, the computation complexity is high while repeated operations and some improper cost evaluation setup are observed. Therefore, in this paper, hybrid frame rate upconversion is proposed with two features. First, the cost evaluation for particle-based motion trajectory calibration is modified based on the possible noise sources and video resolution variations. Second, repeated operations in particle-based motion trajectory calibration are observed. Therefore, original particle-based motion trajectory calibration is replaced by initial motion vector assignment and subsequent motion vector mapping to achieve computation complexity reduction, while the effective search range is relatively expanded. According to the experiment results, the visual quality is enhanced by 1.87 dB on average, compared with state-of-the-art unidirectional-based frame rate upconversion approaches. On the other hand, the computation complexity of the proposed design is reduced by 25-90% based on target video resolution, concluding a high visual quality and low computation complexity frame rate upconversion design.