Hybrid Frame Rate Upconversion Method Based on Motion Vector Mapping

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

Motion estimation (ME) has the highest computational complexity in motion-compensated frame rate up-conversion (MC-FRUC). For the real-time implementation of FRUC, a fast ME algorithm is required. In this paper, a new hierarchical ME algorithm for MC-FRUC is proposed. It constructs an image pyramid by dividing the frame into several sub-images according to resolution, and performs ME at the top level to reduce complexity while improving accuracy by selecting multiple motion vector candidates. These candidates are refined at the lower levels, and the final motion vector is selected at the bottom level. Thus, the proposed algorithm obtains an average peak signal-to-noise ratio gain of upto 0.85 dB compared to conventional algorithms with lower computational complexity and yields interpolated images with better visual quality than other methods.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

In this paper, a novel, low-complexity motion vector processing algorithm at the decoder is proposed for motion-compensated frame interpolation or frame rate up-conversion. We address the problems of having broken edges and deformed structures in an interpolated frame by hierarchically refining motion vectors on different block sizes. Our method explicitly considers the reliability of each received motion vector and has the capability of preserving the structure information. This is achieved by analyzing the distribution of residual energies and effectively merging blocks that have unreliable motion vectors. The motion vector reliability information is also used as a prior knowledge in motion vector refinement using a constrained vector median filter to avoid choosing identical unreliable one. We also propose using chrominance information in our method. Experimental results show that the proposed scheme has better visual quality and is also robust, even in video sequences with complex scenes and fast motion.

This paper presents a motion compensated frame rate up-conversion algorithm that uses the expanded range of the motion trajectory to enhance the accuracy of motion estimation. In the proposed algorithm, an adaptive motion vector refinement is also proposed to correct false motion vectors. In addition, a recursive motion vector smoothing is proposed to smooth motion vectors considering true neighboring motion vectors only. Finally, we propose the weighted index-based bidirectional motion compensated interpolation to reduce artifacts. In the experiments using benchmark test sequences, the proposed algorithm improves the average PSNR of interpolated frames by up to 4.84 dB, when compared to the conventional algorithm.

This paper presents a novel motion-compensated frame rate up-conversion algorithm that uses dual criterion motion estimation to obtain accurate motion vectors in the circumstance of scene luminance changing. In the proposed algorithm, we also present a selective vector median filtering method and a motion vector refinement method to address block mismatching problem. Finally, a motion-compensated interpolation algorithm based on adaptive overlapped block motion compensation is proposed to reduce block artifact. Experimental results demonstrate that the proposed algorithm improves visual qualities compared with the existing algorithms.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

In Multimedia Internet of Things (IoT), in order to reduce the bandwidth consumption of wireless channels, Motion-Compensated Frame Rate Up-Conversion (MC-FRUC) is often used to support the low-bitrate video communication. In this paper, we propose a spatial predictive algorithm which is used to improve the performance of MC-FRUC. The core of the proposed algorithm is a predictive model to split a frame into two kinds of blocks: basic blocks and absent blocks. Then an improved bilateral motion estimation is proposed to compute the Motion Vectors (MVs) of basic blocks. Finally, with the spatial correlation of Motion Vector Field (MVF), the MV of an absent block is predicted based on the MVs of its neighboring basic blocks. Experimental results show that the proposed spatial prediction algorithm can improve both the objective and the subjective quality of the interpolated frame, with a low computational complexity.

Frame rate up-conversion is a useful post processing technique to enhance the visual quality of low frame rate video materials. In this paper, we propose a novel frame rate up-conversion algorithm utilizing true motion estimation and adaptive motion vector refinement. First, block-matching algorithm (BMA) based on an efficient motion similarity constrained criterion is employed to track the true motion between adjacent frames and produce a preliminary intermediate motion vector field (MVF).We then classify the intermediate motion vectors into two categories, isolated group and connected group adaptively by motion clustering. A smooth resampling method is later applied to connected group to obtain consistent motion, while isolated motion vectors are re-estimated with bilateral edge-weighted block decomposition (BEWBD). Finally, overlapped block motion compensation (OBMC) is adopted to interpolate the intermediate frame. Extensive experimental results show that the proposed algorithm outperforms conventional methods both subjectively and objectively.

본 논문은 EBME(Extended Bilateral Motion Estimation) 알고리듬에서 동일한 움직임 벡터들의 방향과 크기를 고려한 알고리듬을 제안하였다. EBME는 BME(Bilateral Motion Estimation)보다 높은 연산량을 요구하기 때문에 프레임 내의 x, y방향 각각의 평균 움직임 벡터크기를 이용하여 동적프레임과 정적프레임을 판단하고, EBME 수행여부를 판단하여 연산량을 줄인다. 또한 동일한 움직임 벡터들의 방향과 크기를 비교하여 MVS(Motion Vector Smoothing)단계 수행여부를 판단함으로써 연산량을 줄인다. 제안하는 알고리듬을 적용한 실험 결과 기존의 EBME 알고리듬에 비해 수행시간은 단축되고 PSNR(Peak Signal to Noise Ratio)은 향상 되었다. In this paper, frame rate up conversion (FRUC) algorithm considering the direction and magnitude of identical motion vectors is proposed. extended bilateral motion estimation (EBME) has higher complexity than bilateral motion estimation (BME). By using average magnitude of motion vector with x and y direction respectively, dynamic frame and static frame are decided. We reduce complexity to decide EBME. also, After we compare the direction and magnitude of identical motion vectors, We reduce complexity to decide motion vector smoothing(MVS). Experimental results show that this proposed algorithm has fast computation and better peak singnal to noise ratio(PSNR) results compared with EBME.

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Traditionally, the re-use of motion vectors extracted from incoming video bit-stream during transcoding has been widely accepted. However, this simple re-use scheme introduces significant quality degradation in many applications including the situation when the frame-rate conversion is needed. In this paper, we analyzed the quantization errors that cause the extracted motion vectors to be non-optimal and we performed simulations to show the quality degradation due to the inaccurate motion vectors during transcoding. To improve the video quality, we proposed an adaptive motion vector refinement. With a highly reduced computational complexity, the proposed adaptive motion vector refinement achieves significant quality improvement in comparison to the conventional motion vector re-use scheme. In addition, the adaptive motion vector refinement is almost as good as performing a new full-scale motion estimation.

In this paper, we propose a new motion compensated frame interpolation (MCFI) method for occlusion and motion ambiguity regions of frame rate up-conversion (FRUC) assisted by depth information in color-plus-depth video. First, we estimate motion vectors in a similar way to conventional methods and refine them using depth information. In the intermediate frame, we try to solve challenging FRUC problems such as occlusion, boundary, and motion ambiguity using depth information. Finally, interpolation is performed to generate intermediate frames. Experiments were performed to compare the proposed method and conventional color and color-plus-depth FRUC methods. The experiment result shows that the proposed method using depth information can improve the FRUC performance.

비디오 프레임 율 증가 변환은 최근에 가전 분야에서 비디오 정보원과 더불어 디스플레이 포맷의 폭발적인 증가에 힘입어 매우 많은 관심의 대상이 되었다. 대부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 효과적인 움직임 보간 기술을 사용하고 있다. 그러나 수신기의 후처리 분야와 같은 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 달성하기 위해, 먼저, 본 논문에서는 움직임 보상 보간 프레임의 블록 단위로 신뢰도를 측정할 수 있는 비용 함수들을 도입한다. 그런 후에 이러한 함수들을 사용하여 움직임 보상 보간 프레임에서 얼마나 많은 잡음이 포함되어 있는지를 평가할 수 있는 두 가지의 왜곡 예측 모델을 제안한다. 모의실험을 통하여 제안된 왜곡 예측 방식은 움직임 보상 보간 프레임의 잡음을 효과적으로 예측할 수 있음을 보인다. Video FRUC (Frame Rate Up Conversion) is one of the main issues that have arisen in recent years with the explosive growth of video sources and display formats in consumer electronics. Most advanced FRUC algorithms adopt an efficient motion interpolation technique to determine the motion vector field of interpolated frames. But, in some application areas such as post processing in receiver side, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame was reconstructed. In order to achieve this aim, first, this paper introduces some cost functions to estimate the reliability of a block in the MCI frame. Then, by using these functions, this paper proposes two distortion estimation models for evaluating how much noise was produced in the MCI frame. Through computer simulations, it is shown that the proposed estimation methods perform effectively in estimating the noises of the MCI frame.