동일한 움직임 벡터들의 방향과 크기를 고려한 프레임율 증가기법

This paper presents a new method for frame rate up‐conversion using sub‐sampled bilateral motion estimation, motion vector smoothing, simplified bidirectional motion compensated interpolation, and complex motion detection. The proposed method produces improved image quality and requires less computational complexity compared to conventional motion estimation. The average PSNRs of the proposed method for frame rate up‐conversion outperform those of the benchmark frame rate up‐conversion technique by up to 4.39 dB for the test sequences. Furthermore, the motion estimation of the proposed FRUC method reduces the required computational complexity to 12.5% of that required for full‐search block matching motion estimation.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

Liquid crystal displays (LCDs), which serve as receivers of high visual quality video, suffer from motion blur issues. One of the methods in terminating motion blur is motion compensated frame rate upconversion, which is widely adopted in LCDs. In the previous work, i.e., particle-based frame rate upconversion, the computation complexity is high while repeated operations and some improper cost evaluation setup are observed. Therefore, in this paper, hybrid frame rate upconversion is proposed with two features. First, the cost evaluation for particle-based motion trajectory calibration is modified based on the possible noise sources and video resolution variations. Second, repeated operations in particle-based motion trajectory calibration are observed. Therefore, original particle-based motion trajectory calibration is replaced by initial motion vector assignment and subsequent motion vector mapping to achieve computation complexity reduction, while the effective search range is relatively expanded. According to the experiment results, the visual quality is enhanced by 1.87 dB on average, compared with state-of-the-art unidirectional-based frame rate upconversion approaches. On the other hand, the computation complexity of the proposed design is reduced by 25-90% based on target video resolution, concluding a high visual quality and low computation complexity frame rate upconversion design.

A new bit allocation technique between motion vectors and DCT coefficients for MPEG-2 encoding is presented. In MPEG-2 encoding, the differential between two neighboring motion vectors is losslessly encoded by a Huffman table. In order to adjust the bit allocation for the motion vectors, a motion vector smoothing approach is taken. The motion vectors are smoothed by a low pass filter when the bit allocation for the motion vector has to be reduced. An algorithm based on the Lagrange multiplier is used to achieve the optimal balance between motion vector smoothing and DCT coefficients quantization.

In this paper, a novel, low-complexity motion vector processing algorithm at the decoder is proposed for motion-compensated frame interpolation or frame rate up-conversion. We address the problems of having broken edges and deformed structures in an interpolated frame by hierarchically refining motion vectors on different block sizes. Our method explicitly considers the reliability of each received motion vector and has the capability of preserving the structure information. This is achieved by analyzing the distribution of residual energies and effectively merging blocks that have unreliable motion vectors. The motion vector reliability information is also used as a prior knowledge in motion vector refinement using a constrained vector median filter to avoid choosing identical unreliable one. We also propose using chrominance information in our method. Experimental results show that the proposed scheme has better visual quality and is also robust, even in video sequences with complex scenes and fast motion.

In this paper, we present an improved multi-pass true motion estimation algorithm for frame rate up-conversion. In the proposed motion estimation algorithm, the motion vectors of different objects which are parted with depth information will be refined respectively. Therefore, more accurate pixel-based motion vector can be obtained which causes the performance of Frame Rate Up-Conversion (FRUC). For overlapped regions, the motion vector can be chosen in candidates with depth information. Experimental results show that our proposed algorithm effectively enhances the overall quality of the frame rate up-converted video sequence, both subjectively and objectively, even with complex scenes and fast motion.

Frame rate up-conversion is a useful post processing technique to enhance the visual quality of low frame rate video materials. In this paper, we propose a novel frame rate up-conversion algorithm utilizing true motion estimation and adaptive motion vector refinement. First, block-matching algorithm (BMA) based on an efficient motion similarity constrained criterion is employed to track the true motion between adjacent frames and produce a preliminary intermediate motion vector field (MVF).We then classify the intermediate motion vectors into two categories, isolated group and connected group adaptively by motion clustering. A smooth resampling method is later applied to connected group to obtain consistent motion, while isolated motion vectors are re-estimated with bilateral edge-weighted block decomposition (BEWBD). Finally, overlapped block motion compensation (OBMC) is adopted to interpolate the intermediate frame. Extensive experimental results show that the proposed algorithm outperforms conventional methods both subjectively and objectively.

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

This paper investigates the dynamic frame-skipping strategy in video transcoding. To speed up the operation, a video transcoder usually reuses the decoded motion vectors to reencode the video sequences at a lower bit-rate. When frame-skipping is allowed in a transcoder, those motion vectors can not be reused because the motion vectors of the current frame is no longer estimated from the immediate past frame. To reduce the computational complexity of motion vectors re-estimation, a bilinear interpolation approach is developed to overcome this problem. Based on these interpolated motion vectors, the search range can be much reduced. Furthermore, we propose a frame rate control scheme which can dynamically adjust the number of skipped frames according to the accumulated magnitude of motion vectors. As a result, the decoded sequence can present much smoother motion.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

본 논문에서는 JVT에서 표준화가 진행 중인 다시점 비디오 부호화의 부호화 효율을 높이기 위해 다시점 비디오의 예측구조의 특성을 고려하여 움직임벡터 예측값을 선택하는 방법을 제안한다. 다시점 비디오 부호화에서 시간 방향과 뷰 방향의 예측을 수행하면서 서로 다른 성향의 움직임벡터가 발생된다. 또한 주변 블록에서 시간 방향과 뷰 방향 참조의 움직임벡터가 혼재하는 상황에서 동일한 성향의 움직임벡터가 선택되지 않는 경우가 발생되어 움직임벡터의 예측 효율이 저하된다. 이에 다시점 비디오 부호화의 예측구조에 따라 움직임벡터 예측값을 선택할 수 있는 방법을 제안한다. 제안하는 움직임벡터 예측값 선택 방법을 통하여 다시점 비디오 부호화의 압축 효율을 높일 수 있었으며, JMVM 3.6에서 움직임벡터 예측값을 선택하는 방법을 사용하는 다시점 비디오 부호화와 비교하여 PSNR(Peak Signal-to-Noise Ratio)이 약<TEX>$0.03{\sim}0.1$</TEX> dB 정도 향상되었다. In this paper, we propose a method to select motion vector predictor by considering prediction structure of a multi view content for coding efficiency of multi view coding which is being standardized in JVT. Motion vector of a different tendency is happened while carrying out temporal and view reference prediction of multi-view video coding. Also, due to the phenomena of motion vectors being searched in both temporal and view order, the motion vectors do not agree with each other resulting a decline in coding efficiency. This paper is about how the motion vector predictor are selected with information of prediction structure. By using the proposed method, a compression ratio of the proposed method in multi-view video coding is increased, and finally <TEX>$0.03{\sim}0.1$</TEX> dB PSNR(Peak Signal-to-Noise Ratio) improvement was obtained compared with the case of JMVM 3.6 method.

Frame rate conversion requires interpolation of image frames at time instances, where the original sequence has not been sampled. This can be done in high quality only by means of motion-compensated algorithms, therefore the knowledge of motion present in the sequence is essential. This motion information has to be determined from the image sequence itself. In this paper a motion-based image segmentation algorithm is proposed, and its application for motion- compensated (MC) frame rate conversion is presented. The segmentation algorithm that can trace multiple rigid objects with translational movement, is based on vector quantization of the estimated motion field determining a set of global motion vectors and segmenting the images into multiple moving areas. Then, the spatial order of the objects (which one is in front of the other) is determined. Interpolation is performed based on the results of the segmentation, the set of motion vectors, and the proper handling of covered and uncovered areas. Furthermore, an accelerated motion model developed previously by the authors is applied, in order to further improve the performance of the MC frame rate converter.

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.

Traditionally, the re-use of motion vectors extracted from incoming video bit-stream during transcoding has been widely accepted. However, this simple re-use scheme introduces significant quality degradation in many applications including the situation when the frame-rate conversion is needed. In this paper, we analyzed the quantization errors that cause the extracted motion vectors to be non-optimal and we performed simulations to show the quality degradation due to the inaccurate motion vectors during transcoding. To improve the video quality, we proposed an adaptive motion vector refinement. With a highly reduced computational complexity, the proposed adaptive motion vector refinement achieves significant quality improvement in comparison to the conventional motion vector re-use scheme. In addition, the adaptive motion vector refinement is almost as good as performing a new full-scale motion estimation.

This paper presents a novel motion-compensated frame rate up-conversion algorithm that uses dual criterion motion estimation to obtain accurate motion vectors in the circumstance of scene luminance changing. In the proposed algorithm, we also present a selective vector median filtering method and a motion vector refinement method to address block mismatching problem. Finally, a motion-compensated interpolation algorithm based on adaptive overlapped block motion compensation is proposed to reduce block artifact. Experimental results demonstrate that the proposed algorithm improves visual qualities compared with the existing algorithms.