A Performance Comparison of Block-Based Matching Cost Evaluation Models for FRUC Techniques

In motion compensated frame interpolation (MCFI), an interpolated frame is produced by placing objects according to their motion estimated between two neighbouring frames. However, multiple motion vectors associated with the background (BG) and foreground (FG) objects can pass through a single location in the interpolated frame. In such motion ambiguity (MA) regions, annoying artefacts can be induced by adopting the motion of the BG object instead of the FG object. Introduced is a novel MCFI algorithm which can accurately infer the motion of the FG object at the MA region with low complexity. Simulation results show that the proposed method achieves higher visual quality for the interpolated frame without annoying artefacts as compared with conventional MCFI algorithms.

비디오 프레임 율 증가 변환은 최근에 가전 분야에서 비디오 정보원과 더불어 디스플레이 포맷의 폭발적인 증가에 힘입어 매우 많은 관심의 대상이 되었다. 대부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 효과적인 움직임 보간 기술을 사용하고 있다. 그러나 수신기의 후처리 분야와 같은 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 달성하기 위해, 먼저, 본 논문에서는 움직임 보상 보간 프레임의 블록 단위로 신뢰도를 측정할 수 있는 비용 함수들을 도입한다. 그런 후에 이러한 함수들을 사용하여 움직임 보상 보간 프레임에서 얼마나 많은 잡음이 포함되어 있는지를 평가할 수 있는 두 가지의 왜곡 예측 모델을 제안한다. 모의실험을 통하여 제안된 왜곡 예측 방식은 움직임 보상 보간 프레임의 잡음을 효과적으로 예측할 수 있음을 보인다. Video FRUC (Frame Rate Up Conversion) is one of the main issues that have arisen in recent years with the explosive growth of video sources and display formats in consumer electronics. Most advanced FRUC algorithms adopt an efficient motion interpolation technique to determine the motion vector field of interpolated frames. But, in some application areas such as post processing in receiver side, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame was reconstructed. In order to achieve this aim, first, this paper introduces some cost functions to estimate the reliability of a block in the MCI frame. Then, by using these functions, this paper proposes two distortion estimation models for evaluating how much noise was produced in the MCI frame. Through computer simulations, it is shown that the proposed estimation methods perform effectively in estimating the noises of the MCI frame.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and also Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase-based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

Frame rate up-conversion is a postprocessing tool to convert the frame rate from a lower number to a higher one. It is a useful technique for a lot of practical applications, such as display format conversion, low bit rate video coding and slow motion playback. Unlike traditional approaches, such as frame repetition or linear frame interpolation, motion-compensated frame interpolation (MCFI) technique is more efficient since it takes block motion into account. In this paper, by considering the deficiencies of previous works, new criteria and coding schemes for improving motion derivation and interpolation processes are proposed. Next, for video coding applications, adaptive frame skip is executed at the encoder side to maximize the power of MCFI so that the quality of interpolated frames is guaranteed. Experimental results show that our proposal effectively enhances the overall quality of the frame rate up-converted video sequence, both subjectively and objectively.

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

In this paper, we propose a new motion compensated frame interpolation (MCFI) method for occlusion and motion ambiguity regions of frame rate up-conversion (FRUC) assisted by depth information in color-plus-depth video. First, we estimate motion vectors in a similar way to conventional methods and refine them using depth information. In the intermediate frame, we try to solve challenging FRUC problems such as occlusion, boundary, and motion ambiguity using depth information. Finally, interpolation is performed to generate intermediate frames. Experiments were performed to compare the proposed method and conventional color and color-plus-depth FRUC methods. The experiment result shows that the proposed method using depth information can improve the FRUC performance.

This paper proposes a new motion-compensated frame (field) interpolation algorithm for frame (field) rate upconversion, which allows us to interpolate frame (or pairs of fields) between two originally continuous frames (fields) of a digital television sequence by preserving the stationary background. First, for a interlace format, the de-interlacing process was used to reduce the motion range after converting the interlace format to progressive one. A video scene can be temporally categorized by the change detector into changed and unchanged regions. Each changed region is further separated into moving objects, covered and uncovered regions. To interpolate the intermediate field (frame), we have developed direct motion interpolation method and indirect motion interpolation method to fill the moving object areas in the changed regions, and then apply the forward/backward motion extrapolation method to fill the covered/uncovered regions. Finally the hybrid repetition is used to interpolate the unchanged regions. In the experiment, we will show the interpolated fields and frames for two standard image sequences.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

One of the most efficient approaches to generate the side information (SI) in distributed video codecs is through motion compensated frame interpolation where the current frame is estimated based on past and future reference frames. However, this approach leads to significant spatial and temporal variations in the correlation noise between the source at the encoder and the SI at the decoder. In such scenario, it would be useful to design an architecture where the SI can be more robustly generated at the block level, avoiding the creation of SI frame regions with lower correlation, largely responsible for some coding efficiency losses. In this paper, a flexible framework to generate SI at the block level in two modes is presented: while the first mode corresponds to a motion compensated interpolation (MCI) technique, the second mode corresponds to a motion compensated quality enhancement (MCQE) technique where a low quality Intra block sent by the encoder is used to generate the SI by doing motion estimation with the help of the reference frames. The novel MCQE mode can be overall advantageous from the rate-distortion point of view, even if some rate has to be invested in the low quality Intra coding blocks, for blocks where the MCI produces SI with lower correlation. The overall solution is evaluated in terms of RD performance with improvements up to 2 dB, especially for high motion video sequences and long Group of Pictures (GOP) sizes.

This paper presents a new frame-rate up conversion method for video plus depth representation. The proposed method improves the quality of video using additional depth information. The depth information is utilized in three ways to improve the accuracy of motion estimation. One usage is that depth frame is added in block matching criterion to estimate initial motion. Another usage is to detect object boundary region combining depth value variation and local depth distribution adjustment technique. Finally, motion vector field is refined adaptively by segmenting macroblocks into object regions. Any existing block-based motion-compensated frame interpolation methods may utilize our method to refine motion vector field by using depth frames. Experimental results verify that the proposed method outperforms the conventional motion-compensated frame interpolation algorithms, while preserving object structure.

Frame rate upconversion (FRUC) is an important post-processing technique to enhance the visual quality of low frame rate video. A major, recent advance in this area is FRUC based on trilateral filtering which novelty mainly derives from the combination of an edge-based motion estimation block matching criterion with the trilateral filter. However, there is still room for improvement, notably towards reducing the size of the uncovered regions in the initial estimated frame, this means the estimated frame before trilateral filtering. In this context, proposed is an improved motion estimation block matching criterion where a combined luminance and edge error metric is weighted according to the motion vector components, notably to regularise the motion field. Experimental results confirm that significant improvements are achieved for the final interpolated frames, reaching PSNR gains up to 2.73 dB, on average, regarding recent alternative solutions, for video content with varied motion characteristics.

Distributed video coding (DVC) architecture designs, based on distributed source coding principles, have benefitted from significant progresses lately, notably in terms of achievable rate-distortion performances. However, a significant performance gap still remains when compared to prediction-based video coding schemes such as H.264/AVC. This is mainly due to the non-ideal exploitation of the video sequence temporal correlation properties during the generation of side information (SI). In fact, the decoder side motion estimation provides only an approximation of the true motion. In this paper, a progressive DVC architecture is proposed, which exploits the spatial correlation of the video frames to improve the motion-compensated temporal interpolation (MCTI). Specifically, Wyner-Ziv (WZ) frames are divided into several spatially correlated groups that are then sent progressively to the receiver. SI refinement (SIR) is performed as long as these groups are being decoded, thus providing more accurate SI for the next groups. It is shown that the proposed progressive SIR method leads to significant improvements over the Discover DVC codec as well as other SIR schemes recently introduced in the literature.