Frame Rate Up Conversion Based on Variational Image Fusion

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

Imaging system are routinely deployed for underwater search, inspection and scientific surveys of manmade and natural structures, etc. Optical cameras, while providing high resolution and target details, have range limitation according to water visibility and become ineffective in turbid environments. In comparison, high-frequency (MHz) 2-D imaging sonar video systems, introduced to the commercial market in recent years, image targets at distances of 10's of meters in highly turbid waters. Visibility permitting, the integration of visual cues in 2-D optical and sonar data would enable better performance compared to deploying either imaging system alone. We address the problem of motion estimation- e.g., for vision-based navigation and target-based positioning of a mobile submersible platform- from 2-D optical and sonar images. The application of structure from motion paradigm in this multimodal imaging scenario also enables the 3-D reconstruction of scene features. We rely on the tracking of features in the sonar and optical motion sequences independently, without the need to establish multi-modal association between corresponding optical and sonar features. In addition to improving the motion estimation accuracy, advantages of the proposed method comprise overcoming certain inherent ambiguities of monocular vision, e.g., the scale-factor ambiguity, and although rare, up to three interpretations for certain scene structures and camera motion. Experiment with synthetic and real data are presented in support of our technical contribution.

Liquid crystal displays (LCDs), which serve as receivers of high visual quality video, suffer from motion blur issues. One of the methods in terminating motion blur is motion compensated frame rate upconversion, which is widely adopted in LCDs. In the previous work, i.e., particle-based frame rate upconversion, the computation complexity is high while repeated operations and some improper cost evaluation setup are observed. Therefore, in this paper, hybrid frame rate upconversion is proposed with two features. First, the cost evaluation for particle-based motion trajectory calibration is modified based on the possible noise sources and video resolution variations. Second, repeated operations in particle-based motion trajectory calibration are observed. Therefore, original particle-based motion trajectory calibration is replaced by initial motion vector assignment and subsequent motion vector mapping to achieve computation complexity reduction, while the effective search range is relatively expanded. According to the experiment results, the visual quality is enhanced by 1.87 dB on average, compared with state-of-the-art unidirectional-based frame rate upconversion approaches. On the other hand, the computation complexity of the proposed design is reduced by 25-90% based on target video resolution, concluding a high visual quality and low computation complexity frame rate upconversion design.

The NP-Completeness theory states that exact and efficient algorithms are unlikely to exist for the class of NP-difficult problems. One way to deal with NP hardness is to relax the optimality requirement and look for solutions instead that are close to the optimum. This is the main idea behind the approximation algorithms, which are called heuristic or metaheuristic. The problem of motion estimation is a process with a high degree of computational complexity, it requires sufficient memory space and execution time. It represents the cost of static, dynamic and video image sequence coding. The main task is to minimize the distortion rate and improve visual quality. This makes research in the field of coding, image compression and video focus on finding efficient algorithms to carry out the estimation of movement in a reasonable time. If a list of images of n elements is analyzed, there are feasible solutions. So, an exhaustive search is too slow, even for small values of the solution space. Therefore, from a practical point of view, it is crucial to have efficient and fast heuristic algorithms that avoid thorough search. In this investigation we design and implement heuristic algorithms, based on the frequency domain, which are applied on the coefficients of the discrete transform of the cosine and wavelets. Also, we propose temporal domain algorithms such as block-matching algorithms, which focus your search on the maximum coincidence of the current image with the reference one. The algorithms used during the implementation of this research work were written with the mathematical programming language MATLAB. In addition, we review the basic concepts of image processing, video, compression algorithms and motion estimation frequently used. The evaluation of the algorithms was carried out with a set of images provided by a previous acquisition system. We show the improvement of visual quality, the amount of compressed or reconstructed information and the behavior of the methods in the search for similarities between pixels or images. Finally, we contribute to the dissemination of new lines of scientific research that lead to the expansion and improvement of the study, the generation of new knowledge, since it is a young area within the Education discipline of Nicaragua.

Block matching algorithms (BMAs) are often employed for motion estimation (ME) in video coding. Most conventional BMAs treat the ME problem as an optimization problem and employ certain search schemes to find a solution. Except the time-consuming full search algorithms (FS), other fast algorithms such as the three step search (TSS), searching on a reduced search range, cannot guarantee optimal solutions, i.e., search is often trapped at local minima; the ME results are thus usually unsatisfactory. Few of them makes use of the information inherent in the images explicitly. We propose a new ME algorithm which can reduce the search range while guaranteeing global optimality in most cases. Microblock visual patterns are designed to extract edge information to guide block matching: searching is only carried out at places where the real match most likely happens; that is where similar edge features present. The proposed algorithm obtains a speed about 7 - 8 times as fast as that of FS with same search range. The prediction quality is very close to that of FS and much better than that of TSS. It can produce MPEG-1 or MPEG-2 compatible motion vectors, and can be extended to model-based ME. It is suitable for parallel implementation too. Moreover, the visual patterns are potential resources for video indexing to facilitate content-based information retrieval, which is important for applications such as VOD.

In this paper, we propose a new motion compensated frame interpolation (MCFI) method for occlusion and motion ambiguity regions of frame rate up-conversion (FRUC) assisted by depth information in color-plus-depth video. First, we estimate motion vectors in a similar way to conventional methods and refine them using depth information. In the intermediate frame, we try to solve challenging FRUC problems such as occlusion, boundary, and motion ambiguity using depth information. Finally, interpolation is performed to generate intermediate frames. Experiments were performed to compare the proposed method and conventional color and color-plus-depth FRUC methods. The experiment result shows that the proposed method using depth information can improve the FRUC performance.

A new bilateral frame rate up-conversion (FRUC) algorithm based on the bi-directional comparison of the texture complexity in the reference frame is proposed. In the proposed algorithm, the accuracy of the motion estimation (ME) process is determined by the comparison of the prediction size in the reference region of the bi-directional ME process. Base on this comparison, the adaptive refinement process is adjusted. In the experimental results, the proposed algorithm outperforms the conventional bilateral FRUC algorithm by 0.27 dB.

This paper presents an application of mean field annealing (MFA) theory for motion estimation in medical image sequences. More precisely, we present the adaptation of the MFA algorithm to solve the motion estimation problem in NMR (nuclear magnetic resonance) image sequences. The proposed algorithm ensures a satisfying trade-off between precision and computational time, in other words, it ensures a good efficiency in comparison with stochastic algorithms. The results are compared in term of precision and computational time with some basic algorithms, such as the basic block-matching and Horn & Schunck. The results are illustrated on both synthetic images (obtained using the free form deformation principle) and real NMR medical image sequences.

An efficient motion estimator with application to medical image registration

비디오 프레임 율 증가 변환은 최근에 가전 분야에서 비디오 정보원과 더불어 디스플레이 포맷의 폭발적인 증가에 힘입어 매우 많은 관심의 대상이 되었다. 대부분의 진보된 FRUC 알고리즘은 보간된 프레임들의 움직임 벡터장을 결정하는 효과적인 움직임 보간 기술을 사용하고 있다. 그러나 수신기의 후처리 분야와 같은 응용 분야에서는 움직임 보상 보간 프레임이 얼마나 잘 복원되었는지에 대한 정보를 필요로 한다. 이와 같은 목적을 달성하기 위해, 먼저, 본 논문에서는 움직임 보상 보간 프레임의 블록 단위로 신뢰도를 측정할 수 있는 비용 함수들을 도입한다. 그런 후에 이러한 함수들을 사용하여 움직임 보상 보간 프레임에서 얼마나 많은 잡음이 포함되어 있는지를 평가할 수 있는 두 가지의 왜곡 예측 모델을 제안한다. 모의실험을 통하여 제안된 왜곡 예측 방식은 움직임 보상 보간 프레임의 잡음을 효과적으로 예측할 수 있음을 보인다. Video FRUC (Frame Rate Up Conversion) is one of the main issues that have arisen in recent years with the explosive growth of video sources and display formats in consumer electronics. Most advanced FRUC algorithms adopt an efficient motion interpolation technique to determine the motion vector field of interpolated frames. But, in some application areas such as post processing in receiver side, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame was reconstructed. In order to achieve this aim, first, this paper introduces some cost functions to estimate the reliability of a block in the MCI frame. Then, by using these functions, this paper proposes two distortion estimation models for evaluating how much noise was produced in the MCI frame. Through computer simulations, it is shown that the proposed estimation methods perform effectively in estimating the noises of the MCI frame.

Pose tracking is a sensitive procedure in different computer vision applications. One of the problems concerning sign language recognition is that the 3-D trajectory and the orientation could not be carefully detected using a single camera. This paper analyses the aforementioned problem from a sequence of stereo images. In this paper, we present a fast and effective algorithm for the estimation of the 3-D trajectory and orientation of a signerpsilas hand. The so-called algorithm utilizes two different procedures in order to extract the 3-D motion model, the registration step and the motion estimation step using limited feature points. Many papers which deal with this kind of problem assume that an accurate match between the corresponding points have been made, and so, deal with only the 3-D motion estimation problem. In our approach, we tackle the registration procedure with two different registration techniques. The registration of the right and left image sequences are done by means of a simple but effective technique but the registration of the feature points in two different frames are done using a more complex correlation based routine. After the registrations of feature points are completed, the handpsilas 3-D trajectory and orientation could be easily computed. This method is applied to both synthetic and real image sequences and the results obtained utilizing our method confirms this approach.

We consider the motion estimation problem in the case of very closely spaced views. We revisit the differential epipolar equation providing an interpretation of it. On the basis of this interpretation we introduce a cost function to estimate the parameters of the differential epipolar equation, which enables us to compute the camera extrinsics and some intrinsics. In the synthetic tests performed we compare this continuous method with traditional discrete motion estimation and, contrary to previous findings by Vieville et al. (1996), show that the continuous method did not perceive any computational advantage.