SIGNAL TRANSMISSION OPTIMIZATION MOVING-VECTOR CLASSIFICATION MPEG STANDARD VIDEO CODING

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

Frame rate up-conversion is a postprocessing tool to convert the frame rate from a lower number to a higher one. It is a useful technique for a lot of practical applications, such as display format conversion, low bit rate video coding and slow motion playback. Unlike traditional approaches, such as frame repetition or linear frame interpolation, motion-compensated frame interpolation (MCFI) technique is more efficient since it takes block motion into account. In this paper, by considering the deficiencies of previous works, new criteria and coding schemes for improving motion derivation and interpolation processes are proposed. Next, for video coding applications, adaptive frame skip is executed at the encoder side to maximize the power of MCFI so that the quality of interpolated frames is guaranteed. Experimental results show that our proposal effectively enhances the overall quality of the frame rate up-converted video sequence, both subjectively and objectively.

In this paper, we propose a new motion compensated frame interpolation (MCFI) method for occlusion and motion ambiguity regions of frame rate up-conversion (FRUC) assisted by depth information in color-plus-depth video. First, we estimate motion vectors in a similar way to conventional methods and refine them using depth information. In the intermediate frame, we try to solve challenging FRUC problems such as occlusion, boundary, and motion ambiguity using depth information. Finally, interpolation is performed to generate intermediate frames. Experiments were performed to compare the proposed method and conventional color and color-plus-depth FRUC methods. The experiment result shows that the proposed method using depth information can improve the FRUC performance.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

Motion-compensated frame rate up-conversion (MC-FRUC) increases the frame rate of a video by interpolating a new frame from temporally consecutive frames. MC-FRUC consists of two key operations: motion estimation and motion compensated frame interpolation. Motion compensation along unidirectional motion trajectories directly causes overlaps and holes issues. To solve these issues, this paper presents a new algorithm for bidirectional motion compensated frame interpolation. Firstly, the proposed method generates a bidirectional motion vector field from two unidirectional motion vector fields (forward and backward) obtained from the motion estimation. It is done by projecting the forward and backward motion vectors into the interpolated frame. A comprehensive metric as an extension of the distance between a projected block and an interpolated block is proposed to compute weighted coefficients in case the interpolated block has multiple projected ones. Holes are filled based on vector median filter of non-hole available neighbor blocks. The proposed method outperforms existing MC-FRUC methods and removes block artifacts significantly.

DVC (Distributed Video Coding) and FRUC (Frame Rate Up Conversion) techniques need to have an efficient motion compensated frame interpolation algorithms. Conventional works of these applications have mainly focused on the performance improvement of overall system. But, in some applications, it is necessary to evaluate how well the MCI (Motion Compensated Interpolation) frame matches the original frame. For this aim, this paper deals with the modeling methods for evaluating the block-based matching cost. First, several matching criteria, which have already been dealt with the motion compensated frame interpolation, are introduced and then combined to make estimate models for the size of MSE (Mean Square Error) noise of the MCI frame to original one. Through computer simulations, it is shown that the block-based matching criteria are evaluated and the proposed model can be effectively used for estimating the MSE noise.

The frame rate (FR) of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second (fps), at low resolutions like CIF or QCIF. As the video frame rate and resolution advance, we reconsider this issue and investigate the relationship between frame rate and the perceptual video quality under high frame rate and high resolution. In this paper, we discuss the impact of frame rate on the perceptual quality of High Definition (HD) video with high frame rates (up to 120 fps) considered. Firstly, we design and conduct subjective experiment to construct the video dataset, which includes video sequences at different frame rates and the corresponding mean opinion scores (MOS) which represent the perceptual video quality. Based on the MOS results, we analyze how perceptual video quality changes as frame rate varies among different video sequences and propose some meaningful findings. The video dataset will be made publicly available. We deem that this study will enrich video quality assessment and benefit the development of high frame rate and high definition video business.KeywordsVideo quality assessmentSubjective experienceHigh frame rateHigh definition

Motion-compensated frame interpolation (MCFI), a frame-interpolation technique to increase the motion continuity of low frame-rate video, can be utilized by counterfeiters for faking high bitrate video or splicing videos with different frame rates. For existing MCFI detectors, their performances are degraded under real-world scenarios such as H.264/AVC compression, noise, or blur. To address this issue, a robust MCFI detector is proposed to locate interpolated frames. By analyzing the distribution of residual energies within interpolated frames, we observe that there exist strong correlations between artifact regions and high residual energies. Thus, an artifact indicated map is introduced to select candidate artifact regions. Then, Tchebichef moments (TMs) are exploited to characterize the blurring effects or deformed structures among these regions. Specifically, the mean value of absolute high-order TMs of selected regions is used to model these temporal inconsistencies. Finally, a sliding window is adopted to locate interpolated frames, which are further refined by three post-processing operations. Chrominance information is also integrated with luminance information for robust identification of interpolated frames. Extensive experimental results show that compared with the state-of-the-art MCFI detectors, the proposed approach is more robust for compressed videos under various real-world scenarios.

Most consumer cameras use rolling shutter (RS) exposure, the captured videos often suffer from distortions (e.g., skew and jelly effect). Also, these videos are impeded by the limited bandwidth and frame rate, which inevitably affect the video streaming experience. In this paper, we excavate the potential of event cameras as they enjoy high temporal resolution. Accordingly, we propose a framework to recover the global shutter (GS) high frame rate (i.e., slow motion) video without RS distortion from an RS camera and event camera. One challenge is the lack of real-world datasets for supervised training. Therefore, we explore self-supervised learning with the key idea of estimating the displacement field-a non-linear and dense 3D spatiotemporal representation of all pixels during the exposure time. This allows for a mutual reconstruction between RS and GS frames and facilitates slow-motion video recovery. We then combine the input RS frames with the DF to map them to the GS frames (RS-to-GS). Given the under-constrained nature of this mapping, we integrate it with the inverse mapping (GS-to-RS) and RS frame warping (RS-to-RS) for self-supervision. We evaluate our framework via objective analysis (i.e., quantitative and qualitative comparisons on four datasets) and subjective studies (i.e., user study). The results show that our framework can recover slow-motion videos without distortion, with much lower bandwidth (94% drop) and higher inference speed ($ 16\; {\rm ms}/{\rm frame}$16 ms / frame ) under $32 \times$32× frame interpolation.

The purpose of this study was to assess whether or not high frame rate (HFR) videos recorded using high-speed digital imaging (HSDI) improve the visual recognition of the motions of the laryngopharyngeal structures during pharyngeal swallow in fiberoptic endoscopic evaluation of swallowing (FEES). Five healthy subjects were asked to swallow 0.5ml water under fiberoptic nasolaryngoscopy. The endoscope was connected to a high-speed camera, which recorded the laryngopharyngeal view throughout the swallowing process at 4000 frames/s (fps). Each HFR video was then copied and downsampled into a standard frame rate (SFR) video version (30 fps). Fifteen otorhinolaryngologists observed all of the HFR/SFR videos in random order and rated the four-point ordinal scale reflecting the degree of visual recognition of the rapid laryngopharyngeal structure motions just before the 'white-out' phenomenon. Significantly higher scores, reflecting better visibility, were seen for the HFR videos compared with the SFR videos for the following laryngopharyngeal structures: the posterior pharyngeal wall (p=0.001), left pharyngeal wall (p=0.015), right lateral pharyngeal wall (p=0.035), tongue base (p=0.005), and epiglottis tilting (p=0.005). However, when visualized with HFR and SFR, 'certainly clear observation' of the laryngeal structures was achieved in <50% of cases, because all the motions were not necessarily captured in each video. These results demonstrate the use of HSDI in FEES makes the motion perception of the laryngopharyngeal structures during pharyngeal swallow easier in comparison to SFR videos with equivalent image quality due to the ability of HSDI to depict the laryngopharyngeal motions in a continuous manner.

This paper proposes a new motion-compensated frame (field) interpolation algorithm for frame (field) rate upconversion, which allows us to interpolate frame (or pairs of fields) between two originally continuous frames (fields) of a digital television sequence by preserving the stationary background. First, for a interlace format, the de-interlacing process was used to reduce the motion range after converting the interlace format to progressive one. A video scene can be temporally categorized by the change detector into changed and unchanged regions. Each changed region is further separated into moving objects, covered and uncovered regions. To interpolate the intermediate field (frame), we have developed direct motion interpolation method and indirect motion interpolation method to fill the moving object areas in the changed regions, and then apply the forward/backward motion extrapolation method to fill the covered/uncovered regions. Finally the hybrid repetition is used to interpolate the unchanged regions. In the experiment, we will show the interpolated fields and frames for two standard image sequences.

High frame rate (HFR) videos are becoming increasingly common with the tremendous popularity of live, high-action streaming content such as sports. Although HFR contents are generally of very high quality, high bandwidth requirements make them challenging to deliver efficiently, while simultaneously maintaining their quality. To optimize trade-offs between bandwidth requirements and video quality, in terms of frame rate adaptation, it is imperative to understand the intricate relationship between frame rate and perceptual video quality. Towards advancing progression in this direction we designed a new subjective resource, called the LIVE-YouTube-HFR (LIVE-YT-HFR) dataset, which is comprised of 480 videos having 6 different frame rates, obtained from 16 diverse contents. In order to understand the combined effects of compression and frame rate adjustment, we also processed videos at 5 compression levels at each frame rate. To obtain subjective labels on the videos, we conducted a human study yielding 19,000 human quality ratings obtained from a pool of 85 human subjects. We also conducted a holistic evaluation of existing state-of-the-art Full and No-Reference video quality algorithms, and statistically benchmarked their performance on the new database. The LIVE-YT-HFR database has been made available online for public use and evaluation purposes, with hopes that it will help advance research in this exciting video technology direction. It may be obtained at <uri>https://live.ece.utexas.edu/research/LIVE_YT_HFR/LIVE_YT_HFR/index.html</uri>.

In this paper, we present an improved multi-pass true motion estimation algorithm for frame rate up-conversion. In the proposed motion estimation algorithm, the motion vectors of different objects which are parted with depth information will be refined respectively. Therefore, more accurate pixel-based motion vector can be obtained which causes the performance of Frame Rate Up-Conversion (FRUC). For overlapped regions, the motion vector can be chosen in candidates with depth information. Experimental results show that our proposed algorithm effectively enhances the overall quality of the frame rate up-converted video sequence, both subjectively and objectively, even with complex scenes and fast motion.