Efficient Video Frame Interpolation Using Generative Adversarial Networks

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and also Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase-based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

The rise of useful multimedia applications has sparked a rush to perfect video compression methods. Over the last decade, there has been a lot of interest in using motion-compensated frame interpolation (MCFI) methods to boost video frame rate during playback in both the academic and consumer electronics sectors. Interpolation refers to the process of defining the values of a function at places that are not represented by any of the examples. It accomplishes this process by fitting a constant function across the discrete input samples. We based our methodology on the fact that color cameras function similarly to the human eye. We have been used to black and white photographs, yet the real world is full of color. Grayscale images were the sole option until the advent of color monitors and cameras. Energy assessment calls for photo capture using a camera. Light's electromagnetic waves provide the observable energy in this scheme. For motion-compensated video vector frame interpolation or frame rate up-conversion and Phase frame interpolation Method, the authors present a novel approach with low vector dispensation method in the conclusion. Using optical flow or other conventional methods, it is necessary to have precise pixel correspondences between pictures in order to compute interpolated frames in a video series. We provide a practical replacement by capitalizing on advances in phase-based approaches, which use the phase shift of individual pixels to express motion. In challenging interpolation conditions, such as major appearance changes, flow-based algorithms frequently create severe visual distortions, yet our solution fails gently. Our approach is particularly useful for retiming and interpolating frames in high-resolution video shot at a high frame rate. Estimating motion for the purpose of improving the quality of the decompressed video requires more research on the nature of the compression problem. a novel way to estimating motion that is analogous to full-search 3ss,4ss, block-matching, and parametric techniques. The primary focus will be research into both error-free compression and the development of novel motion estimation algorithms. The results of the research indicate that the suggested system has the potential to increase visual superiority and is also robust, especially when applied to video sequences with quick movements and challenging portions. By combining phase based and conventional Lagrangian, which is utilized to get better quality results and deal with variations in light more gracefully.

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme.

In this paper, we propose a new motion compensated frame interpolation (MCFI) method for occlusion and motion ambiguity regions of frame rate up-conversion (FRUC) assisted by depth information in color-plus-depth video. First, we estimate motion vectors in a similar way to conventional methods and refine them using depth information. In the intermediate frame, we try to solve challenging FRUC problems such as occlusion, boundary, and motion ambiguity using depth information. Finally, interpolation is performed to generate intermediate frames. Experiments were performed to compare the proposed method and conventional color and color-plus-depth FRUC methods. The experiment result shows that the proposed method using depth information can improve the FRUC performance.

Adaptive multi-text union for stable text-to-image synthesis learning

Owing to their great success in unsupervised tasks, generative adversarial networks (GANs) are widely adopted for supervised (conditional) image-generation tasks such as in-painting. Unfortunately, designing an objective function using GANs is not trivial. For supervised tasks, the generator trained only with GAN loss does not yield the corresponding target for the given input because GAN is trained to match the data distribution, not to find the exact answer. Therefore, the loss function of a generator is often formulated by linearly combining supervised loss and GAN loss as similar to multi-task learning, expecting that each loss's weakness is complemented. Contrary to expectation, both losses cause a conflict in practice due to different optimum of each loss, yielding low objective and subjective image qualities. To address this problem, we empirically investigated the conflict caused by using a conventional GAN with pixel-wise losses; we then propose a novel (relativistic) accuracy-aware discriminator. Based on the proposed discriminator, we developed an accuracy-aware GAN (AAGAN) and proved its optimality under an ideal assumption. We then propose a relativistic accuracy-aware GAN (RAAGAN) by considering practical assumptions. Experimental results on supervised tasks demonstrated that the proposed schemes alleviated the competition between losses and outperformed conventional GANs in terms of both objective and subjective qualities.

In recent studies of generative adversarial networks (GAN), researchers have attempted to combine adversarial perturbation with data hiding in order to protect the privacy and authenticity of the host image simultaneously. However, most of the studied approaches can only achieve adversarial perturbation through a visible watermark; the quality of the host image is low, and the concealment of data hiding cannot be achieved. In this work, we propose a true data hiding method with adversarial effect for generating high-quality covers. Based on GAN, the data hiding area is selected precisely by limiting the modification strength in order to preserve the fidelity of the image. We devise a genetic algorithm that can explore decision boundaries in an artificially constrained search space to improve the attack effect as well as construct aggressive covert adversarial samples by detecting “sensitive pixels” in ordinary samples to place discontinuous perturbations. The results reveal that the stego-image has good visual quality and attack effect. To the best of our knowledge, this is the first attempt to use covert data hiding to generate adversarial samples based on GAN.

The effect of loss function on conditional generative adversarial networks

Distributed optical fiber acoustic sensing (DAS) is a new and rapid-developing detection technology in seismic exploration. Unfortunately, due to the weak energy of scattered optical signals and the inferior coupling between DAS cable and receiving interface, the seismic data received by DAS are often characterized by low signal-to-noise ratio (SNR); this low SNR is likely to affect some subsequent analysis, such as inversion, imaging, and interpretation. In addition, the noise caused by the inferior coupling is a new kind of noise not presented on conventional seismic data. To enhance the SNR of DAS seismic data and suppress the DAS noise effectively, we propose a convolutional adversarial denoising network (CADN) based on the basic strategy of generative adversarial network (GAN) and the usage of a denoiser to replace the original generator in GAN. In CADN, the performance of denoiser is significantly strengthened via its own mean square error (MSE) loss and the adversarial loss between it and the discriminator. To balance the two losses and thus ensure the optimization of denoiser, we construct a novel loss function, where the optimal ratio of MSE and adversarial losses is determined by quantifying the denoising performance. Both real and synthetic examples are included to testify the denoising performance of CADN. Experimental results have demonstrated that CADN can suppress most of the DAS noise and enhance the SNR of DAS seismic data; also, it can recover the effective signals completely, even the extremely weak effective signals reflected by deep layers.

In practical application, the research of synthetic aperture radar (SAR) target recognition has fallen into a bottle-neck due to the lack of samples. Data argumentation methods based on generative adversarial networks (GAN) have received widespread attention in solving this type of few-shot sample problem. However, the generated images suffer from various shortcomings, such as lack of diversity, low signal-to-noise ratio, blur, etc. In this article, the VAE-WGANGP is proposed, which combines GAN and variational autoencoder (VAE) to alleviate these shortcomings. The innovations of this paper are as follows: firstly, the generator of GAN is replaced with VAE, which constructs an asymmetric network ensuring the stability of GAN training; secondly, the asymmetric loss function is composed of four parts, including reconstruction loss, divergence loss, adversarial loss, and gradient penalty. In this way, the problem of gradient explosion or gradient disappearance is alleviated. The experimental results with the MSTAR dataset show that the images generated by the proposed model outperform the advanced technology with many similar deep features and achieve significant improvement in the target recognition accuracy rate.

To achieve a higher compression ratio for very low bit rate (VLBR) video compression, motion estimation has to be more reliable and easier to code. Many segmentation based techniques try to approximate several motion vectors by a single global motion vector so that fewer vectors are required. However, a global motion vector may not be a good representation for the original vectors and the bit saving is not worth the loss. This paper proposes a new scheme for motion vector representation. Many candidates from full search block matching are selected for merging possibility. Blocks which experience similar displacement are combined to form a single object. Upon compensation for object translation or global compensation, accurate motion of each block can be identified by local compensation within an object. Instead of presenting object movement as a vector, a specific index is assigned to reduce the overhead bits. Based on extensive simulation, the proposed method with an appropriate threshold shows an improvement over standard block based hybrid schemes.

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

Deep radio signal clustering with interpretability analysis based on saliency map

Frame rate up-conversion is a postprocessing tool to convert the frame rate from a lower number to a higher one. It is a useful technique for a lot of practical applications, such as display format conversion, low bit rate video coding and slow motion playback. Unlike traditional approaches, such as frame repetition or linear frame interpolation, motion-compensated frame interpolation (MCFI) technique is more efficient since it takes block motion into account. In this paper, by considering the deficiencies of previous works, new criteria and coding schemes for improving motion derivation and interpolation processes are proposed. Next, for video coding applications, adaptive frame skip is executed at the encoder side to maximize the power of MCFI so that the quality of interpolated frames is guaranteed. Experimental results show that our proposal effectively enhances the overall quality of the frame rate up-converted video sequence, both subjectively and objectively.