DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting

Nowadays, image inpainting methods based on deep learning would lead to information loss when acquiring deep features, which is not conducive to the restoration of texture details and ignores the inpainting of semantic features. Besides, great majority of them generate inpainting results with unreasonable structures. In response to the above problems, an image inpainting network based on a multi-scale feature joint attention model is proposed. First of all, an image inpainting network using multi-scale feature joint attention model is advanced. When acquiring image depth features, multi-scale feature fusion is used to reduce the loss of information in the convolution process. Afterwards, a joint attention mechanism not only strengthens the model’s ability to repair image semantics, but also ensures the model could generate inpainting results with distinct texture. Last but not least, the style loss and perceptual loss are introduced into the network for the purpose of ensuring the consistency of the detail and style of the inpainting results. The qualitative experimental results on the CelebA-HQ and Places2 datasets and commonly used evaluation indicators such as PSNR and SSIM verify the method is superior to the existing image inpainting methods. Compared with the comparison methods, the proposed method improves PSNR and SSIM by 0.4%−6% and 0.4%−3% respectively.

Traditional image inpainting algorithms aim to obtain satisfactory results for the relatively small and regular missing regions. However, the problem of the image with the large and irregular holes has always been challenging. Our work focus is not on filling holes in the corrupted image, but on the more difficult task of semantic repair, which aims to predict the details of the large regions according to the context of surrounding pixels. In this paper, we propose a perceptual generative adversarial network for image generation and inpainting method. In the training stage, VGG pre-trained model is introduced to GAN network to make the generated image have more high-frequency features. This framework can not only synthesize images with perceptual reality but also make a better prediction of the corrupted images by using the encoding of images. The weighted context loss is adopted to repair the missing regions in image inpainting, and the adversarial loss is used to punish the perceptually unrealistic image. Experiments on natural scene data set CelebA show that our proposed method can produce higher quality repair results.

Recently, learning-based image inpainting has gained much attention. It widely utilizes an auto-encoder structure and can obtain compact feature representation in the encoder to achieve high-quality image inpainting. Although this approach has achieved encouraging inpainting results, it inevitably reduces the high-resolution representation due to interval downsampling. In order to solve this problem and achieve an excellent image inpainting effect, this paper proposes a brand-new generative network, Interactive Separation Network, which retains the high-resolution information and extracts the semantic features from corrupted images. Furthermore, this paper also discusses network designs with different complexity in different application scenarios. Finally, to improve the effectiveness and robustness of our proposal to large corrupted regions in the inpainting image, we further propose a flexible and highly reusable reconstruction scheme to complete the inpainting in the prediction process gradually. Experiments show that our proposed generation network and reconstruction scheme can significantly improve the quality of repaired images. The proposed method significantly outperforms the state-of-the-art image inpainting approaches in image quality.

Recently, deep learning has become a mainstream method of image inpainting. It can not only restore the image texture, obtain high-level abstract features of images, but also restore semantic images such as human face images. Among these methods, generative adversarial networks (GANs) using autoencoder as the generator have become the promising model for image inpainting. These models implement the end-to-end image inpainting and also generate visually reasonable and clear image structures and textures. However, GANs often have problems with gradient vanishing and model collapse during training, so we propose a Generative Adversarial Network with Evolutionary Generators (EG-GAN) and apply it in face image inpainting. To stabilize the model training process, EG-GAN trains the generator network by evolution, combines two mutation functions as a training objective to update the parameter of generator networks, and produces offspring generators through crossover, using the matcher assists the discriminator to criticize the generated image. Experiments on various face image datasets such as CelebA-HQ and CelebA show that EG-GAN successfully overcomes the gradient vanishing problem, achieves stable and efficient training, and generates visually reasonable images.

We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of any pre-trained deep generative network (DGN). Leveraging the fact that DGNs are, or can be ap-proximated by, continuous piecewise affine splines, we derive the analytical DGN output space distribution as a function of the product of the DGN's Jacobian singular values raised to a power p. We dub p the polarity param-eter and prove that p focuses the DGN sampling on the modes (p < 0) or anti-modes (p > 0) of the DGN output-space probability distribution. We demonstrate that nonzero polarity values achieve a better precision-recall (quality-diversity) Pareto frontier than standard methods, such as truncation, for a number of state-of-the-art DGNs. We also present quantitative and qualitative results on the improve-ment of overall generation quality (e.g., in terms of the Fréchet Inception Distance) for a number of state-of-the-art DGNs, including StyleGAN3, BigGAN-deep, NVAE, for different conditional and unconditional image generation tasks. In particular, Polarity Sampling redefines the state-of-the-art for StyleGAN2 on the FFHQ Dataset to FID 2.57, StyleGAN2 on the LSUN Car Dataset to FID 2.27 and Style-GAN3 on the AFHQv2 Dataset to FID 3.95. Colab Demo.

A popular method for training a machine learning model is to use a data-driven approach. This research contributes to expanding the dataset of urban road images without vehicles. Using this method will benefit a variety of existing and new research projects that require an empty road to train their data-driven model. To achieve the desired result, the method combines image segmentation and image inpainting. The model detects the vehicle with Mask RCNN and removes the detected object with image inpainting. Morphological transformation was used to improve the method's efficiency. Using dilation operation of morphological transformation, the mask generated from mask RCNN is enlarged. The results of the experiment support the method's efficacy.

In order to solve the issue mismatching and structure disconnecting in exemplar-based image inpainting, an image completion algorithm based on improved total variation minimization method had been proposed in the paper, refer as ETVM. The structure of image had been extracted using improved total variation minimization method, and the known information of image is sufficiently used by existing methods. The robust filling mechanism can be achieved according to the direction of image structure and it has less noise than original image. The priority term had been redefined to eliminate the product effect and ensure data term had always effective. The priority of repairing patch and the best matching patch are determined by the similarity of the known information and the consistency of the unknown information in the repairing patch. The comparisons with cognitive computing image algorithms had been shown that the proposed method can ensure better selection of candidate image pixel to fill with, and it is achieved better global coherence of image completion than others. The inpainting results of noisy images show that the proposed method has good robustness and can also get good inpainting results for noisy images.

Image inpainting aims to complete missing regions in images, effectively serves imagery processes like historical image restoration or photo editing. This task is challenging because the completion should maintain visual coherence throughout the image. This paper’s contribution lies in an architecture that comprises multiple generators and discriminators to achieve better inpainting results. The two generators work sequentially, in which the first model coarsely reconstructs the missing regions, and the latter completes these regions following the given prior knowledge. Meanwhile, the discriminator stage includes two parallel, global and local branches, allowing for more significant discrimination. We further suggest using dilated convolution, which effectively broadens the receptive field, and WGAN-GP to mitigate gradient vanishing. Both quantitative and qualitative experiments on standard datasets have shown that our method provides more plausible results than current baselines.

Image denoising, a classic ill-posed problem, aims to recover a latent image from a noisy measurement. Over the past few decades, a considerable number of denoising methods have been studied extensively. Among these methods, supervised deep convolutional networks have garnered increasing attention, and their superior performance is attributed to their capability to learn realistic image priors from a large amount of paired noisy and clean images. However, if the image to be denoised is significantly different from the training images, it could lead to inferior results, and the networks may even produce hallucinations by using inappropriate image priors to handle an unseen noisy image. Recently, deep image prior (DIP) was proposed, and it overcame this drawback to some extent. The structure of the DIP generator network is capable of capturing the low-level statistics of a natural image using an unsupervised method with no training images other than the image itself. Compared with a supervised denoising model, the unsupervised DIP is more flexible when processing image content that must be denoised. Nevertheless, the denoising performance of DIP is usually inferior to the current supervised learning-based methods using deep convolutional networks, and it is susceptible to the over-fitting problem. To solve these problems, we propose a novel deep generative network with multiple target images and an adaptive termination condition. Specifically, we utilized mainstream denoising methods to generate two clear target images to be used with the original noisy image, enabling better guidance during the convergence process and improving the convergence speed. Moreover, we adopted the noise level estimation (NLE) technique to set a more reasonable adaptive termination condition, which can effectively solve the problem of over-fitting. Extensive experiments demonstrated that, according to the denoising results, the proposed approach significantly outperforms the original DIP method in tests on different databases. Specifically, the average peak signal-to-noise ratio (PSNR) performance of our proposed method on four databases at different noise levels is increased by 1.90 to 4.86 dB compared to the original DIP method. Moreover, our method achieves superior performance against state-of-the-art methods in terms of popular metrics, which include the structural similarity index (SSIM) and feature similarity index measurement (FSIM). Thus, the proposed method lays a good foundation for subsequent image processing tasks, such as target detection and super-resolution.

History matching is an important part of reservoir numerical simulation research and the basis for predicting oilfield development performance. At present, integrated smoother and ensemble Kalman filter methods are the mainstream methods of history matching. However, these data assimilation algorithms need to be based on Gaussian assumptions and cannot guarantee the geological authenticity of the modified model. Therefore, to achieve end-to-end processing of dynamic data to modify the reservoir model, while ensuring that the distribution of data does not need to be restricted by Gaussian distribution, this paper proposes the use of the deep conditional generative adversarial network (CGAN) method for automatic reservoir history matching for the first time. The mapping relationship between static geological reservoir parameters and dynamic production data is found through the training generative network, which reduces the uncertainty of the reservoir model obtained by automatic history matching and explains parallel offline processing. The numerical simulation results show that the prediction results on the 2D two-phase flow model are consistent with the results of the proposed deep conditional generative adversarial network method.

The mid-infrared spectra of galaxies are rich in features such as the Polycyclic Aromatic Hydrocarbon (PAH) and silicate dust features which give valuable information about the physics of galaxies and their evolution. For example they can provide information about the relative contribution of star formation and accretion from a supermassive black hole to the power output of galaxies. However, the mid-infrared spectra are currently available for a very small fraction of galaxies that have been detected in deep multi-wavelength surveys of the sky. In this paper we explore whether Deep Generative Network methods can be used to reconstruct mid-infrared spectra in the 5–35μm range using the limited multi-wavelength photometry in ∼20 bands from the ultraviolet to the submillimeter which is typically available in extragalactic surveys. For this purpose we use simulated spectra computed with a combination of radiative transfer models for starbursts, active galactic nucleus (AGN) tori and host galaxies. We find that our method using Deep Generative Networks, namely Generative Adversarial Networks and Generative Latent Optimization models, can efficiently produce high quality reconstructions of mid-infrared spectra in ∼ 60% of the cases. We discuss how our method can be improved by using more training data, photometric bands, model parameters or by employing other generative networks.

Image inpainting, which is the repair of pixels in damaged areas of an image to make it look as much like the original image as possible. Deep learning-based image inpainting technology is a prominent area of current research interest. This paper focuses on a systematic and comprehensive study of GAN-based image inpainting and presents an analytical summary. Firstly, this paper introduces GAN, which includes the principle of GAN and its mathematical expression. Secondly, the recent GAN-based image inpainting algorithms are summarized, and the advantages and disadvantages of each algorithm are listed. After that, the evaluation metrics, and common datasets of deep learning-based image inpainting are listed. Finally, the existing image inpainting methods are summarized and summarized, and the ideas for future key research directions are presented and prospected.

All existing image inpainting methods based on neural network models are affected by structural distortions and blurred textures on visible connectivity, such that overfitting and overlearning phenomena can easily emerge in the image inpainting processing procedure. Accordingly, in an attempt to address the defects of image inpainting algorithm, such as long iteration time, poor adaptability and unsatisfactory repairing effects, the image inpainting algorithm of improved Generative Adversarial Networks based on deep learning method of Two-Discriminations Network has been proposed in the paper. The proposed method uses image inpainting network, global discrimination network and local discrimination network to create a fusion network to apply computational images. In the training procedure of proposed algorithm, the network of image inpainting algorithm uses similar patching method to fill the broken area in image and set it as input training objects, which greatly improves the speed and quality of image inpainting. The global discrimination network uses global structure with marginal information and feature information to judge the completed image, meaning that it comprehensively achieves visible connectivity. As local discrimination network can judge the computational images, it has also been trained with assisted feature patches found on multiple images. Furthermore, the proposed method can enhance the discriminant capability and solve the problem that the image inpainting network has easily been overfitting when the features are too concentrated and limited in number to process. Our results of designed experiments demonstrate that proposed algorithm has better adaptive capability on several image categories than those state-of-the-arts.

Estimating the 3D pose of a target object using particle filter has an important problem of high dimensional search space. Because the objects probably appear anywhere in the search space along the camera ray in the 3D world space, a huge number of samples covering the whole search space are required, which necessitates costly expensive computation time and iterations until convergence. For this reason, we propose a particle filter base on back projection sampling on saliency technique. We obtain the object boundaries as foreground regions using saliency segmentation based on color and depth information that is robust to complex environments. Moreover, we apply the particle filter with sampling, which is based on the back projection technique, using the concept of a relationship between 3D world space and the 2D image plane. The sampling dimension of whole samples along the camera ray can be omitted by generating the samples in the 2D image plane on saliencies before they are back projected into 3D world space using depth information. The required number of samples and iterations are drastically decreased. In addition, our method can perceive the salient regions that may be the region of the target object. Most of the samples will be predicted into these promising regions that make the algorithm converges rapidly.

Image inpainting attempts to fill the missing areas of an image with plausible content that is visually coherent with the image context. Semantic image inpainting has remained a challenging task even with the emergence of deep learning-based approaches. We propose a deep semantic inpainting model built upon a generative adversarial network and a dense U-Net network. Such a design helps achieve feature reuse while avoiding feature explosion along the upsampling path of the U-Net. The model also uses a composite loss function for the generator network to enforce a joint global and local content consistency constraint. More specifically, our new loss function combines the global reconstruction loss characterizing the semantic similarity between the missing and known image regions with the local total variation loss characterizing the natural transitions among adjacent regions. Experimental results on CelebA-HQ and Paris StreetView datasets have demonstrated encouraging performance when compared with other state-of-the-art methods in terms of both quantitative and qualitative metrics. For the CelebA-HQ dataset, the proposed method can more faithfully infer the semantics of human faces; for the StreetView dataset, our method achieves improved inpainting results in terms of more natural texture transitions, better structural consistency, and enriched textural details.