Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering.

Image inpainting that completes large free-form missing regions in images is a promising yet challenging task. State-of-the-art approaches have achieved significant progress by taking advantage of generative adversarial networks (GAN). However, these approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512×512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release codes and models in https://github.com/researchmm/AOT-GAN-for-Inpainting.

Most deep learning-based image inpainting approaches adopt autoencoder or its variants to fill missing regions in images. Encoders are usually utilized to learn powerful representational spaces, which are important for dealing with sophisticated learning tasks. Specifically, in image inpainting tasks, masks with any shapes can appear anywhere in images (i.e., free-form masks) which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. To tackle this problem, we propose a self-supervised Siamese inference network to improve the robustness and generalization. It can encode contextual semantics from full-resolution images and obtain more discriminative representations. we further propose a multi-scale decoder with a novel dual attention fusion module (DAF), which can combine both the restored and known regions in a smooth way. This multi-scale architecture is beneficial for decoding discriminative representations learned by encoders into images layer by layer. In this way, unknown regions will be filled naturally from outside to inside.

Image inpainting aims to complete missing regions in images, effectively serves imagery processes like historical image restoration or photo editing. This task is challenging because the completion should maintain visual coherence throughout the image. This paper’s contribution lies in an architecture that comprises multiple generators and discriminators to achieve better inpainting results. The two generators work sequentially, in which the first model coarsely reconstructs the missing regions, and the latter completes these regions following the given prior knowledge. Meanwhile, the discriminator stage includes two parallel, global and local branches, allowing for more significant discrimination. We further suggest using dilated convolution, which effectively broadens the receptive field, and WGAN-GP to mitigate gradient vanishing. Both quantitative and qualitative experiments on standard datasets have shown that our method provides more plausible results than current baselines.

In this paper, we tackle the problem of image in painting which aims at removing objects from an image or repairing damaged pictures by replacing the missing regions using the information in the rest of the scene. The image in painting method proposed here builds on an exemplar-based perspective so as to improve the local consistency of the in painted region. This is done by selecting the optimal patch which maximises the local consistency with respect to abutting candidate patches. The similarity computation generates weights based upon an edge prior and the structural differences between in painting exemplar candidates. This treatment permits the generation of an in painting sequence based on a list of factors. The experiments show that the proposed method delivers a margin of improvement as compared to alternative methods.

Image inpainting is an important problem of image processing that has many applications. The goal of the image inpainting problem is to restore or fill the corrupted or missing regions of image. In this paper, we propose an image inpainting method based on the adaptive fuzzy switching median. The adaptive fuzzy switching median is to provide an accurate estimation for the values of corrupted pixels when there are many corrupted pixels on image. In the experiments, we implement the proposed method on an open dataset of real natural images. We utilize standard image quality assessment metrics such as the peak signal-to-noise ratio metric and the structured similarity metric to compare the inpainting result of the proposed method with other similar inpainting methods to prove its effectiveness. The proposed inpainting method is also extended to process colorful images.

We study the robust estimation of missing regions in images and video using adaptive, sparse reconstructions. Our primary application is on missing regions of pixels containing textures, edges, and other image features that are not readily handled by prevalent estimation and recovery algorithms. We assume that we are given a linear transform that is expected to provide sparse decompositions over missing regions such that a portion of the transform coefficients over missing regions are zero or close to zero. We adaptively determine these small magnitude coefficients through thresholding, establish sparsity constraints, and estimate missing regions in images using information surrounding these regions. Unlike prevalent algorithms, our approach does not necessitate any complex preconditioning, segmentation, or edge detection steps, and it can be written as a sequence of denoising operations. We show that the region types we can effectively estimate in a mean-squared error sense are those for which the given transform provides a close approximation using sparse nonlinear approximants. We show the nature of the constructed estimators and how these estimators relate to the utilized transform and its sparsity over regions of interest. The developed estimation framework is general, and can readily be applied to other nonstationary signals with a suitable choice of linear transforms. Part I discusses fundamental issues, and Part II is devoted to adaptive algorithms with extensive simulation examples that demonstrate the power of the proposed techniques.

Inpainting high-resolution images with large holes challenges existing deep learning-based image inpainting methods. We present a novel framework—PyramidFill for high-resolution image inpainting, which explicitly disentangles the task into two sub-tasks: content completion and texture synthesis. PyramidFill attempts to complete the content of unknown regions in a lower-resolution image, and synthesize the textures of unknown regions in a higher-resolution image, progressively. Thus, our model consists of a pyramid of fully convolutional GANs, wherein the content GAN is responsible for completing contents in the lowest-resolution masked image, and each texture GAN is responsible for synthesizing textures in a higher-resolution image. Since completing contents and synthesizing textures demand different abilities from generators, we customize different architectures for the content GAN and texture GAN. Experiments on multiple datasets including CelebA-HQ, Places2 and a new natural scenery dataset (NSHQ) with different resolutions demonstrate that PyramidFill generates higher-quality inpainting results than the state-of-the-art methods.

In this paper, we propose an efficient exemplar-based inpainting algorithm via sparse modeling and local similarity constraint. The inpainting procedure contains two steps: calculating the filling order and reconstructing the target patch. The filling order is decided by patch priority, which privileges the patch located at edge or corner. The target patch is then estimated by a combination of candidate patches. In the proposed method, three regularization terms are introduced to improve the patch reconstruction step. The first term ensures the compatibility between the target patch and the estimated one. The second term assigns larger combination coefficients for the candidate patches which are most similar with the target patch. The third term penalties the combination coefficients for the outliers in the candidate patches. Finally, the three regularization terms are incorporated into a unified sparse representation framework for reconstructing the target patch. Experiments show that the proposed algorithm can effectively fill in missing pixels in a visually plausible way.

We propose a low rank structured matrix completion algorithm for image inpainting problems originated from scanning microscopy. The proposed method exploits the annihilation property observed in Gaussian Markov Random Field (GMRF) or partial differential equation (PDE)-based inpainting approaches. By utilizing the commutative property of the convolution, the annihilation property is embodied into rank-deficient block Hankel structure data matrices and the image inpainting problem is converted into low-rank structured matrix completion problem. To solve the structured low-rank matrix completion problem, an alternating direction method of multiplier (ADMM) method is used with factorization matrix initialization using the low rank matrix fitting (LMaFit) algorithm. Experimental results showed that the proposed method outperforms the existing state-of-the-art image inpainting methods.

Restoring missing areas without leaving visible traces has become a trivial task with Photoshop inpainting tools. However, such tools have potentially illegal or unethical uses, such as removing specific objects in images to deceive the public. Despite the emergence of many forensics methods of image inpainting, their detection ability is still insufficient when attending to professional Photoshop inpainting. Motivated by this, we propose a novel method termed primary-secondary network (PS-Net) to localize the Photoshop inpainted regions in images. To the best of our knowledge, this is the first forensic method devoted specifically to Photoshop inpainting. The PS-Net is designed to deal with the problems of delicate and professional inpainted images. It consists of two subnetworks: the primary network (P-Net) and the secondary network (S-Net). The P-Net aims at mining the frequency clues of subtle inpainting features through the convolutional network and further identifying the tampered region. The S-Net enables the model to mitigate compression and noise attacks to some extent by increasing the co-occurring feature weights and providing features that are not captured by the P-Net. Furthermore, the dense connection, Ghost modules, and channel attention blocks (C-A blocks) are adopted to further strengthen the localization ability of PS-Net. Extensive experimental results illustrate that PS-Net can successfully distinguish forged regions in elaborate inpainted images, outperforming several state-of-the-art solutions. The proposed PS-Net is also robust against some postprocessing operations commonly used in Photoshop.

Multi-focus image fusion aims to integrate clear segments from different partially focused images, creating an ‘all-in-focus’ composite. Due to the lack of ground-truth for multi-focus image fusion, supervised deep learning methods are deemed inappropriate for this task. In this paper, we present an unsupervised approach for multi-focus image fusion, named Fusion2Void. Fusion2Void ingeniously tackles the challenge of missing ground-truth by framing image inpainting as an auxiliary task. Specifically, Fusion2Void utilizes a fusion network to merge focused regions from multiple source images. Following the fusion process, image patches in the source images are randomly dropped to construct an additional image inpainting task. Subsequently, an image inpainting network uses the fused image as a guide to restore the missing content in the source images. The missing content in the source images includes both focused and defocused regions. Restoring focused image patches is significantly more challenging than restoring their defocused counterparts due to their inclusion of more high-frequency details. If the focused image patches are effectively restored, the repair of the defocused image patches becomes notably easier. Therefore, the image inpainting network implicitly compels the fused image to incorporate all focused content from the source images, as these can be utilized to restore the missing focused regions in the source images perfectly. Based on image inpainting, the fusion network generates ‘all-in-focus’ images in an unsupervised manner. Experiments on several synthetic and real-world datasets highlight Fusion2Void’s state-of-the-art performance relative to other methods.

Recently, learning-based image inpainting has gained much attention. It widely utilizes an auto-encoder structure and can obtain compact feature representation in the encoder to achieve high-quality image inpainting. Although this approach has achieved encouraging inpainting results, it inevitably reduces the high-resolution representation due to interval downsampling. In order to solve this problem and achieve an excellent image inpainting effect, this paper proposes a brand-new generative network, Interactive Separation Network, which retains the high-resolution information and extracts the semantic features from corrupted images. Furthermore, this paper also discusses network designs with different complexity in different application scenarios. Finally, to improve the effectiveness and robustness of our proposal to large corrupted regions in the inpainting image, we further propose a flexible and highly reusable reconstruction scheme to complete the inpainting in the prediction process gradually. Experiments show that our proposed generation network and reconstruction scheme can significantly improve the quality of repaired images. The proposed method significantly outperforms the state-of-the-art image inpainting approaches in image quality.

With the development of digital images processing, image inpainting is become one of the most impressive and useful technique. Based on partial derivative equations or texture synthesis, many image inpainting techniques have been proposed. Amano et al. proposed a method that based on eigenspace analyzes, which is called Back Projection for Lost Pixels (BPLP). It obtains the eigenspace from a set of learning samples from original image, then estimating the missing region by inverse projection and a linear combination of the eigenspace. But it remains have some obvious discomfort surrounding the damage region after restoration treatment. In this paper, we have proposed an adaptation of BPLP algorithms, which are performing well on images with natural defects.

This paper presents a sparse representation based image inpainting method using local patch analysis and geometric structure based feature extraction. In local patch analysis, we approximate the target region by weighted average of some local patches which are frequently occurred within a neighborhood. Local patch statistics is applied to find the most relevant neighbors for each target patch. Further we extract local steering kernel (LSK) based feature to preserve geometric structure and texture sharpness in the target region. The advantage of non local self similarity as redundancy of similar patches in natural images is introduced to find the candidate patches from the whole source region. Based on these local and non local prior information we propose a sparse representation framework for image inpainting. Our proposed method is tested on wide range of natural images. The experimental results show the superiority of the proposed method compared to some of the previous approaches.

Image inpainting is a challenging computer vision task that aims to fill in missing regions of corrupted images with realistic contents. With the development of convolutional neural networks, many deep learning models have been proposed to solve image inpainting issues by learning information from a large amount of data. In particular, existing algorithms usually follow an encoding and decoding network architecture in which some operations with standard schemes are employed, such as static convolution, which only considers pixels with fixed grids, and the monotonous normalization style (e.g., batch normalization). However, these techniques are not well-suited for the image inpainting task because the random corrupted regions in the input images tend to mislead the inpainting process and generate unreasonable content. In this paper, we propose a novel dynamic selection network (DSNet) to solve this problem in image inpainting tasks. The principal idea of the proposed DSNet is to distinguish the corrupted region from the valid ones throughout the entire network architecture, which may help make full use of the information in the known area. Specifically, the proposed DSNet has two novel dynamic selection modules, namely, the validness migratable convolution (VMC) and regional composite normalization (RCN) modules, which share a dynamic selection mechanism that helps utilize valid pixels better. By replacing vanilla convolution with the VMC module, spatial sampling locations are dynamically selected in the convolution phase, resulting in a more flexible feature extraction process. Besides, the RCN module not only combines several normalization methods but also normalizes the feature regions selectively. Therefore, the proposed DSNet can illustrate realistic and fine-detailed images by adaptively selecting features and normalization styles. Experimental results on three public datasets show that our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively.