Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Traditional image inpainting algorithms aim to obtain satisfactory results for the relatively small and regular missing regions. However, the problem of the image with the large and irregular holes has always been challenging. Our work focus is not on filling holes in the corrupted image, but on the more difficult task of semantic repair, which aims to predict the details of the large regions according to the context of surrounding pixels. In this paper, we propose a perceptual generative adversarial network for image generation and inpainting method. In the training stage, VGG pre-trained model is introduced to GAN network to make the generated image have more high-frequency features. This framework can not only synthesize images with perceptual reality but also make a better prediction of the corrupted images by using the encoding of images. The weighted context loss is adopted to repair the missing regions in image inpainting, and the adversarial loss is used to punish the perceptually unrealistic image. Experiments on natural scene data set CelebA show that our proposed method can produce higher quality repair results.

Recently, deep learning has become a mainstream method of image inpainting. It can not only restore the image texture, obtain high-level abstract features of images, but also restore semantic images such as human face images. Among these methods, generative adversarial networks (GANs) using autoencoder as the generator have become the promising model for image inpainting. These models implement the end-to-end image inpainting and also generate visually reasonable and clear image structures and textures. However, GANs often have problems with gradient vanishing and model collapse during training, so we propose a Generative Adversarial Network with Evolutionary Generators (EG-GAN) and apply it in face image inpainting. To stabilize the model training process, EG-GAN trains the generator network by evolution, combines two mutation functions as a training objective to update the parameter of generator networks, and produces offspring generators through crossover, using the matcher assists the discriminator to criticize the generated image. Experiments on various face image datasets such as CelebA-HQ and CelebA show that EG-GAN successfully overcomes the gradient vanishing problem, achieves stable and efficient training, and generates visually reasonable images.

Image inpainting, a pivotal technology for restoring damaged regions of images, has emerged as a significant research focus in computer vision. This review systematically surveys recent advances in deep learning-based image inpainting. We begin by categorizing prevailing methods into three groups based on their core architectures: Convolutional Neural Networks (CNNs), Generative Models, and Transformers. Through a comparative analysis of their symmetric versus asymmetric network architectures, applicable scenarios, and performance bottlenecks, we provide a critical discussion of the strengths and limitations inherent to each approach. The evolution of underlying design principles, such as symmetry, and the corresponding solutions to core challenges are also discussed. Furthermore, we introduce key benchmark datasets and commonly used image quality assessment metrics, offering a multidimensional framework for evaluation. We highlight that mainstream datasets collectively foster a greenhouse-like evaluation environment detached from real-world complexities and that existing metrics are critically misaligned with the fundamental objective of inpainting: generating plausible new content. Finally, we summarize the prevailing challenges in current deep learning-based inpainting research and outline promising future directions. We highlight critical issues, such as enhancing restoration quality, reducing computational costs, and broadening application scenarios, thereby providing valuable insights for subsequent research.

Multi-focus image fusion aims to integrate clear segments from different partially focused images, creating an ‘all-in-focus’ composite. Due to the lack of ground-truth for multi-focus image fusion, supervised deep learning methods are deemed inappropriate for this task. In this paper, we present an unsupervised approach for multi-focus image fusion, named Fusion2Void. Fusion2Void ingeniously tackles the challenge of missing ground-truth by framing image inpainting as an auxiliary task. Specifically, Fusion2Void utilizes a fusion network to merge focused regions from multiple source images. Following the fusion process, image patches in the source images are randomly dropped to construct an additional image inpainting task. Subsequently, an image inpainting network uses the fused image as a guide to restore the missing content in the source images. The missing content in the source images includes both focused and defocused regions. Restoring focused image patches is significantly more challenging than restoring their defocused counterparts due to their inclusion of more high-frequency details. If the focused image patches are effectively restored, the repair of the defocused image patches becomes notably easier. Therefore, the image inpainting network implicitly compels the fused image to incorporate all focused content from the source images, as these can be utilized to restore the missing focused regions in the source images perfectly. Based on image inpainting, the fusion network generates ‘all-in-focus’ images in an unsupervised manner. Experiments on several synthetic and real-world datasets highlight Fusion2Void’s state-of-the-art performance relative to other methods.

Image inpainting attempts to fill the missing areas of an image with plausible content that is visually coherent with the image context. Semantic image inpainting has remained a challenging task even with the emergence of deep learning-based approaches. We propose a deep semantic inpainting model built upon a generative adversarial network and a dense U-Net network. Such a design helps achieve feature reuse while avoiding feature explosion along the upsampling path of the U-Net. The model also uses a composite loss function for the generator network to enforce a joint global and local content consistency constraint. More specifically, our new loss function combines the global reconstruction loss characterizing the semantic similarity between the missing and known image regions with the local total variation loss characterizing the natural transitions among adjacent regions. Experimental results on CelebA-HQ and Paris StreetView datasets have demonstrated encouraging performance when compared with other state-of-the-art methods in terms of both quantitative and qualitative metrics. For the CelebA-HQ dataset, the proposed method can more faithfully infer the semantics of human faces; for the StreetView dataset, our method achieves improved inpainting results in terms of more natural texture transitions, better structural consistency, and enriched textural details.

Image inpainting is a challenging computer vision task that aims to fill in missing regions of corrupted images with realistic contents. With the development of convolutional neural networks, many deep learning models have been proposed to solve image inpainting issues by learning information from a large amount of data. In particular, existing algorithms usually follow an encoding and decoding network architecture in which some operations with standard schemes are employed, such as static convolution, which only considers pixels with fixed grids, and the monotonous normalization style (e.g., batch normalization). However, these techniques are not well-suited for the image inpainting task because the random corrupted regions in the input images tend to mislead the inpainting process and generate unreasonable content. In this paper, we propose a novel dynamic selection network (DSNet) to solve this problem in image inpainting tasks. The principal idea of the proposed DSNet is to distinguish the corrupted region from the valid ones throughout the entire network architecture, which may help make full use of the information in the known area. Specifically, the proposed DSNet has two novel dynamic selection modules, namely, the validness migratable convolution (VMC) and regional composite normalization (RCN) modules, which share a dynamic selection mechanism that helps utilize valid pixels better. By replacing vanilla convolution with the VMC module, spatial sampling locations are dynamically selected in the convolution phase, resulting in a more flexible feature extraction process. Besides, the RCN module not only combines several normalization methods but also normalizes the feature regions selectively. Therefore, the proposed DSNet can illustrate realistic and fine-detailed images by adaptively selecting features and normalization styles. Experimental results on three public datasets show that our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively.

Nonlocal texture similarity and local intensity smoothness are both essential for solving most image inpainting problems. In this paper, we propose a novel image inpainting algorithm that is capable of reproducing the underlying textural details using a nonlocal texture measure and also smoothing pixel intensity seamlessly in order to achieve natural-looking inpainted images. For matching texture, we propose a Gaussian-weighted nonlocal texture similarity measure to obtain multiple candidate patches for each target patch. To compute the pixel intensity, we apply the -trimmed mean filter to the candidate patches to inpaint the target patch pixel-by-pixel. The proposed algorithm is compared with four current image inpainting algorithms under different scenarios, including object removal, texture synthesis, and error concealment. Experimental results show that the proposed algorithm outperforms the existing algorithms when inpainting large missing regions in images with texture and geometric structures.

Most deep learning-based image inpainting approaches adopt autoencoder or its variants to fill missing regions in images. Encoders are usually utilized to learn powerful representational spaces, which are important for dealing with sophisticated learning tasks. Specifically, in image inpainting tasks, masks with any shapes can appear anywhere in images (i.e., free-form masks) which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. To tackle this problem, we propose a self-supervised Siamese inference network to improve the robustness and generalization. It can encode contextual semantics from full-resolution images and obtain more discriminative representations. we further propose a multi-scale decoder with a novel dual attention fusion module (DAF), which can combine both the restored and known regions in a smooth way. This multi-scale architecture is beneficial for decoding discriminative representations learned by encoders into images layer by layer. In this way, unknown regions will be filled naturally from outside to inside.

Restoring missing areas without leaving visible traces has become a trivial task with Photoshop inpainting tools. However, such tools have potentially illegal or unethical uses, such as removing specific objects in images to deceive the public. Despite the emergence of many forensics methods of image inpainting, their detection ability is still insufficient when attending to professional Photoshop inpainting. Motivated by this, we propose a novel method termed primary-secondary network (PS-Net) to localize the Photoshop inpainted regions in images. To the best of our knowledge, this is the first forensic method devoted specifically to Photoshop inpainting. The PS-Net is designed to deal with the problems of delicate and professional inpainted images. It consists of two subnetworks: the primary network (P-Net) and the secondary network (S-Net). The P-Net aims at mining the frequency clues of subtle inpainting features through the convolutional network and further identifying the tampered region. The S-Net enables the model to mitigate compression and noise attacks to some extent by increasing the co-occurring feature weights and providing features that are not captured by the P-Net. Furthermore, the dense connection, Ghost modules, and channel attention blocks (C-A blocks) are adopted to further strengthen the localization ability of PS-Net. Extensive experimental results illustrate that PS-Net can successfully distinguish forged regions in elaborate inpainted images, outperforming several state-of-the-art solutions. The proposed PS-Net is also robust against some postprocessing operations commonly used in Photoshop.

The degree of difficulty in image inpainting depends on the types and sizes of the missing parts. Existing image inpainting approaches usually encounter difficulties in completing the missing parts in the wild with pleasing visual and contextual results as they are trained for either dealing with one specific type of missing patterns (mask) or unilaterally assuming the shapes and/or sizes of the masked areas. We propose a deep generative inpainting network, named DeepGIN, to handle various types of masked images. We design a Spatial Pyramid Dilation (SPD) ResNet block to enable the use of distant features for reconstruction. We also employ Multi-Scale Self-Attention (MSSA) mechanism and Back Projection (BP) technique to enhance our inpainting results. Our DeepGIN outperforms the state-of-the-art approaches generally, including two publicly available datasets (FFHQ and Oxford Buildings), both quantitatively and qualitatively. We also demonstrate that our model is capable of completing masked images in the wild.

Super-resolution reconstruction of pneumocystis carinii pneumonia images based on generative confrontation network

In this paper, we propose a method that removes raindrops with light field image using image inpainting. We first use the depth map generated from light field image to detect raindrop regions which are then expressed as a binary mask. The original image with raindrops is improved by refocusing on the far regions and filtering by a high-pass filter. With the binary mask and the enhanced image, image inpainting is then utilized to eliminate raindrops from the original image. We compare pre-trained models of several deep learning based image inpainting methods. A light field raindrop dataset is released to verify our method. Image quality analysis is performed to evaluate the proposed image restoration method. The recovered images are further applied to object detection and visual localization tasks.

Image inpainting is an important problem of image processing that has many applications. The goal of the image inpainting problem is to restore or fill the corrupted or missing regions of image. In this paper, we propose an image inpainting method based on the adaptive fuzzy switching median. The adaptive fuzzy switching median is to provide an accurate estimation for the values of corrupted pixels when there are many corrupted pixels on image. In the experiments, we implement the proposed method on an open dataset of real natural images. We utilize standard image quality assessment metrics such as the peak signal-to-noise ratio metric and the structured similarity metric to compare the inpainting result of the proposed method with other similar inpainting methods to prove its effectiveness. The proposed inpainting method is also extended to process colorful images.

Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, due to two stacked generative networks, the coarse-to-fine network needs numerous computational resources, such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called parallel extended-decoder path for semantic inpainting (PEPSI) network, which aims at reducing the hardware costs and improving the inpainting performance. PEPSI consists of a single shared encoding network and parallel decoding networks called coarse and inpainting paths. The coarse path produces a preliminary inpainting result to train the encoding network for the prediction of features for the CAM. Simultaneously, the inpainting path generates higher inpainting quality using the refined features reconstructed via the CAM. In addition, we propose Diet-PEPSI that significantly reduces the network parameters while maintaining the performance. In Diet-PEPSI, to capture the global contextual information with low hardware costs, we propose novel rate-adaptive dilated convolutional layers that employ the common weights but produce dynamic features depending on the given dilation rates. Extensive experiments comparing the performance with state-of-the-art image inpainting methods demonstrate that both PEPSI and Diet-PEPSI improve the qualitative scores, i.e., the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as significantly reduce hardware costs, such as computational time and the number of network parameters.

Computer-aided diagnosis of prostate ultrasound images is gradually being combined with deep learning to help detect and treat diseases. However, ultrasound images of the prostate have problems such as low resolution and unbalanced categories. In addition, the current image classification algorithms have difficulty with classification network performance due to insufficient data volume. To solve these problems, inspired by CycleGAN, we propose an enhanced multiscale generation and depth-perceptual loss-based super-resolution (SR) network for prostate ultrasound images (EGDL-CycleGAN). We study and improve the generative network and perceptual loss of CycleGAN. In this work we achieve multiscale feature extraction through an improved generator, and utilize full-scale skip connections between encoder and decoder to capture fine-grained details and coarse-grained semantics at full scale. This effectively improves the performance of the generative network and makes the reconstruction effect better. We also use the residual structure for deep extraction of features to obtain perceptual loss, and add this to the network loss function for training the model. This enables the model to learn the global and local differences between the real and generated images. This approach pays more attention to the edge information and spatial information of the image, and provides relevant spatial information feedback to the generator to improve the generator’s ability to perceive consistent super-resolution. The method can enhance the prostate ultrasound image dataset and provide rich images for the next step in intelligence-assisted classification and diagnosis of prostate cancer ultrasound images. The evaluation of peak signal-to-noise ratio/structural similarity and visual effects against the benchmark of our datasets illustrates that our proposed approach is effective and superior to the bicubic classic image SR reconstruction algorithm, the SRGAN perception-driven method and the CycleGAN method applied to ultrasound images. In addition, the method of using the original dataset combined with the SR reconstruction image dataset can effectively improve the accuracy of the classification network in intelligence-assisted classification diagnosis of prostate cancer ultrasound images. In EfficientNetV2 the accuracy is improved from 0.843 to 0.867 and in Swin Transformer the accuracy is improved from 0.893 to 0.917.