Aggregated Contextual Transformations for High-Resolution Image Inpainting.

Nonlocal texture similarity and local intensity smoothness are both essential for solving most image inpainting problems. In this paper, we propose a novel image inpainting algorithm that is capable of reproducing the underlying textural details using a nonlocal texture measure and also smoothing pixel intensity seamlessly in order to achieve natural-looking inpainted images. For matching texture, we propose a Gaussian-weighted nonlocal texture similarity measure to obtain multiple candidate patches for each target patch. To compute the pixel intensity, we apply the -trimmed mean filter to the candidate patches to inpaint the target patch pixel-by-pixel. The proposed algorithm is compared with four current image inpainting algorithms under different scenarios, including object removal, texture synthesis, and error concealment. Experimental results show that the proposed algorithm outperforms the existing algorithms when inpainting large missing regions in images with texture and geometric structures.

We study the robust estimation of missing regions in images and video using adaptive, sparse reconstructions. Our primary application is on missing regions of pixels containing textures, edges, and other image features that are not readily handled by prevalent estimation and recovery algorithms. We assume that we are given a linear transform that is expected to provide sparse decompositions over missing regions such that a portion of the transform coefficients over missing regions are zero or close to zero. We adaptively determine these small magnitude coefficients through thresholding, establish sparsity constraints, and estimate missing regions in images using information surrounding these regions. Unlike prevalent algorithms, our approach does not necessitate any complex preconditioning, segmentation, or edge detection steps, and it can be written as a sequence of denoising operations. We show that the region types we can effectively estimate in a mean-squared error sense are those for which the given transform provides a close approximation using sparse nonlinear approximants. We show the nature of the constructed estimators and how these estimators relate to the utilized transform and its sparsity over regions of interest. The developed estimation framework is general, and can readily be applied to other nonstationary signals with a suitable choice of linear transforms. Part I discusses fundamental issues, and Part II is devoted to adaptive algorithms with extensive simulation examples that demonstrate the power of the proposed techniques.

Image inpainting aims to complete missing regions in images, effectively serves imagery processes like historical image restoration or photo editing. This task is challenging because the completion should maintain visual coherence throughout the image. This paper’s contribution lies in an architecture that comprises multiple generators and discriminators to achieve better inpainting results. The two generators work sequentially, in which the first model coarsely reconstructs the missing regions, and the latter completes these regions following the given prior knowledge. Meanwhile, the discriminator stage includes two parallel, global and local branches, allowing for more significant discrimination. We further suggest using dilated convolution, which effectively broadens the receptive field, and WGAN-GP to mitigate gradient vanishing. Both quantitative and qualitative experiments on standard datasets have shown that our method provides more plausible results than current baselines.

Most deep learning-based image inpainting approaches adopt autoencoder or its variants to fill missing regions in images. Encoders are usually utilized to learn powerful representational spaces, which are important for dealing with sophisticated learning tasks. Specifically, in image inpainting tasks, masks with any shapes can appear anywhere in images (i.e., free-form masks) which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. To tackle this problem, we propose a self-supervised Siamese inference network to improve the robustness and generalization. It can encode contextual semantics from full-resolution images and obtain more discriminative representations. we further propose a multi-scale decoder with a novel dual attention fusion module (DAF), which can combine both the restored and known regions in a smooth way. This multi-scale architecture is beneficial for decoding discriminative representations learned by encoders into images layer by layer. In this way, unknown regions will be filled naturally from outside to inside.

Based on a generative adversarial network (GAN), a novel version named Task-Oriented GAN is proposed to tackle difficulties in PolSAR image interpretation, including PolSAR data analysis and small sample problem. Besides two typical parts in GAN, i.e., generator (G-Net) and discriminator (D-Net), there is a third part named TaskNet (T-Net) in the Task-Oriented GAN, where T-Net is employed to accomplish a certain task. Two tasks, PolSAR image classification and clustering, are studied in this paper, where T-Net acts as a Classifier and a Clusterer, respectively. The learning procedure of Task-Oriented GAN consists of two main stages. In the first stage, G-Net and D-Net vie with each other like that in a general GAN; in the second stage, G-Net is adjusted and oriented by T-Net so that more samples, which are benefit for the task and called fake data, are generated. As a result, Task-Oriented GAN not only has the advantage of GAN (no-assumption data modeling) but also overcomes the disadvantage of GAN (task-free). After learning, fake data are employed to enrich training set and avoid overfitting; so Task-Oriented GAN performs well even if the manual-labeled data are small. To verify the effectiveness of T-Net, a visualized comparison is provided, where some fake digits generated from Task-Oriented GAN are illustrated along with that from GAN. What is more, considering that there is a great difference between PolSAR data and general data, in our PolSAR image classification and clustering tasks, the specific PolSAR information is inserted into the structure of the Task-Oriented GAN. This enables researchers to mine inherent information in PolSAR data without any data hypothesis and find ways for small sample problem at the same time. Experiment results tested on three PolSAR images show that the proposed method performs well in dealing with PolSAR image classification and clustering.

Inpainting high-resolution images with large holes challenges existing deep learning-based image inpainting methods. We present a novel framework—PyramidFill for high-resolution image inpainting, which explicitly disentangles the task into two sub-tasks: content completion and texture synthesis. PyramidFill attempts to complete the content of unknown regions in a lower-resolution image, and synthesize the textures of unknown regions in a higher-resolution image, progressively. Thus, our model consists of a pyramid of fully convolutional GANs, wherein the content GAN is responsible for completing contents in the lowest-resolution masked image, and each texture GAN is responsible for synthesizing textures in a higher-resolution image. Since completing contents and synthesizing textures demand different abilities from generators, we customize different architectures for the content GAN and texture GAN. Experiments on multiple datasets including CelebA-HQ, Places2 and a new natural scenery dataset (NSHQ) with different resolutions demonstrate that PyramidFill generates higher-quality inpainting results than the state-of-the-art methods.

Video summarization based on generative adversarial networks (GANs) has been shown to easily produce more realistic results. However, most summary videos are composed of multiple key components. If the selection of some video frames changes during the training process, the information carried by these frames may not be reasonably reflected in the identification results. In this paper, we propose a video summarization method based on selecting keyframes over GANs. The novelty of the proposed method is the discriminator not only identifies the completeness of the video, but also takes into account the value judgment of the candidate keyframes, thus enabling the influence of keyframes on the result value. Given GANs are mainly designed to generate continuous real values, it is generally challenging to generate discrete symbol sequences during the summarization process directly. However, if the generated sample is based on discrete symbols, the slight guidance change of the discrimination network may be meaningless. To better use the advantages of GANs, the study also adopts the video summarization optimization method of GANs under a collaborative reinforcement learning strategy. Experimental results show the proposed method gets a significant summarization effect and character compared with the existing cutting-edge methods.

Image inpainting is an important problem of image processing that has many applications. The goal of the image inpainting problem is to restore or fill the corrupted or missing regions of image. In this paper, we propose an image inpainting method based on the adaptive fuzzy switching median. The adaptive fuzzy switching median is to provide an accurate estimation for the values of corrupted pixels when there are many corrupted pixels on image. In the experiments, we implement the proposed method on an open dataset of real natural images. We utilize standard image quality assessment metrics such as the peak signal-to-noise ratio metric and the structured similarity metric to compare the inpainting result of the proposed method with other similar inpainting methods to prove its effectiveness. The proposed inpainting method is also extended to process colorful images.

Most zero-shot learning (ZSL) methods aim to learn a mapping from visual feature space to semantic feature space or from both visual and semantic feature spaces to a common joint space and align them. However, in these methods the visual and semantic information are not utilized sufficiently and the useless information is not excluded. Moreover, there exists a strong bias problem that the instances from unseen classes always tend to be predicted as some seen classes in most ZSL methods. In this paper, combining the advantages of generative adversarial networks (GANs), a method based on bidirectional projections between the visual and semantic feature spaces is proposed. GANs are used to perform bidirectional generations and alignments between the visual and semantic features. In addition, cycle mapping structure ensures that the important information are kept in the alignments. Furthermore, in order to better solve the bias problem, pseudo-labels are generated for unseen instances and the model is adjusted along with them iteratively. We conduct extensive experiments at traditional ZSL and generalized ZSL settings, respectively. Experiment results confirm that our method achieves the state-of-the-art performances on the popular datasets AWA2, aPY and SUN.

BackgroundFor the growing patient population with congenital heart disease (CHD), improving clinical workflow, accuracy of diagnosis, and efficiency of analyses are considered unmet clinical needs. Cardiovascular magnetic resonance (CMR) imaging offers non-invasive and non-ionizing assessment of CHD patients. However, although CMR data facilitates reliable analysis of cardiac function and anatomy, clinical workflow mostly relies on manual analysis of CMR images, which is time consuming. Thus, an automated and accurate segmentation platform exclusively dedicated to pediatric CMR images can significantly improve the clinical workflow, as the present work aims to establish.MethodsTraining artificial intelligence (AI) algorithms for CMR analysis requires large annotated datasets, which are not readily available for pediatric subjects and particularly in CHD patients. To mitigate this issue, we devised a novel method that uses a generative adversarial network (GAN) to synthetically augment the training dataset via generating synthetic CMR images and their corresponding chamber segmentations. In addition, we trained and validated a deep fully convolutional network (FCN) on a dataset, consisting of 64 pediatric subjects with complex CHD, which we made publicly available. Dice metric, Jaccard index and Hausdorff distance as well as clinically-relevant volumetric indices are reported to assess and compare our platform with other algorithms including U-Net and cvi42, which is used in clinics.ResultsFor congenital CMR dataset, our FCN model yields an average Dice metric of 91.0mathrm{%} and 86.8mathrm{%} for LV at end-diastole and end-systole, respectively, and 84.7mathrm{%} and 80.6mathrm{%} for RV at end-diastole and end-systole, respectively. Using the same dataset, the cvi42, resulted in 73.2mathrm{%}, 71.0mathrm{%}, 54.3mathrm{%} and 53.7mathrm{%} for LV and RV at end-diastole and end-systole, and the U-Net architecture resulted in 87.4mathrm{%}, 83.9mathrm{%}, 81.8mathrm{%} and 74.8mathrm{%} for LV and RV at end-diastole and end-systole, respectively.ConclusionsThe chambers’ segmentation results from our fully-automated method showed strong agreement with manual segmentation and no significant statistical difference was found by two independent statistical analyses. Whereas cvi42 and U-Net segmentation results failed to pass the t-test. Relying on these outcomes, it can be inferred that by taking advantage of GANs, our method is clinically relevant and can be used for pediatric and congenital CMR segmentation and analysis.

As the basis of image processing, single image deraining has always been a significant and challenging issue. Due to the lack of real rainy images and corresponding clean images, most deraining networks are trained by synthetic datasets, which makes the output images unsatisfactory in real applications. Besides, note that a heavy rainfall is typically accompanied with some fog. Although some deraining networks have been proposed to remove the rain streaks in the rainy images, the output images may still be blurred due to the accompanied fog. In this paper, these problems existing in single image deraining is comprehensively considered, and propose a Cycle‐Derain network based on an unsupervised attention‐guided mechanism. Specifically, the Cycle‐Derain network takes advantage of generative adversarial networks with two mappings and the cycle consistency loss to train both unpaired rainy images and rain‐free images. Moreover, it introduces an unsupervised attention‐guided mechanism and exploits the loop‐search positioning algorithm to deal with the details of rain and fog in images. Extensive experiments have been carried out, and the results show that the proposed Cycle‐Derain network is preferable compared with other deraining networks, especially in term of rainy image restoration.

Addressing inpainting challenges in high-resolution images remains a complex task. The most recent image inpainting techniques rely on machine learning models; however, a major limitation of supervised methods is their dependence on end-to-end training. Even minor changes to the input often necessitate retraining, making the process inefficient. As a result, unsupervised learning approaches have gained prominence in image inpainting. State-of-the-art methods, particularly those using generative adversarial networks (GANs), have achieved promising results. However, generating photorealistic outputs for high-resolution images with arbitrary large-region masks remains difficult. Inpainted images often suffer from deformed structures and blurry textures, compromising quality. Additionally, building a model capable of handling a diverse range of images presents further challenges. These challenges are addressed by proposing a novel probabilistic model that utilizes picture priors to learn prominent features within StyleGAN3. The priors are constructed using cosine similarity, mean, and intensity, where intensity is computed using the improved Papoulis–Gerchberg algorithm. The image is reconstructed using the probabilistic maximum a posteriori estimate. Variational inference is then applied to obtain the optimal solution using a modified Bayes-by-Backprop approach. The model is evaluated on 70,000 images from the Flickr-Faces-HQ, DIV2K, and brain datasets and surpasses state-of-the-art techniques in reconstruction quality.

Due to limited spatial bandwidth, one has to compromise between large field of view and high spatial resolution in both photography and microscopy. This dilemma largely hampers revealing fine details and global structures of the target scene simultaneously. Recently, a mainstream method is formed by utilizing multiple sensors for synchronous acquisition across different sub-FOVs with high resolution and stitching the patches according to the spatial position of the cameras. Various inpainting algorithms have been proposed to eliminate the intensity discontinuities, but conventional optimization methods are prone to misalignment, seaming artifacts or long processing time, and thus unable to achieve dynamic gap elimination. By taking advantage of generative adversarial networks (GANs) on image generation and padding, we propose a conditional GAN-based deep neural network for seamless gap inpainting. Specifically, a short series of displaced images are acquired to characterize the system configuration, under which we generate patch pairs with and without gap for deep network training. After supervised learning, we can achieve seamless inpainting in gap regions. To validate the proposed approach, we apply our approach on real data captured by large-scale imaging systems and demonstrate that the missing information at gaps can be retrieved successfully. We believe the proposed method holds potential for all-round observation in various fields including urban surveillance and systems biology.

Traditional image inpainting algorithms aim to obtain satisfactory results for the relatively small and regular missing regions. However, the problem of the image with the large and irregular holes has always been challenging. Our work focus is not on filling holes in the corrupted image, but on the more difficult task of semantic repair, which aims to predict the details of the large regions according to the context of surrounding pixels. In this paper, we propose a perceptual generative adversarial network for image generation and inpainting method. In the training stage, VGG pre-trained model is introduced to GAN network to make the generated image have more high-frequency features. This framework can not only synthesize images with perceptual reality but also make a better prediction of the corrupted images by using the encoding of images. The weighted context loss is adopted to repair the missing regions in image inpainting, and the adversarial loss is used to punish the perceptually unrealistic image. Experiments on natural scene data set CelebA show that our proposed method can produce higher quality repair results.

Adaptation to Natural Binocular Disparities in Primate V1 Explained by a Generalized Energy Model