Learning Degradation Uncertainty for Unsupervised Real-world Image Super-resolution

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

Deep Feature Blend Attention: A New Frontier in Super Resolution Image Generation

Image super-resolution is of great interest in medical imaging field. However, different from natural images studied in computer vision field, the low-resolution (LR) medical imaging data is often a stack of high-resolution (HR) 2D slices with large slice thickness. Consequently, the goal of super-resolution for medical imaging data is to reconstruct the missing slice(s) between any two consecutive slices. Since some modalities (e.g., T1-weighted MR image) are often acquired with high-resolution (HR) image, it is intuitive to harness the prior self-similarity information in the HR image for guiding the super-resolution of LR image (e.g., T2-weighted MR image). The conventional way is to find the profile of patchwise self-similarity in the HR image and then use it to reconstruct the missing information at the same location of LR image. However, the local morphological patterns could vary significantly across the LR and HR images, due to the use of different imaging protocols. Therefore, such direct (un-supervised) adaption of self-similarity profile from HR image is often not effective in revealing the actual information in the LR image. To this end, we propose to employ the existing image information in the LR image to supervise the estimation of self-similarity profile by requiring it not only being optimal in representing patches in the HR image, but also producing less reconstruction errors for the existing image information in the LR image. Moreover, to make the anatomical structures spatially consistent in the reconstructed image, we simultaneously estimate the self-similarity profiles for a stack of patches across consecutive slices by solving a group sparse patch representation problem. We have evaluated our proposed super-resolution method on both simulated brain MR images and real patient images with multiple sclerosis lesion, achieving promising results with more anatomical details and sharpness.

In this paper, we demonstrate simple algorithms that project low resolution (LR) images differing in subpixel shifts on a high resolution (HR) also called super resolution (SR) grid. The algorithms are very effective in accuracy as well as time efficiency. A number of spatial interpolation techniques using nearest neighbor, inverse-distance weighted averages, Radial Basis Functions (RBF) etc. are used in projection. For best accuracy of reconstructing SR image by a factor of two requires four LR images differing in four independent subpixel shifts. The algorithm has two steps: i) registration of low resolution images and (ii) shifting the low resolution images to align with reference image and projecting them on high resolution grid based on the shifts of each low resolution image using different interpolation techniques. Experiments are conducted by simulating low resolution images by subpixel shifts and subsampling of original high resolution image and the reconstructing the high resolution images from the simulated low resolution images. The results of accuracy of reconstruction are compared by using mean squared error measure between original high resolution image and reconstructed image. The algorithm was tested on remote sensing images and found to outperform previously proposed techniques such as Iterative Back Projection algorithm (IBP), Maximum Likelihood (ML) algorithms. The algorithms are robust and are not overly sensitive to the registration inaccuracies.

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

Single image super-resolution based on compressive sensing and improved TV minimization sparse recovery

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

Image upscaling aims to increase the resolution and size of a low resolution (LR) image in order to generate a high resolution (HR) image of high frequency (HF). There are several polynomial methods for obtaining a sharpened, upscaled HR image. The interpolated pixel is measured using a weighted average of the neighboring pixels within the image grid that blur at HF regions in these methods. Edge degradation is also caused by other edge-directed and learning-based upscaling methods, which produce blurring artifacts. A novel approach is proposed to fill these gaps. Using the concept of unsharp masking (USM), the LR image is blurred adaptively based on the region’s local variance. The sharpened high pass filtered (HPF) image is then obtained by subtracting the adaptively blurred image from the LR image. According to USM, the HPF image is combined with the LR image via a gain factor optimized using the cuckoo search (CS) algorithm. To compensate for the loss caused by upscaling, this pre-processing step is performed prior to interpolation. Aside from that, the edge of the B-spline interpolated image is detected and expanded. Edge expansion of the upscaled image is performed to further restore the HF details and reduce zigzag artifacts introduced by upscaling while also preserving the edge boundary. The proposed method outperforms the Lanczos, Bicubic, and Bilinear schemes in terms of peak signal to noise ratio (PSNR) gain of 3.475, 8.3839, and 8.075 dB, respectively. In terms of performance, this method outperforms state-of-the-art techniques both objectively (PSNR and SSIM) and subjectively (visual quality).

In this paper we propose a fast approach for edge preserving super-resolution (SR) based on learning of contourlet coefficients. Given a low resolution test image, we first obtain an initial HR estimate i.e., a close approximation to SR image by learning the contourlet coefficients from a training database consisting of low resolution (LR) and high resolution (HR) images. The final SR image is obtained by using a regularization framework in which both the SR and the LR images are modeled as separate homogeneous Markov Random Fields (MRFs). The LR image formation process is modeled as a decimated and noisy version of the SR image and the final cost function is minimized by using a gradient descent method. Novelty of our approach lies in preserving the edges in the final SR image while using a non edge preserving MRF prior. This is definitely advantageous since it avoids the use of discontinuity preserving prior and hence the computationally taxing optimization methods. The edges in the final SR correspond to those learned from the initial HR estimate. The use of MRF on the low resolution image imposes an additional constraint on the final solution and hence we expect a better solution. In addition, we use the initial HR image for estimating the decimation matrix entries as well as for learning the corresponding MRF parameter. We show the effectiveness of the proposed approach by conducting the experiments on images captured using a real camera.

Most current super-resolution methods rely on low and high resolution image pairs to train a network in a fully supervised manner. However, such image pairs are not available in real-world applications. Instead of directly addressing this problem, most works employ the popular bicubic downsampling strategy to artificially generate a corresponding low resolution image. Unfortunately, this strategy introduces significant artifacts, removing natural sensor noise and other real-world characteristics. Super-resolution networks trained on such bicubic images therefore struggle to generalize to natural images. In this work, we propose an unsupervised approach for image super-resolution. Given only unpaired data, we learn to invert the effects of bicubic downsampling in order to restore the natural image characteristics present in the data. This allows us to generate realistic image pairs, faithfully reflecting the distribution of real-world images. Our super-resolution network can therefore be trained with direct pixel-wise supervision in the high resolution domain, while robustly generalizing to real input. We demonstrate the effectiveness of our approach in quantitative and qualitative experiments.

Image super-resolution (SR) can be seen as finding a nonlinear mapping between low-resolution (LR) image and high-resolution (HR) image. Convolutional neural networks (CNN) based image SR methods have achieved great success based on features extracted from pretrained networks. However, the previous works less consider multiscale features. In this paper, we propose a SR cascaded multi-column convolutional neural network (SRCMCN) to recover HR image by learning multi-scale features from the corresponding LR image. Compared to existing CNN based image SR methods, the proposed method directly takes the original LR image as input to learn the residual between interpolated LR image and corresponding HR image, which will reduce the computational cost of convolution due to the small input size. Besides, in order to achieve high-quality reconstruction, we optimize our networks using mean absolute error (MAE) loss function. Experimental results demonstrate that our proposed method outperforms several state-of-the-art image SR methods in terms of both reconstructed image quality and computational efficiency.

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.

The image super-resolution algorithm can overcome the imaging system's hardware limitation and obtain higher resolution and clearer images. Existing super-resolution methods based on convolutional neural networks(CNN) can learn the mapping relationship between high-resolution(HR) and low-resolution(LR) images. However, when the reconstruction target is a face image, the reconstruction results often have problems that the face area is too smooth and lacks details. We propose a guided cascaded face super-resolution network, called guided cascaded super-resolution network (GCFSRnet). GCFSRnet takes the LR image and a high-quality guided image as inputs, and it consists of a pose deformation module and a super-resolution network. Firstly, the pose deformation module converts the guide image's posture into the same as the low-resolution face image based on 3D fitting and 3D morphable model (3DMM). Then, the LR image and the deformed guide image are used as input of the super-resolution network. The super-resolution networks are formed by a cascade of two layers of networks, which extract different features. During the reconstruction process, the guide image can provide real facial details and help generate subtle facial textures. The cascade structure of a super-resolution network can gradually extract features and restore different levels of image details. The experimental results on the CASIA Web Face and CelebA datasets show that the proposed method can generate facial images with clear outlines and rich details, which are superior to other state-of-the-art methods such as SRResNet, SRGAN, VDSR, DBPN, etc.

Deep convolutional neural networks (CNNs), trained on corresponding pairs of high- and low-resolution images, achieve state-of-the-art performance in single-image super- resolution and surpass previous signal-processing based approaches. However, their performance is limited when applied to real photographs. The reason lies in their train- ing data: low-resolution (LR) images are obtained by bicu- bic interpolation of the corresponding high-resolution (HR) images. The applied convolution kernel significantly differs from real-world camera-blur. Consequently, while current CNNs well super-resolve bicubic-downsampled LR images, they often fail on camera-captured LR images. To improve generalization and robustness of deep super- resolution CNNs on real photographs, we present a ker- nel modeling super-resolution network (KMSR) that incor- porates blur-kernel modeling in the training. Our pro- posed KMSR consists of two stages: we first build a pool of realistic blur-kernels with a generative adversarial net- work (GAN) and then we train a super-resolution network with HR and corresponding LR images constructed with the generated kernels. Our extensive experimental vali- dations demonstrate the effectiveness of our single-image super-resolution approach on photographs with unknown blur-kernels.

To reduce external disturbances and achieve high vertical resolution, the scanning time for white-light interference microscopy is very short. Because capturing high-resolution (HR) images is time consuming, low-resolution (LR) images are acquired instead. However, HR images are more desirable because they contain more details. To ensure high vertical resolution and high image resolution, one feasible solution is to process the scanned LR images to HR images by single image super-resolution (SISR). In this paper, an interference image super-resolution (IISR) model based on a generative adversarial network (GAN) is proposed. The generator is based on the enhanced super-resolution generative adversarial network (ESRGAN) architecture. With the aim of acquiring more realistic images, the discriminator network is designed using a modified DenseNet architecture, in which the pooling layers are replaced with dilated convolutional layers. The perceptual loss is optimized, and the content loss is upgraded to a continuously differentiable piecewise function. Various microscopy images are tested, including images with and without interference fringes. The IISR model has been proven to restore LR images to HR images. The comparative experiments prove that the proposed model achieves better visual quality than other models, preserving more realistic details.