Image Super-Resolution by Supervised Adaption of Patchwise Self-similarity from High-Resolution Image.

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

Cross-Resolution Person Re-Identification (re-ID) aims to match images with disparate resolutions arising from variations in camera hardware and shooting distances. Most conventional works utilize Super-Resolution (SR) models to recover Low Resolution (LR) images to High Resolution (HR) images. However, because the SR models cannot completely compensate for the missing information in the LR images, there is still a large gap between the HR image recovered from the LR images and the real HR images. To tackle this challenge, we propose a novel Multi-Scale Image- and Feature-Level Alignment (MSIFLA) framework to align the images on multiple resolution scales at both the image and feature level. Specifically, (i) we design a Cascaded Multi-Scale Resolution Reconstruction (CMSR2) module, which is composed of three cascaded Image Reconstruction (IR) networks, and can continuously reconstruct multiple variables of different resolution scales from low to high for each image, regardless of image resolution. The reconstructed images with specific resolution scales are of similar distribution; therefore, the images are aligned on multiple resolution scales at the image level. (ii) We propose a Multi-Resolution Representation Learning (MR2L) module which consists of three-person re-ID networks to encourage the IR models to preserve the ID-discriminative information during training separately. Each re-ID network focuses on mining discriminative information from a specific scale without the disturbance from various resolutions. By matching the extracted features on three resolution scales, the images with different resolutions are also aligned at the feature-level. We conduct extensive experiments on multiple public cross-resolution person re-ID datasets to demonstrate the superiority of the proposed method. In addition, the generalization of MSIFLA in handling cross-resolution retrieval tasks is verified on the UAV vehicle dataset.

In this paper, we demonstrate simple algorithms that project low resolution (LR) images differing in subpixel shifts on a high resolution (HR) also called super resolution (SR) grid. The algorithms are very effective in accuracy as well as time efficiency. A number of spatial interpolation techniques using nearest neighbor, inverse-distance weighted averages, Radial Basis Functions (RBF) etc. are used in projection. For best accuracy of reconstructing SR image by a factor of two requires four LR images differing in four independent subpixel shifts. The algorithm has two steps: i) registration of low resolution images and (ii) shifting the low resolution images to align with reference image and projecting them on high resolution grid based on the shifts of each low resolution image using different interpolation techniques. Experiments are conducted by simulating low resolution images by subpixel shifts and subsampling of original high resolution image and the reconstructing the high resolution images from the simulated low resolution images. The results of accuracy of reconstruction are compared by using mean squared error measure between original high resolution image and reconstructed image. The algorithm was tested on remote sensing images and found to outperform previously proposed techniques such as Iterative Back Projection algorithm (IBP), Maximum Likelihood (ML) algorithms. The algorithms are robust and are not overly sensitive to the registration inaccuracies.

Due to the limitations of communication bandwidth, the transmission of high-resolution images or videos between spacecrafts such as satellites and planetary landers and ground stations has always been a challenging task. To address the issue of high-resolution image transmission, we propose a novel image transmission paradigm based on reference-based super resolution (Ref-SR). This approach involves the simultaneous transmission of both high-resolution and low-resolution images between the spacecraft and ground stations. The low-resolution images are transmitted at a high frame rate, while the high-resolution images are transmitted at a lower frame rate. The high-resolution images serve as references to enhance the quality of the low-resolution images, thereby optimizing bandwidth usage while maintaining image clarity. Compared to directly applying super resolution to low-resolution images, which introducing image distortion due to the domain bias between the space image and its training images, our proposed method keeps the consistency of the super-resolved images and the ground truth images. By using high-resolution images as references, the super resolution process is guided to produce more accurate and reliable high-resolution images. Previous ground-based experiments have demonstrated the feasibility of Ref-SR in image and video restoration, and we believe this method has great potential for lunar communication scenarios.

Image super-resolution (SR) can be seen as finding a nonlinear mapping between low-resolution (LR) image and high-resolution (HR) image. Convolutional neural networks (CNN) based image SR methods have achieved great success based on features extracted from pretrained networks. However, the previous works less consider multiscale features. In this paper, we propose a SR cascaded multi-column convolutional neural network (SRCMCN) to recover HR image by learning multi-scale features from the corresponding LR image. Compared to existing CNN based image SR methods, the proposed method directly takes the original LR image as input to learn the residual between interpolated LR image and corresponding HR image, which will reduce the computational cost of convolution due to the small input size. Besides, in order to achieve high-quality reconstruction, we optimize our networks using mean absolute error (MAE) loss function. Experimental results demonstrate that our proposed method outperforms several state-of-the-art image SR methods in terms of both reconstructed image quality and computational efficiency.

Deep Feature Blend Attention: A New Frontier in Super Resolution Image Generation

Single image super-resolution based on compressive sensing and improved TV minimization sparse recovery

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.

We propose a new high-quality up-scaling technique that extends the existing example based super-resolution (SR) framework. Our approach is based on the fundamental idea that a low-resolution (LR) image could be generated from any of the multiple possible high-resolution (HR) images. Therefore it would be more natural to use multiple predictors of HR patch from LR patch instead of single one. In this work we build a generic framework to estimate an HR image from LR one using an adaptive prior (select the predictor locally) based on the local statistics of LR images. We use natural image patch prior as the HR image statistics. We partition the natural images into documents and group them to discover the inherent topics using probabilistic Latent Semantic Analysis (pLSA) and also learn the dual dictionaries of HR and LR image patch pairs for each of the topics using sparse dictionary learning technique. Then for test image we infer locally which topic it corresponds to and then we use the corresponding learned dual dictionary to generate HR image. Experimental results show the effectiveness of our method over existing state-of-art methods.KeywordsSparse RepresentationSparse CodeSuper ResolutionProbabilistic Latent Semantic AnalysisBicubic InterpolationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

In this paper, an undecimated double density dual tree discrete wavelet transform (UDDDT-DWT) based image resolution enhancement technique is proposed. The critically sampled discrete wavelet transform (DWT) suffers from the drawbacks of being shift-variant and lacking the capacity to process directional information in images. The double density dual tree wavelet transform (DDDT-WT) is an approximately shift-invariant transform capturing directional information. The UDDDT-DWT is an improvement of the DDDT-DWT, making it exactly shift-invariant. The method uses a forward and inverse (UDDDT-DWT) to construct a high-resolution (HR) image from the given lower-resolution (LR) image. The HR image is reconstructed from the LR image using the inverse UDDDT-DWT. Results are presented and discussed through comparisons between state-of-the-art resolution enhancement methods.

Due to the factors like processing power limitations and channel capabilities images are often down sampled and transmitted at low bit rates resulting in a low resolution compressed image. High resolution images can be reconstructed from several blurred, noisy and down sampled low resolution images using a computational process know as super resolution reconstruction. The problem of recovering a high resolution image from a sequence of low resolution compressed images is considered. In this paper, we propose lifting schemes for intentionally introducing down sampling of the high resolution image sequence before compression and then utilize super resolution techniques for generating a high resolution image at the decoder. Lifting wavelet transform has its advantages over the ordinary wavelet transform by way of reduction in memory required for its implementation. This is possible because lifting transform uses in-place computation. The lifting coefficients replace the image samples present in the respective memory locations. In our proposed approach the forward lifting is applied to the high resolution images which are compressed using Set Portioning in Hierarchical Trees (SPHIT), the compressed images are transmitted which results in low resolution images at the encoder, at the decoder super resolution techniques are applied; lifting scheme based fusion is performed, then decoded using DSPIHT and Inverse lifting is applied. In order to remove noise from the reconstructed image soft thresholding is performed and the blur is removed using blind deconvolution, and finally interpolated using our novel interpolation technique. We have performed both objective and subjective analysis of the reconstructed image, and the resultant image has better super resolution factor, and a higher ISNR and PSNR.

Acquiring degraded images with paired high-resolution (HR) images is often challenging, impeding the advance of image super-resolution in real-world applications. By generating realistic low-resolution (LR) images with degradation similar to that in real-world scenarios, simulated paired LR-HR data can be constructed for supervised training. However, most of the existing work ignores the degradation uncertainty of the generated realistic LR images, since only one LR image has been generated given an HR image. To address this weakness, we propose learning the degradation uncertainty of generated LR images and sampling multiple LR images from the learned LR image (mean) and degradation uncertainty (variance) and construct LR-HR pairs to train the super-resolution (SR) networks. Specifically, uncertainty can be learned by minimizing the proposed loss based on Kullback-Leibler (KL) divergence. Furthermore, the uncertainty in the feature domain is exploited by a novel perceptual loss; and we propose to calculate the adversarial loss from the gradient information in the SR stage for stable training performance and better visual quality. Experimental results on popular real-world datasets show that our proposed method has performed better than other unsupervised approaches.

A new multi-frame image super resolution (SR) algorithm via Bayesian modeling with natural image prior modeled by fields of experts (FoE) is proposed. Multi-frame SR can be used to obtain a high resolution (HR) image from a set of degraded low resolution (LR) images without changing any hardware device. However, SR is well known to be an ill-posed problem. So state-of-the-art solutions usually formulate the problem with Bayesian modeling techniques, which infer the HR image based on not only the LR input images but also on prior information about the HR image. Current Bayesian SR approaches typically use simple prior models such as L1 norm, TV prior and Laplacian prior, which cannot exploit the statistics of natural scenes well. In this paper, a Bayesian multi-frame image SR approach using a FOE model as the prior for natural images is presented. The Maximum a Posteriori (MAP) framework is used for estimating the HR image. The proposed method cannot only capture the statistics of natural images well, but also require less computational power than the other Bayesian modelling methods such as Sampling methods and Approximate inference. The proposed method shows superior or comparable results to the state-of-art multi-frame SR methods.

Image super-resolution is a research endeavour that has gained notoriety in computer vision. The research goal is to increase the spatial dimensions of an image using corresponding low-resolution and high-resolution image pairs to enhance the perceptual quality. The challenge of maintaining such perceptual quality lies in developing appropriate algorithms that learn to reconstruct higher-quality images from their lower-resolution counterparts. Recent methods employ deep learning algorithms to reconstruct textural details prevalent in low-resolution images. Since corresponding image pairs are non-trivial to collect, researchers attempt super-resolution by creating synthetic low-resolution representations of high-resolution images. Unfortunately, such methods employ ineffective downscaling operations to achieve synthetic low-resolution images. These methods fail to generalize well on real-world images that may suffer different degradations. A different angle is offered to solve the task of image super-resolution by investigating the plausibility of learning the degradational operation using generative adversarial networks. A two-stage generative adversarial network along with two architectural variations is proposed to solve the task of real-world super-resolution from low-resolution images with unknown degradations. It is demonstrated that learning to downsample images in a weakly supervised manner is an impactful and viable approach for super-resolution.