Deep Feature Blend Attention: A New Frontier in Super Resolution Image Generation

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.

In this paper we propose a fast approach for edge preserving super-resolution (SR) based on learning of contourlet coefficients. Given a low resolution test image, we first obtain an initial HR estimate i.e., a close approximation to SR image by learning the contourlet coefficients from a training database consisting of low resolution (LR) and high resolution (HR) images. The final SR image is obtained by using a regularization framework in which both the SR and the LR images are modeled as separate homogeneous Markov Random Fields (MRFs). The LR image formation process is modeled as a decimated and noisy version of the SR image and the final cost function is minimized by using a gradient descent method. Novelty of our approach lies in preserving the edges in the final SR image while using a non edge preserving MRF prior. This is definitely advantageous since it avoids the use of discontinuity preserving prior and hence the computationally taxing optimization methods. The edges in the final SR correspond to those learned from the initial HR estimate. The use of MRF on the low resolution image imposes an additional constraint on the final solution and hence we expect a better solution. In addition, we use the initial HR image for estimating the decimation matrix entries as well as for learning the corresponding MRF parameter. We show the effectiveness of the proposed approach by conducting the experiments on images captured using a real camera.

Image super-resolution is of great interest in medical imaging field. However, different from natural images studied in computer vision field, the low-resolution (LR) medical imaging data is often a stack of high-resolution (HR) 2D slices with large slice thickness. Consequently, the goal of super-resolution for medical imaging data is to reconstruct the missing slice(s) between any two consecutive slices. Since some modalities (e.g., T1-weighted MR image) are often acquired with high-resolution (HR) image, it is intuitive to harness the prior self-similarity information in the HR image for guiding the super-resolution of LR image (e.g., T2-weighted MR image). The conventional way is to find the profile of patchwise self-similarity in the HR image and then use it to reconstruct the missing information at the same location of LR image. However, the local morphological patterns could vary significantly across the LR and HR images, due to the use of different imaging protocols. Therefore, such direct (un-supervised) adaption of self-similarity profile from HR image is often not effective in revealing the actual information in the LR image. To this end, we propose to employ the existing image information in the LR image to supervise the estimation of self-similarity profile by requiring it not only being optimal in representing patches in the HR image, but also producing less reconstruction errors for the existing image information in the LR image. Moreover, to make the anatomical structures spatially consistent in the reconstructed image, we simultaneously estimate the self-similarity profiles for a stack of patches across consecutive slices by solving a group sparse patch representation problem. We have evaluated our proposed super-resolution method on both simulated brain MR images and real patient images with multiple sclerosis lesion, achieving promising results with more anatomical details and sharpness.

In this paper, trade-off between effect of registration error and number of images used in the process of super resolution image reconstruction is studied. Super Resolution image reconstruction is three phase process, of which registration is of at most importance. Super resolution image reconstruction uses set of low resolution images to reconstruct high resolution image during registration. The study demonstrates the effects of registration error and benefit of more number of low resolution images on the quality of reconstructed image. Study reveals that the registration error degrades the reconstructed image and without better registration methodology, a better super resolution method is still not of any use. It is noticed that without further improvement in the registration technique, not much improvement can be achieved by increasing number of input low resolution images.

In this paper, we demonstrate simple algorithms that project low resolution (LR) images differing in subpixel shifts on a high resolution (HR) also called super resolution (SR) grid. The algorithms are very effective in accuracy as well as time efficiency. A number of spatial interpolation techniques using nearest neighbor, inverse-distance weighted averages, Radial Basis Functions (RBF) etc. are used in projection. For best accuracy of reconstructing SR image by a factor of two requires four LR images differing in four independent subpixel shifts. The algorithm has two steps: i) registration of low resolution images and (ii) shifting the low resolution images to align with reference image and projecting them on high resolution grid based on the shifts of each low resolution image using different interpolation techniques. Experiments are conducted by simulating low resolution images by subpixel shifts and subsampling of original high resolution image and the reconstructing the high resolution images from the simulated low resolution images. The results of accuracy of reconstruction are compared by using mean squared error measure between original high resolution image and reconstructed image. The algorithm was tested on remote sensing images and found to outperform previously proposed techniques such as Iterative Back Projection algorithm (IBP), Maximum Likelihood (ML) algorithms. The algorithms are robust and are not overly sensitive to the registration inaccuracies.

In real-life scenarios, the accuracy of person re-identification (Re-ID) is subject to the limitation of camera hardware conditions and the change of image resolution caused by factors such as camera focusing errors. People call this problem cross-resolution person Re-ID. In this paper, we improve the recognition accuracy of cross-resolution person Re-ID by enhancing the image enhancement network and feature extraction network. Specifically, we treat cross-resolution person Re-ID as a two-stage task: the first stage is the image enhancement stage, and we propose a Super-Resolution Dual-Stream Feature Fusion sub-network, named SR-DSFF, which contains SR module and DSFF module. The SR-DSFF utilizes the SR module recovers the resolution of the low-resolution (LR) images and then obtains the feature maps of the LR images and super-resolution (SR) images, respectively, through the dual-stream feature fusion with learned weights extracts and fuses feature maps from LR and SR images in the DSFF module. At the end of SR-DSFF, we set a transposed convolution to visualize the feature maps into images. The second stage is the feature acquisition stage. We design a global-local feature extraction network guided by human pose estimation, named FENet-ReID. The FENet-ReID obtains the final features through multistage feature extraction and multiscale feature fusion for the Re-ID task. The two stages complement each other, making the final pedestrian feature representation has the advantage of accurate identification compared with other methods. Experimental results show that our method improves significantly compared with some state-of-the-art methods.

Zoom based image super-resolution using DCT with LBP as characteristic model

Due to the factors like processing power limitations and channel capabilities images are often down sampled and transmitted at low bit rates resulting in a low resolution compressed image. High resolution images can be reconstructed from several blurred, noisy and down sampled low resolution images using a computational process know as super resolution reconstruction. Super-resolution is the process of combining multiple aliased low-quality images to produce a high resolution, high-quality image. The problem of recovering a high resolution image progressively from a sequence of low resolution compressed images is considered. In this paper we propose a novel DCT based progressive image display algorithm by stressing on the encoding and decoding process. At the encoder we consider a set of low resolution images which are corrupted by additive white Gaussian noise and motion blur. The low resolution images are compressed using 8 by 8 blocks DCT and noise is filtered using our proposed novel zonal filter. Multiframe fusion is performed in order to obtain a single noise free image. At the decoder the image is reconstructed progressively by transmitting the coarser image first followed by the detail image. And finally a super resolution image is reconstructed by applying our proposed novel adaptive interpolation technique. We have performed both objective and subjective analysis of the reconstructed image, and the resultant image has better super resolution factor, and a higher ISNR and PSNR. A comparative study done with Iterative Back Projection (IBP) and Projection on to Convex Sets (POCS),Papoulis Grechberg, FFT based Super resolution Reconstruction shows that our method has out performed the previous contributions.

Super-resolution (SR) techniques produce a high resolution (HR) image from a set of low-resolution (LR) undersampled images. Usually, SR problems are posed as estimation problems where the LR images are contaminated by stationary noise. However, in real SR problems is very common to have non-stationary noise due to problems in the registration of the images or outliers. SR methods that address this type of problems are called robust. In this paper we propose a novel robust SR method that employs a median filter directly in the data from the LR images, before proceeding to the interpolation and deblurring steps that are common in SR. We compare this new method with other robust SR methods with synthetic and real data, proving that it outperforms the other methods in both cases.KeywordsProjective MethodMedian FilterRegistration ErrorPepper NoiseIrregular SampleThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Super-resolution (SR) of facial images from video suffers from facial expression changes. Most of the existing SR algorithms for facial images make an unrealistic assumption that the ?perfect? registration has been done prior to the SR process. However, the registration is a challenging task for SR with expression changes. This paper proposes a new method for enhancing the resolution of low-resolution (LR) facial image by handling the facial image in a non-rigid manner. It consists of global tracking, local alignment for precise registration and SR algorithms. A B-spline based resolution aware incremental free form deformation (RAIFFD) model is used to recover a dense local non-rigid flow field. In this scheme, low-resolution image model is explicitly embedded in the optimization function formulation to simulate the formation of low resolution image. The results achieved by the proposed approach are significantly better as compared to the SR approaches applied on the whole face image without considering local deformations. The results are also compared with two state-of-the-art SR algorithms to show the effectiveness of the approach in super-resolving facial images with local expression changes.

Example learning-based image super-resolution (SR) is recognized as an effective way to produce a high-resolution (HR) image with the help of an external training set. The effectiveness of learning-based SR methods, however, depends highly upon the consistency between the supporting training set and low-resolution (LR) images to be handled. To reduce the adverse effect brought by incompatible high-frequency details in the training set, we propose a single image SR approach by learning multiscale self-similarities from an LR image itself. The proposed SR approach is based upon an observation that small patches in natural images tend to redundantly repeat themselves many times both within the same scale and across different scales. To synthesize the missing details, we establish the HR-LR patch pairs using the initial LR input and its down-sampled version to capture the similarities across different scales and utilize the neighbor embedding algorithm to estimate the relationship between the LR and HR image pairs. To fully exploit the similarities across various scales inside the input LR image, we accumulate the previous resultant images as training examples for the subsequent reconstruction processes and adopt a gradual magnification scheme to upscale the LR input to the desired size step by step. In addition, to preserve sharper edges and suppress aliasing artifacts, we further apply the nonlocal means method to learn the similarity within the same scale and formulate a nonlocal prior regularization term to well pose SR estimation under a reconstruction-based SR framework. Experimental results demonstrate that the proposed method can produce compelling SR recovery both quantitatively and perceptually in comparison with other state-of-the-art baselines.

Acquiring degraded images with paired high-resolution (HR) images is often challenging, impeding the advance of image super-resolution in real-world applications. By generating realistic low-resolution (LR) images with degradation similar to that in real-world scenarios, simulated paired LR-HR data can be constructed for supervised training. However, most of the existing work ignores the degradation uncertainty of the generated realistic LR images, since only one LR image has been generated given an HR image. To address this weakness, we propose learning the degradation uncertainty of generated LR images and sampling multiple LR images from the learned LR image (mean) and degradation uncertainty (variance) and construct LR-HR pairs to train the super-resolution (SR) networks. Specifically, uncertainty can be learned by minimizing the proposed loss based on Kullback-Leibler (KL) divergence. Furthermore, the uncertainty in the feature domain is exploited by a novel perceptual loss; and we propose to calculate the adversarial loss from the gradient information in the SR stage for stable training performance and better visual quality. Experimental results on popular real-world datasets show that our proposed method has performed better than other unsupervised approaches.

Social media network is inseparable from image recognition, and image super-resolution (SR) reconstruction plays an important role in image recognition. The changes of scale and geometry are rarely considered in the image super-resolution reconstruction based on deep learning over the years, we introduce a super-resolution reconstruction network based on deformable convolutional network. We replace the ordinary convolution with the deformable convolution to pretend the geometric deformation and extract abundant local features. The image super-resolution reconstruction is usually based on the conventional convolutional neural network (CNN). Most CNN-based SR models do not utilize the features of the original low resolution (LR) image as much as possible, resulting in lower performance. After introducing the idea of deformable convolution, though the complexity is increased, the recognition accuracy is obviously raised.