Image Super Resolution using Generative Adverbial Networks

Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images which could affect the clinical interpretation and diagnosis, and (3) the vanilla GAN architecture may suffer from unstable training and collapse mode that can also affect the SISR results. In this paper, we propose a novel lesion focused SR (LFSR) method, which incorporates GAN to achieve perceptually realistic SISR results for brain tumour MRI images. More importantly, we test and make comparison using recently developed GAN variations, e.g., Wasserstein GAN (WGAN) and WGAN with Gradient Penalty (WGAN-GP), and propose a novel multi-scale GAN (MS-GAN), to achieve a more stabilised and efficient training and improved perceptual quality of the super-resolved results. Based on both quantitative evaluations and our designed mean opinion score, the proposed LFSR coupled with MS-GAN has performed better in terms of both perceptual quality and efficiency.

Abstract. Single image super resolution (SISR) technology has been attracted much attention from remote sensing community due to its proven potentials in remote sensing applications. Existing SISR techniques varying from conventional interpolation methods to different network architectures. Generative adversarial networks (GANs) are one of the latest network architectures proven a greater potential as a SISR method whereas least attention has been given by the remote sensing community. Several studies have already been carried out on this context. However, yet there is no generalized GAN based approach to super resolve remote sensing imageries. Therefore, this study investigated the potentials of enhanced super resolution generative adversarial (ESRGAN) model to super resolve very high to medium resolution images from high to coarse resolution images for remote sensing applications. Two models were trained and Worldview-3 (WV3) images used as for very high resolution images. Whereas, down sampled WV3 and Sentinel-2(S2) were used as low resolution counterparts. Model performances were qualitatively and quantitatively analysed using standard metrics such as PSNR, SSIM, UIQI, CC, SAM, SID. Evaluation results emphasised super resolved images were preserved the original quality of the satellite images to a greater extent while improving its ground resolution.

Recent studies have shown that Deep Learning (DL) algorithms can significantly improve Super Resolution (SR) performance. Single image SR is useful in producing High Resolution (HR) images from their Low Resolution (LR) counterparts. The motivation for SR is the potential to assist algorithms such as object detection, localization, and classification. Insufficient work has been conducted using Generative Adversarial Networks (GANs) for SR on infrared (IR) images despite its promising ability to increase object detection accuracy by extracting more precise features from a given image. This work adopts the idea of a relativistic GAN that utilizes Residual in Residual Dense blocks (RRDBs) for feature ex- traction, a novel residual image addition, and a Pixel Transposed Convolutional Layer (PixelTCL) for up-sampling. Recent work has validated the use of GANs for Visible Light (VL) images, making them a strong candidate. The inclusion of these components produce more realistic and natural features while also receiving superior metric values.

In recent years, the study of a single image super-resolution (SISR) is crucial to improving image resolution and using hardware technology to improve image resolution. SISR is widely used in satellite remote sensing, video surveillance, and medical image processing because it mainly relies on deep learning algorithms to realize the conversion from low-resolution (LR) images to high-resolution images. It has the advantages of low cost, simple operation, and high efficiency. This paper proposes an image super-resolution method based on a generative adversarial network named text localization generative adversarial nets (TLGAN) model. The method is improved based on super-resolution generative adversarial networks (SRGAN), and the batch normalization layer is removed, which significantly reduces the computational burden of the model. In TLGAN model, we used the transfer learning method to pre-trained the model on the large dataset ImageNet, and then apply the pre-trained model to the cartoon image data set animes to achieve image super-resolution. Experimental results report that the proposed method has the advantages of fast running speed and excellent visual perception of super-resolution images compared with bicubic interpolation and SRGAN method.

To reduce external disturbances and achieve high vertical resolution, the scanning time for white-light interference microscopy is very short. Because capturing high-resolution (HR) images is time consuming, low-resolution (LR) images are acquired instead. However, HR images are more desirable because they contain more details. To ensure high vertical resolution and high image resolution, one feasible solution is to process the scanned LR images to HR images by single image super-resolution (SISR). In this paper, an interference image super-resolution (IISR) model based on a generative adversarial network (GAN) is proposed. The generator is based on the enhanced super-resolution generative adversarial network (ESRGAN) architecture. With the aim of acquiring more realistic images, the discriminator network is designed using a modified DenseNet architecture, in which the pooling layers are replaced with dilated convolutional layers. The perceptual loss is optimized, and the content loss is upgraded to a continuously differentiable piecewise function. Various microscopy images are tested, including images with and without interference fringes. The IISR model has been proven to restore LR images to HR images. The comparative experiments prove that the proposed model achieves better visual quality than other models, preserving more realistic details.

Images created using the Super Resolution method can generate more information compared to their low resolution counterparts. A super-resolved image, which is created using an original image captured by an imaging source is not only more meaningful to human perception but also has advantages on downstream tasks such as object detection and pattern recognition. In this work, we aim to apply the Super Resolution method to the Aerial Images captured for surveillance to enable more information about the original scenes. To achieve this Super Resolution Generative Adversarial Network (SRGAN), which is based on the Generative Adversarial Networks architecture is used. We also applied transfer learning methodology to achieve better image quality. Public xView and DOTA datasets which contain images mostly captured by satellites around the world are used to train a generative model via SRGAN architecture. Furthermore, DIV2K dataset is used to pre-train a generative model, and then the transfer learning technique is used to train separate models on xView and DOTA validation datasets. Perceptual Index (PI) and Root Mean Squared Error (RMSE) which are used on European Conference on Computer Vision -Perceptual Image Restoration and Manipulation Workshop 2018 are computed as the performance metrics. We have seen that the model which gives the best PI results, i.e. better perceptual quality, on xView and DOTA validation datasets is the one trained using the DIV2K dataset and the model which gives the best RMSE results, i.e. better reconstruction quality, is the one trained using the transfer learning technique.

Medical image super-resolution using a relativistic average generative adversarial network

The single image super resolution recovers missing high resolution details so as to reconstruct a high resolution image from a single low resolution image. This paper proposes a novel directionally adaptive, learning-based, single image super resolution method using multiple direction wavelet transform, called directionlets. Here, critically sampled directionlets are used to capture directional features effectively and to extract edge information along different directions from a set of available high resolution images. This information is used as the training set for super resolving a low resolution input image. The directionlet coefficients at finer scales of its high resolution image are learned locally from this training set and the inverse directionlet transform recovers the super resolved high resolution image. The simulation results showed that the proposed directionlet approach outperforms standard interpolation techniques like cubic spline interpolation as well as standard wavelet-based learning, both visually and in terms of the mean squared error (MSE) values. The SNR scores for cubic spline interpolation, wavelet and directionlet method are 13.6998 dB, 23.8324 dB and 30.8654 dB respectively for Barbara.

This paper provides an in-depth review of recent advancements in image super-resolution (SR) techniques using deep learning, with a specific focus on their applications in medical imaging. In the medical field, high-resolution images are crucial for accurate diagnosis and treatment planning. However, due to hardware limitations, it is often challenging to obtain high-resolution (HR) medical images. Traditional image enhancement techniques are not always sufficient to recover fine details. Deep learning-based methods, particularly those leveraging convolutional neural networks (CNNs), generative adversarial networks (GANs), and transformer-based models, have shown remarkable success in overcoming these challenges. The review explores the evolution of these techniques, their advantages, and the challenges involved in medical image super-resolution. The paper also highlights the future directions of research to improve SR for medical imaging, addressing concerns related to image quality, data availability, and computational efficiency.

Medical imaging plays an essential role in modern healthcare, providing non-invasive tools for diagnosing and monitoring various medical conditions. However, the resolution limitations of imaging hardware often result in suboptimal images, which can hinder the precision of clinical decision-making. Single image super-resolution (SISR) techniques offer a solution by reconstructing high-resolution (HR) images from low-resolution (LR) counterparts, enhancing the visual quality of medical images. In this paper, we propose an enhanced Residual Feature Learning Network (RFLN) tailored specifically for medical imaging. Our contributions include replacing the residual local feature blocks with standard residual blocks, increasing the model depth for improved feature extraction, and incorporating enhanced spatial attention (ESA) mechanisms to refine the feature selection. Extensive experiments on medical imaging datasets demonstrate that the proposed model achieves superior performance in terms of both quantitative metrics, such as PSNR and SSIM, and qualitative visual quality compared to existing state-of-the-art models. The enhanced RFLN not only effectively mitigates noise but also preserves critical anatomical details, making it a promising solution for high-precision medical imaging applications.

SRDiff: Single image super-resolution with diffusion probabilistic models

Nowadays, satellite images are used in various governmental applications, such as urbanization and monitoring the environment. Spatial resolution is an element of crucial impact on the usage of remote sensing imagery. As such, increasing the spatial resolution of an image is an important pre-processing step that can improve the performance of various image processing tasks, such as segmentation. Once a satellite is launched, the more practical solution to improve the resolution of its captured images is to use Single Image Super Resolution (SISR) techniques. In the recent years, Deep Convolutional Neural Networks (DCNNs) have been recognized as a highly effective tool to reconstruct a High Resolution (HR) image from its Low Resolution (LR) counterpart, which is an open problem due to the inherent difficulty of estimating the missing high frequency components. The aim of this research paper is to design and implement a satellite image SISR algorithm by estimating high frequency details through training Deep Convolutional Neural Network (DCNNs) with respect to wavelet analysis. The goal is to improve the spatial resolution of multispectral remote sensing images captured by DubaiSat-2 satellite. The accuracy of the proposed algorithm is assessed using several metrics such as Peak Signal-to-Noise Ratio (PSNR), Wavelet-based Signal-to-Noise Ratio (WSNR) and Structural Similarity Index Measurement (SSIM).

Tremendous progress has been witnessed on single image super-resolution (SR), where existing deep SR models achieve impressive performance in objective criteria, e.g., PSNR and SSIM. However, most of the SR methods are limited in visual perception, for example, they look too smooth. Generative adversarial network (GAN) favors SR visual effects over most of the deep SR models but is poor in objective criteria. In order to trade off the objective and subjective SR performance, we design a joint-attention discriminator with which GAN improves the SR performance in PSNR and SSIM, as well as maintaining the visual effect compared with non-attention GAN based SR models. The joint-attention discriminator contains dense channel-wise attention and cross-layer attention blocks. The former is applied in the shallow layers of the discriminator for channel-wise weighting combination of feature maps. The latter is employed to select feature maps in some middle and deep layers for effective discrimination. Extensive experiments are conducted on six benchmark datasets and the experimental results show that our proposed discriminator combining with different generators can achieve more realistic visual performances.

Abstract: Restoring damaged character photographs (DCPs) is essential for preserving cultural and personal memories, yet traditional methods are labor-intensive and often require significant manual intervention. The degradation of photographs due to factors such as creases, spots, water damage, and exposure to light poses a considerable challenge to conventional restoration techniques. Manual restoration, typically performed using software like Photoshop, is not only time-consuming but also demands a high level of expertise and meticulous effort, making it impractical for large-scale applications. In this study, we propose an innovative solution to automate the restoration process by employing a practical generative adversarial network (GAN) architecture. Our approach addresses the limitations of traditional methods and aims to significantly reduce the time and effort required for photo restoration while improving the quality of the restored images. The proposed method comprises a novel three-step pipeline designed to tackle the complexities of DCPs effectively. First, we initiate the process by collecting a diverse dataset, including clear character photographs (CCPs), real DCPs, and dirty masks that illustrate common damage patterns. This comprehensive dataset forms the foundation for training our models. Second, we utilize a Residual U-Net GAN (RUGAN) to learn the spoilage patterns of DCPs and generate realistic fake DCPs from the CCPs and dirty masks. RUGAN leverages the structural and textural information from the clear images and dirty masks to produce fake damaged images that closely mimic real-world damage. Finally, we train a restoration model known as the Residual U-Net conditional GAN (RUCGAN) using the paired fake DCPs and CCPs.

Image super-resolution with sparsity prior provides promising performance. However, traditional sparse-based super resolution methods transform a two dimensional (2D) image into a one dimensional (1D) vector, which ignores the intrinsic 2D structure as well as spatial correlation inherent in images. In this paper, we propose the first image super-resolution method which reconstructs a high resolution image from its low resolution counterpart via a two dimensional sparse model. Correspondingly, we present a new dictionary learning algorithm to fully make use of the corresponding relationship of two pairs of 2D dictionaries of low and high resolution images, respectively. Experimental results demonstrate that our proposed image super-resolution with 2D sparse model outperforms state-of-the-art 1D sparse model based super resolution methods in terms of both reconstruction ability and memory usage.