An Efficient Deep Unsupervised Superresolution Model for Remote Sensing Images

In many practical situations, images are not only down sampled but also compressed for efficient transmission and storage. JPEG and MPEG-2 compressions often introduce blocking artifacts because they process the data as 8 × 8 blocks. Many of the existing super resolution (SR) methods assume low resolution images as a down sampled version of high resolution (HR) image, and neglect the degradation due to compression. This exacerbates artifacts in the SR image and reduces the user experience. To address the joint deblocking and SR (DbSR), a novel deep network with dense skip connections and dilated convolutions is proposed in this paper, and we name it as DenseDbSR. Recently, many researchers have proposed deeper networks and achieved improvement in the SR performance. However, training deeper networks is very challenging because of the problem of vanishing gradients. Simply increasing the depth of the network leads to cumbersome computational costs. To enlarge the field-of-view (FOV) without increasing the computational cost, the dilated convolution is used. The dilated convolution exponentially expands the FOV and helps to exploit the contextual information efficiently. Moreover, the dense skip connections create short paths for gradients to be back-propagated efficiently and alleviates the problem of vanishing gradients. Furthermore, the network is relieved from the training burden by learning residuals of the SR image instead of learning raw images. From the conducted extensive experimentation, the proposed DenseDbSR network produced better performance in terms of PSNR and SSIM than the compared state-of-the-art methods.

In recent years, research on increasing the spatial resolution and enhancing the quality of satellite images using the deep learning-based super-resolution (SR) method has been actively conducted. In a remote sensing field, conventional SR methods required high-quality satellite images as the ground truth. However, in most cases, high-quality satellite images are difficult to acquire because many image distortions occur owing to various imaging conditions. To address this problem, we propose an adaptive image quality modification method to improve SR image quality for the KOrea Multi-Purpose Satellite-3 (KOMPSAT-3). The KOMPSAT-3 is a high performance optical satellite, which provides 0.7-m ground sampling distance (GSD) panchromatic and 2.8-m GSD multi-spectral images for various applications. We proposed an SR method with a scale factor of 2 for the panchromatic and pan-sharpened images of KOMPSAT-3. The proposed SR method presents a degradation model that generates a low-quality image for training, and a method for improving the quality of the raw satellite image. The proposed degradation model for low-resolution input image generation is based on Gaussian noise and blur kernel. In addition, top-hat and bottom-hat transformation is applied to the original satellite image to generate an enhanced satellite image with improved edge sharpness or image clarity. Using this enhanced satellite image as the ground truth, an SR network is then trained. The performance of the proposed method was evaluated by comparing it with other SR methods in multiple ways, such as edge extraction, visual inspection, qualitative analysis, and the performance of object detection. Experimental results show that the proposed SR method achieves improved reconstruction results and perceptual quality compared to conventional SR methods.

Wavelet based medical image super resolution using cross connected residual-in-dense grouped convolutional neural network

Most of the previous sparse coding (SC) based super resolution (SR) methods partition the image into overlapped patches, and process each patch separately. These methods, however, ignore the consistency of pixels in overlapped patches, which is a strong constraint for image reconstruction. In this paper, we propose a convolutional sparse coding (CSC) based SR (CSC-SR) method to address the consistency issue. Our CSC-SR involves three groups of parameters to be learned: (i) a set of filters to decompose the low resolution (LR) image into LR sparse feature maps, (ii) a mapping function to predict the high resolution (HR) feature maps from the LR ones, and (iii) a set of filters to reconstruct the HR images from the predicted HR feature maps via simple convolution operations. By working directly on the whole image, the proposed CSC-SR algorithm does not need to divide the image into overlapped patches, and can exploit the image global correlation to produce more robust reconstruction of image local structures. Experimental results clearly validate the advantages of CSC over patch based SC in SR application. Compared with state-of-the-art SR methods, the proposed CSC-SR method achieves highly competitive PSNR results, while demonstrating better edge and texture preservation performance.

The thermal camera systems can be used in all kinds of applications that require the detection of heat change, but thermal imaging systems are highly costly systems. In recent years, developments in the field of deep learning have increased the success by obtaining quality results compared to traditional methods. In this paper, thermal images of neonates (healthy - unhealthy) obtained from a high-resolution thermal camera were used and these images were evaluated as high resolution (ground truth) images. Later, these thermal images were downscaled at 1/2, 1/4, 1/8 ratios, and three different datasets consisting of low-resolution images in different sizes were obtained. In this way, super-resolution applications have been carried out on the deep network model developed based on generative adversarial networks (GAN) by using three different datasets. The successful performance of the results was evaluated with PSNR (peak signal to noise ratio) and SSIM (structural similarity index measure). In addition, healthy - unhealthy classification application was carried out by means of a classifier network developed based on convolutional neural networks (CNN) to evaluate the super-resolution images obtained using different datasets. The obtained results show the importance of combining medical thermal imaging with super-resolution methods.

Arbitrary scale super-resolution diffusion model for brain MRI images

This paper presents a new quality assessment parameter for the evaluation of deep learning based super resolution techniques applied on thermal images. Three widely used deep learning-based models namely Super-Resolution Convolutional Neural Network (SRCNN), Thermal Enhancement Network (TEN) and Very Deep Super Resolution (VDSR) have been implemented for achieving super resolution of thermal images. Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) have been the most widely used conventional evaluation metrics for performance measurement of super resolution algorithms. Since these parameters require a reference image for the evaluation of the resultant images, we propose a new quality assessment parameter based on strength of the edges. Edge detection of the super resoluted image is performed utilizing Canny Edge Detection method and the entropy of the edge detection image is computed to provide a new parameter, Edge Detection Entropy Score (EDES). For the comparison and validation of the proposed image quality assessment techniques, Mean Opinion Score (MOS) of the target images have been obtained to be used as a benchmark. The obtained results indicate that the proposed EDES of the super resoluted images has high degree of correlation with the MOS as well as PSNR and SSIM.

Full waveform inversion (FWI) is a powerful tool for estimating the underground velocity model. However, it is computationally expensive and the resulting models tend to be not accurate enough. Thus, to improve the efficiency and accuracy of FWI, we propose a super-resolution (SR) method based on deep learning to enhance the resolution of the seismic velocity model. Since the edge images of the seismic velocity model are also widely used in geophysics, a multitask learning (MTL) network with hard parameter sharing is applied to perform the SR of the seismic velocity model and its edge images. The proposed MTL model dubbed M-RUDSR includes a global residual skip connection, an encoder-decoder structure of U-Net, and a dense skip connection structure. Besides, two networks for comparison, namely, RUDSR and M-RUSR, are proffered. RUDSR is a single-task version of M-RUDSR, whereas M-RUSR is a simplified version of M-RUDSR without a dense skip connection structure. Compared with RUDSR and M-RUSR, M-RUDSR produced the best results for all kinds of blurring levels and achieved better visual details. We found that FWI followed by SR can help reduce the computational cost of FWI in the high-frequency part of the spectrum, as well as achieve better high-frequency details recovery. The experimental results show that M-RUDSR is a practical recovery scheme in SR of the seismic velocity model and can be applied to a real data set efficiently.

Tremendous progress has been witnessed on single image super-resolution (SR), where existing deep SR models achieve impressive performance in objective criteria, e.g., PSNR and SSIM. However, most of the SR methods are limited in visual perception, for example, they look too smooth. Generative adversarial network (GAN) favors SR visual effects over most of the deep SR models but is poor in objective criteria. In order to trade off the objective and subjective SR performance, we design a joint-attention discriminator with which GAN improves the SR performance in PSNR and SSIM, as well as maintaining the visual effect compared with non-attention GAN based SR models. The joint-attention discriminator contains dense channel-wise attention and cross-layer attention blocks. The former is applied in the shallow layers of the discriminator for channel-wise weighting combination of feature maps. The latter is employed to select feature maps in some middle and deep layers for effective discrimination. Extensive experiments are conducted on six benchmark datasets and the experimental results show that our proposed discriminator combining with different generators can achieve more realistic visual performances.

Abstract. As deep learning models advance, their use in Super Resolution (SR) tasks has become pivotal for enhancing remote sensing low-resolution (LR) satellite images. However, the decision making processes within these models remain opaque, especially in remote sensing applications where transparency is critical. This paper focuses on applying Explainable Artificial Intelligence (XAI) techniques, particularly Local Attribution Maps (LAMs), to analyze and interpret the internal behavior of both general purpose and remote sensing specific SR neural networks. General purpose models like Generative Adversarial Network for Super Resolution (SRGAN), Enhanced Deep Super-Resolution Network (EDSR), Efficient Super-Resolution Transformer (ESRT), and Hybrid-Attention Transformer (HAT), although highly effective in SR tasks, were originally designed for broader image enhancement challenges; in contrast, Hybrid-Scale Self-Similarity Exploitation Network (HSENet) and Multi-scale Enhanced Network (MEN) are tailored for the unique complexities of remote sensing, such as varied textures and intricate scene features. By leveraging LAMs, we highlight how different networks prioritize and process features such as edges, textures, and high frequency details to generate super resolved outputs. The comparative study between general purpose and remote sensing specific networks outlines each model’s strengths and weaknesses in managing the data present in remote sensing imagery. This approach addresses the previously unexplored application of LAMs in remote sensing for SR, contributing to the research in this field and providing deeper insights into the interpretability and transparency of widely used SR models. Furthermore, by drawing parallels between feature attribution in SR and classification tasks, we suggest new pathways for integrating semantic information to refine model transparency and performance in remote sensing applications.

Image explicit prior has made breakthrough progress in the super-resolution (SR) due to the additional supervisory information provided. However, existing explicit prior-guided SR methods directly use the Gaussian gradient or Laplacian gradient prior, which cannot fit the gradient distribution of remote sensing images. Through the statistics of gradient probability density distribution of the remote sensing image dataset, we found that the hyper-Laplacian prior can fit the heavy-tailed distribution better, which aroused us to use the hyper-Laplacian before facilitating the SR reconstruction. We propose a novel hyper-Laplacian prior SR method for remote sensing images in this manuscript. Specifically, our model consists of three components: rough reconstruction subnetwork (RRS), hyper-Laplacian prior subnetwork (HPS), and image refinement enhancement subnetwork (RES). In the RRS, we reconstruct low-resolution (LR) images into rough SR images by a set of resblocks. In the HPS, we first introduce the hyper-Laplacian prior for LR images to provide an additional texture. Hereafter, we set up a prior loss which imposes a second-order supervision on the SR image. Like the previous image space loss function, it helps the model to gather the geometric structure of the image. Finally, the outputs of the RRS and HPS are fused and then fed to the RES for high-quality image reconstruction. Numerous studies of SR reconstruction and segmentation on UCMerced, PatternNet, and OpenBayes datasets confirm that our method is superior compared to state-of-the-art methods.

Today, Big Data brings benefits to many areas of scientific research. However, processing these large amounts of data often requires extensive computing time and a large storage space. Global feature analysis is considered to be universal in traditional super resolution methods, but it is not applicable to Big Data. There remains a viewpoint that it is not necessary to address all data equally and impartially. Focusing on useful information can make the massive data analysis possible and more effective. In this paper, we consider the significant regions, and thus, we propose a new super resolution approach that uses significant and general information. Under the framework of a convolutional neural network, the training process is performed on the significant parts of the training data set, and the reconstruction process considers significant parts separately; then, a super resolution image will be obtained according to each different demand. This concept is easy to understand, but it can be achieved only via the Big Data approach with many similar images on the Internet and the effective deep learning algorithm. Experiments show that our new approach can reduce the testing time and obtain a high-quality reconstructed image.

Deep Feature Blend Attention: A New Frontier in Super Resolution Image Generation

Super-resolution (SR) brings an excellent opportunity to improve a wide range of different remote sensing applications. SR techniques are concerned about increasing the image resolution while providing finer spatial details than those captured by the original acquisition instrument. Therefore, SR techniques are particularly useful to cope with the increasing demand remote sensing imaging applications requiring fine spatial resolution. Even though different machine learning paradigms have been successfully applied in SR, more research is required to improve the SR process without the need of external high-resolution (HR) training examples. This paper proposes a new convolutional generator model to super-resolve low-resolution (LR) remote sensing data from an unsupervised perspective. That is, the proposed generative network is able to initially learn relationships between the LR and HR domains throughout several convolutional, downsampling, batch normalization, and activation layers. Then, the data are symmetrically projected to the target resolution while guaranteeing a reconstruction constraint over the LR input image. An experimental comparison is conducted using 12 different unsupervised SR methods over different test images. Our experiments reveal the potential of the proposed approach to improve the resolution of remote sensing imagery.

Convolutional neural network (CNN) has made significant progress in image superresolution (SR), which could thrash the limits of image spatial resolution. Recently, abundant CNN-based methods have been proposed for the remote sensing image SR; however, the usages of complex structures and coarse manners could introduce excessive learnable parameters and ignorance of heterogeneous image details, respectively. In this article, we propose a recurrently complicated lightweight network (RCL-Net) for SR image recovery, through procedures of recurrent fluctuated complexity. We design a serial of the progressive complicated block (PC-Block) in the RCL-Net, and each PC-Block is composed of three complicated lightweight branches (CL-Branches) with increasing complexities in order, for recovering heterogeneous image details. Meanwhile, the CL-Branch is integrated with a multireceptive field module (MRF-Module) to more efficiently recover intact images through forward propagation paths of heterogeneous routes and lengths, where the excessive interactive calculations between feature subparts are constrained to reduce learnable parameters. In this manner, the proposed RCL-Net achieves a tradeoff between model complexities of traditional powerful structures, such as coarse-to-fine manners, and SR performances. Plentiful experiments with excellent results grounded on popular datasets exactly demonstrate the superiority of our proposed network, which even surpasses the advanced large SR model with less than 3% learnable parameters, compared to the state-of-the-art lightweight methods.