DHBSR: A deep hybrid representation-based network for blind image super resolution

Abstract. As deep learning models advance, their use in Super Resolution (SR) tasks has become pivotal for enhancing remote sensing low-resolution (LR) satellite images. However, the decision making processes within these models remain opaque, especially in remote sensing applications where transparency is critical. This paper focuses on applying Explainable Artificial Intelligence (XAI) techniques, particularly Local Attribution Maps (LAMs), to analyze and interpret the internal behavior of both general purpose and remote sensing specific SR neural networks. General purpose models like Generative Adversarial Network for Super Resolution (SRGAN), Enhanced Deep Super-Resolution Network (EDSR), Efficient Super-Resolution Transformer (ESRT), and Hybrid-Attention Transformer (HAT), although highly effective in SR tasks, were originally designed for broader image enhancement challenges; in contrast, Hybrid-Scale Self-Similarity Exploitation Network (HSENet) and Multi-scale Enhanced Network (MEN) are tailored for the unique complexities of remote sensing, such as varied textures and intricate scene features. By leveraging LAMs, we highlight how different networks prioritize and process features such as edges, textures, and high frequency details to generate super resolved outputs. The comparative study between general purpose and remote sensing specific networks outlines each model’s strengths and weaknesses in managing the data present in remote sensing imagery. This approach addresses the previously unexplored application of LAMs in remote sensing for SR, contributing to the research in this field and providing deeper insights into the interpretability and transparency of widely used SR models. Furthermore, by drawing parallels between feature attribution in SR and classification tasks, we suggest new pathways for integrating semantic information to refine model transparency and performance in remote sensing applications.

In recent years, the field of image super-resolution has mainly focused on the single-image super-resolution (SISR) task, which is to estimate an HR image from a single LR input. Due to the ill-posed ness of the SISR problem, these methods are limited to increasing the high-frequency details of the image by learning the a priori of the image. And multi-frame super-resolution (MFSR) provides the possibility to reconstruct rich details using the spatial and temporal difference information between images. With the increasing popularity of array camera technology, this key advantage makes MFSR an important issue for practical applications. We propose a new structure to complete the task of multi-frame image super-resolution. Our network takes multiple noisy images as input and generates a denoised, super-resolution RGB image as output. First, we align the multi-frame images by estimating the dense pixel optical flow between the images, and construct an adaptive fusion module to fuse the information of all frames. Then we build a feature fusion network to simultaneously fuse the depth feature information of multiple LR images and the internal features of the initial high-resolution image. In order to evaluate real-world data, We use the BurstSR data set, which includes real images of smartphones and highresolution SLR cameras, to prove the effectiveness of the proposed multiframe image super-resolution algorithm.

Deep learning techniques not only have surpassed humans in several computer vision tasks, but also have achieved state-of-the-art performance on the super-resolution (SR) task, which enhances the resolution of reconstructed images from the observed low-resolution (LR) images. Very Deep Super- Resolution (VDSR) is a popular architecture with decent performance on the SR task. In general, the deeper the VDSR network, the better the performance. Say, on a computing device with limited computational resources, the SR task must support multiple resolutions or multiple output rates. For this multi-rate SR task, the state-of-the-art architectures require multiple networks with varying depths. However as the number of supported rates increases, so does the number of networks imposing incremental computational burden on the computing device. We propose a depth-controllable network and training principles for the multi-rate SR task. The proposed network is configured as a single network regardless of the number of supported rates. The inverse auxiliary loss and contiguous/progressive skip connections are presented to train the network end-to-end throughout the varying number of layers without biasing the performances of specific depths. With three data sets and three scaling factors, the proposed network is compared to the baseline network VDSR, along with the Super-Resolution Convolutional Neural Network (SRCNN). Our network not only requires a single network at varying rates, but also performs as well as the baseline networks in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). Our model outperforms the baseline networks when the depth of layers is more than eleven.

Recently, significant progress has been made for the task of image super-resolution (SR). However, existing methods show that stacking convolution layers blindly leads to overwhelming parameters and high computational complexities. Besides, the conventional feed-forward architectures can hardly fully exploit the mutual dependencies between low- and high-resolution images. Motivated by these observations, we first propose a novel architecture by taking advantage of recursive learning. Based on dual-path block (DPB), we enhance the network performance simply by increasing the number of DPB recursions and avoiding additional parameters. Moreover, in contrast to most convolutional neural network (CNN) methods with one state, our network is endowed with two states (low-resolution state and high-resolution state) transformed mutually. We exploit the relationship between the two states by introducing a back-projection operation which calculates the differences between the two states for a better result. Extensive experiments show that our method outperforms the state-of-the-art methods.

Single image super-resolution (SISR) with deep convolutional neural networks has recently attracted increasing attention due to its potentials to generate rich details. To obtain better fidelity and visual quality, most of existing methods are of heavy design with the depth of network. However, the lack of high-frequency information in the deep network and the limit of non-local context relations of pixels remain the core issue. In addition, there are many arbitrary-scale image super-resolution tasks, so it is a focus that provides an effective and efficient model for super resolution with arbitrary-scale. To this end, we propose a Feature Preserving and Enhancing Network (FPEN) based on implicit representation, which is aims to preserving high-frequency information in deep network and enhancing non-local contextual features of pixels to output more realistic images at arbitrary scales. In particular, our proposed High Frequency Preserving Block (HFPB) can divide the features into high-frequency components and low-frequency components, and allocate more operations to high-frequency components to ensure that high-frequency components can be preserved in the network. Since high-frequency information contains the details and texture of the image, it can restore the finer details of the image. Moreover, Pixel Continuity Attention (PiCA) module our proposed utilizes visual cues observed from pixels to adaptively recalibrate the pixel range that needs attention in the image to better achieve feature enhancement and generate smoother images. Extensive experiments conducted on benchmark SISR models and datasets show that Feature Preserving and Enhancing Network can be employed for various SISR tasks with arbitrary-scale to obtain the better visual quality than other state-of-the-art SR algorithms.

High-resolution optical images will provide planetary geology researchers with finer and more microscopic image data information. In order to maximize scientific output, it is necessary to further increase the resolution of acquired images, so image super-resolution (SR) reconstruction techniques have become the best choice. Aiming at the problems of large parameter quantity and high computational complexity in current deep learning-based image SR reconstruction methods, we propose a novel Recursive Swin Transformer Super-Resolution Network (RSTSRN) for SR applied to images. The RSTSRN improves upon the LapSRN, which we use as our backbone architecture. A Residual Swin Transformer Block (RSTB) is used for more efficient residual learning, which consists of stacked Swin Transformer Blocks (STBs) with a residual connection. Moreover, the idea of parameter sharing was introduced to reduce the number of parameters, and a multi-scale training strategy was designed to accelerate convergence speed. Experimental results show that the proposed RSTSRN achieves superior performance on 2×, 4× and 8×SR tasks to state-of-the-art methods with similar parameters. Especially on high-magnification SR tasks, the RSTSRN has great performance superiority. Compared to the LapSRN network, for 2×, 4× and 8× Mars image SR tasks, the RSTSRN network has increased PSNR values by 0.35 dB, 0.88 dB and 1.22 dB, and SSIM values by 0.0048, 0.0114 and 0.0311, respectively.

Deep convolutional neural networks (CNNs) have recently achieved great success for single image super-resolution (SISR) task due to their powerful feature representation capabilities. The most recent deep learning based SISR methods focus on designing deeper / wider models to learn the non-linear mapping between low-resolution (LR) inputs and high-resolution (HR) outputs. These existing SR methods do not take into account the image observation (physical) model and thus require a large number of network's trainable parameters with a great volume of training data. To address these issues, we propose a deep Iterative Super-Resolution Residual Convolutional Network (ISRResCNet) that exploits the powerful image regularization and large-scale optimization techniques by training the deep network in an iterative manner with a residual learning approach. Extensive experimental results on various super-resolution benchmarks demonstrate that our method with a few trainable parameters improves the results for different scaling factors in comparison with the state-of-art methods.

Convolutional neural network (CNN) has achieved unprecedented success in image super-resolution tasks in recent years. However, the network's performance depends on the distribution of the training sets and degrades on out-of-distribution samples. This paper adopts a Bayesian approach for estimating uncertainty associated with output and applies it in a deep image super-resolution model to address the concern mentioned above. We use the uncertainty estimation technique using the batch-normalization layer, where stochasticity of the batch mean and variance generate Monte-Carlo (MC) samples. The MC samples, which are nothing but different super-resolved images using different stochastic parameters, reconstruct the image, and provide a confidence or uncertainty map of the reconstruction. We propose a faster approach for MC sample generation, and it allows the variable image size during testing. Therefore, it will be useful for image reconstruction domain. Our experimental findings show that this uncertainty map strongly relates to the quality of reconstruction generated by the deep CNN model and explains its limitation. Furthermore, this paper proposes an approach to reduce the model's uncertainty for an input image, and it helps to defend the adversarial attacks on the image super-resolution model. The proposed uncertainty reduction technique also improves the performance of the model for out-of-distribution test images. To the best of our knowledge, we are the first to propose an adversarial defense mechanism in any image reconstruction domain.

Improved dual-scale residual network for image super-resolution

Traditional image super-resolution and deblurring methods typically handle resolution enhancement and blur removal separately, and often assume a single type of degradation. However, in real-world scenarios, images frequently suffer from multiple simultaneous degradations, such as motion blur and resolution loss, which these methods struggle to address effectively. Specifically, classical super-resolution methods tend to amplify blur artifacts when increasing resolution, while deblurring methods may lose structural details during restoration. To overcome these limitations, this paper proposes a novel Structure Preservation and Interactive Fusion Network (SPIFNet) that integrates semantic priors from deep learning with edge-aware structural guidance. First, based on the prior knowledge that edge information is crucial for image super-resolution and deblurring tasks, a structure-preserving module based on edge maps is proposed. This module utilizes edge-aided information and multi-scale edge constraints to enhance the structural and detail preservation capabilities of the reconstruction network. Additionally, to obtain accurate edge information of blurry images for building the structure-preserving module, a multi-branch fusion-structured edge extraction network is introduced. Furthermore, to fully utilize the relationship between image deblurring and super-resolution tasks, an edge-guided interactive fusion module is proposed to adaptively fuse feature information from deblurring and super-resolution tasks. To improve the network’s capacity to represent features, local and global attention methods are used. The suggested approach for super-resolution of fuzzy images has been rigorously validated by both quantitative and qualitative testing findings.

Hyperspectral images (HSI) contain rich spectral information and are highly valuable in applications such as remote sensing monitoring and target identification. However, due to limitations in imaging technology and cost constraints, existing HSI generally suffer from low spatial resolution. Recently, several hyperspectral vision foundation models have been proposed, employing large-scale network architectures and newly constructed large-scale datasets, achieving remarkable performance in HSI super resolution (SR) and other hyperspectral vision tasks. However, as these general-purpose foundation models are designed to accommodate diverse vision tasks, their architectures exhibit inherent limitations when applied specifically to SR as a low-level vision task. To address this architectural limitation, we propose HyperAdapter, an enhanced network based on HyperSIGMA that employs task-specific architectural optimization to fully exploit the performance potential of foundation models for SR tasks. Specifically, we introduce a spatial-spectral prior module to improve the model’s ability to capture local spatial context and spectral correlations in HSI, along with an adapter module to effectively integrate and refine the spatial and spectral features of the base model, ensuring better adaptation to the HSI SR task. Experiments on three public HSI SR datasets demonstrate that the proposed method significantly improves reconstruction performance, validating its superiority and practical utility.

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

Recently, Transformer-based methods have gained prominence in image super-resolution (SR) tasks, addressing the challenge of long-range dependence through the incorporation of cross-layer connectivity and local attention mechanisms. However, the analysis of these networks using local attribution maps has revealed significant limitations in leveraging the spatial extent of input information. To unlock the inherent potential of Transformer in image SR, we propose the Multi-Attention Fusion Transformer (MAFT), a novel model designed to integrate multiple attention mechanisms with the objective of expanding the number and range of pixels activated during image reconstruction. This integration enhances the effective utilization of input information space. At the core of our model lies the Multi-attention Adaptive Integration Groups, which facilitate the transition from dense local attention to sparse global attention through the introduction of Local Attention Aggregation and Global Attention Aggregation blocks with alternating connections, effectively broadening the network's receptive field. The effectiveness of our proposed algorithm has been validated through comprehensive quantitative and qualitative evaluation experiments conducted on benchmark datasets. Compared to state-of-the-art methods (e.g. HAT), the proposed MAFT achieves 0.09 dB gains on Urban100 dataset for × 4 SR task while containing 32.55% and 38.01% fewer parameters and FLOPs, respectively.

Image super-resolution is a research endeavour that has gained notoriety in computer vision. The research goal is to increase the spatial dimensions of an image using corresponding low-resolution and high-resolution image pairs to enhance the perceptual quality. The challenge of maintaining such perceptual quality lies in developing appropriate algorithms that learn to reconstruct higher-quality images from their lower-resolution counterparts. Recent methods employ deep learning algorithms to reconstruct textural details prevalent in low-resolution images. Since corresponding image pairs are non-trivial to collect, researchers attempt super-resolution by creating synthetic low-resolution representations of high-resolution images. Unfortunately, such methods employ ineffective downscaling operations to achieve synthetic low-resolution images. These methods fail to generalize well on real-world images that may suffer different degradations. A different angle is offered to solve the task of image super-resolution by investigating the plausibility of learning the degradational operation using generative adversarial networks. A two-stage generative adversarial network along with two architectural variations is proposed to solve the task of real-world super-resolution from low-resolution images with unknown degradations. It is demonstrated that learning to downsample images in a weakly supervised manner is an impactful and viable approach for super-resolution.

Recently, convolutional neural network (CNN) based models have shown great potential in the task of single image super-resolution (SISR). However, many state-of-the-art SISR solutions are reproducing some tricks proven effective in other vision tasks, such as pursuing a deeper model. In this paper, we propose a new solution (named as Multi-Receptive-Field Network - MRFN), which outperforms existing SISR solutions in three different aspects. First, from receptive field: a novel multi-receptive-field (MRF) module is proposed to extract and fuse features in different receptive fields from local to global. Integrating these hierarchical features can generate better mappings on recovering high-fidelity details at different scales. Second, from network architectures: both dense skip connections and deep supervision are utilized to combine features from the current MRF module and preceding ones for training more representative features. Moreover, a deconvolution layer is embedded at the end of the network to avoid artificial priors induced by numerical data pre-processing (e.g., bicubic stretching), and speed up the restoration process. Finally, from error modeling: different from $L1$ and $L2$ loss functions, we proposed a novel two-parameter training loss called Weighted Huber loss function which can adaptively adjust the value of back-propagated derivative according to the residual value, thus fit the reconstruction error more effectively. Extensive qualitative and quantitative evaluation results on benchmark datasets demonstrate that our proposed MRFN can achieve more accurate recovering results than most state-of-the-art methods with significantly less complexity.