Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

The progress on Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. HSIs usually have a high number of spectral bands, so accurately modeling spectral band interaction for HSI SR is hard. Also, training data for HSI SR is hard to obtain so the dataset is usually rather small. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance. In order to better support our test-time training method, we also propose a new network architecture to learn HSI SR without modeling spectral band interaction and propose a new data augmentation method Spectral Mixup to increase the diversity of the training data at test time. We also collect a new HSI dataset with a diverse set of images of interesting objects ranging from food to vegetation, to materials, and to general scenes. Extensive experiments on multiple datasets show that our method can improve the performance of pre-trained models significantly after test-time training and outperform competing methods significantly for HSI SR.

Super-resolution (SR) is significant for hyperspectral image (HSI) applications. In single-frame HSI SR, how to reconstruct detailed image structures in high resolution (HR) HSI is challenging since there is no auxiliary image (e.g., HR multispectral image) providing structural information. Wavelet could capture image structures in different orientations, and emphasis on predicting high-frequency wavelet sub-bands is helpful for recovering the detailed structures in HSI SR. In this study, we propose a multi-scale wavelet 3D convolutional neural network (MW-3D-CNN) for HSI SR, which predicts the wavelet coefficients of HR HSI rather than directly reconstructing the HR HSI. To exploit the correlation in the spectral and spatial domains, the MW-3D-CNN is built with 3D convolutional layers. An embedding subnet and a predicting subnet constitute the MW-3D-CNN, the embedding subnet extracts deep spatial-spectral features from the low resolution (LR) HSI and represents the LR HSI as a set of feature cubes. The feature cubes are then fed to the predicting subnet. There are multiple output branches in the predicting subnet, each of which corresponds to one wavelet sub-band and predicts the wavelet coefficients of HR HSI. The HR HSI can be obtained by applying inverse wavelet transform to the predicted wavelet coefficients. In the training stage, we propose to train the MW-3D-CNN with L1 norm loss, which is more suitable than the conventional L2 norm loss for penalizing the errors in different wavelet sub-bands. Experiments on both simulated and real spaceborne HSI demonstrate that the proposed algorithm is competitive with other state-of-the-art HSI SR methods.

Hyperspectral images (HSI) contain rich spectral information and are highly valuable in applications such as remote sensing monitoring and target identification. However, due to limitations in imaging technology and cost constraints, existing HSI generally suffer from low spatial resolution. Recently, several hyperspectral vision foundation models have been proposed, employing large-scale network architectures and newly constructed large-scale datasets, achieving remarkable performance in HSI super resolution (SR) and other hyperspectral vision tasks. However, as these general-purpose foundation models are designed to accommodate diverse vision tasks, their architectures exhibit inherent limitations when applied specifically to SR as a low-level vision task. To address this architectural limitation, we propose HyperAdapter, an enhanced network based on HyperSIGMA that employs task-specific architectural optimization to fully exploit the performance potential of foundation models for SR tasks. Specifically, we introduce a spatial-spectral prior module to improve the model’s ability to capture local spatial context and spectral correlations in HSI, along with an adapter module to effectively integrate and refine the spatial and spectral features of the base model, ensuring better adaptation to the HSI SR task. Experiments on three public HSI SR datasets demonstrate that the proposed method significantly improves reconstruction performance, validating its superiority and practical utility.

In this paper, a hyperspectral (HS) image super-resolution (SR) approach based on attention convolutional bi-long short-term memory (ConvBiLSTM) network is proposed, aiming to explore the collaborative spatial and spectral attention characteristics, thereby enhancing the spatial resolution of HS image. ConvBiLSTM network combines the spatial feature mining and sequential predicting abilities of convolutional neural network and recurrent neural network, respectively. We adapt the ConvBiLSTM network for our super-resolution purpose by regarding each band as a single frame of sequential data, and propose a band-sharing spatial-channel attention-combined ConvBiLSTM SR method to intensify the saliency features. Moreover, a spatial-regularized loss function is presented to further promote the fidelity of the super-resolved HS image. Experiments on four HS data sets show that the proposed approach outperforms some state-of-the-art HS image SR techniques, from the aspect of spectral fidelity.

The production of a high spatial resolution (HR) hyperspectral image (HSI) through the fusion of a low spatial resolution (LR) HSI with an HR multispectral image (MSI) has underpinned much of the recent progress in HSI super-resolution. The premise of these signs of progress is that both the degeneration from the HR HSI to LR HSI in the spatial domain and the degeneration from the HR HSI to HR MSI in the spectral domain are assumed to be known in advance. However, such a premise is difficult to achieve in practice. To address this problem, we propose to incorporate degeneration estimation into HSI super-resolution and present an unsupervised deep framework for "blind" HSIs super-resolution where the degenerations in both domains are unknown. In this framework, we model the latent HR HSI and the unknown degenerations with deep network structures to regularize them instead of using handcrafted (or shallow) priors. Specifically, we generate the latent HR HSI with an image-specific generator network and structure the degenerations in spatial and spectral domains through a convolution layer and a fully connected layer, respectively. By doing this, the proposed framework can be formulated as an end-to-end deep network learning problem, which is purely supervised by those two input images (i.e., LR HSI and HR MSI) and can be effectively solved by the backpropagation algorithm. Experiments on both natural scene and remote sensing HSI data sets show the superior performance of the proposed method in coping with unknown degeneration either in the spatial domain, spectral domain, or even both of them.

Hyperspectral image super-resolution (HSISR) is crucial for enhancing the spatial detail of hyperspectral images (HSIs). However, existing single HSI super-resolution (SR) methods face challenges with effective use of spectral context and generalization across varying spectral bands. We propose a dynamic recurrent self-refinement network (DRSN) to address these limitations. DRSN innovatively models the HSI SR process as a non-linear dynamic system evolving along the spectral dimension, where the spectral groups are treated as state variables. This formulation inherently allows DRSN to process HSIs with arbitrary band counts, thereby broadening the application scope of HSISR. The key contributions in DRSN can be summarized into two novel modules: adaptive state prediction-updating (ASPU) and bidirectional cross-state alignment (BCSA). The ASPU employs an uncertainty-guided adaptive activation mechanism to dynamically refine the current state by selectively integrating inter-state complementary information while suppressing irrelevant context. The BCSA utilizes an efficient back-and-forth caching strategy and masked inter&intra-state attention to mitigate information imbalance and achieve effective bidirectional contextual alignment without substantial computational overhead. Experiments on benchmark HSI datasets demonstrate that DRSN achieves state-of-the-art performance, exhibits wide applicability, and maintains a lightweight and computationally efficient structure. The source code is available at https://github.com/hgpftd/DRSN.

Limited by the existing imagery sensors, hyperspectral images are characterized by high spectral resolution but low spatial resolution. The super-resolution (SR) technique aiming at enhancing the spatial resolution of the input image is a hot topic in computer vision. In this paper, we present a hyperspectral image (HSI) SR method based on a deep information distillation network (IDN) and an intra-fusion operation. Specifically, bands are firstly selected by a certain distance and super-resolved by an IDN. The IDN employs distillation blocks to gradually extract abundant and efficient features for reconstructing the selected bands. Second, the unselected bands are obtained via spectral correlation, yielding a coarse high-resolution (HR) HSI. Finally, the spectral-interpolated coarse HR HSI is intra-fused with the input HSI to achieve a finer HR HSI, making further use of the spatial-spectral information these unselected bands convey. Different from most existing fusion-based HSI SR methods, the proposed intra-fusion operation does not require any auxiliary co-registered image as the input, which makes this method more practical. Moreover, contrary to most single-based HSI SR methods whose performance decreases significantly as the image quality gets worse, the proposal deeply utilizes the spatial-spectral information and the mapping knowledge provided by the IDN, which achieves more robust performance. Experimental data and comparative analysis have demonstrated the effectiveness of this method.

The hyperspectral image (HSI) distinguishes itself in material identification through its exceptional spectral resolution. However, its spatial resolution is constrained by hardware limitations, prompting the evolution of HSI super-resolution (SR) techniques. Single HSI SR endeavors to reconstruct high-spatial-resolution HSI from low-spatial-resolution inputs, and recent progress in deep learning-based algorithms has significantly advanced the quality of reconstructed images. However, convolutional methods struggle to extract comprehensive spatial and spectral features. Transformer-based models have yet to harness long-range dependencies across both dimensions fully, thus inadequately integrating spatial and spectral data. To solve the above problem, in this paper, we propose a new HSI SR method, SSAformer, which merges the strengths of CNNs and Transformers. It introduces specially designed attention mechanisms for HSI, including spatial and spectral attention modules, and overcomes the previous challenges in extracting and amalgamating spatial and spectral information. Evaluations on benchmark datasets show that SSAformer surpasses contemporary methods in enhancing spatial details and preserving spectral accuracy, underscoring its potential to expand HSI’s utility in various domains, such as environmental monitoring and remote sensing.

Hyperspectral images (HSIs) have very high spectral resolution, which can reflect the characteristics of different materials well. However, compared with RGB image or multispectral image (MSI), the spatial resolution of HSI is much lower, which limits its applications. Therefore, many super-resolution (SR) techniques have been proposed to reconstruct HSI with high spatial resolution image. To the best of our knowledge, there has not, to date, that been a study aimed at expatiating and summarizing the current research situation. Therefore, this is our motivation in this survey. In view of the promising development prospects in this field, this paper systematically reviews the existing SR methods of HSI. Specifically, two major categories are summarized, one is fusion-based methods, and the other is single HSI SR methods. At the end of the paper, several future development directions for HSI SR are given.

Fusion-based hyperspectral image (HSI) super-resolution (SR) aims to recover high-resolution HSI (hr-HSI) from its two degraded modalities, that is, low-resolution HSI (lr-HSI) and high-resolution multispectral image (hr-MSI). The resulting HSI-SR image can be explained from two viewpoints, an inverse problem solution or a spatial–spectral information fusion product. Recent methods focusing on the former point are limited by demands of accurate degradation parameters and precise prior assumptions. On the contrary, the other research line avoids these drawbacks and offers a more flexible design. However, recent methods implicitly handle information fusion. The interaction between lr-HSI and hr-MSI only lies in the extracted feature domain, not directly on the data. Moreover, the proposed modules in recent methods promote information fusion from the perspective of deep learning and neglect the specialty of the HSI domain, leading to weak interpretability and poor reliability in practice. Considering the essence of the HSI-SR problem and the inherent property of HSIs, in this article, we propose a maximum mutual information (MMI) strategy. At first, we model the HSI-SR problem in a new variance inference (VI) architecture. This new VI model simulates the physical process of HSI pairs imaging and provides convenience for the MMI strategy. Then, we insert the MMI strategy in the VI model. The MMI strategy promotes information fusion with quantitative constraints on the information interaction between lr-HSI and hr-MSI. Finally, we implement the VI architecture with a neural network. Benefitting from the MMI strategy, a simple network structure can achieve efficient fusion performance, which indicates that MMI frees DL-based HSI-SR methods from the complicated structure design. The experimental results on synthetic and real datasets demonstrate the superiority of our method to state-of-the-art methods in terms of effectiveness, generalization, and interpretability.

Hyperspectral imaging is beneficial in a diverse range of applications from diagnostic medicine, to agriculture, to surveillance to name a few. However, hyperspectral images often suffer from degradation such as noise and low resolution. In this paper, we propose an effective model for hyperspectral image (HSI) restoration, specifically image denoising and super-resolution. Our model considers three underlying characteristics of HSIs: sparsity across the spatial-spectral domain, high correlation across spectra, and non-local self-similarity over space. We first exploit high correlation across spectra and non-local self-similarity over space in the degraded HSI to learn an adaptive spatial-spectral dictionary. Then, we employ the local and non-local sparsity of the HSI under the learned spatial-spectral dictionary to design an HSI restoration model, which can be effectively solved by an iterative numerical algorithm with parameters that are adaptively adjusted for different clusters and different noise levels. In experiments on HSI denoising, we show that the proposed method outperforms many state-of-the-art methods under several comprehensive quantitative assessments. We also show that our method performs well on HSI super-resolution.

Hierarchical spatio-spectral fusion for hyperspectral image super resolution via sparse representation and pre-trained deep model

Limited by the existing imagery sensors, a hyperspectral image (HSI) is characterized by its high spectral resolution but low spatial resolution. HSI super-resolution (SR) aims to enhance the spatial resolution of the HSIs without modifying the equipment and has become a hot issue for HSI processing. In this paper, inspired by two important observations, a gated content-oriented residual dense network (GCoRDN) is designed for the HSI SR. To be specific, based on the observation that the structure and texture exhibit different sensitivities to the spatial degradation, a content-oriented network with two branches is designed. Meanwhile, a weight-sharing strategy is merged in the network to preserve the consistency in the structure and the texture. In addition, based on the observation of the super-resolved results, a gating mechanism is applied as a form of post-processing to further enhance the SR performance. Experimental results and data analysis on both ground-based HSIs and airborne HSIs have demonstrated the effectiveness of the proposed method.

Low spatial resolution is the mainly drawback of hyperspectral imaging. Image super-resolution techniques can be applied to overcome the limits. This paper presents a new framework for improving the spatial resolution of hyperspectral images base by combing high-resolution spectral information and high-resolution spatial information by image fusion and compressed sensing. Based on the compressed sensing theory, small patches of hyperspectral observations from different wavelengths can be represented as weighted linear combinations of a small number of atoms in dictionary which is trained by using panchromatic images. Then hyperspectral image super-resolution is treated as a special image fusion problem with sparse constraints. To make the super-resolution reconstruction more accurate, local manifold projection is used as a regulation term. Extensive experiments on image super-resolution validate that proposed method achieves much better results.