Hyperspectral image super-resolution through clustering-based sparse representation

Super-resolution (SR) is significant for hyperspectral image (HSI) applications. In single-frame HSI SR, how to reconstruct detailed image structures in high resolution (HR) HSI is challenging since there is no auxiliary image (e.g., HR multispectral image) providing structural information. Wavelet could capture image structures in different orientations, and emphasis on predicting high-frequency wavelet sub-bands is helpful for recovering the detailed structures in HSI SR. In this study, we propose a multi-scale wavelet 3D convolutional neural network (MW-3D-CNN) for HSI SR, which predicts the wavelet coefficients of HR HSI rather than directly reconstructing the HR HSI. To exploit the correlation in the spectral and spatial domains, the MW-3D-CNN is built with 3D convolutional layers. An embedding subnet and a predicting subnet constitute the MW-3D-CNN, the embedding subnet extracts deep spatial-spectral features from the low resolution (LR) HSI and represents the LR HSI as a set of feature cubes. The feature cubes are then fed to the predicting subnet. There are multiple output branches in the predicting subnet, each of which corresponds to one wavelet sub-band and predicts the wavelet coefficients of HR HSI. The HR HSI can be obtained by applying inverse wavelet transform to the predicted wavelet coefficients. In the training stage, we propose to train the MW-3D-CNN with L1 norm loss, which is more suitable than the conventional L2 norm loss for penalizing the errors in different wavelet sub-bands. Experiments on both simulated and real spaceborne HSI demonstrate that the proposed algorithm is competitive with other state-of-the-art HSI SR methods.

Hyperspectral-multispectral image fusion aims to generate high-resolution hyperspectral (HRHS) images with rich spectral details, by mining the complementary information between low-resolution hyperspectral (LRHS) images and high resolution multispectral (HRMS) images. However, in practical scenarios, LRHS images are susceptible to noise interference due to their narrow spectral bandwidth, resulting in limited energy absorption of surface reflectance in each band. Various noise disrupt the correlations of LRHS and HRMS images, thus leading to the spatial details loss or spectral distortion. To address these challenges, we propose an anti-noise hyperspectral and multispectral image fusion network based on State Space Model (SSM), namely AHMFNet. To mitigate noise in LRHS images, the spectral denoising and refinement module is designed by combining low-rank representation with the discrete wavelet transform, while simultaneously extracting key spectral features. Then, the SSM is utilized for global modeling of spectral information to enhance essential features. Finally, although spectral and spatial features have differences, the spatial-spectral cross Mamba is designed to fuse them by mining their relationships, thus improving the fusion quality. Experiments on four public datasets show that, compared to existing state-of-the-art methods, the proposed AHMFNet generates images with superior spatial quality and more accurate spectral characteristics under various noise conditions.

Fusing a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI), aiming to produce a super-resolution hyperspectral image, has recently attracted increasing research interest. In this paper, a novel approach based on coupled non-negative tensor decomposition is proposed. The proposed method performs a tucker tensor factorization of a low resolution hyperspectral image and a high resolution multispectral image under the constraint of non-negative tensor decomposition (NTD). The conventional matrix factorization methods essentially lose spatio-spectral structure information when stacking the 3D data structure of a hyperspectral image into a matrix form. Moreover, the spectral, spatial, or their joint structural features have to be imposed from the outside as a constraint to well pose the matrix factorization problem. The proposed method has the advantage of preserving the spatio-spectral structure of hyperspectral images. In this paper, the NTD is directly imposed on the coupled tensors of the HSI and MSI. Hence, the intrinsic spatio-spectral structure of the HSI is represented without loss, and spatial and spectral information can be interdependently exploited. Furthermore, multilinear interactions of different modes of the HSIs can be exactly modeled with the core tensor of the Tucker tensor decomposition. The proposed method is straightforward and easy to implement. Unlike other state-of-the-art approaches, the complexity of the proposed approach is linear with the size of the HSI cube. Experiments on two well-known datasets give promising results when compared with some recent methods from the literature.

Fusing a low-resolution (LR) hyperspectral image (HSI) with a high-resolution (HR) multispectral image (MSI) has emerged as a promising strategy for reconstructing high-quality HSIs that combine rich spectral and fine spatial information. However, most existing HSI fusion methods operate under the restrictive assumption that the LR HSI and HR MSI are spatially aligned and fully consistent on the field-of-view (FoV), which significantly limits their applicability in real-world scenarios when such alignment is unavailable. To overcome these limitations, we propose IrregFusion, a generalized HSI fusion framework capable of handling both FoV-consistent and inconsistent fusion scenarios. Specifically, IrregFusion incorporates a Transformer-based reconstruction module that captures both intra- and inter-modal correlations between the diverse spectra data and the MSI, enhancing the model’s ability to perceive and reconstruct non-local spectral–spatial structures. To further address the challenges posed by FoV inconsistencies, we introduce a spectral propagation strategy that diffuses observed spectral information into adjacent spectral-blank regions, thereby easing the reconstruction of missing spectral content. Additionally, a self-supervised adaptation mechanism is integrated into the framework, enabling robust spectral–spatial representation learning and enhancing generalization across diverse and challenging conditions. Extensive experiments conducted on benchmark datasets demonstrate that IrregFusion effectively addresses the challenges of diverse spectral data fusion and consistently outperforms state-of-the-art methods in both reconstruction accuracy and visual fidelity. The source code will be released in https://github.com/JiangtaoNie/IrregFusion.git.

Hyperspectral imaging has many applications from agriculture and astronomy to surveillance and mineralogy. However, it is often challenging to obtain high-resolution (HR) hyperspectral images using existing hyperspectral imaging techniques due to various hardware limitations. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) image and a HR reference image of the same scene. The estimation of the HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the prior knowledge of the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary representing prototype reflectance spectra vectors of the scene is first learned from the input LR image. Specifically, an efficient non-negative dictionary learning algorithm using the block-coordinate descent optimization technique is proposed. Then, the sparse codes of the desired HR hyperspectral image with respect to learned hyperspectral basis are estimated from the pair of LR and HR reference images. To improve the accuracy of non-negative sparse coding, a clustering-based structured sparse coding method is proposed to exploit the spatial correlation among the learned sparse codes. The experimental results on both public datasets and real LR hypspectral images suggest that the proposed method substantially outperforms several existing HR hyperspectral image recovery techniques in the literature in terms of both objective quality metrics and computational efficiency.

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

Recently, many convolutional networks have been built to fuse a low spatial resolution (LR) hyperspectral image (HSI) and a high spatial resolution (HR) multispectral image (MSI) to obtain HR HSIs. However, most deep learning-based methods are supervised methods, which require sufficient HR HSIs for supervised training. Collecting plenty of HR HSIs is laborious and time-consuming. In this paper, a self-supervised spectral-spatial residual network (SSRN) is proposed to alleviate dependence on a mass of HR HSIs. In SSRN, the fusion of HR MSIs and LR HSIs is considered a pixel-wise spectral mapping problem. Firstly, this paper assumes that the spectral mapping between HR MSIs and HR HSIs can be approximated by the spectral mapping between LR MSIs (derived from HR MSIs) and LR HSIs. Secondly, the spectral mapping between LR MSIs and LR HSIs is explored by SSRN. Finally, a self-supervised fine-tuning strategy is proposed to transfer the learned spectral mapping to generate HR HSIs. SSRN does not require HR HSIs as the supervised information in training. Simulated and real hyperspectral databases are utilized to verify the performance of SSRN.

Due to the difficulties of directly obtaining high-resolution hyperspectral images (HR-HSI), the fusion of low-resolution hyperspectral images (LR-HSI) and high-resolution multispectral images (HR-MSI) has emerged as an effective approach. While existing methods leverage image-level priors from HR-MSI, they often lack explicit semantic guidance for precise detail reconstruction. Recognizing that textual scene descriptions encapsulate valuable object attributes and contextual information, we introduce the first Language-Bridging framework for Hyperspectral and Multispectral image fusion (CO²IF). CO²IF leverages language semantics as prior knowledge to explicitly guide the reconstruction process. To bridge the modality gap between textual descriptions and high-dimensional hyperspectral data, we design a Cross-modal Optimal Transport (COT) module. COT establishes precise semantic correspondences between language features and the visual cues of individual spectral bands. Building upon this semantic alignment, we develop a Multimodal Coordinated State Space Model (CoMamba). CoMamba effectively integrates the language-derived priors with spatial information from HR-MSI and spectral information from LR-HSI. This language-guided reconstruction significantly enhances the extraction of crucial spatial-spectral details, leading to superior fidelity in the generated HR-HSI. In addition, this paper adds text descriptions for three widely used datasets. Both qualitative and quantitative experimental results on the public datasets confirm the superiority of the proposed method compared to the SOTA methods.

Fusion-based hyperspectral image (HSI) super-resolution (SR) aims to recover high-resolution HSI (hr-HSI) from its two degraded modalities, that is, low-resolution HSI (lr-HSI) and high-resolution multispectral image (hr-MSI). The resulting HSI-SR image can be explained from two viewpoints, an inverse problem solution or a spatial–spectral information fusion product. Recent methods focusing on the former point are limited by demands of accurate degradation parameters and precise prior assumptions. On the contrary, the other research line avoids these drawbacks and offers a more flexible design. However, recent methods implicitly handle information fusion. The interaction between lr-HSI and hr-MSI only lies in the extracted feature domain, not directly on the data. Moreover, the proposed modules in recent methods promote information fusion from the perspective of deep learning and neglect the specialty of the HSI domain, leading to weak interpretability and poor reliability in practice. Considering the essence of the HSI-SR problem and the inherent property of HSIs, in this article, we propose a maximum mutual information (MMI) strategy. At first, we model the HSI-SR problem in a new variance inference (VI) architecture. This new VI model simulates the physical process of HSI pairs imaging and provides convenience for the MMI strategy. Then, we insert the MMI strategy in the VI model. The MMI strategy promotes information fusion with quantitative constraints on the information interaction between lr-HSI and hr-MSI. Finally, we implement the VI architecture with a neural network. Benefitting from the MMI strategy, a simple network structure can achieve efficient fusion performance, which indicates that MMI frees DL-based HSI-SR methods from the complicated structure design. The experimental results on synthetic and real datasets demonstrate the superiority of our method to state-of-the-art methods in terms of effectiveness, generalization, and interpretability.

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

Recently, fusing a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) of different satellites has become an effective way to improve the resolution of an HSI. However, due to different imaging satellites, different illumination, and adjacent imaging time, the LR-HSI and HR-MSI may not satisfy the observation models established by existing works, and the LR-HSI and HR-MSI are hard to be registered. To solve the above problems, we establish new observation models for LR-HSIs and HR-MSIs from different satellites, then a deep-learning-based framework is proposed to solve the key steps in multi-satellite HSI fusion, including image registration, blur kernel learning, and image fusion. Specifically, we first construct a convolutional neural network (CNN), called RegNet, to produce pixel-wise offsets between LR-HSI and HR-MSI, which are utilized to register the LR-HSI. Next, according to the new observation models, a tiny network, called BKLNet, is built to learn the spectral and spatial blur kernels, where the BKLNet and RegNet can be trained jointly. In the fusion part, we further train a FusNet by downsampling the registered data with the learned spatial blur kernel. Extensive experiments demonstrate the superiority of the proposed framework in HSI registration and fusion accuracy.

The production of a high spatial resolution (HR) hyperspectral image (HSI) through the fusion of a low spatial resolution (LR) HSI with an HR multispectral image (MSI) has underpinned much of the recent progress in HSI super-resolution. The premise of these signs of progress is that both the degeneration from the HR HSI to LR HSI in the spatial domain and the degeneration from the HR HSI to HR MSI in the spectral domain are assumed to be known in advance. However, such a premise is difficult to achieve in practice. To address this problem, we propose to incorporate degeneration estimation into HSI super-resolution and present an unsupervised deep framework for "blind" HSIs super-resolution where the degenerations in both domains are unknown. In this framework, we model the latent HR HSI and the unknown degenerations with deep network structures to regularize them instead of using handcrafted (or shallow) priors. Specifically, we generate the latent HR HSI with an image-specific generator network and structure the degenerations in spatial and spectral domains through a convolution layer and a fully connected layer, respectively. By doing this, the proposed framework can be formulated as an end-to-end deep network learning problem, which is purely supervised by those two input images (i.e., LR HSI and HR MSI) and can be effectively solved by the backpropagation algorithm. Experiments on both natural scene and remote sensing HSI data sets show the superior performance of the proposed method in coping with unknown degeneration either in the spatial domain, spectral domain, or even both of them.

The fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) is a crucial technology for producing high-resolution hyperspectral images. Most existing image fusion algorithms based on deep learning do not fully utilize the ability of neural network to extract and process multi-scale features, which leads to the problem of difficulty in fully learning features and ambiguity of features. In order to overcome these issues, a residual selective kernel attention-based U-net named RSKAU-net is designed for LR-HSI and HR-MSI fusion. RSKAU-net is constructed by a residual selective kernel module with an attention mechanism and a channel attention block. The residual selective kernel attention-based (RSKA) module is designed to process images of different resolutions, which adaptively extracts multi-scale features and efficiently emphasizes significant features through the attention mechanism. The channel attention (CA) module retains important spectral information by assigning different weights to each channel of LR-HSI. The proposed network can enhance the spatial information of LR-HSI while preserving its spectral information. Meanwhile, it effectively fuses the features from the source image to obtain the HR-HSI with rich details. The experimental results demonstrate that the proposed network has advantages in terms of both visual effect and objective quantitative indices when compared to existing HSI-MSI fusion approaches.

The image super-resolution (SR) technique, which aims at reconstructing a high-resolution (HR) image from a single low-resolution (LR) image, is a classical problem in computer vision. Limited by the imaging hardware, the spatial resolution of a hyperspectral images (HSI) is usually very coarse. Meanwhile, the spectral information of the HSI is extremely important for its applications and cannot be severely distorted. This paper presents a spatial constraint (SCT) strategy with combination of a deep learning method for HSI SR. The SCT strategy restraints the LR HSI generated by the reconstructed HR HSI should be spatially close to the input LR HSI. The deep learning method learns an end-to-end mapping between the spectral difference of the LR HSI and that of the HR HSI. The mapping is represented as a deep convolutional neural network (CNN). The CNN learned spectral difference is utilized to super-resolve the LR HSI while preserve the important spectral information of the desired HR HSI. Experiments have been conducted on three databases that contains both indoor scenes and outdoor scenes. Comparative analyses have verified the effectiveness of the overall method.

For many remote sensing applications it is preferable to have images with both high spectral and spatial resolutions. On this regards, hyperspectral and multispectral images have complementary characteristics in terms of spectral and spatial resolutions. In this paper we propose an approach for the fusion of low spatial resolution hyperspectral images with high spatial resolution multispectral images in order to obtain superresolution (spatial and spectral) hyperspectral images. The proposed approach is based on the assumption that, since both hyperspectral and multispectral images acquired on the same scene, the corresponding endmembers should be the same. On a first step the hyperspectral image is spectrally down-sampled in order to match the multispectral one. Then an endmember extraction algorithm is performed on the down-sampled hyperspectral image and the successive abundance estimation is performed on the multispectral one. Finally, the extracted endmembers are up-sampled back to the original hyperspectral space and then used to reconstruct the superresolution hyperspectral image according to the abundances obtained from the multispectral image.