Unregistered Hyperspectral and Multispectral Image Fusion with Synchronous Nonnegative Matrix Factorization

The production of a high spatial resolution (HR) hyperspectral image (HSI) through the fusion of a low spatial resolution (LR) HSI with an HR multispectral image (MSI) has underpinned much of the recent progress in HSI super-resolution. The premise of these signs of progress is that both the degeneration from the HR HSI to LR HSI in the spatial domain and the degeneration from the HR HSI to HR MSI in the spectral domain are assumed to be known in advance. However, such a premise is difficult to achieve in practice. To address this problem, we propose to incorporate degeneration estimation into HSI super-resolution and present an unsupervised deep framework for "blind" HSIs super-resolution where the degenerations in both domains are unknown. In this framework, we model the latent HR HSI and the unknown degenerations with deep network structures to regularize them instead of using handcrafted (or shallow) priors. Specifically, we generate the latent HR HSI with an image-specific generator network and structure the degenerations in spatial and spectral domains through a convolution layer and a fully connected layer, respectively. By doing this, the proposed framework can be formulated as an end-to-end deep network learning problem, which is purely supervised by those two input images (i.e., LR HSI and HR MSI) and can be effectively solved by the backpropagation algorithm. Experiments on both natural scene and remote sensing HSI data sets show the superior performance of the proposed method in coping with unknown degeneration either in the spatial domain, spectral domain, or even both of them.

This paper studies a new Bayesian optimization algorithm for fusing hyperspectral and multispectral images. The hyperspectral image is supposed to be obtained by blurring and subsampling a high spatial and high spectral target image. The multispectral image is modeled as a spectral mixing version of the target image. By introducing appropriate priors for parameters and hyperparameters, the fusion problem is formulated within a Bayesian estimation framework, which is very convenient to model the noise and the target image. The high spatial resolution hyperspectral image is then inferred from its posterior distribution. To compute the Bayesian maximum a posteriori estimator associated with this posterior, an alternating direction method of multipliers within block coordinate descent algorithm is proposed. Simulation results demonstrate the efficiency of the proposed fusion method when compared with several state-of-the-art fusion techniques.

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

The hyperspectral and multispectral image fusion process, commonly known as the multisharpening process (also known as the hypersharpening process), results in the creation of unobservable high spectral and spatial resolution hyperspectral remote sensing data, by combining (or fusing) a hyperspectral image, characterized by a high spectral resolution, with, especially, a high spatial resolution multispectral one. There are several approaches in the literature performing such a process, in particular, those based on the spectral unmixing concept. The latter fusion methods aim to extract pure-material spectra from the high spectral resolution hyperspectral image and combine them with their corresponding proportions extracted from the high spatial resolution multispectral image. However, most of these methods, which mainly consider the standard linear mixing model, do not address the spectral variability phenomenon, precisely due to the considered data mixing model, which does not handle variations in illumination and atmospheric conditions or material composition inducing spectral variability. Neglecting such a phenomenon may lead to errors in the fusion process, thus decreasing the spatial and spectral qualities of sharpened products. In this paper, a new hyperspectral and multispectral data fusion method, based on the linear spectral unmixing concept that deals with spectral variability, is proposed. This approach, based on a multiplicative Nonnegative Matrix Factorization (NMF) technique, substantially extends the multiplicative Coupled NMF (CNMF) algorithm by considering a linear data mixing model that addresses the considered phenomenon. Indeed, the introduced approach minimizes new cost functions involving matrices with specific structures in order to deal with the spectral variability phenomenon. The proposed method is applied to synthetic and real data, and its effectiveness is evaluated with established performance criteria. Experimental results show that the designed algorithm globally yields sharpened hyperspectral data with fine spectral and spatial fidelities. Also, these results demonstrate that the proposed technique globally outperforms the tested literature approaches.

Hyperspectral (HS) remote sensing image with finer spectral information has great advantages in feature identification and classification. However, the spatial resolution of HS image is usually low due to practical limitations. In this paper, the low-spatial-resolution HS image is fused with the high-spatial-resolution multispectral (MS) image of the same observation scene to improve its spatial resolution. A novel spectral unmixing based HS and MS image fusion approach (VSC-CNMF) is proposed, in which CNMF with minimum endmember simplex volume and abundance sparsity constraints is employed for coupled unmixing of HS and MS images. Simulative experiments are employed for verification and comparison. The experimental results illustrate that the newly proposed VSC-CNMF based HS and MS fusion algorithm outperforms several state-of-the-art unmixing based fusion approaches in cases with moderate number of endmembers.

VTT Technical Research Centre of Finland has developed a Fabry-Perot Interferometer (FPI) based hyperspectral imager compatible with light weight UAV (Unmanned Aerial Vehicle) platforms (SPIE Proc. 7474¹, 8186B²). The FPI based hyperspectral imager was used in a UAV imaging campaign for forest and agriculture tests during the summer 2011 (SPIE Proc. 8174³). During these tests high spatial resolution Color-Infrared (CIR) images and hyperspectral images were recorded on separate flights. The spectral bands of the CIR camera were 500 - 580 nm for the green band, 580 - 700 nm for the red band and 700 - 1000 nm for the near infrared band. For the summer 2012 flight campaign a new hyperspectral imager is currently being developed. A custom made CIR camera will also be used. The system which includes both the high spatial resolution Color-Infrared camera and a light weight hyperspectral imager can provide all necessary data with just one UAV flight over the target area. The new UAV imaging system contains a 4 Megapixel CIR camera which is used for the generation of the digital surface models and CIR mosaics. The hyperspectral data can be recorded in the wavelength range 500 - 900 nm at a resolution of 10 - 30 nm at FWHM. The resolution can be selected from approximate values of 10, 15, 20 or 30 nm at FWHM.

Multispectral and hyperspectral image fusion with spatial-spectral sparse representation

Abstract. Hyperspectral image enhancement has been a concern for the remote sensing society for detailed end member detection. Hyperspectral remote sensor collects images in hundreds of narrow, continuous spectral channels, whereas multispectral remote sensor collects images in relatively broader wavelength bands. However, the spatial resolution of the hyperspectral sensor image is comparatively lower than that of the multispectral. As a result, spectral signatures from different end members originate within a pixel, known as mixed pixels. This paper presents an approach for obtaining an image which has the spatial resolution of the multispectral image and spectral resolution of the hyperspectral image, by fusion of hyperspectral and multispectral image. The proposed methodology also addresses the band remapping problem, which arises due to different regions of spectral coverage by multispectral and hyperspectral images. Therefore we apply algorithms to restore the spatial information of the hyperspectral image by fusing hyperspectral bands with only those bands which come under each multispectral band range. The proposed methodology is applied over Henry Island, of the Sunderban eco-geographic province. The data is collected by the Hyperion hyperspectral sensor and LISS IV multispectral sensor.

Hyperspectral (HS) and multispectral (MS) image fusion aims at producing high-resolution HS (HRHS) images. However, the existing methods could not simultaneously consider the structures in both the spatial and spectral domains of the HS cube. In order to effectively preserve spatial–spectral structures in HRHS images, we propose a new low-resolution HS (LRHS) and high-resolution MS (HRMS) image fusion method based on spatial–spectral-graph-regularized low-rank tensor decomposition (SSGLRTD) in this paper. First, we reformulate the image fusion problem as a low-rank tensor decomposition model to utilize the low-rank property in the HS image. Then, two graphs are constructed in spatial and spectral domains, respectively. One of them is derived from the HRMS image for the spatial consistency, and the other is inferred from the LRHS image for spectral smoothness. Finally, the SSGLRTD fusion framework is established by combining all regularizers. With these two graphs, the spatial correlation and the spectral structure in the fused HRHS images are efficiently preserved. The experimental results on different datasets reveal that the proposed fusion method outperforms several existing fusion methods in terms of visual analysis and numerical comparison.

HSI fusion approaches aims to recover high spatial resolution hyperspectral image (HSI) with the preservation of original spectral information. The existed fusion framework are designed to establish a spatial feature mapping relationship between low/high resolution HSIs, which is always an under-determined problem affected by artifacts and distortions. In this paper, a multi-layer residual learning fusion approach is proposed for HSI resolution enhancement. It is believed that there are residuals between recovered result and ideal ground truth HSI. Our method focuses on reducing the residuals to gradually modify the recovered HSI and iteratively make it approximate to the ground truth. Firstly, the back propagation neural network (BPNN) is employed to acquire an intermediate recovered result. Then, a series of online residual dictionaries are trained from residuals to update modified recovered HSIs in successive iterations. Therefore, the accurate recovered high resolution HSI less errors and spectral distortions.can be achieved. Experimental results on two different HSI datasets show that the proposed method can acquire better or comparable evaluation metric and visual fusion result with the current state-of-the-art methods.

In this paper, unconstrained least square unmixing (ULSU) scheme is proposed for the fusion of hyperspectral (HS), multispectral (MS), and panchromatic (PAN) data. Based on unmixing formulation, the spectral and the spatial features are extracted from these three data in order to produce high spatial resolution hyperspectral image. we conduct experiments on simulated data which is generated from real HYDICE dataset. The proposed approach delivers improved fusion performances in both spectral and spatial domains with smaller spectral distortion in comparison with CNMF method.

Recent advances and new guidelines on hyperspectral and multispectral image fusion

In recent years, the fusion of hyperspectral and multispectral images in remote sensing image processing still faces challenges, primarily due to their complexity and multimodal characteristics. Diffusion models, known for their stable training process and exceptional image generation capabilities, have shown good application potential in this field. However, when dealing with multimodal data, it may prove challenging for the models to fully capture the intricate relationships between the modalities, which may result in incomplete information integration and a small amount of remaining noise in the generated images. To address these problems, we propose a new model, KanDiff, for hyperspectral and multispectral image fusion. To address the differences in modal information between multispectral and hyperspectral images, KANDiff incorporates Kolmogorov–Arnold Networks (KAN) to guide the inputs. It helps the model understand the complex relationships between the modalities by replacing the fixed activation function in the traditional MLP with a learnable activation function. Furthermore, the image generated by the diffusion model may exhibit a small amount of the remaining noise. Convolutional Neural Networks (CNNs) effectively extract local features through their convolutional layers and achieve noise suppression via layer-by-layer feature representation. Therefore, the MergeCNN module is further introduced to enhance the fusion effect, resulting in smoother and more accurate outcomes. Experimental results on the public CAVE and Harvard datasets indicate that KanDiff has achieved improvements over current high-performance methods across several metrics, particularly showing significant enhancements in the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM), thus demonstrating superior performance. Additionally, we have created an image fusion dataset of the lunar surface, and KANDiff exhibits robust performance on this dataset as well. This work introduces an effective solution for addressing the challenges posed by missing high-resolution hyperspectral images (HRHS) data, which is essential for tasks including landing site selection and resource exploration within the realm of deep space exploration.

For many remote sensing applications it is preferable to have images with both high spectral and spatial resolutions. On this regards, hyperspectral and multispectral images have complementary characteristics in terms of spectral and spatial resolutions. In this paper we propose an approach for the fusion of low spatial resolution hyperspectral images with high spatial resolution multispectral images in order to obtain superresolution (spatial and spectral) hyperspectral images. The proposed approach is based on the assumption that, since both hyperspectral and multispectral images acquired on the same scene, the corresponding endmembers should be the same. On a first step the hyperspectral image is spectrally down-sampled in order to match the multispectral one. Then an endmember extraction algorithm is performed on the down-sampled hyperspectral image and the successive abundance estimation is performed on the multispectral one. Finally, the extracted endmembers are up-sampled back to the original hyperspectral space and then used to reconstruct the superresolution hyperspectral image according to the abundances obtained from the multispectral image.