Deep Blind Hyperspectral Image Super-Resolution.

Recently, many convolutional networks have been built to fuse a low spatial resolution (LR) hyperspectral image (HSI) and a high spatial resolution (HR) multispectral image (MSI) to obtain HR HSIs. However, most deep learning-based methods are supervised methods, which require sufficient HR HSIs for supervised training. Collecting plenty of HR HSIs is laborious and time-consuming. In this paper, a self-supervised spectral-spatial residual network (SSRN) is proposed to alleviate dependence on a mass of HR HSIs. In SSRN, the fusion of HR MSIs and LR HSIs is considered a pixel-wise spectral mapping problem. Firstly, this paper assumes that the spectral mapping between HR MSIs and HR HSIs can be approximated by the spectral mapping between LR MSIs (derived from HR MSIs) and LR HSIs. Secondly, the spectral mapping between LR MSIs and LR HSIs is explored by SSRN. Finally, a self-supervised fine-tuning strategy is proposed to transfer the learned spectral mapping to generate HR HSIs. SSRN does not require HR HSIs as the supervised information in training. Simulated and real hyperspectral databases are utilized to verify the performance of SSRN.

Multispectral and hyperspectral image fusion with spatial-spectral sparse representation

Super-resolution (SR) is significant for hyperspectral image (HSI) applications. In single-frame HSI SR, how to reconstruct detailed image structures in high resolution (HR) HSI is challenging since there is no auxiliary image (e.g., HR multispectral image) providing structural information. Wavelet could capture image structures in different orientations, and emphasis on predicting high-frequency wavelet sub-bands is helpful for recovering the detailed structures in HSI SR. In this study, we propose a multi-scale wavelet 3D convolutional neural network (MW-3D-CNN) for HSI SR, which predicts the wavelet coefficients of HR HSI rather than directly reconstructing the HR HSI. To exploit the correlation in the spectral and spatial domains, the MW-3D-CNN is built with 3D convolutional layers. An embedding subnet and a predicting subnet constitute the MW-3D-CNN, the embedding subnet extracts deep spatial-spectral features from the low resolution (LR) HSI and represents the LR HSI as a set of feature cubes. The feature cubes are then fed to the predicting subnet. There are multiple output branches in the predicting subnet, each of which corresponds to one wavelet sub-band and predicts the wavelet coefficients of HR HSI. The HR HSI can be obtained by applying inverse wavelet transform to the predicted wavelet coefficients. In the training stage, we propose to train the MW-3D-CNN with L1 norm loss, which is more suitable than the conventional L2 norm loss for penalizing the errors in different wavelet sub-bands. Experiments on both simulated and real spaceborne HSI demonstrate that the proposed algorithm is competitive with other state-of-the-art HSI SR methods.

A variety of methods have been proposed to restore high resolution hyperspectral image (HSI) from a hybrid camera system, which captures high spatial resolution RGB images and low spatial resolution HSI. They focused unanimously on HSI super-resolution via fusion, yet did not explore the potential of this kind of system for camera spectral sensitivity (CSS), illumination spectrum, and high spatial resolution spectral reflectance recovery. In this paper, we present a sparse representation based method to estimate the CSS of the RGB camera under unknown illumination for the hybrid camera system. Furthermore, the illumination and high spatial resolution spectral reflectance are simultaneously recovered. Experimental results show the effectiveness of the proposed methods on camera spectral sensitivity, illumination spectrum and spectral reflectance recovery.

For many remote sensing applications it is preferable to have images with both high spectral and spatial resolutions. On this regards, hyperspectral and multispectral images have complementary characteristics in terms of spectral and spatial resolutions. In this paper we propose an approach for the fusion of low spatial resolution hyperspectral images with high spatial resolution multispectral images in order to obtain superresolution (spatial and spectral) hyperspectral images. The proposed approach is based on the assumption that, since both hyperspectral and multispectral images acquired on the same scene, the corresponding endmembers should be the same. On a first step the hyperspectral image is spectrally down-sampled in order to match the multispectral one. Then an endmember extraction algorithm is performed on the down-sampled hyperspectral image and the successive abundance estimation is performed on the multispectral one. Finally, the extracted endmembers are up-sampled back to the original hyperspectral space and then used to reconstruct the superresolution hyperspectral image according to the abundances obtained from the multispectral image.

ABSTRACTIn this article, we propose a method for extracting spatio-spectral features from high spatial resolution hyperspectral (HS) images. The method is based on extracting two-dimensional moments from neighbourhoods of pixels. Three different types of moments are considered: geometric, complex Zernike and Legendre. Moments of a given type are extracted from a few principal components (PC) of HS data, and are stacked on the original HS data to form a joint spatio-spectral feature space. These features are classified using a support vector machine (SVM) classifier. The influence of the moments orders and the size of the neighbourhood window on the quality of the extracted features are analysed. A few experiments are conducted on two widely used HS data sets, Pavia University and Salinas. The results demonstrate high capabilities of the proposed method in comparison with some state-of-the-art spatio-spectral HS classification methods.

Recently, many methods have been proposed to generate a high spatial resolution (HR) hyperspectral image (HSI) by fusing HSI and multispectral image (MSI). Most methods need a precondition that HSI and MSI are well registered. However, in practice, it is hard to acquire registered HSI and MSI. In this paper, a synchronous nonnegative matrix factorization (SNMF) is proposed to directly fuse unregistered HSI and MSI. The proposed SNMF does not require the registration operation by modeling the abundances of unregistered HSI and MSI independently. Moreover, to exploit both HSI and MSI in the endmember optimization of the desired HR HSI, the unregistered HSI and MSI fusion is formulated as a bound-constrained optimization problem. A synchronous projected gradient method is proposed to solve this bound-constrained optimization problem. Experiments on both simulated and real data demonstrate that the proposed SNMF outperforms the state-of-the-art methods.

Fusing a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI), aiming to produce a super-resolution hyperspectral image, has recently attracted increasing research interest. In this paper, a novel approach based on coupled non-negative tensor decomposition is proposed. The proposed method performs a tucker tensor factorization of a low resolution hyperspectral image and a high resolution multispectral image under the constraint of non-negative tensor decomposition (NTD). The conventional matrix factorization methods essentially lose spatio-spectral structure information when stacking the 3D data structure of a hyperspectral image into a matrix form. Moreover, the spectral, spatial, or their joint structural features have to be imposed from the outside as a constraint to well pose the matrix factorization problem. The proposed method has the advantage of preserving the spatio-spectral structure of hyperspectral images. In this paper, the NTD is directly imposed on the coupled tensors of the HSI and MSI. Hence, the intrinsic spatio-spectral structure of the HSI is represented without loss, and spatial and spectral information can be interdependently exploited. Furthermore, multilinear interactions of different modes of the HSIs can be exactly modeled with the core tensor of the Tucker tensor decomposition. The proposed method is straightforward and easy to implement. Unlike other state-of-the-art approaches, the complexity of the proposed approach is linear with the size of the HSI cube. Experiments on two well-known datasets give promising results when compared with some recent methods from the literature.

Hyperspectral (HS) pansharpening seeks to integrate low spatial resolution HS (LRHS) images with connected panchromatic (PAN) images to produce high spatial resolution HS (HRHS) images. Traditional pansharpening convolutional neural networks (CNNs) directly map LRHS and PAN images into HRHS images under fixed network parameters, which imply static pansharpening rules. However, real-world HS data are often characterized by spatial variations and intuitively the pansharpening rules should be dynamic. To deal with the dilemma, in this paper we develop dynamic HS pansharpening CNNs. We first specify the concepts of dynamic pansharpening and static pansharpening. Then, we propose a learn-to-learn oriented pansharpening CNN paradigm, which aims to learn a how-to-learn rule to produce spatially adaptive pansharpening rules and comprises three stages of preliminary fusion, scene-sensitive modulation and spectral reconstruction. Finally, following the paradigm, we design two groups of dynamic pansharpening CNNs (DyPNNs), i.e., internal-connection-based and external-connection-based. They involve various spatial modulations, including spatial affine transform (AT), spatial dynamic convolution (DC) or improved spatial attention (SA), and thus consist of six specific DyPNNs: IC-AT-DyPNN, IC-DC-DyPNN, IC-SA-DyPNN, EC-AT-DyPNN, EC-DC-DyPNN and EC-SA-DyPNN. Experimental results on several HS datasets verify the effectiveness of the proposed DyPNNs in terms of both the spatial reconstruction and spectral fidelity.

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

HSI fusion approaches aims to recover high spatial resolution hyperspectral image (HSI) with the preservation of original spectral information. The existed fusion framework are designed to establish a spatial feature mapping relationship between low/high resolution HSIs, which is always an under-determined problem affected by artifacts and distortions. In this paper, a multi-layer residual learning fusion approach is proposed for HSI resolution enhancement. It is believed that there are residuals between recovered result and ideal ground truth HSI. Our method focuses on reducing the residuals to gradually modify the recovered HSI and iteratively make it approximate to the ground truth. Firstly, the back propagation neural network (BPNN) is employed to acquire an intermediate recovered result. Then, a series of online residual dictionaries are trained from residuals to update modified recovered HSIs in successive iterations. Therefore, the accurate recovered high resolution HSI less errors and spectral distortions.can be achieved. Experimental results on two different HSI datasets show that the proposed method can acquire better or comparable evaluation metric and visual fusion result with the current state-of-the-art methods.

In this paper, unconstrained least square unmixing (ULSU) scheme is proposed for the fusion of hyperspectral (HS), multispectral (MS), and panchromatic (PAN) data. Based on unmixing formulation, the spectral and the spatial features are extracted from these three data in order to produce high spatial resolution hyperspectral image. we conduct experiments on simulated data which is generated from real HYDICE dataset. The proposed approach delivers improved fusion performances in both spectral and spatial domains with smaller spectral distortion in comparison with CNMF method.

Hyperspectral (HS) and multispectral (MS) image fusion aims at producing high-resolution HS (HRHS) images. However, the existing methods could not simultaneously consider the structures in both the spatial and spectral domains of the HS cube. In order to effectively preserve spatial–spectral structures in HRHS images, we propose a new low-resolution HS (LRHS) and high-resolution MS (HRMS) image fusion method based on spatial–spectral-graph-regularized low-rank tensor decomposition (SSGLRTD) in this paper. First, we reformulate the image fusion problem as a low-rank tensor decomposition model to utilize the low-rank property in the HS image. Then, two graphs are constructed in spatial and spectral domains, respectively. One of them is derived from the HRMS image for the spatial consistency, and the other is inferred from the LRHS image for spectral smoothness. Finally, the SSGLRTD fusion framework is established by combining all regularizers. With these two graphs, the spatial correlation and the spectral structure in the fused HRHS images are efficiently preserved. The experimental results on different datasets reveal that the proposed fusion method outperforms several existing fusion methods in terms of visual analysis and numerical comparison.

Hyperspectral images (HSIs) usually have high spectral and low spatial resolution. Conversely, multispectral images (MSIs) usually have low spectral and high spatial resolution. The fusion of HSI and MSI aims to create spectral images with high spectral and spatial resolution. In this paper, we propose a fusion algorithm by combining linear spectral unmixing with the local low-rank property. By taking advantage of the local low-rank property, we first partition the corresponding spectral image into patches. For each patch pair, we cast the fusion problem as a coupled spectral unmixing problem that extracts the abundance and the endmembers of MSI and HSI, respectively. It then updates the abundance and the endmember through an alternating update algorithm. In fact, the convergence of the alternative update algorithm can be mathematically and empirically supported. We also propose a multiscale postprocessing procedure to combine fusion results obtained under different patch sizes. In experiments on three data sets, the proposed fusion algorithms outperformed state-of-the-art fusion algorithms in both spatial and spectral domains.

In this paper, we address the issue of hyperspectral pansharpening, which consists in fusing a (low spatial resolution) hyperspectral image HX and a (high spatial resolution) panchromatic image P to obtain a high spatial resolution hyperspectral image. The problem is addressed under a variational convex constrained formulation. The objective favors high resolution spectral bands with level lines parallel to those of the panchromatic image. This term is balanced with a total variation term as regularizer. Fit-to-P data and fit-to-HX data constraints are effectively considered as mathematical constraints, which depend on the statistics of the data noise measurements. The developed Alternate Direction Method of Multipliers (ADMM) optimization scheme enables us to solve this problem efficiently despite the non differentiabilities and the huge number of unknowns.