Wavelet-Based Multi-stream Spatial-Frequency Fusion Network for Few-Shot Hyperspectral Image Classification

Adaptive spectral-spatial feature fusion network for hyperspectral image classification using limited training samples

The lack of training samples remains one of the major obstacles in applying convolutional neural networks (CNNs) to the hyperspectral image (HSI) classification. In this letter, the accurate classification of HSI with limited training samples is investigated. Due to the advantages of minimizing the distance between samples in the same class and maximizing the distance between samples in different classes, siamese CNN is used for HSI classification with limited training samples. After that, to improve the classification performance, data augmentation is investigated for siamese CNN-based HSI classification. Specifically, pair data augmentation based on CutMix is proposed to generate the training pairs of the same or different classes and the new generated training pairs are used to train siamese CNN. Traditional data augmentation methods simply generate new training samples. It is not proper when data augmentation methods keep the loss function and change the content of the input at the same time. Therefore, a soft-loss-based siamese CNN, which changes its loss according to the coupled replacement data augmentation, is proposed to further address the HSI classification with limited training samples. In the experimental part, on two widely used hyperspectral datasets, the influences of different training samples and window sizes are discussed, and the experimental results reveal that the proposed soft-augmentation-based siamese CNN provides competitive results with limited training samples compared with state-of-the-art methods.

Deep learning-based methods have made significant progress in hyperspectral image (HSI) classification in recent years. However, deep learning-based methods usually rely on a large number of samples, and in many cases, it is difficult to label HSI and only limited training samples are available. To solve this problem, an HSI classification method based on adaptive spatial pyramid constraint (ASPC) is proposed to make full use of the global spatial neighborhood information of the labeled samples, which can improve the generalization ability of the classification model. The main steps of the proposed method are as follows. First, an HSI complexity evaluation method based on edge detection is proposed to assess the homogeneity of the objects in the HSI. Second, an HSI pyramid segmentation method based on spatial pyramid is proposed to generate multiscale subregions, where HSI complexity is used to adaptively determine the scale of the segmentation. Third, a spatial supervised constraint is proposed to generate the loss function of labeled subregions. Fourth, a spatial unsupervised constraint is proposed to generate the loss function of unlabeled subregions. The proposed method fully explores the spatial-spectral correlation between unlabeled samples and labeled samples, and add corresponding constraints to the training objective according to the correlation. By adding the ASPC, the trained model becomes more robust and can make full use of the limited training samples. To verify the effectiveness of the proposed method, three benchmark hyperspectral datasets are used to verify the performance of the proposed method. Experimental results show that the performance of this method is better than the existing state-of-the-art methods.

Recently, deep learning methods, for example, convolutional neural networks (CNNs), have achieved high performance in hyperspectral image (HSI) classification. The limited training samples of HSI images make it hard to use deep learning methods with many layers and a large number of convolutional kernels as in large scale imagery tasks, and CNN-based methods usually need long training time. In this paper, we present a wide sliding window and subsampling network (WSWS Net) for HSI classification. It is based on layers of transform kernels with sliding windows and subsampling (WSWS). It can be extended in the wide direction to learn both spatial and spectral features more efficiently. The learned features are subsampled to reduce computational loads and to reduce memorization. Thus, layers of WSWS can learn higher level spatial and spectral features efficiently, and the proposed network can be trained easily by only computing linear weights with least squares. The experimental results show that the WSWS Net achieves excellent performance with different hyperspectral remotes sensing datasets compared with other shallow and deep learning methods. The effects of ratio of training samples, the sizes of image patches, and the visualization of features in WSWS layers are presented.

Recently, deep learning-based methods have made great progress in hyperspectral image (HSI) classification (HSIC). Different from ordinary images, the intrinsic complexity of HSIs data still limits the performance of many common convolutional neural network (CNN) models. Thus, the network architecture becomes more and more complex to extract discriminative spectral-spatial features. For instance, 3-D CNN usually has a large number of trainable parameters, thus increasing the computational complexity of the HSIC. In this letter, we designed a cooperative spectral-spatial attention dense network (CS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ADN) that takes raw 3-D HSI data as input data. Specifically, the attention module consists of spectral and spatial axes, by which the salient spectral-spatial features will be emphasized. Furthermore, we combined these attention modules with the dense connection, which is termed as the lightweight dense block; it has a lower computation cost and achieves better classification performance. At the same time, we introduced the center loss, by jointly using the supervision of the center loss and the softmax loss, where the discriminative features could be clearly observed, particularly for small data sets. Experimental results on the biased and unbiased HSI data show that our method outperforms several state-of-the-art methods in HSIC with small training samples.

Recently, hyperspectral image (HSI) classification methods based on deep learning have demonstrated excellent performance. However, these deep learning methods still face two major challenges. One is that they require a large number of labeled samples, and the other is that training parameters takes a lot of time. In this paper, we propose a dual-branch deep stochastic adaptive Fourier decomposition (SAFD) network (DSAFDNet) to alleviate the aforementioned two issues in HSI classification applications. SAFD is a newly developed signal processing tool with solid mathematical foundation. It can be used to find common filters (i.e. convolution kernels) of a set of random signals or multi-signals. Since the convolution kernels obtained by SAFD decomposition are complex numbers, few deep learning methods directly deal with such complex convolution kernels. To this end, we propose a dual-branch network to extract deep features from hyperspectral images using both real and imaginary parts of convolutional kernels. After deep feature extraction using DSAFDNet, we further investigate the classification performance of different classifiers on the extracted features. Experimental results show that the proposed method outperforms some HSI classification methods with similar principles. Moreover, compared with other state-of-the-art deep learning methods, the proposed method can achieve better classification performance.

Hyperspectral image (HSI) classification has become one of the most significant tasks in the field of hyperspectral analysis. However, classifying each pixel in HSI accurately is challenging due to the curse of dimensionality and limited training samples. In this paper, we present an HSI classification architecture called camera spectral response network (CSR-Net), which can learn the optimal camera spectral response (CSR) function for HSI classification problems and effectively reduce the spectral dimensions of HSI. Specifically, we design a convolutional layer to simulate the capturing process of cameras, which learns the optimal CSR function for HSI classification. Then, spectral and spatial features are further extracted by spectral and spatial attention modules. On one hand, the learned CSR can be implemented physically and directly used to capture scenes, which makes the image acquisition process more convenient. On the other hand, compared with ordinary HSIs, we only need images with far fewer bands, without sacrificing the classification precision and avoiding the curse of dimensionality. The experimental results of four popular public hyperspectral datasets show that our method, with only a few image bands, outperforms state-of-the-art HSI classification methods which utilize the full spectral bands of images.

Hyperspectral images (HSI) classification has been a research hotspot in the remote sensing field. Deep learning methods such as Cube-CNN have been applied to address the HSI classification problem. However, mainstream frameworks exist a performance gap to train Cube-CNN, since they are not designed for processing high dimensional data like HSI. To close this gap, we propose a Multi-GPU Training Framework (MTFC) for Cube-CNN-based HSI classification. We first design a Parallel Neighbor Pixel Extraction (PNPE) algorithm for efficiently generating 3-dimensional cube samples from raw data. Then, to fully exploit massive GPU parallelism and realize unique characteristics of HSI and Cube-CNN, we employ optimizations in MTFC such as task division, fine-grained mapping between tasks and GPU thread blocks, shared memory usage reduction, etc. Finally, to further improve training speed, we take advantage of CUDA streams and multiple GPUs to train a mini-batches of data samples simultaneously. An extensive set of experiments highlights that MTFC constantly outperforms the two baselines Caffe and Theano for all measured metrics across all system configurations, while offering the same level of classification accuracy. The speedup is up to <inline-formula><tex-math notation="LaTeX">$3.6x$</tex-math></inline-formula> when using a single GPU and MTFC can achieve a rough linear scaling on multiple GPUs.

Deep learning methods, especially convolutional neural networks(CNN), have been widely used in hyperspectral image(HSI) classification. Recently, graph convolutional networks (GCN) have shown great potential in HSI classification problem. However, the existing GCN-based methods have several problems. First, the existing methods rely too much on the adjacency matrix, which cannot be changed during training. Furthermore, most of them can only use a single kind of feature, and fail to extract the spectral-spatial information from the HSI. Finally, for the existing GCN-based methods, it is difficult to achieve the same accuracy as the mature CNN methods. In this paper, we propose a spectral-spatial hypergraph convolutional neural network (S²HCN) for HSI classification. Compared with the existing GCN-based methods, S²HCN has the following advantages. Different from the adjacency matrix that is fixed during training of GCN, S²HCN can dynamically update the weight of the hyperedge during training, which reduces the reliance on prior information to a certain extent. In addition, S²HCN generates hyperedges from the spectral and spatial features independently, and adopts the incidence matrix composed of all hyperedges to replace the adjacency matrix in GCN. In this way, the spectral and spatial features can be better integrated. Finally, compared to a simple graph structure, the hypergraph structure can express the high-dimensional relationships in the data, which is beneficial to classification problems. Sufficient experiments on two popular HSI datasets have proved the effectiveness of S²HCN.

Due to a lack of labeled samples, deep learning methods generally tend to have poor classification performance in practical applications. Few-shot learning (FSL), as an emerging learning paradigm, has been widely utilized in hyperspectral image (HSI) classification with limited labeled samples. However, the existing FSL methods generally ignore the domain shift problem in cross-domain scenes and rarely explore the associations between samples in the source and target domain. To tackle the above issues, a graph-based domain adaptation FSL (GDAFSL) method is proposed for HSI classification with limited training samples, which utilizes the graph method to guide the domain adaptation learning process in a uniformed framework. First, a novel deep residual hybrid attention network (DRHAN) is designed to extract discriminative embedded features efficiently for few-shot HSI classification. Then, a graph-based domain adaptation network (GDAN), which combines graph construction with domain adversarial strategy, is proposed to fully explore the domain correlation between source and target embedded features. By utilizing the fully explored domain correlations to guide the domain adaptation process, a domain invariant feature metric space is learned for few-shot HSI classification. Comprehensive experimental results conducted on three public HSI datasets demonstrate that GDAFSL is superior to the state-of-the-art with a small sample size.

Hyperspectral image (HSI) classification has become a hot topic in the field of remote sensing. In general, the complex characteristics of hyperspectral data make the accurate classification of such data challenging for traditional machine learning methods. In addition, hyperspectral imaging often deals with an inherently nonlinear relation between the captured spectral information and the corresponding materials. In recent years, deep learning has been recognized as a powerful feature-extraction tool to effectively address nonlinear problems and widely used in a number of image processing tasks. Motivated by those successful applications, deep learning has also been introduced to classify HSIs and demonstrated good performance. This survey paper presents a systematic review of deep learning-based HSI classification literatures and compares several strategies for this topic. Specifically, we first summarize the main challenges of HSI classification which cannot be effectively overcome by traditional machine learning methods, and also introduce the advantages of deep learning to handle these problems. Then, we build a framework that divides the corresponding works into spectral-feature networks, spatial-feature networks, and spectral-spatial-feature networks to systematically review the recent achievements in deep learning-based HSI classification. In addition, considering the fact that available training samples in the remote sensing field are usually very limited and training deep networks require a large number of samples, we include some strategies to improve classification performance, which can provide some guidelines for future studies on this topic. Finally, several representative deep learning-based classification methods are conducted on real HSIs in our experiments.

Hyperspectral image (HSI) classification is an important technology for hyperspectral data analysis, which has the challenge of high dimensionality and limited training samples. Moreover, due to the spatial continuity of material distribution, it is reasonable to improve the performance of HSI classification from the local similarity viewpoint, i.e., pixels in the same local region have a high probability to share similar features. Therefore, in this paper, a novel method based on local similarity and kernel low-rank representation is proposed to improve the classification accuracy for HSI. First, taking into consideration that pixels in the same homogeneous region share similar spectral features (termed local similarity), the HSI is segmented by the superpixel segmentation algorithm to obtain the homogeneous regions. Then, a model of kernel low-rank representation based on local similarity (KLRRLS) is proposed for HSI classification. In the model of KLRRLS, the nonlinear structure characteristics of HSI are extracted by employing the kernel trick, and the low-rank constraint is used as a prior to restrict the recovery coefficient matrix in each segmentation region, thus efficiently solving the linear non-separable problem and exploiting the contextual information of HSI. Finally, the classification results are generated by fusing the result of each segmented map via the maximum voting mechanism. Experimental results on three real HSI datasets demonstrate that the proposed method can effectively improve the classification accuracy.

Traditional hyperspectral image (HSI) classification methods mainly include machine learning and convolutional neural network. However, they extremely depend on the large training samples. To obtain high accuracy on limited training samples, we propose a novel end-to-end spectral-spatial multi-scale network (SSMSN) for HSI classification. The SSMSN uses the multi-scale spectral module and the multi-scale spatial module to extract discriminative multi-scale spectral and multi-scale spatial features separately. In the multi-scale spectral module and spatial module, the multi-scale Res2Net block structure can learn multi-scale features at a granular level and increase the range of receptive fields by constructing the hierarchical residual-like connection within one single residual block. To alleviate the overfitting problem and further improve the classification accuracy on limited training samples, we adopt a simple but effective hinge cross-entropy loss function to train the SSMSN at the dynamic learning rate. A large number of experimental results demonstrate that on the Indiana Pines, University of Pavia, Kennedy Space Center, and Salinas Scene data sets, the proposed SSMSN achieves higher classification accuracy than state-of-the-art methods on limited training samples. Meanwhile, our SSMSN obtains less training and testing time than the popular AUSSC method.

Deep learning methods have demonstrated excellent performance in hyperspectral image (HSI) classification. However, these methods mainly focus on improving the classification accuracy while ignoring their high complexity. By considering that the data formats of both HSIs and network weights can be represented in the form of tensors, we develop a new lightweight tensorized neural network for HSI classification that takes advantage of low-rank tensor decomposition techniques to reduce complexity. Firstly, inspired by tensor train (TT)-based tensorized convolutional layers, a new tensorized 2D convolutional layer based on chain calculation (with better expression ability) is introduced. Based on this innovation, a new lightweight 2D tensorized neural network (2D-TNN) is designed for HSI classification. Furthermore, to better preserve the intrinsic structure of HSI data, a new lightweight 3D tensorized neural network (3D-TNN) is proposed by extending the tensorized 2D convolutional layers to their 3D versions. Quantitative and comparative experiments on three widely used data sets show that the proposed models are able to achieve state-of-the-art performance (with a low number of model parameters) for different training sample sizes, especially for very small training sets.

In recent years, deep-learning-based hyperspectral image (HSI) classification networks have become one of the most dominant implementations in HSI classification tasks. Among these networks, convolutional neural networks (CNNs) and attention-based networks have prevailed over other HSI classification networks. While convolutional neural networks with perceptual fields can effectively extract local features in the spatial dimension of HSI, they are poor at capturing the global and sequential features of spectral–spatial information; networks based on attention mechanisms, for example, Transformer, usually have better ability to capture global features, but are relatively weak in discriminating local features. This paper proposes a fusion network of convolution and Transformer for HSI classification, known as FusionNet, in which convolution and Transformer are fused in both serial and parallel mechanisms to achieve the full utilization of HSI features. Experimental results demonstrate that the proposed network has superior classification results compared to previous similar networks, and performs relatively well even on a small amount of training data.