Dynamic gated fusion network with hierarchical multi-scale attention for hyperspectral image classification

In the field of hyperspectral image classification, deep learning technology, especially convolutional neural networks, has achieved remarkable progress. However, convolutional neural network models encounter challenges in hyperspectral image classification due to limitations in their receptive fields. Conversely, the global modeling capability of Transformers has garnered attention in hyperspectral image classification. Nevertheless, the high computational cost and inadequate local feature extraction hinder its widespread application. In this study, we propose a novel fusion model of convolutional neural networks and Transformers to enhance performance in hyperspectral image classification, namely the dual-branch multi-granularity convolutional cross-substitution Transformer (DMCCT). The proposed model adopts a dual-branch structure to separately extract spatial and spectral features, thereby mitigating mutual interference and information loss between spectral and spatial data during feature extraction. Moreover, a multi-granularity embedding module is introduced to facilitate multi-scale and multi-level local feature extraction for spatial and spectral information. In particular, the improved convolutional cross-substitution Transformer module effectively integrates convolution and Transformer, reducing the complexity of attention operations and enhancing the accuracy of hyperspectral image classification tasks. Subsequently, the proposed method is evaluated against existing approaches using three classical datasets, namely Pavia University, Kennedy Space Center, and Indian Pines. Experimental results demonstrate the efficacy of the proposed method, achieving significant classification results on these datasets with overall classification accuracies of 98.57%, 97.96%, and 96.59%, respectively. These results establish the superiority of the proposed method in the context of hyperspectral image classification under similar experimental conditions.

Spectral features cannot effectively reflect the differences among the ground objects and distinguish their boundaries in hyperspectral image (HSI) classification. Multi-scale feature extraction can solve this problem and improve the accuracy of HSI classification. The Gaussian pyramid can effectively decompose HSI into multi-scale structures, and efficiently extract features of different scales by stepwise filtering and downsampling. Therefore, this paper proposed a Gaussian pyramid based multi-scale feature extraction (MSFE) classification method for HSI. First, the HSI is decomposed into several Gaussian pyramids to extract multi-scale features. Second, we construct probability maps in each layer of the Gaussian pyramid and employ edge-preserving filtering (EPF) algorithms to further optimize the details. Finally, the final classification map is acquired by a majority voting method. Compared with other spectral-spatial classification methods, the proposed method can not only extract the characteristics of different scales, but also can better preserve detailed structures and the edge regions of the image. Experiments performed on three real hyperspectral datasets show that the proposed method can achieve competitive classification accuracy.

Recently, deep learning (DL)-based hyperspectral image classification (HSIC) has attracted substantial attention. Many works based on the convolutional neural network (CNN) model have been certificated to be significantly successful for boosting the performance of HSIC. However, most of these methods extract features by using a fixed convolutional kernel and ignore multiscale features of the ground objects of hyperspectral images (HSIs). Although some recent methods have proposed multiscale feature extraction schemes, more computing and storage resources were consumed. Moreover, when using CNN to implement HSI classification, many methods only use the high-level semantic information extracted from the end of the network, ignoring the edge information extracted from shallow layers of the network. To settle the preceding two issues, a novel HSIC method based on hierarchical shrinkage multiscale network and the hierarchical feature fusion is proposed, with which the newly proposed classification framework can fuse features generated by both of multiscale receptive field and multiple levels. Specifically, multidepth and multiscale residual block (MDMSRB) is constructed by superposition dilated convolution to realize multiscale feature extraction. Furthermore, according to the change of feature size in different stages of the neural networks, we design a hierarchical shrinkage multiscale feature extraction network by pruning MDMSRB to reduce the redundancy of network structure. In addition, to make full use of the features extracted in each stage of the network, the proposed network hierarchically integrates low-level edge features and high-level semantic features effectively. Experimental results demonstrate that the proposed method achieves more competitive performance with a limited computational cost than other state-of-the-art methods.

The convolutional neural network (CNN) is widely used in the task of hyperspectral image (HSI) classification. However, for the HSI of three-dimensional characteristics, the 2D CNN-based methods will result in losing spatial-spectral information. To solve this problem, this paper proposes a multi-branch 3D-densely connected network for HSI classification. This network is able to reuse features to fully exploit the shallow spatial-spectral information of HSI. Meanwhile, the convolutional kernels of different sizes are used to extract multi-scale spatial-spectral features. Subsequently, spatial attention mechanisms are used to emphasize spatial features and increase the diversity of features. By introducing global average pooling instead of a fully connected layer, the number of parameters in the whole network will be reduced. In order to verify the performance of the proposed method, the experiment results conducted in the Indian Pines, the University of Pavia, Salinas Valley and Houston 2013 datasets show that the proposed method is better than the state-of-the-art methods.

In this research, a deep learning approach for hyperspectral image (HSI) classification was developed, incorporating attention mechanisms, multiscale feature learning, and utilization of unsampled pixels. The proposed model, multiscale attention-based hybrid spectral network and UNet (MSA-HybridSN-U), was evaluated on three benchmark datasets: Indian Pines, University of Pavia, and University of Houston using Overall Accuracy (OA), Average Accuracy (AA), and Kappa index (K). The proposed model MSA-HybridSN-U had high accuracy on Indian Pines (IP), Pavia University (PU), and Houston University (HU) datasets. It achieved the highest average accuracy compared with other models like 3D CNN, HybridSN, M3D-DCNN, DBDA, and ACA-HybridSN and higher than 2D CNN and SVM. For the IP dataset, the OA and AA scores are both 99.71% and 99.65% respectively, indicating a high level of accuracy in classifying the samples. The Kappa statistic, which measures inter-annotator agreement, is also high at 0.997, suggesting that the method is consistent in its predictions. For the PU dataset, the results are even higher with OA and AA scores of 99.97% and 99.89% respectively, and a Kappa statistic of 0.999, indicating even higher accuracy and consistency. Finally, for the HU dataset, the results are similarly high with OA and AA scores of 99.47% and 99.92% respectively, and a Kappa statistic of 0.994. The use of attention mechanisms, multiscale features, and unsampled pixels has improved the classification performance of the model. The combination of spectral and spatial attention modules improved the accuracy the most, with the highest accuracy of 99.97% on the PU dataset. Utilizing unsampled pixels in the classification process resulted in a noticeable improvement in accuracy, particularly on the HU dataset with an improvement of 0.29%. The results show the effectiveness of combining multiscale features and attention modules in improving the accuracy of HSI classification.

Hyperspectral image (HSI) classification has been extensively applied for analyzing remotely sensed images. HSI data consist of multiple bands that provide abundant spatial information. Convolutional neural networks (CNNs) have emerged as powerful deep learning methods for processing visual data. In recent work, CNNs have shown impressive results in HSI classification. In this paper, we propose a hierarchical neural network architecture called feature extraction with hybrid spectral CNN (FE-HybridSN) to extract superior spectral–spatial features. FE-HybridSN effectively captures more spectral–spatial information while reducing computational complexity. Considering the prevalent issue of class imbalance in experimental datasets (IP, UP, SV) and real-world hyperspectral datasets, we apply the focal loss to mitigate these problems. The focal loss reconstructs the loss function and facilitates effective achievement of the aforementioned goals. We propose a framework (FEHN-FL) that combines FE-HybridSN and the focal loss for HSI classification and then conduct extensive HSI classification experiments using three remote sensing datasets: Indian Pines (IP), University of Pavia (UP), and Salinas Scene (SV). Using cross-entropy loss as a baseline, we assess the hyperspectral classification performance of various backbone networks and examine the influence of different spatial sizes on classification accuracy. After incorporating focal loss as our loss function, we not only compare the classification performance of the FE-HybridSN backbone network under different loss functions but also evaluate their convergence rates during training. The proposed classification framework demonstrates satisfactory performance compared to state-of-the-art end-to-end deep-learning-based methods, such as 2D-CNN, 3D-CNN, etc.

Most of the existing spatial-spectral-based hyperspectral image classification (HSIC) methods mainly extract the spatial-spectral information by combining the pixels in a small neighborhood or aggregating the statistical and morphological characteristics. However, those strategies can only generate shallow appearance features with limited representative ability for classes with high interclass similarity and spatial diversity and therefore reduce the classification accuracy. To this end, we present a novel HSIC framework, named deep multiscale spatial-spectral feature extraction algorithm, which focuses on learning effective discriminant features for HSIC. First, the well pretrained deep fully convolutional network based on VGG-verydeep-16 is introduced to excavate the potential deep multiscale spatial structural information in the proposed hyperspectral imaging framework. Then, the spectral feature and the deep multiscale spatial feature are fused by adopting the weighted fusion method. Finally, the fusion feature is put into a generic classifier to obtain the pixelwise classification. Compared with the existing spectral-spatial-based classification techniques, the proposed method provides the state-of-the-art performance and is much more effective, especially for images with high nonlinear distribution and spatial diversity.

Sleep spindles have emerged as valuable biomarkers for assessing cognitive abilities and related disorders, underscoring the importance of their detection in clinical research. However, template matching-based algorithms using fixed templates may not be able to fully adapt to spindles of different durations. Moreover, inspired by the multiscale feature extraction of images, the use of multiscale feature extraction methods can be used to better adapt to spindles of different frequencies and durations. Therefore, this study proposes a novel automatic spindle detection algorithm based on elastic time windows and spatial pyramid pooling (SPP) for extracting multiscale features. The algorithm utilizes elastic time windows to segment electroencephalogram (EEG) signals, enabling the extraction of features across multiple scales. This approach accommodates significant variations in spindle duration and polarization positioning during different EEG epochs. Additionally, spatial pyramid pooling is integrated into a depthwise separable convolutional (DSC) network to perform multiscale pooling on the segmented spindle signal features at different scales. Compared with existing template matching algorithms, this algorithm's spindle wave polarization positioning is more consistent with the real situation. Experimental results conducted on the public dataset DREAMS show that the average accuracy of this algorithm reaches 95.75%, with an average negative predictive value (NPV) of 96.55%, indicating its advanced performance. The effectiveness of each module was verified through thorough ablation experiments. More importantly, the algorithm shows strong robustness when faced with changes in different experimental subjects. This feature makes the algorithm more accurate at identifying sleep spindles and is expected to help experts automatically detect spindles in sleep EEG signals, reduce the workload and time of manual detection, and improve efficiency.

Convolutional neural networks (CNNs), a kind of feedforward neural network with a deep structure, are one of the representative methods in hyperspectral image (HSI) classification. However, redundant information and interclass interference are common and challenging problems in HSI classification. In addition, if the spectral and spatial information is not properly extracted and analyzed, it will affect the classification performance of the network to a great extent. Aiming at these issues, this article proposes an HSI classification method based on an adaptive hash attention mechanism and a lower triangular network (AHA-LT). First, the attention mechanism is introduced in the preprocessing stage, which is composed of the spectral attention module and the adaptive hash spatial attention module in series. Then, the data processed by the attention mechanism are introduced into the lower triangular network (LTNet) to obtain the fused high-dimensional semantic features. Finally, we compress the features and obtain the output classification results through several fully connected layers. Among them, LTNet is composed of 2-D–3-D CNN and multiscale features. The network integrates the characteristics of multibranch, feature fusion, feature compression, and skip connections. Extensive experiments on four widely used HSI data sets show that the proposed method can obtain a great improvement in performance compared with the existing methods.

Gait recognition is a distinctive biometric technique that can identify pedestrians by their walking patterns from considerable distances. A critical challenge in gait recognition lies in effectively acquiring discriminative spatial-temporal representations from silhouettes that exhibit invariance to disturbances. In this paper, we present a novel gait recognition network by aggregating features in the spatial-temporal and view domains, which consists of two-path spatial-temporal feature fusion module and view embedding module. Specifically, two-path spatial-temporal feature fusion module firstly utilizes multi-scale feature extraction (MSFE) to enrich the input features with multiple convolution kernels of various sizes. Then, frame-level spatial feature extraction (FLSFE) and multi-scale temporal feature extraction (MSTFE) are parallelly constructed to capture spatial and temporal gait features of different granularities and these features are fused together to obtain muti-scale spatial-temporal features. FLSFE is designed to extract both global and local gait features by employing a specially designed residual operation. Simultaneously, MSTFE is applied to adaptively interact multi-scale temporal features and produce suitable motion representations in temporal domain. Taking into account the view information, we introduce a view embedding module to reduce the impact of differing viewpoints. Through the extensive experimentation over CASIA-B and OU-MVLP datasets, the proposed method has achieved superior performance to the other state-of-the-art gait recognition approaches.

Detection of surface defects in strip steel is crucial in industrial automation to ensure steel quality. Recently, learning-based detection methods have gained prominence among researchers for their robust capabilities. However, accurately delineating clear area boundaries remains a challenge, particularly in the complex backgrounds of steel surfaces. In this paper, we propose the edge-aware Interactive Refinement Network (EAIRNet) to address this challenge. EAIRNet comprises three core modules: a Multiscale Feature Extractor, a Sal-Edge Interactive Module, and a regional attention guided module (RAGM). Firstly, our network extracts multiscale features using ResNet from the steel surface environment. Subsequently, features from encoder’s stage 3–5 are integrated to derive edge features. The SEIM facilitates interaction between salient and edge features in the decoder stage, thereby enhancing the network’s focus on defect area boundaries and refining detection precision. Additionally, we introduce the RAGM to improve the focus of shallow decoder layers on defect areas. Our EAIRNet’s effectiveness is proven through comprehensive experimental results on the public dataset named SD-Saliency-900, in comparison to state-of-the-art methods.

Adaptive spectral-spatial feature fusion network for hyperspectral image classification using limited training samples

Emission computed tomography provides information on the growth and metabolic status of brain gliomas, but its use of radiopharmaceuticals brings radiation risks and financial costs to patients. These problems will be improved by reducing the dose of radiopharmaceuticals, but it will reduce the quality of brain emission computed tomography (ECT) images, which will lead to low resolution, unclear lesion areas, and blurred boundaries of parts of the lesion areas in the magnetic resonance imaging-emission computed tomography (MRI-ECT) fusion image of gliomas. Therefore, a high-definition fusion of multi-modal images of brain glioma based on a super-resolution reconstruction network based on the adaptive kernel selection (AKS-SRNet) is proposed. First, to solve the problem of loss of detailed structural information in the disease region of fusion images, a correlation-driven feature decomposition fusion network based on multi-scale and fine-grained feature extraction was constructed. The multi-scale feature extraction module and the outlook attention module are introduced into the fusion network to extract multi-scale features and retain more detailed features. Second, to solve the problems of low resolution and unclear lesion areas of MRI and ECT fusion images, a super-resolution reconstruction network based on adaptive kernel selection was constructed. A selective kernel attention module is added to the generator of PAUP-ESRGAN to improve the efficiency of the super-resolution reconstruction network generator. Finally, the experimental results show that the fusion image generated by the proposed high-definition fusion network has a less burr-prone and a clearer contour of the lesion area, which is helpful for doctors in analyzing and diagnosing the glioma accurately.

Hyperspectral image (HSI) classification remains challenging due to the high dimensionality of spectral data, the strong coupling between spatial and spectral information, and the difficulty of learning robust representations under limited annotated samples. To address this, we propose 3D W-Net, a novel deep learning framework that integrates 3D convolution into the U-Net architecture and introduces a parallel–serial multi-scale feature learning strategy tailored for HSIs. The encoder incorporates a parallel spectraldimension multi-scale feature processing module, which employs three 3D convolution kernels with different spectral receptive fields to capture diverse spectral-scale features. An adjacent-scale selective fusion strategy is then applied to enrich spectral representations while reducing redundancy. The decoder features a serial spatial-dimension multiscale feature restoration module that progressively merges high-level semantic and lowlevel spatial details, preserving spatial–spectral correlations and mitigating information loss typically caused by flattening operations. This design forms a distinctive “W”-shaped feature propagation path that enhances multi-scale feature interaction. Extensive experiments on five public datasets(Botswana, Pavia University, Chikusei, Houston 2013, and WHU-Hi-HongHu) demonstrate that 3D W-Net achieves superior performance compared to six state-of-the-art methods, attaining overall accuracies of 99.6%, 99.1%, 100%, 99.3%, and 99.5%, respectively. The results highlight the effectiveness of the proposed parallel–serial multi-scale strategy in improving classification accuracy and generalization capability for complex HSI scenarios.

Semantic segmentation is a crucial method for recognizing and classifying objects in high-resolution remote sensing images (HRRSIs). However, due to the problems of varying target scale and difficulty in determining the edges of small-scale targets in remote sensing images, traditional semantic segmentation models perform poorly. To address this issue, we propose a multi-scale feature enhancement network (MFENet) to improve the segmentation accuracy of small-scale objects in HRRSIs. MFENet considers the differences between objects of different scales and selects more suitable receptive fields to enhance the extraction of multi-scale semantic features. We propose a composite atrous multi-scale feature fusion (CAMFF) module to enhance the extraction of spatial detail and semantic information of features at different scales. In addition, we propose an improved composite atrous spatial pyramid pooling (C-ASPP) module to enhance the network feature extraction capability across multiple scales. We also propose a network structure that combines the C-ASPP module with the efficient channel attention (ECA) module in parallel, which performs better to extract contextual information. Our experimental evaluations on the Potsdam and Vaihingen datasets demonstrate the effectiveness of our Network, It F1 score reaching 93.33% and 94.66% respectively.