A Deformable Convolutional Neural Network with Spatial-Channel Attention for Remote Sensing Scene Classification

Storing and processing Remote Sensing (RS) images require large amounts of memory space and computing resources. Consequently, RS images are compressed and stored in various compression formats, such as JPEG2000. However, the processing of RS images for machine interpretation and understanding still necessitates the deployment of an image decompression stage in its entirety, followed by a computationally demanding image analysis pipeline. The image analysis stage is commonly composed of machine learning techniques, such as Deep Convolutional Neural Network (DCNN) models. Classification of remote sensing images is among the most common image analysis tasks. In the scope of this paper, we propose a sub-band image based classification method for the Remote Sensing Scene Classification (RSSC) task in the JPEG2000 compressed domain. The proposed approach exploits the already available sub-band image coefficients to classify RS images without needing for full decompression. Our study shows that our method increases the high frequency information in the LL sub-band and allows the image to contain more detail, leading to improved classifier performance while taking advantage of the partial decompression method.

Remote sensing scene classification is an important technology, which is widely used in military and civil applications. However, it is still a challenging problem due to the complexity of scene images. Recently, the development of remote sensing satellite and sensor devices has greatly improved the spatial resolution and semantic information of remote sensing images. Therefore, we propose a novel remote sensing scene classification approach to enhance the performance of scene classification. First, a spatial and channel-wise attention module is proposed to adequately utilize the spatial and feature information. Compare with other methods, channel-wise module works on the feature maps with diverse levels and pays more attention to semantic-level features. On the other hand, spatial attention module promotes correlation between foreground and classification result. Second, a novel classifier named broad-classifier is designed to enhance the discriminability. It greatly reduces the cost of computing in the meantime by broad learning system. The experimental results have show that our classification method can effectively improve the average accuracies on remote sensing scene classification data sets.

Accurate classification of remote sensing (RS) images is perennial topic of interest in the RS community. Recently, transfer learning, especially for fine-tuning pre-trained convolutional neural networks (CNNs), has been proposed as a feasible strategy for RS scene classification. However, because the target domain (i.e., the RS images) and the source domain (e.g., ImageNet) are quite different, simply using the model pre-trained on an ImageNet dataset presents some difficulties. The RS images and the pre-trained models need to be properly adjusted to build a better classification system. In this study, an adaptive learning strategy for transferring a CNN-based model is proposed. First, an adaptive transform is used to adjust the original size of the RS image to a certain size, which is tailored to the input of the subsequent pre-trained model. Then, an adaptive transferring model is proposed to automatically learn what knowledge from the pre-trained model should be transferred to the RS scene classification model. Finally, in combination with a label smoothing approach, adaptive label is presented to generate soft labels based on the statistics of the classification model predictions for each category, which is beneficial for learning the relationships between the target and non-target categories of scenes. In general, the proposed methods adaptively manage the input, model, and label simultaneously, which leads to better classification performance for RS scene classification. The proposed methods are tested on three widely-used data sets and the obtained results show that the proposed methods provide competitive classification accuracy compared to the state-of-the-art methods.

Because of the challenges associated with the difficulty of accurately labeling the remote sensing (RS) scene images and the need to identify new scene classes, few-shot learning has shown significant advantages in addressing the remote sensing scene classification (RSSC) tasks, leading to a growing interest. However, due to the scale variations of targets and irrelevant complex background in scene images, the current few-shot methods exist the following problems: the problem of the extraction capability of feature extractor in the few-shot mechanism; the problem of the separability of few-shot RS scene images classifier. To solve the above problems, an approach, called few-shot RSSC via subspace based on multiscale feature learning is introduced in this work. We first design a multiscale feature learning technique to address scale variations of the targets in the scene images. Concretely, different branches are utilized to learn scene features at various scales. The self-attention mechanism is embedded in each branch to incorporate the understanding of the global information in the different scale features. After that, a multiscale feature fusion operation, incorporating channel attention, will be devised to effectively merge the different scale features, so as to obtain a more precise feature representation of RS scene images. Furthermore, the subspace is utilized to capture the shared characteristics of each category, to reduce the impact of the complex irrelevant backgrounds in the scene images. The results of our experiments conducted on the public available RS scene datasets demonstrate the strong competitiveness of our approach.

Learning powerful discriminative features is the key for remote sensing scene classification. Most existing approaches based on convolutional neural network (CNN) have achieved great results. However, they mainly focus on global-based visual features while ignoring object-based location features, which is important for large-scale scene classification. There are a large number of scene-related ground objects in remote sensing images, as well as Graph convolutional network (GCN) has the potential to capture the dependencies among objects. This article introduces a novel two-stream architecture that combines global-based visual features and object-based location features, so as to improve the feature representation capability. First, we extract appearance visual features from whole scene image based on CNN. Second, we detect ground objects and construct a graph to learn the spatial location features based on GCN. As a result, the network can jointly capture appearance visual information and spatial location information. To the best of authors' knowledge, we are the first to investigate the dependencies among objects in remote sensing scene classification task. Extensive experiments on two datasets show that our framework improves the discriminative ability of features and achieves competitive accuracy against other state-of-the-art approaches.

Convolutional neural networks (CNNs) have recently shown remarkable performance in remote sensing scene image classification. However, long-range dependencies (e.g. non-local similarities) within the scene are often ignored by CNNs. To address this issue, in this paper we develop a new high-order self-attention network (HoSA) for remote sensing scene classification. Specifically, we embed two novel modules, i.e., a self-attention module and a high order pooling module, into off-the-shelf CNN models and then fine-tune the whole network. The advantages of our newly proposed HoSA network are twofold. Firstly, with the self-attention module, the HoSA can capture long-range dependencies within the scenes for high-level semantic feature extraction. Secondly, by means of its high-order pooling mechanism, our newly developed HoSA can further explore high-order information contained in the features. Our experiments with a widely used remote sensing scene data set demonstrate that the proposed HoSA network exhibits better classification performance than the baseline and several well-known methods.

This study presents an improved data augmentation technique for the classification of remote sensing (RS) scenes. The method is called Quality-based Sample Selection (QSS) data augmentation technique. It is based on generating a large number of samples using geometric transformations and then selecting the best ones based on a quality criterion. Sample images are generated online, i.e. during training of the convolutional neural network (CNN) model. For each training sample image, a few images are generated randomly using a set of geometric transformations. Then, the generated images are passed to the CNN model being trained, and the predicted probabilities are used to evaluate the quality of the images. The motivation behind this research is that we aim to augment the training set with new images that will benefit the learning of the CNN model. Therefore, an objective method to evaluate the quality of the new images is the CNN model itself. The image with the best criteria score is used to augment the training set. Several quality criteria based on the prediction probabilities of the CNN model are considered including entropy, breaking-ties and our own proposed criteria called True Label Probability (TLP). QSS is tested on five common RS scene classification datasets: UCMerced, Optimal31, RSSCN7, AID and NWPU-RS45. It has outperformed all previous methods except for one case involving NWPU-RS45 dataset. For the 10%–90% train–test split, QSS achieved 94.51, 81.72, 89.89, 94.09 and 93.98, respectively, for the mentioned datasets. While for the 20%–80% train–test split, it achieved 97.09, 88.37, 94.01, 95.71 and 94.71, respectively, for the listed datasets. Thus, this work presents a novel data augmentation method that uses data with higher quality instead of randomly selected. The paper proves experimentally that the proposed method improves the classification accuracy of RS scene classification.

Remote sensing scene classification (RSSC) is to accurately assign semantic labels to remote sensing images by analyzing scene contents. Recently, many algorithms have made significant progress in improving the classification accuracy of RSSC. However, a large number of parameters and floating point operations are needed to achieve that end in those approaches, resulting in high complexity. To address the issue, we propose a novel RSSC algorithm, dubbed FishermaskFormer, which aggressively decimates features in the convolutional backbone via a novel masking operation with a proposed Fisher Discriminant Analysis Criterion, and then designs a lightweight transformer block to drive the classification loss. This is aimed at offering a flexible and effective framework for preserving classification accuracy while significantly reducing the complexity. The proposed transformer design employs a new grouping index that assigns multi-headed transformer groups by maximizing the information interactions in each group. Compared with leading lightweight RSSC methods, experimental results show this proposed framework achieves higher classification accuracy while having the similar low complexity.

Despite deep neural networks (DNNs) have been widely applied in remote sensing (RS) scene classification and achieved satisfying performance, the vulnerability of DNNs towards adversarial examples significantly degrades their performance. Moreover, the relatively limited labeled samples of RS scene classification make DNNs more likely to overfit, leading to weak generalizability and noise sensitivity. This may result in DNNs being more vulnerable to adversarial examples. Consequently, the defense of adversarial examples is of crucial importance to improve both the generalizability and robustness of DNNs in the RS scene classification task. However, few studies have been conducted on defense for RS scene classification, especially ignoring the intrinsic characteristics of RS images. In this paper, an effective defense framework for RS scene classification, named reconstruction-assisted and distance-optimized adversarial training (RDAT), is proposed to defend adversarial examples. In order to solve the problems caused by high interclass similarity, a distance-optimized (DO) strategy is designed for adversarial training to strengthen the learning of underfitting content, increase the interclass distance, and improve the robustness of the networks. Furthermore, in order to generate high quality samples for adversarial training, a reconstruction-assisted (RA) block is proposed to eliminate adversarial perturbations in adversarial examples. Specifically, in this block, by swin transformer (SwinT) block and multi-scale convolution (MSC) block, SwinT-MSC-UNet (SMUNet) is constructed to fully extract global and multi-scale local features to adapt to the characteristics of RS images with large variance of ground object scales. Extensive experiments on the benchmark datasets, i.e., UC Merced (UCM) and Aerial Image Dataset (AID), have demonstrate that the proposed RDAT can effectively resist multiple adversarial attacks and yield superior results than other defense methods for RS scene classification.

Manual and automatic annotation of Remote Sensing (RS) scenes are rather complex tasks which may unavoidably introduce some degree of mislabeled data in large-scale archives. In this regard, noisy annotations become an important constraint for deep metric learning-based RS characterization methods since most of them are trained in a supervised way. To address this problem, here we investigate the use of deep metric learning for characterizing RS scenes with noisy labels. Specifically, we consider the Normalized Softmax Loss and develop a robust extension, i.e., the Robust Normalized Softmax Loss (RNSL), in order to effectively capture the semantic relationships among RS scenes with mislabeled ground-truth information. The conducted experiments, using the K-NN classifier and two benchmark RS image archives, show the potential of the proposed approach with respect to other state-of-the-art methods.

The existing convolutional neural network (CNN) models have shown excellent performance in remote sensing scene classification. However, the structure of such models is becoming more and more complex, and the learning of low-level features is difficult to interpret. To address this problem, in this study, we introduce lie group machine learning into the CNN model, try to combine both approaches to extract more distinguishing ability and effective features, and propose a novel network model, namely, the lie group regional influence network (LGRIN). First, manifold space samples of the lie group are obtained by mapping, and then, the features of the lie group are extracted after the operations of image decomposition and integral image calculation. Second, the multidilation pooling is integrated into the CNN architecture. At the same time, the image regional influence network module is designed to guide the attention of the classification model by using the regional-level supervision of the decomposition. Finally, the fusion features are classified, and the predicted results are obtained. Our model takes full advantage of regional influence, lie group kernel function, and lie group feature learning. Moreover, our model produces satisfactory performance on three public and challenging data sets: Aerial Image Dataset (AID), UC Merced, and NWPU-RESISC45. The experimental results verify that, compared with the state-of-the-art methods, this method is more explanatory and achieves higher accuracy.

Remote sensing scene classification (RSSC) is a fundamental but challenging task in the discipline of remote sensing (RS). Significant recent advancements have been achieved in RSSC utilizing convolution neural network. The increasing amount of RS images has resulted in two tough problems: intra-class diversity and inter-class similarity. Existing approaches cannot effectively concentrate on both global and local key features simultaneously. Therefore, in this letter, we propose a dual-branch global-local attention network (DBGA-Net) to solve this problem. Specifically, effective complementary features are first extracted from two distinct models. Following that, it was designed that global-local attention module, which concurrently focuses on both the entire image global features and local pertinent parts within it, would be necessary to successfully extract valid information from RS images. In the end, the fusion module efficiently combines the extracted complementary features to yield more comprehensive scene information. To verify the effectiveness of DBGA-Net, experiments are conducted on three scene classification datasets, 99.86%, 97.60% and 96.06% experimental results are obtained, respectively, with significant classification performance of the proposed method compared with the state-of-the-art (SOTA) method. Our code and images are available at https://github.com/ZhouYao1020/DBGANet.

The fusion of convolutional neural networks (CNNs) and vision transformers (ViTs) is a powerful strategy for remote sensing scene classification (RSSC), leveraging their complementary strengths in local texture and global context modeling. To fully exploit these complementary strengths for addressing the inter-class similarity and high intra-class variability caused by the complex spatial arrangements of RS scenes, this work introduces the hierarchically aligned bidirectional CNN-Transformer network (HAB-Net), which fosters deeper, two-way integration through two novel components. First, Hierarchical Attention Gating (HAG) distills multi-level attention from the ViT into spatial gates, compelling the CNN to focus on semantically critical regions. In turn, CNN-driven Feedback Attention (CFA) completes the loop by feeding these enhanced, detail-rich CNN features back into the ViT, refining its tokens via cross-attention. Ultimately, a dual-stream head merges these mutually reinforced features for scene classification. Extensive experiments on the UCM, AID, and NWPU-RESISC45 benchmarks demonstrate that HAB-Net achieves state-of-the-art performance. Ablation studies confirm the indispensable contributions of both HAG and CFA, validating the effectiveness of the proposed deep bidirectional integration strategy.

Scene classification has become an important research topic in remote sensing (RS) field. Typical solution relies on labeling a large enough set of the RS scenes manually using expert opinion if needed, then training the algorithm on this set to learn how to correctly classify other new scenes. The best performance deep learning models required a large labeled dataset for training. Accordingly, there is great need to develop intelligent machine learning algorithm that can learn to classify RS datasets containing new unseen classes from few labeled samples only. This problem is known as few-shot machine learning. In this work we develop a deep few-shot learning method for the classification of RS scenes. The proposed method is based on prototypical deep neural networks combined with SqueezeNet pre-trained CNN for image embedding. In this paper, we report preliminary results using the two RS scene datasets UC Merced and optimal31.

With the popular use of high-resolution satellite images, remote sensing scene classification has always been a hot research topic in its related areas. However, limited to the issues of remote sensing datasets including the small scale of scene classes, the lack of rich label information and so on, it is quite challenging for deep learning methods to learn powerful feature representation. To overcome this problem, we propose a rotation-invariant feature learning and joint decision-making method based on Siamese convolutional neural networks with the combination of identification and verification models. Firstly, a novel data augmentation strategy is proposed specially for the Siamese model to learning rotation-invariant features. Secondly, a joint decision mechanism is introduced in our method, which is realized by the identification and verification model to better improve the classification performance. The proposed method can not only suppress problems caused by lack of rich label samples but also improve the robustness of Siamese convolutional neural networks. Experimental results demonstrate that the proposed method is effective and efficient for remote sensing scene classification.