Level merging attention based on dense network for remote sensing image scene classification

Remote sensing image scene classification is a challenging task. With the development of deep learning, methods based on convolutional neural networks (CNNs) have made great achievements in remote sensing image scene classification. Since the training of a CNN requires a large number of labeled samples, a generative adversarial network (GAN) for sample generation represents a new opportunity to solve the problem of the limited samples. However, most of the existing GAN-based sample generation methods can only generate unlabeled samples, instead of samples labeled with the corresponding scene category. In this article, to solve the problem, a supervised progressive growing generative adversarial network (SPG-GAN) is proposed for remote sensing image scene classification. The proposed method can generate labeled samples for the remote sensing image scene classification, significantly improving the classification accuracy in the case of limited samples. The SPG-GAN method has two main improvements. First, a conditional generative framework for labeled samples is proposed, in which the label information is added in the channel dimension as the input. By considering the constraints of the label information in the loss function, the network can be trained in the direction of a specific category. As a result, the network can generate remote sensing image scene classification samples with label categories. Second, a progressive growing sample generation method is introduced. In order to ensure that the generated samples have more spatial details, they are generated by progressively adding modules to the generator and discriminator, thereby ensuring that the generated sample is of better quality. After testing on two benchmark datasets and carrying out a large-scale experiment in the central area of the city of Wuhan in China, it was found that the proposed method can obtain a superior scene classification accuracy in the case of limited samples.

In recent years, with the rapid development of computer vision, increasing attention has been paid to remote sensing image scene classification. To improve the classification performance, many studies have increased the depth of convolutional neural networks (CNNs) and expanded the width of the network to extract more deep features, thereby increasing the complexity of the model. To solve this problem, in this paper, we propose a lightweight convolutional neural network based on attention-oriented multi-branch feature fusion (AMB-CNN) for remote sensing image scene classification. Firstly, we propose two convolution combination modules for feature extraction, through which the deep features of images can be fully extracted with multi convolution cooperation. Then, the weights of the feature are calculated, and the extracted deep features are sent to the attention mechanism for further feature extraction. Next, all of the extracted features are fused by multiple branches. Finally, depth separable convolution and asymmetric convolution are implemented to greatly reduce the number of parameters. The experimental results show that, compared with some state-of-the-art methods, the proposed method still has a great advantage in classification accuracy with very few parameters.

Unsupervised domain adaptation (UDA) is an important solution to reduce the bias between the labeled source domain and the unlabeled target domain. It has attracted more attention for optical remote sensing image scene classification and retrieval. Currently, most of the previous work is devoted to closed-set UDA. In fact, the target domain often contains unknown classes. Moreover, some open UDA methods mine structural information of the target domain directly from the type knowledge of the source domain, and less directly from the unlabeled data of the target domain. In this paper, we propose a new self-supervised-driven open-set UDA method combining contrastive self-supervised learning with consistency self-training for optical remote sensing scene classification and retrieval. Specifically, a contrastive self-supervised learning network is introduced to learn discriminative features from the unlabeled target domain data. Moreover, a novel open-set class learning module is developed based on two-level confidence rules and the consistency self-training strategy, which can obtain reliable unknown class samples for co-training. Finally, an open-set dataset including six cross-domain scenarios is constructed based on three public datasets and several experiments are conducted with eleven state-of-the-art domain adaptation methods. Experimental results demonstrate that our proposed method achieves superior performances on the six open-set cross-domain scenarios in both scene classification and retrieval. Especially, our method improves the overall classification accuracies by 9.72% to 24.06% and improve mean average retrieval precisions by 8.06% to 16.21% on the complex UCMD (source domain) → NWPU (target domain) scenario, compared with the other eleven state-of-the-art methods.

Remote sensing image scene classification has a high application value in the agricultural, military, as well as other fields. A large amount of remote sensing data is obtained every day. After learning the new batch data, scene classification algorithms based on deep learning face the problem of catastrophic forgetting, that is, they cannot maintain the performance of the old batch data. Therefore, it has become more and more important to ensure that the scene classification model has the ability of continual learning, that is, to learn new batch data without forgetting the performance of the old batch data. However, the existing remote sensing image scene classification datasets all use static benchmarks and lack the standard to divide the datasets into a number of sequential learning training batches, which largely limits the development of continual learning in remote sensing image scene classification. First, this study gives the criteria for training batches that have been partitioned into three continual learning scenarios, and proposes a large-scale remote sensing image scene classification database called the Continual Learning Benchmark for Remote Sensing (CLRS). The goal of CLRS is to help develop state-of-the-art continual learning algorithms in the field of remote sensing image scene classification. In addition, in this paper, a new method of constructing a large-scale remote sensing image classification database based on the target detection pretrained model is proposed, which can effectively reduce manual annotations. Finally, several mainstream continual learning methods are tested and analyzed under three continual learning scenarios, and the results can be used as a baseline for future work.

Remote sensing image scene classification faces challenges, such as the difference in semantic granularity of different scene categories and the imbalance of the number of samples, which cause the wrong features learning for deep convolutional networks (DCNs). This article proposes a multiple granularity semantic learning network (MGSN), including multiple granularity semantic learning (MGSL) and nonuniform sampling augmentation (NUA) modules. Specifically, the MGSL module makes full use of different granularities of semantic information of scenes, guiding the network to learn global and local features simultaneously. And, the relationship between semantic features of different granularity has been explored, based on which the learning of coarse-grained features helps to improve the learning of fine-grained semantic features. It shows that learning fine-grain semantics can inhibit learning coarse-grain semantic features. The NUA module combines sampling and sample augmentation to balance the sample distribution, which can avoid overfitting caused by oversampling. The proposed MGSN achieved state-of-the-art classification accuracy on two large-scale remote sensing image scene classification datasets, Million-AID and NWPU-RESISC45. Under 10<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and 20<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> training samples of the NWPU-RESISC45 dataset, MGSN achieves 91.92<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and 94.33<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> top-1 accuracy, respectively. In experiments conducted on the Million-AID dataset, the proposed MGSN performed best among 18 DCNs. In comparison to the baseline, FixEfficientNet, MGSN improved the accuracy of top-1 and top-5 by 10.63<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and 5.47<inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula>, respectively, with low complexity costs.

Remote sensing image scene classification is to label remote sensing images as a specific scene category by understanding the semantic information of the images. It is an essential link in remote sensing image analysis and interpretation and has important research value. Convolutional neural networks (CNNs) have been dominant in remote sensing image scene classification due to their powerful feature extraction capabilities. The general trend has been to make deeper and wider CNN architectures to achieve higher classification accuracy. However, these advances to improve accuracy enlarge the network, creating too many parameters and high computational costs. Large models are difficult to deploy on resource-constrained edge devices for practical applications. Furthermore, CNNs can effectively capture local information but are weak in extracting global features. To overcome these drawbacks, we propose a novel knowledge distillation (KD) based method by employing Swin Transformer as a teacher network for guiding MobileNetV2 with Parameter-Free Attention (MobileNetV2-PFA). First, we modify MobileNetV2 by introducing PFA into the inverted bottleneck block; this improvement helps the model learn more latent and robust features without extra parameters. Second, Swin Transformer is an excellent architecture for capturing long-range dependencies via shifted window-based attention. So, we utilize the long-range dependency information from the Swin Transformer to assist MobileNetV2-PFA training through KD. Experimental results on the challenging NWPU-RESISC45 dataset show that the proposed method outperforms the original MobileNetV2 in classification accuracy with low computational consumption.

Remote sensing image (RSI) scene classification is the foundation and important technology of ground object detection, land use management and geographic analysis. During recent years, convolutional neural networks (CNNs) have achieved significant success and are widely applied in RSI scene classification. However, crafted images that serve as adversarial examples can potentially fool CNNs with high confidence and are hard for human eyes to interpret. For the increasing security and robust requirements of RSI scene classification, the adversarial example problem poses a serious problem for the classification results derived from systems using CNN models, which has not been fully recognized by previous research. In this study, to explore the properties of adversarial examples of RSI scene classification, we create different scenarios by testing two major attack algorithms (i.e., the fast gradient sign method (FGSM) and basic iterative method (BIM)) trained on different RSI benchmark datasets to fool CNNs (i.e., InceptionV1, ResNet and a simple CNN). In the experiment, our results show that CNNs of RSI scene classification are also vulnerable to adversarial examples, and some of them have a fooling rate of over 80%. These adversarial examples are affected by the architecture of CNNs and the type of RSI dataset. InceptionV1 has a fooling rate of less than 5%, which is lower than the others. Adversarial examples generated on the UCM dataset are easier than other datasets. Importantly, we also find that the classes of adversarial examples have an attack selectivity property. Misclassifications of adversarial examples of RSIs are related to the similarity of the original classes in the CNN feature space. Attack selectivity reveals potential classes of adversarial examples and provides insights into the design of defensive algorithms in future research.

SummarySince last decade, deep learning has made exceptional progress in various fields of artificial intelligence including image and voice recognition, natural language processing. Inspired by these successes, researchers are now applying deep learning techniques to classification of scenes in remote sensing images. The purpose of remote sensing image scene classification is to classify remote sensing scenes according to their content. These images display a complex structure due to the variety of landforms as well as the distance between the image collection instrument and earth. In our review, we discussed 76 relevant papers published on this topic over the past 6 years. The review conducts a comparison analysis based on the overall accuracy parameter to provide insight into the effectiveness of different methods on different proportions of the dataset. The five classes of techniques we describe are convolutional neural networks, autoencoders, generative adversarial networks, vision transformers, and few‐shot learning. Future directions are discussed in this review in order to enhance the effectiveness of deep learning‐based scene classification approaches. This article concludes with an overview of the proposed method to enhance the accuracy in classifying remote sensing images.

Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various datasets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning datasets and methods for scene classification is still lacking. In addition, almost all existing datasets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale dataset, termed "NWPU-RESISC45", which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This dataset contains 31,500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 (i) is large-scale on the scene classes and the total image number, (ii) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion, and (iii) has high within-class diversity and between-class similarity. The creation of this dataset will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed dataset and the results are reported as a useful baseline for future research.

In recent times, many efforts have been made to improve remote sensing image scene classification, especially using popular deep convolutional neural networks. However, most of these methods do not consider the specific scene orientation of the remote sensing images. In this letter, we propose the improved oriented response network (IORN), which is based on the ORN, to handle the orientation problem in remote sensing image scene classification. We propose average active rotating filters (A-ARFs) in the IORN. While IORNs are being trained, A-ARFs are updated by a method that is different from the ARFs of the ORN, without additional computations. This change helps IORN improve its ability to encode orientation information and speeds up optimization during training. We also propose Squeeze-ORAlign (S-ORAlign) by adding a squeeze layer to ORAlign of ORN. With the squeeze layer, S-ORAlign can address large-scale images, unlike ORAlign. An ablation study and comparison experiments are designed on a public remote sensing image scene classification data set. The experimental results demonstrate the effectiveness and better performance of the proposed model over that of other state-of-the-art models.

This work proposes a remote sensing image scene classification method based on stackable attention structure. This method utilizes a stackable attention structure to represent the feature information in remote sensing images as a linear function, changing the previous processing methods for global and local information in remote sensing images to avoid the need for memory intensive attention maps. Through experiments, it has been shown that the scene classification method we proposed based on stackable attention structure has higher accuracy and smaller model size than ResNet152. This provides a new approach for constructing remote sensing image scene classification models, especially for processing large-sized remote sensing images.

The classification of aerial scenes has been extensively studied as the basic work of remote sensing image processing and interpretation. However, the performance of remote sensing image scene classification based on deep neural networks is limited by the number of labeled samples. In order to alleviate the demand for massive labeled samples, various methods have been proposed to apply semi-supervised learning to train the classifier using labeled and unlabeled samples. However, considering the complex contextual relationship and huge spatial differences, the existing semi-supervised learning methods bring different degrees of incorrectly labeled samples when pseudo-labeling unlabeled data. In particular, when the number of labeled samples is small, it affects the generalization performance of the model. In this article, we propose a novel semi-supervised learning method with early labeled and small loss selection. First, the model learns the characteristics of simple samples in the early stage and uses multiple early models to screen out a small number of unlabeled samples for pseudo-labeling based on this characteristic. Then, the model is trained in a semi-supervised manner by combining labeled samples, pseudo-labeled samples, and unlabeled samples. In the training process of the model, small loss selection is used to further eliminate some of the noisy labeled samples to improve the recognition accuracy of the model. Finally, in order to verify the effectiveness of the proposed method, it is compared with several state-of-the-art semi-supervised classification methods. The results show that when there are only a few labeled samples in remote sensing image scene classification, our method is always better than previous methods.

Remote sensing image scene classification (RSISC) is an active task in the remote sensing community and has attracted great attention due to its wide applications. Recently, the deep convolutional neural networks (CNNs)-based methods have witnessed a remarkable breakthrough in performance of remote sensing image scene classification. However, the problem that the feature representation is not discriminative enough still exists, which is mainly caused by the characteristic of inter-class similarity and intra-class diversity. In this paper, we propose an efficient end-to-end local-global-fusion feature extraction (LGFFE) network for a more discriminative feature representation. Specifically, global and local features are extracted from channel and spatial dimensions respectively, based on a high-level feature map from deep CNNs. For the local features, a novel recurrent neural network (RNN)-based attention module is first proposed to capture the spatial layout information and context information across different regions. Gated recurrent units (GRUs) is then exploited to generate the important weight of each region by taking a sequence of features from image patches as input. A reweighed regional feature representation can be obtained by focusing on the key region. Then, the final feature representation can be acquired by fusing the local and global features. The whole process of feature extraction and feature fusion can be trained in an end-to-end manner. Finally, extensive experiments have been conducted on four public and widely used datasets and experimental results show that our method LGFFE outperforms baseline methods and achieves state-of-the-art results.

Fine-tuning of pretrained convolutional neural networks (CNNs) has been proven to be an effective strategy for remote sensing image scene classification, particularly when a limited number of labeled data sets are available for training purposes. However, such a fine-tuning process often needs that the input images are resized into a fixed size to generate input vectors of the size required by fully connected layers (FCLs) in the pretrained CNN model. Such a resizing process often discards key information in the scenes and thus deteriorates the classification performance. To address this issue, in this paper, we introduce a scale-free CNN (SF-CNN) for remote sensing scene classification. Specifically, the FCLs in the CNN model are first converted into convolutional layers, which not only allow the input images to be of arbitrary sizes but also retain the ability to extract discriminative features using a traditional sliding-window-based strategy. Then, a global average pooling (GAP) layer is added after the final convolutional layer so that input images of arbitrary size can be mapped to feature maps of uniform size. Finally, we utilize the resulting feature maps to create a new FCL that is fed to a softmax layer for final classification. Our experimental results conducted using several real data sets demonstrate the superiority of the proposed SF-CNN method over several well-known classification methods, including pretrained CNN-based ones.

Remote sensing image scene classification with deep learning (DL) is a rapidly growing field that has gained significant attention in the past few years. While previous review papers in this domain have been confined to 2020, an up-to-date review to show the progression of research extending into the present phase is lacking. In this review, we explore the recent articles, providing a thorough classification of approaches into three main categories: Convolutional Neural Network (CNN)-based, Vision Transformer (ViT)-based, and Generative Adversarial Network (GAN)-based architectures. Notably, within the CNN-based category, we further refine the classification based on specific methodologies and techniques employed. In addition, a novel and rigorous meta-analysis is performed to synthesize and analyze the findings from 50 peer-reviewed journal articles to provide valuable insights in this domain, surpassing the scope of existing review articles. Our meta-analysis shows that the most adopted remote sensing scene datasets are AID (41 articles) and NWPU-RESISC45 (40). A notable paradigm shift is seen towards the use of transformer-based models (6) starting from 2021. Furthermore, we critically discuss the findings from the review and meta-analysis, identifying challenges and future opportunities for improvement in this domain. Our up-to-date study serves as an invaluable resource for researchers seeking to contribute to this growing area of research.