A Remote Sensing Image Classification Method Based on Detail Attention Sampling and Teacher-Student Network

With the growing demand for better performance of remote sensing (RS) image classification, a variety of methods have been proposed in RS image classification field in recent years. In general, there are two categories of RS image classification methods: pixel-based (PB) approach and object-based (OB) approach. In this paper, RS image classification methods are reviewed from the perspective of PB approach and OB approach and, specifically, the development and characteristics of a promising methodology for RS image classification named support vector machine (SVM) are surveyed. SVM is particularly popular in the RS field since it can deal with small-sized training dataset and provide higher classification accuracy than some traditional methods like maximum likelihood classifier. Besides, SVM has advantages of high memory-efficiency and strong generalization. However, SVM-based approaches also suffer from some problems. For instance, SVM-based methods tend to overfit due to inappropriate choice of kernel functions and it is inefficient for them to determine the optimum kernel function parameters as well as to process hyperspectral images. This paper also proposes the improvement of SVM-based methods aiming to address the limitations and improve the performance of SVM in RS image classification field. Moreover, future directions for SVM in RS image classification field are presented, expecting to help researchers to find possible research focuses in the future.

In this paper, we propose a complementary self-supervised mask model based on teacher-student networks. This model contains a student network, a teacher network, and a mask prediction module. The student's network is an encoder structure, and the teacher's network consists of encoders and decoders. The teacher and student network encoders are used for learning image representations and have the same network structure and model parameters. The pre-training has two pre-tasks: First, the mask image block representation predicted by the decoder in the teacher network predicts the actual image pixels through the mask prediction module. Then, we introduce a comparative learning loss function to compare the outputs of the teacher and student modules in representation space. This paper proposes a complementary masking mechanism to reduce the gap between upstream and downstream mismatches in the pre-training model based on mask image modeling (MIM). For example, a complete picture is an input into the teacher and the student network. For the teacher network, the input picture is randomly masked off, for example, 75 %; the student network masks the remaining part of the input picture, 25 %. The student network masks the rest (25%) of the input image. The pre-trained model proposed in this paper has been pre-trained on COCO and other data sets, and downstream tasks are performed on four conventional data sets. By comparing with some of the latest self-supervised pre-trained models, it is proved that the pre-trained model proposed in this paper can learn better representational information.

Remote sensing classification, as an important means of urban planning and construction, has been widely concerned. Urban land use classification is extremely challenging tasks because of some land covers are spectrally too similar to be separated using only the spectral information of remote sensing image. Object-oriented remote sensing image classification method overcomes the drawbacks of traditional pixel-based classification method. It combines the spectral, special structure and texture features of the images, can effectively avoid the phenomenon of "different objects share the same spectrum" or "the same objects differ in spectrum. Support Vector Machine (SVM) is an excellent tool for remote sensing classification. Combination of both can develop their own advantages to do high-resolution remote sensing image classification. Using a public image in Harbin city as an example, classification based on object-oriented method and SVM has achieved better results than traditional pixel-based classification method.

Specific emitter identification (SEI) is a non-cryptographic authentication technique to provide an extra security layer for wireless devices, which has promising applications. However, the traditional methods of SEI are only available in limited equipments. In actual application scenarios, new devices (as new classes) are constantly appearing. In this paper, an effective class incremental learning (CIL) method is proposed for SEI, named class-incremental with self-training and prototype augmentation (CISP). It is a teacher-student network. Firstly, the teacher network trained by the old-class data is utilized to instruct the student network to adapt the new classes while retaining the old-class knowledge through the knowledge distillation (KD) techniques. Secondly, in order to mitigate the problem of favoring the new classes, weight aligning (WA) method is introduced to balance the weights of the new-class and old-class classification layers in the student network. Lastly, the old-class samples are recalled from the unlabeled dataset by the student network and input into the teacher network. Then the feature prototypes of the old classes are constructed and augmented. This would further ease the imbalance between the old and new classes and alleviate the problem of noisy pseudo-labels. Experiment results on the real AIS-100 dataset and ADS-B-100 dataset with the number of the initial classes being 20 and 20 classes per incremental step demonstrate that the proposed method can achieve an average accuracy of 95.29% and 95.84%, respectively. It effectively mitigates the catastrophic forgetting of the model and is superior to the state-of-the-art incremental learning approaches of not saving the old-class samples.

Traditional remote sensing image surveys rely on field observations, which have shortcomings such as low efficiency and limited coverage, making it difficult to meet the needs of urban planning and management. In response to this, this article conducts in-depth research on remote sensing image classification methods based on graph neural networks (GNN). This study adopts a joint framework of GAT (Graph Attention Network)+SLC (Simple Linear Iterative Clustering)+Hybrid Pooling. Use SLIC algorithm to perform superpixel segmentation on hyperspectral data and construct a graph structure that integrates spectral spatial features. Utilizing GAT's attention mechanism to enhance feature interaction among neighboring nodes, combined with a hybrid pooling strategy to balance feature dimensionality reduction and key information preservation. The experiment was conducted on the public datasets WHU-RS19 and UC Merced, using traditional CNN, random forest, and minimum distance methods as baseline models. The significance of the differences was verified through independent sample t-test. By expanding the sample coverage, optimizing the model regularization strategy, and continuously iterating training, the generalization performance of the model has been significantly improved. Therefore, from the comparison of classification performance, graph NN classification is superior to minimum distance classification. This method utilizes perspective remote sensing data to achieve accurate ground target classification and dynamic change information extraction, providing reliable data support for urban land survey and management.

Numerous transformer-based medical image segmentation methods have been proposed and achieved good segmentation results. However, it is still a challenge to train and deploy transformer networks to mobile medical devices due to a large number of model parameters. To resolve the training and model parameter problems, in this paper, we propose a Transformer-based network for Medical Image Segmentation using Knowledge Distillation named MISTKD. The MISTKD consists of a teacher network and a student network. It achieves comparable performance to state-of-the-art transformer works using fewer parameters by employing the teacher network to train the student network. The training can be implemented by extracting the sequence in the teacher and student encoder networks during the training procedure. The losses between sequences are further calculated, thus the student network can learn from the teacher network. The experimental results on Synapse show that the proposed work achieves competitive performance using only one-eighth parameters.

Knowledge distillation is a widely used neural network model compression technique. In general, the knowledge distillation transfer the knowledge from a large pre-trained teacher network with superior performance to a small student network enables the student network to achieve better performance. This paper proposes a triplet knowledge distillation framework (abbreviated as TKD), which introduces a smaller assistant network into the knowledge distillation structure. The performance of the assistant network is lower than that of the student network. During the training of the TKD, by minimizing the Mean Squared Error(MSE) loss function, the output of the student network will closer to the output of the teacher network and further from that of the assistant network. Therefore, the student network can learn more expressive knowledge from the teacher network while throwing away mistaken knowledge in the assistant network. Finally, the student network achieves a surprising performance even superior to the teacher network. We have demonstrated the effectiveness of TKD by extensive experiments on benchmark datasets(CIFAR-10, CIFAR-100, SVHN, STL-10). When using VGGNet as an experimental model, the student network VGGNet13 achieving 94.29%, 75.30%, 95.53%, and 87.61% accuracy on the CIFAR-10, CIFAR-100, SVHN, and STL-10 datasets, improved by 1.24%, 2.81%, 0.40%, and 2.32%, respectively.

Knowledge Distillation (KD) aims to distill the dark knowledge of a high-powered teacher network into a student network, which can improve the capacity of student network and has been successfully applied to semantic segmentation. However, the standard knowledge distillation approaches merely represent the supervisory signal of teacher network as the dark knowledge, while ignoring the impact of network architecture during distillation. In this paper, we found that the student network with a more similar architecture against the teacher network obtains more performance gain from distillation. Therefore, a more generalized paradigm for knowledge distillation is to distill both the soft-label and the structure of the teacher network. We propose a novel Structural Distillation (SD) method which introduces the structural similarity constraints into vanilla knowledge distillation. We leverage Neural Architecture Search technique to search optimal student structure for semantic segmentation from a well-designed search space, which mimics the given teacher both in terms of soft-label and network structure. Experiment results make clear that our proposed method outperforms both the NAS with conventional Knowledge Distillation and human-designed methods, and achieves sota performance on the Cityscapes dataset under various platform-aware latency constraints. Furthermore, the best architecture discovered on Cityscapes also transfers well to the PASCAL VOC2012 dataset.

Knowledge distillation is a deep learning method that mimics the way that humans teach, i.e., a teacher network is used to guide the training of a student one. Knowledge distillation can generate an efficient student network to facilitate deployment in resource-constrained edge computing devices. Existing studies have typically mined knowledge from a teacher network and transferred it to a student one. The latter can only passively receive knowledge but cannot understand how the former acquires the knowledge, thus limiting the latter’s performance improvement. Inspired by the old Chinese saying “Give a man a fish and you feed him for a day; teach a man how to fish and you feed him for a lifetime,” this work proposes a Skill-transferring Knowledge Distillation (SKD) method to boost a student network’s ability to create new valuable knowledge. SKD consists of two main meta-learning networks: Teacher Behavior Teaching and Teacher Experience Teaching. The former captures the process of a teacher network’s learning behavior in the hidden layers and can predict the teacher network’s subsequent behavior based on previous ones. The latter models the optimal empirical knowledge of a teacher network’s output layer at each learning stage. With their help, a teacher network can provide its actions to a student one in the subsequent behavior and its optimal empirical knowledge in the current stage. The overall performance of SKD is verified through its application to multiple object recognition tasks and comparison with the state of the art.

Knowledge distillation (KD) is an effective model compression technique where a compact student network is taught to mimic the behavior of a complex and highly trained teacher network. In contrast, Mutual Learning (ML) provides an alternative strategy where multiple simple student networks benefit from sharing knowledge, even in the absence of a powerful but static teacher network. Motivated by these findings, we propose a single-teacher, multi-student framework that leverages both KD and ML to achieve better performance. Furthermore, an online distillation strategy is utilized to train the teacher and students simultaneously. To evaluate the performance of the proposed approach, extensive experiments were conducted using three different versions of teacher-student networks on benchmark biomedical classification (MSI vs. MSS) and object detection (Polyp Detection) tasks. Ensemble of student networks trained in the proposed manner achieved better results than the ensemble of students trained using KD or ML individually, establishing the benefit of augmenting knowledge transfer from teacher to students with peer-to-peer learning between students.

A text corpus typically contains two types of context information -- global context and local context. Global context carries topical information which can be utilized by topic models to discover topic structures from the text corpus, while local context can train word embeddings to capture semantic regularities reflected in the text corpus. This encourages us to exploit the useful information in both the global and the local context information. In this paper, we propose a unified language model based on matrix factorization techniques which 1) takes the complementary global and local context information into consideration simultaneously, and 2) models topics and learns word embeddings collaboratively. We empirically show that by incorporating both global and local context, this collaborative model can not only significantly improve the performance of topic discovery over the baseline topic models, but also learn better word embeddings than the baseline word embedding models. We also provide qualitative analysis that explains how the cooperation of global and local context information can result in better topic structures and word embeddings.

A tucker decomposition based knowledge distillation for intelligent edge applications

A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve model performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on knowledge distillation and self-distillation. The method consists of two parts: learning in the teacher network and self-teaching in the student network. Learning in the teacher network allows the student network to use knowledge from the teacher network. Self-teaching in the student network is to build a multi-exit network based on self-distillation and provide deep features as supervised information for training. In the inference stage, we use ensembles to vote on the classification results of multiple sub-models in the student network. The experimental results demonstrate the superior performance of our method compared with a traditional knowledge distillation method and a self-distillation-based multi-exit network.

Knowledge distillation, in which the parameter values learned in a large teacher network are transferred to a smaller student network, is a popular and effective network compression method. Recently, researchers have proposed methods to improve the performance of a student network by using a Generative Adverserial Network (GAN). However, because a GAN is an architecture that is ideally used to create realistic synthetic images, a pure GAN architecture may not be ideally suited for knowledge distillation. In knowledge distillation for image signal processing, synthetic images do not need to be realistic, but instead should include features that help the training of the student network. In the proposed Generative Image Processing (GIP) method, this is accomplished by using only the generator portion of a GAN and utilizing special techniques to capture the distinguishing feature capability of the teacher network. Experimental results show that the GIP method outperforms knowledge distillation using GANs as well as training using only knowledge distillation.

Knowledge distillation guided by multiple homogeneous teachers