Domain-Collaborative Contrastive Learning for Hyperspectral Image Classification

Sensors are devices that output signals for sensing physical phenomena and are widely used in all aspects of our social production activities. The continuous recording of physical parameters allows effective analysis of the operational status of the monitored system and prediction of unknown risks. Thanks to the development of deep learning, the ability to analyze temporal signals collected by sensors has been greatly improved. However, models trained in the source domain do not perform well in the target domain due to the presence of domain gaps. In recent years, many researchers have used deep unsupervised domain adaptation techniques to address the domain gap between signals collected by sensors in different scenarios, i.e., using labeled data in the source domain and unlabeled data in the target domain to improve the performance of models in the target domain. This survey first summarizes the background of recent research on unsupervised domain adaptation with time series sensor data, the types of sensors used, the domain gap between the source and target domains, and commonly used datasets. Then, the paper classifies and compares different unsupervised domain adaptation methods according to the way of adaptation and summarizes different adaptation settings based on the number of source and target domains. Finally, this survey discusses the challenges of the current research and provides an outlook on future work. This survey systematically reviews and summarizes recent research on unsupervised domain adaptation for time series sensor data to provide the reader with a systematic understanding of the field.

Bionic limb control through myoelectric pattern recognition, offering intuitive decoding of motor intent, can improve the quality of life for individuals with amputations. However, most work on pattern recognition only uses a small subset of the myoelectric data generated during daily life, to train an Artificial Neural Network (ANN) via Supervised Learning (SL). Scenarios substantially different from the recording session e.g. different limb positions, can lead to misclassifications by the ANN during everyday usage of the bionic limb. Recording labeled data from all scenarios encountered in daily life could alleviate the problem, but would be prohibitively time consuming. Unsupervised Domain Adaptation (UDA) offers a solution by leveraging unlabeled data from a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">target domain</i>, not represented in the labeled dataset i.e. the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">source domain</i>, to calibrate ANNs for improved performance. In this study we explore the potential of two UDA algorithms for domain shifts in myoelectric pattern recognition: Domain Adversarial Neural Networks (DANN) and Sliced Wasserstein Discrepancy (SWD). Offline evaluation identified SWD as the best-performing algorithm, which was subsequently validated in online experiments with 11 participants. Using UDA improved the performance on the target domain by 19% compared to an ANN trained through SL on data from the source domain only. Indeed, it nearly matched the performance of an ANN trained through SL on labeled data from both the source and target domain. This is the first time UDA was validated in online experiments as a viable approach to overcome domain shift problems caused by changes in limb position.

With the rapid development of the remote sensing monitoring and computer vision technology, the deep learning method has made a great progress to achieve applications such as earth observation, climate change and even space exploration. However, the model trained on existing data cannot be directly used to handle the new remote sensing data, and labeling the new data is also time-consuming and labor-intensive. Unsupervised Domain Adaptation (UDA) is one of the solutions to the aforementioned problems of labeled data defined as the source domain and unlabeled data as the target domain, i.e., its essential purpose is to obtain a well-trained model and tackle the problem of data distribution discrepancy defined as the domain shift between the source and target domain. There are a lot of reviews that have elaborated on UDA methods based on natural data, but few of these studies take into consideration thorough remote sensing applications and contributions. Thus, in this paper, in order to explore the further progress and development of UDA methods in remote sensing, based on the analysis of the causes of domain shift, a comprehensive review is provided with a fine-grained taxonomy of UDA methods applied for remote sensing data, which includes Generative training, Adversarial training, Self-training and Hybrid training methods, to better assist scholars in understanding remote sensing data and further advance the development of methods. Moreover, remote sensing applications are introduced by a thorough dataset analysis. Meanwhile, we sort out definitions and methodology introductions of partial, open-set and multi-domain UDA, which are more pertinent to real-world remote sensing applications. We can draw the conclusion that UDA methods in the field of remote sensing data are carried out later than those applied in natural images, and due to the domain gap caused by appearance differences, most of methods focus on how to use generative training (GT) methods to improve the model’s performance. Finally, we describe the potential deficiencies and further in-depth insights of UDA in the field of remote sensing.

Accurate evaluation of orchard areas from remote sensing images is of great importance in economic and ecological aspects. In practice, the differences in distributions between remote sensing images and the lack of data labels make the semantic segmentation model impossible to use in new data. Unsupervised domain adaptation (UDA) methods can improve the performance of the model in the target domain by aligning the source domain and the target domain. However, due to the class mismatch problem and the interference of high-dimensional feature complexity, most UDA methods cannot achieve satisfactory results in orchard areas segmentation task. To address these issues, we propose an UDA model for orchard areas segmentation by developing a class feature aggregate discriminator. The class feature aggregate discriminator is designed to distinguish intra-domain classes and align inter-domain classes, and class feature aggregate can represent class information of different domains, which helps the model to avoid the interference of complex information. In addition, adversarial loss reweighting is introduced to the novel model, which makes the segmentation model pay more attention to the orchard areas. To verify the effectiveness of the proposed method, we conducted extensive experiments in three different remote sensing images around Yichang City. Compared to the baseline model, the proposed approach improves IoU by 27.68%, and we achieve high gains of 6.07% in IoU over other UDA methods. The larger gain indicates that our proposed method has great potential in cross-domain orchard areas segmentation.

Unsupervised domain adaptation (UDA) techniques, extensively studied in hyperspectral image (HSI) classification, aim to use labeled source domain data and unlabeled target domain data to learn domain invariant features for cross-scene classification. Compared to natural images, numerous spectral bands of HSIs provide abundant semantic information, but they also increase the domain shift significantly. In most existing methods, both explicit alignment and implicit alignment simply align feature distribution, ignoring domain information in the spectrum. We noted that when the spectral channel between source and target domains is distinguished obviously, the transfer performance of these methods tends to deteriorate. Additionally, their performance fluctuates greatly owing to the varying domain shifts across various datasets. To address these problems, a novel shift-sensitive spatial-spectral disentangling learning (S4DL) approach is proposed. In S4DL, gradient-guided spatial-spectral decomposition (GSSD) is designed to separate domain-specific and domain-invariant representations by generating tailored masks under the guidance of the gradient from domain classification. A shift-sensitive adaptive monitor is defined to adjust the intensity of disentangling according to the magnitude of domain shift. Furthermore, a reversible neural network is constructed to retain domain information that lies not only in semantic but also the shallow-level detailed information. Extensive experimental results on several cross-scene HSI datasets consistently verified that S4DL is better than the state-of-the-art UDA methods. Our source code will be available at https://github.com/xdu-jjgs/IEEE_TNNLS_S4DL.

Unsupervised domain adaptation (UDA) has been widely studied with the goal of transferring knowledge from a label-rich source domain to a related but unlabeled target domain. Most UDA techniques achieve this by reducing the feature discrepancies between the two domains to learn domain-invariant feature representations. While domain-invariant feature representations can reduce the differences between the source and target domains, excessively simplifying these differences may cause the model to overlook important domain-specific features, resulting in a decline in transfer learning effectiveness. To address this issue, this paper proposes a novel Gaussian Mixture Model for graph domain adaptation (GMM). This model effectively reduces the distributional bias between the source and target domains by modeling the distribution differences on a graph structure. GMM leverages the local structural information of the graph and the clustering capability of the Gaussian mixture model to automatically learn the latent mapping relationships between the source and target domains. To the best of our knowledge, this is the first work to introduce a Gaussian mixture model into UDA. Extensive experimental results on three standard benchmarks demonstrate that the proposed GMM algorithm outperforms state-of-the-art unsupervised domain adaptation methods in terms of performance.

Unsupervised domain adaptation (UDA) is an important solution to reduce the bias between the labeled source domain and the unlabeled target domain. It has attracted more attention for optical remote sensing image scene classification and retrieval. Currently, most of the previous work is devoted to closed-set UDA. In fact, the target domain often contains unknown classes. Moreover, some open UDA methods mine structural information of the target domain directly from the type knowledge of the source domain, and less directly from the unlabeled data of the target domain. In this paper, we propose a new self-supervised-driven open-set UDA method combining contrastive self-supervised learning with consistency self-training for optical remote sensing scene classification and retrieval. Specifically, a contrastive self-supervised learning network is introduced to learn discriminative features from the unlabeled target domain data. Moreover, a novel open-set class learning module is developed based on two-level confidence rules and the consistency self-training strategy, which can obtain reliable unknown class samples for co-training. Finally, an open-set dataset including six cross-domain scenarios is constructed based on three public datasets and several experiments are conducted with eleven state-of-the-art domain adaptation methods. Experimental results demonstrate that our proposed method achieves superior performances on the six open-set cross-domain scenarios in both scene classification and retrieval. Especially, our method improves the overall classification accuracies by 9.72% to 24.06% and improve mean average retrieval precisions by 8.06% to 16.21% on the complex UCMD (source domain) → NWPU (target domain) scenario, compared with the other eleven state-of-the-art methods.

In unsupervised domain adaptation (UDA), a classifier for a target domain is trained with labeled source data and unlabeled target data. Existing UDA methods assume that the source data come from the same source domain (i.e., single-source scenario) or from multiple source domains, whose feature spaces have the same dimension ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">homogeneous</i> ) but different distributions (i.e., multihomogeneous-source scenario). However, in the real world, for a specific target domain, we probably have multiple different-dimension ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">heterogeneous</i> ) source domains, which do not satisfy the assumption of existing UDA methods. To remove this assumption and move forward to a realistic UDA problem, this article presents a shared-fuzzy-equivalence-relation neural network (SFERNN) for addressing the multisource heterogeneous UDA problem. The SFERNN is a five-layer neural network containing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> source branches and one target branch. The network structure of the SFERNN is first confirmed by a novel fuzzy relation called multisource shared fuzzy equivalence relation. Then, we optimize parameters of the SFERNN via minimizing cross-entropy loss on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> source branches and the distributional discrepancy between each source branch and the target branch. Experiments distributed across eight real-world datasets are conducted to validate the SFERNN. This testing regime demonstrates that the SFERNN outperforms the existing single-source heterogeneous UDA methods, especially when the target domain contains few data.

Hyperspectral image cross-scene classification (HSICC) tasks are confronted with tremendous challenges due to spectral shift phenomena across scenes and the tough work of obtaining labels. Unsupervised domain adaptation has proven its effectiveness in tackling this issue, but it has a fatal limitation of intending to narrow the disparity between source and target domains by utilizing fully labeled source data and unlabeled target data. However, it is costly even to attain labels from source domains in many cases, rendering sufficient labeling as used in prior work impractical. In this work, we investigate an extreme and realistic scenario where unsupervised domain adaptation methods encounter sparsely labeled source data when handling HSICC tasks, namely, few-shot unsupervised domain adaptation. We propose an end-to-end refined bi-directional prototypical contrastive learning (RBPCL) framework for overcoming the HSICC problem with only a few labeled samples in the source domain. RBPCL captures category-level semantic features of hyperspectral data and performs feature alignment through in-domain refined prototypical self-supervised learning and bi-directional cross-domain prototypical contrastive learning, respectively. Furthermore, our framework introduces the class-balanced multicentric dynamic prototype strategy to generate more robust and representative prototypes. To facilitate prototype contrastive learning, we employ a Siamese-style distance metric loss function to aggregate intra-class features while increasing the discrepancy of inter-class features. Finally, extensive experiments and ablation analysis implemented on two public cross-scene data pairs and three pairs of self-collected ultralow-altitude hyperspectral datasets under different illumination conditions verify the effectiveness of our method, which will further enhance the practicality of hyperspectral intelligent sensing technology.

Unsupervised domain adaptation aims to learn domain-invariant features across domains to transfer knowledge from a well-labeled source domain to an unlabeled target domain. Recently, some unsupervised domain adaptation methods focus on semantically aligning data distributions with pseudo-labels of the target domain. However, semantic alignment based on pseudo-labels has potential risks, e.g., inaccurate pseudo-labeling from classifier, and error accumulation from pseudo-label bias. To alleviate these risks, we propose a novel self-supervision based semantic alignment (S3A) approach for unsupervised domain adaptation, which can jointly incorporate the source alignment and cross-domain target alignment for better semantic alignment across domains. S3A consists of a two-stage semantic alignment procedure with self-supervision. One is to capture the discriminative structure of source domain by aligning source data to source class prototypes, and the other is to match each target data to its neighbor in source domain with self-supervision. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our proposed method, compared with the representative adversarial learning and self-supervised learning based unsupervised domain adaptation methods.

Unsupervised cross-domain person re-identification by instance and distribution alignment

Deep models may have disappointing performance in real applications due to the domain shifts in data distributions between the source and target domain. Although a few unsupervised domain adaptation methods have been proposed to make the pre-train models effective on target domain datasets, constraints like data privacy, security, and transmission limits restrict access to VHR remote sensing images, making existing unsupervised domain adaptation methods almost ineffective in specific change detection areas. Therefore, we propose a source-free unsupervised domain adaptation change detection structure to complete specific change detection tasks, using only the pre-trained source model and unlabelled target data. The GAN-based source generation component is designed to generate synthetic source data, which, to some extent, reflects the distribution of the source domain. Moreover, these data can be utilised in model knowledge transfer. The model adaptation component facilitates knowledge transfer between models by minimising the differences between deep features, using AAM (Attention Adaptation Module) to extract the difference between high-level features, meanwhile we proposed ISM (Intra-domain Self-supervised Module) to train target model in a self-supervised strategy in order to improve the knowledge adaptation. Our SFDA-CD framework demonstrates superior accuracy over existing unsupervised domain adaptation change detection methods, which has 0.6% cIoU and 1.5% F1 score up in cross-regional tasks and 1.4% cIoU and 1.9% F1 score up in cross-scenario tasks, proving that it can effectively reduce the domain shift between the source and target domains even without access to source data. Additionally, it can facilitate knowledge transfer from the source model to the target model.

Rethinking unsupervised domain adaptation for semantic segmentation

Reducing bias to source samples for unsupervised domain adaptation

Unsupervised deep domain adaptation algorithm for video based human activity recognition via recurrent neural networks