- Research Article
45
- 10.1016/j.patcog.2021.108238
Universal multi-Source domain adaptation for image classification
- Aug 14, 2021
- Pattern Recognition
- Yueming Yin + 3 more +3
Universal multi-Source domain adaptation for image classification
Learning invariant representation for unsupervised domain adaptive thorax disease classification
Universal multi-Source domain adaptation for image classification
Universal multi-Source domain adaptation for image classification
A two-layer regression network for robust and accurate domain adaptation
A two-layer regression network for robust and accurate domain adaptation
Multi-Source Collaborative Contrastive Learning for Decentralized Domain Adaptation
Unsupervised multi-source domain adaptation aims to obtain a model working well on the unlabeled target domain by reducing the domain gap between the labeled source domains and the unlabeled target domain. Considering the data privacy and storage cost, data from multiple source domains and target domain are isolated and decentralized. This data decentralization scenario brings the difficulty of domain alignment for reducing the domain gap between the decentralized source domains and target domain, respectively. For conducting domain alignment under the data decentralization scenario, we propose Multi-source Collaborative Contrastive learning for decentralized Domain Adaptation (MCC-DA). The models from other domains are used as the bridge to reduce the domain gap. On the source domains and target domain, we penalize the inconsistency of data features extracted from the source domain models and target domain model by contrastive alignment. With the collaboration of source domain models and target domain model, the domain gap between decentralized source domains and target domain is reduced without accessing the data from other domains. The experiment results on multiple benchmarks indicate that our method can reduce the domain gap effectively and outperform the state-of-the-art methods significantly.
Read moreLabel-efficient visual recognition with transfer learning
Visual recognition, encompassing tasks such as image classification, object detection, and semantic segmentation, has been a fundamental challenge in computer vision research. It also serves as the foundation for numerous applications, including autonomous driving, remote sensing, and robotics. The emergence of deep learning has significantly advanced this field, enabling remarkable achievements through the use of end-to-end trainable deep neural networks (DNNs). However, this remarkable progress comes at the cost of requiring vast amounts of annotated training data, the collection of which is often extremely labor-intensive and time-consuming. This constraint presents a significant barrier to the scalability and adaptability of various deep learning approaches. Transfer learning, which aims to transfer the off-the-shelf learned knowledge from one or multiple related source domains to target domains, is a promising way to mitigate this constraint. In this thesis, we explore label-efficient visual recognition with various transfer learning techniques, where the objective is to learn effective visual representations for target domains without any annotations by transferring the off-the-shelf knowledge learned in different but related source domains. Specifically, we investigate three representative types of transfer learning problems, including unsupervised domain adaptation, black-box unsupervised domain adaptation and test-time prompt tuning. As one of the most typical transfer learning problems, unsupervised domain adaptation aims to adapt a visual recognition model that is trained on labeled source-domain data to perform well on a related but unlabeled target-domain data. The primary challenge is to mitigate the distribution differences between the source and target domains, which are also referred to as domain gap or domain discrepancy, such that the model can generalize well on the target domain. In this thesis, we propose three innovative unsupervised domain adaptation techniques designed to efficiently transfer knowledge from labeled source domains to unlabeled target domains across a range of visual recognition tasks. Specifically, we explore diverse approaches to align domains from multiple perspectives (e.g., data or feature space alignment), thereby reducing the domain gap and enabling to perform well on unlabeled target domain. Black-box unsupervised domain adaptation aims to learn visual recognition model for target domain with the initial predictions of target data obtained by black-box source-domain model. In another word, during the adaptation process, only the model predictions of target data are available while either source data or source models are inaccessible. Compared to conventional unsupervised domain adaptation, black-box unsupervised domain adaptation offers notable advantages in terms of data privacy and flexibility, allowing adaptation to different target models regardless of the architecture or details of the source-trained black-box model. In this thesis, we analyze the challenges in black-box domain adaptation and design a novel approach for effectively exploiting the initial predictions of target data, leading to better adapted model on unlabeled target data. We further explore label-efficient visual recognition in the recent era of vision-language models. Despite the impressive zero-shot generalization capabilities of vision-language models, they always face unneglectable gaps while applied to target datasets, i.e., a target dataset typically features unique image styles and text formats tailored to its specific tasks. To circumvent this limitation, we explore an effective transfer learning approach for vision-language models, i.e., test-time prompt tuning, where the prompts are optimized as learnable vectors during the test stage without requiring any annotations for target data. Specifically, we examine the limitation of existing test-time prompt tuning methods, and propose a novel approach that could effectively exploit the learned knowledge in vision-language models and learn prompts for target data on-the-fly. Extensive experiments conducted over various visual recognition benchmarks indicate that our proposed methods achieve superior performance over target datasets, enabling to largely mitigate the dependence of labeled data across various visual recognition tasks.
Read moreUnsupervised Remote Sensing Image Semantic Segmentation Based on Multiscale Contrastive Domain Adaptation
Unsupervised domain adaptation for remote sensing image semantic segmentation aims to train a deep model on the labeled source domain and apply it to the unlabeled target domain. However, resolution and scene inconsistencies of cross-domain remote sensing images lead to great distribution differences, which weakens the semantic segmentation effect. To solve the above issues, an unsupervised remote sensing image semantic segmentation method is proposed based on multi-scale contrastive domain adaptation. Firstly, the mean teacher model is introduced into the unsupervised domain adaptation paradigm to generate pseudo-labels for target domain data, thereby achieving the cross-domain segmentation capability. A dynamic class balance sampling method is proposed to mitigate the class imbalance problem in cross-domain data by increasing the sampling frequency of the categories with fewer samples. Then, a data augmentation method called cross-domain mixup is developed to reduce the gap between the source and target domains. Finally, a multi-scale cross-domain contrastive loss is developed, which introduces the contrastive learning to learn domain-consistent features across the source and target domains, resulting in a more coherent and discriminative feature representation. Experimental results show that the proposed method can yield superior performance for unsupervised remote sensing image semantic segmentation.
Read moreA Self-Supervised-Driven Open-Set Unsupervised Domain Adaptation Method for Optical Remote Sensing Image Scene Classification and Retrieval
Unsupervised domain adaptation (UDA) is an important solution to reduce the bias between the labeled source domain and the unlabeled target domain. It has attracted more attention for optical remote sensing image scene classification and retrieval. Currently, most of the previous work is devoted to closed-set UDA. In fact, the target domain often contains unknown classes. Moreover, some open UDA methods mine structural information of the target domain directly from the type knowledge of the source domain, and less directly from the unlabeled data of the target domain. In this paper, we propose a new self-supervised-driven open-set UDA method combining contrastive self-supervised learning with consistency self-training for optical remote sensing scene classification and retrieval. Specifically, a contrastive self-supervised learning network is introduced to learn discriminative features from the unlabeled target domain data. Moreover, a novel open-set class learning module is developed based on two-level confidence rules and the consistency self-training strategy, which can obtain reliable unknown class samples for co-training. Finally, an open-set dataset including six cross-domain scenarios is constructed based on three public datasets and several experiments are conducted with eleven state-of-the-art domain adaptation methods. Experimental results demonstrate that our proposed method achieves superior performances on the six open-set cross-domain scenarios in both scene classification and retrieval. Especially, our method improves the overall classification accuracies by 9.72% to 24.06% and improve mean average retrieval precisions by 8.06% to 16.21% on the complex UCMD (source domain) → NWPU (target domain) scenario, compared with the other eleven state-of-the-art methods.
Read moreUnsupervised Adversarial Domain Adaptation Network for Semantic Segmentation
With the rapid development of deep learning technology, semantic segmentation methods have been widely used in remote sensing data. A pretrained semantic segmentation model usually cannot perform well when the testing images (target domain) have an obvious difference from the training data set (source domain), while a large enough labeled data set is almost impossible to be acquired for each scenario. Unsupervised domain adaptation (DA) techniques aim to transfer knowledge learned from the source domain to a totally unlabeled target domain. By reducing the domain shift, DA methods have shown the ability to improve the classification accuracy for the target domain. Hence, in this letter, we propose an unsupervised adversarial DA network that converts deep features into 2-D feature curves and reduces the discrepancy between curves from the source domain and curves from the target domain based on a conditional generative adversarial networks (cGANs) model. Our proposed DA network is able to improve the semantic labeling accuracy when we apply a pretrained semantic segmentation model to the target domain. To test the effectiveness of the proposed method, experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) 2-D Semantic Labeling data set. Results show that our proposed network is able to stably improve overall accuracy not only when the source and target domains are from the same city but with different building styles but also when the source and target domains are from different cities and acquired by different sensors. By comparing with a few state-of-the-art DA methods, we demonstrate that our proposed method achieves the best cross-domain semantic segmentation performance.
Read moreJoint Distribution Adaptation via Wasserstein Adversarial Training
This paper considers the unsupervised domain adaptation problem, in which we want to find a good prediction function on the unlabeled target domain, by utilizing the information provided in the labeled source domain. A common approach to the domain adaptation problem is to learn a representation space where the distributional discrepancy of the source and target domains is small. Existing methods generally tend to match the marginal distributions of the two domains, while the label information in the source domain is not fully exploited. In this paper, we propose a representation learning approach for domain adaptation, which is addressed as JODAWAT. We aim to adapt the joint distributions of the feature-label pairs in the shared representation space for both domains. In particular, we minimize the Wasserstein distance between the source and target domains, while the prediction performance on the source domain is also guaranteed. The proposed approach results in a minimax adversarial training procedure that incorporates a novel split gradient penalty term. A generalization bound on the target domain is provided to reveal the efficacy of representation learning for joint distribution adaptation. We conduct extensive evaluations on JODAWAT, and test its classification accuracy on multiple synthetic and real datasets. The experimental results justify that our proposed method is able to achieve superior performance compared with various domain adaptation methods.
Read moreHeterogeneous Graph Attention Network for Unsupervised Multiple-Target Domain Adaptation.
Domain adaptation, which transfers the knowledge from label-rich source domain to unlabeled target domains, is a challenging task in machine learning. The prior domain adaptation methods focus on pairwise adaptation assumption with a single source and a single target domain, while little work concerns the scenario of one source domain and multiple target domains. Applying pairwise adaptation methods to this setting may be suboptimal, as they fail to consider the semantic association among multiple target domains. In this work we propose a deep semantic information propagation approach in the novel context of multiple unlabeled target domains and one labeled source domain. Our model aims to learn a unified subspace common for all domains with a heterogeneous graph attention network, where the transductive ability of the graph attention network can conduct semantic propagation of the related samples among multiple domains. In particular, the attention mechanism is applied to optimize the relationships of multiple domain samples for better semantic transfer. Then, the pseudo labels of the target domains predicted by the graph attention network are utilized to learn domain-invariant representations by aligning labeled source centroid and pseudo-labeled target centroid. We test our approach on four challenging public datasets, and it outperforms several popular domain adaptation methods.
Read moreDomain adaptive remote sensing image semantic segmentation with prototype guidance
Domain adaptive remote sensing image semantic segmentation with prototype guidance
Multi-Target Domain Adaptation with Collaborative Consistency Learning
Recently unsupervised domain adaptation for the semantic segmentation task has become more and more popular due to high-cost of pixel-level annotation on real-world images. However, most domain adaptation methods are only restricted to single-source-single-target pair, and can not be directly extended to multiple target domains. In this work, we propose a collaborative learning framework to achieve unsupervised multi-target domain adaptation. An unsupervised domain adaptation expert model is first trained for each source-target pair and is further encouraged to collaborate with each other through a bridge built between different target domains. These expert models are further improved by adding the regularization of making the consistent pixel-wise prediction for each sample with the same structured context. To obtain a single model that works across multiple target domains, we propose to simultaneously learn a student model which is trained to not only imitate the output of each expert on the corresponding target domain, but also to pull different expert close to each other with regularization on their weights. Extensive experiments demonstrate that the proposed method can effectively exploit rich structured information contained in both labeled source domain and multiple unlabeled target domains. Not only does it perform well across multiple target domains but also performs favorably against state-of-the-art unsupervised domain adaptation methods specially trained on a single source-target pair. Code is available at https://github.com/junpan19/MTDA.
Read moreUnlocking Cross-Domain Synergies for Domain Adaptive Semantic Segmentation.
Unsupervised domain adaptation semantic segmentation (UDASS) aims to perform dense prediction on the unlabeled target domain by training the model on a labeled source domain. In this field, self-training approaches have demonstrated strong competitiveness and advantages. However, existing methods often rely on additional training data (such as reference datasets or depth maps) to rectify the unreliable pseudo-labels, ignoring the cross-domain interaction between the target and source domains. To address this issue, in this paper, we propose a novel method for unsupervised domain adaptation semantic segmentation, termed Unlocking Cross-Domain Synergies (UCDS). Specifically, in the UCDS network, we design a new Dynamic Self-Correction (DSC) module that effectively transfers source domain knowledge and generates high-confidence pseudo-labels without additional training resources. Unlike the existing methods, DSC proposes a Dynamic Noisy Label Detection method for the target domain. To correct the noisy pseudo-labels, we design a Dual Bank mechanism that explores the reliable and unreliable predictions of the source domain, and conducts cross-domain synergy through Weighted Reassignment Self-Correction and Negative Correction Prevention strategies. To enhance the discriminative ability of features and amplify the dissimilarity of different categories, we propose Discrepancy-based Contrastive Learning (DCL). The DCL selects positive and negative samples in the source and target domains based on the semantic discrepancies among different categories, effectively avoiding the numerous false negative samples found in existing methods. Extensive experimental results on three commonly used datasets demonstrate the superiority of the proposed UCDS in comparison with the state-of-the-art methods. The project and code are available at https://github.com/wqh011128/UCDS.
Read moreMulti-Source Domain Adaptation with Fuzzy-Rule based Deep Neural Networks
Unsupervised domain adaptation provides a variety of methods to leverage the previously gained knowledge from a labeled source domain to help complete a task from a similar unlabeled target domain. Many existing methods focus on transferring knowledge across single source and single target domains, while few studies deal with multi-source domain adaptation, which is more realistic and challengeable. Existing multi-source domain adaptation methods rarely consider the uncertainty of the transformed knowledge resulting from limited information in target domain. A fuzzy system allows imprecision and ambiguity within transfer, thus it can deal with problems with uncertainty. This work proposes a multi-source domain adaptation method with fuzzy-rule based deep neural networks (MDAFuz). The proposed method first extracts multi-view adapted features and pre-trains source classifiers. Using the learned features and classifiers, training samples are then split into multiple clusters, hence fuzzy rules can be built to learn new classifiers. At the same time, the cluster discriminator is trained to define the membership. Finally, by measuring the similarities among source and target domains using the pseudo target labels and a domain discriminator, the target task is completed by combining all source classifiers with regard to the learned weights. The experiment results on real-world visual datasets show the superiority of the proposed method.
Read moreUniversal Domain Alignment Framework for Classification and Regression Tasks
Unsupervised Domain Adaptation (UDA), a critical technique for transferring knowledge from labeled source domains to unlabeled target domains, holds significant value in both classification and regression tasks. However, existing methods are often designed for a single task, lacking a unified framework that can handle both types of tasks simultaneously. Moreover, while global distribution alignment can reduce the overall discrepancy between the source and target domains, inconsistencies in local feature spaces persist. These local inconsistencies may introduce biases in high-level semantic structures, increasing the variability of feature distributions and significantly weakening the model’s generalization ability in the target domain. To address these challenges, we propose a Universal Domain Alignment Framework (UDAF), which achieves unified modeling for classification and regression tasks while reducing local distribution discrepancies through the Target Space Coupling Module (TSCM) and the Domain Embedding Consistency Loss (DECL). In UDAF, TSCM optimizes the structure of the output space to meet the requirements of both classification and regression tasks. Specifically, for labeled source domain data, we minimize the distance between features and their corresponding class centers to cluster similar samples in the output space. For unlabeled target domain data, pseudo labels are generated using prediction probabilities to estimate target class centers, and the class centers are dynamically updated to optimize the output space structure. Meanwhile, DECL models local feature alignment from a sample-to-domain perspective. By measuring the distance between target domain samples and the orthogonal basis of source domain subspaces, DECL explicitly captures local distribution biases and incorporates them as optimization objectives, further enhancing the fine-grained alignment of cross-domain features. The proposed UDAF is highly extensible and can be seamlessly integrated into existing methods. Comprehensive experiments conducted on seven datasets demonstrate that this approach achieves a performance improvement, especially with an 2.1% improvement on DomainNet dataset.
Read moreACT: Semi-supervised Domain-adaptive Medical Image Segmentation with Asymmetric Co-Training.
Unsupervised domain adaptation (UDA) has been vastly explored to alleviate domain shifts between source and target domains, by applying a well-performed model in an unlabeled target domain via supervision of a labeled source domain. Recent literature, however, has indicated that the performance is still far from satisfactory in the presence of significant domain shifts. Nonetheless, delineating a few target samples is usually manageable and particularly worthwhile, due to the substantial performance gain. Inspired by this, we aim to develop semi-supervised domain adaptation (SSDA) for medical image segmentation, which is largely underexplored. We, thus, propose to exploit both labeled source and target domain data, in addition to unlabeled target data in a unified manner. Specifically, we present a novel asymmetric co-training (ACT) framework to integrate these subsets and avoid the domination of the source domain data. Following a divide-and-conquer strategy, we explicitly decouple the label supervisions in SSDA into two asymmetric sub-tasks, including semi-supervised learning (SSL) and UDA, and leverage different knowledge from two segmentors to take into account the distinction between the source and target label supervisions. The knowledge learned in the two modules is then adaptively integrated with ACT, by iteratively teaching each other, based on the confidence-aware pseudo-label. In addition, pseudo label noise is well-controlled with an exponential MixUp decay scheme for smooth propagation. Experiments on cross-modality brain tumor MRI segmentation tasks using the BraTS18 database showed, even with limited labeled target samples, ACT yielded marked improvements over UDA and state-of-the-art SSDA methods and approached an "upper bound" of supervised joint training.
Read more