Cross Lingual Transfer Learning for Zero-Resource Domain Adaptation

Domain Adaptation (DA) methods are usually carried out by means of simply reducing the marginal distribution differences between the source and target domains, and subsequently using the resultant trained classifier, namely source classifier, for use in the target domain. However, in many cases, the true predictive distributions of the source and target domains can be vastly different especially when their class distributions are skewed, causing the issues of sample selection bias in DA. Hence, DA methods which leverage the source labeled data may suffer from poor generalization in the target domain, resulting in negative transfer. In addition, we observed that many DA methods use either a source classifier or a linear combination of source classifiers with a fixed weighting for predicting the target unlabeled data. Essentially, the labels of the target unlabeled data are spanned by the prediction of these source classifiers. Motivated by these observations, in this paper, we propose to construct many source classifiers of diverse biases and learn the weight for each source classifier by directly minimizing the structural risk defined on the target unlabeled data so as to heal the possible sample selection bias. Since the weights are learned by maximizing the margin of separation between opposite classes on the target unlabeled data, the proposed method is established here as Maximal Margin Target Label Learning (MMTLL), which is in a form of Multiple Kernel Learning problem with many label kernels. Extensive experimental studies of MMTLL against several state-of-the-art methods on the Sentiment and Newsgroups datasets with various imbalanced class settings showed that MMTLL exhibited robust accuracies on all the settings considered and was resilient to negative transfer, in contrast to other counterpart methods which suffered significantly in prediction accuracy.

In real-world hyperspectral image (HSI) classification, the limited annotated examples usually lead to the insufficiently trained classifier, which further generates low classification accuracy. To overcome this challenge, Domain Adaptation (DA) methods have been developed to transfer learnable knowledge from the external HSIs with sufficient labeled examples ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., the source domain) to the interested HSI with scarce labeled examples ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., the target domain). Conventional DA approaches often pseudo-label the examples with high classification confidence and then incorporate them in the training process. However, due to the significant domain gap, relying solely on the confident examples may not be adequate to achieve satisfactory performance. Therefore, this paper proposes an Interactive Easy-to-Hard Domain Adaptation method (IEH-DA) to arrange the adaptation process so that the “easy” examples are adapted ahead of the “hard” ones. In an early stage, the easy examples with high pseudo-labeling confidence are selected for the adversarial learning based DA. In a later stage, the “hard” examples with high informativity are further selected, and they are interactively labeled by human expert to provide accurate supervision information for adaptation. As a result, the examples in target domain are used in an easy-to-hard way, which forms a curriculum sequence for orderly model training. Extensive experiments conducted on typical public datasets demonstrate that IEH-DA outperforms other state-of-the-art DA methods for HSI classification.

Transfer learning and domain adaptation (DA) methods have been utilized in bearing prognostic and health management, but most of the current DA methods do not take into account the feature scale change of degraded features when aligning the feature distribution, and these methods are more suitable for the classification problem, which is more robust to the feature scale change. However, they perform poorly in regression problems. In addition, most of the remaining useful life (RUL) prediction methods require preprocessing such as statistical feature extraction on the signal, which makes the prediction process complicated. To solve the above problems, a DA method based on the representation subspace distance (RSD) is proposed for predicting the bearing RUL under different operating conditions. First, the proposed convolutional neural network (CNN) self-attention (SA) long short term memory network model is utilized to extract the deep features from the original signal, which overcomes the limitations of the CNN in extracting time series. Then, the RSD in the Riemannian geometry of the Grassmann manifold is proposed as a domain transfer loss to learn domain invariant features. The modified method can align the feature distribution of the source domain and the target domain without changing the feature scale. At the same time, the bases mismatch penalization is introduced to avoid destroying the semantic information of the features in the process of domain alignment. Finally, the effectiveness of the proposed method is verified by experiments on four types of transfer tasks, and its superiority is also demonstrated by comparison with other advanced methods.

Importance-weighted conditional adversarial network for unsupervised domain adaptation

Conventional machine learning relies on two presumptions: (1) the training and testing datasets follow the same independent distribution, and (2) an adequate quantity of samples is essential for achieving optimal model performance during training. Nevertheless, meeting these two assumptions can be challenging in real-world scenarios. Domain adaptation (DA) is a subfield of transfer learning that focuses on reducing the distribution difference between the source domain (Ds) and target domain (Dt) and subsequently applying the knowledge gained from the Ds task to the Dt task. The majority of current DA methods aim to achieve domain invariance by aligning the marginal probability distributions of the Ds. and Dt. Recent studies have pointed out that aligning marginal probability distributions alone is not sufficient and that alignment of conditional probability distributions is equally important for knowledge migration. Nonetheless, unsupervised DA presents a more significant difficulty in aligning the conditional probability distributions because of the unavailability of labels for the Dt. In response to this issue, there have been several proposed methods by researchers, including pseudo-labeling, which offer novel solutions to tackle the problem. In this paper, we systematically analyze various pseudo-labeling algorithms and their applications in unsupervised DA. First , we summarize the pseudo-label generation methods based on the single and multiple classifiers and actions taken to deal with the problem of imbalanced samples. Second, we investigate the application of pseudo-labeling in category feature alignment and improving feature discrimination. Finally, we point out the challenges and trends of pseudo-labeling algorithms. As far as we know, this article is the initial review of pseudo-labeling techniques for unsupervised DA.

Deep multi-Wasserstein unsupervised domain adaptation

Joint predictive model and representation learning for visual domain adaptation

All current deep learning-based prediction methods for remaining useful life (RUL) assume that training and testing data have similar distributions, but the existence of various operating conditions, failure modes, and noise lead to insufficient data with similar distributions during the training process, thereby reducing RUL prediction performance. Domain adaptation can effectively solve this problem by learning the cross-domain invariant features of the source domain and target domain to reduce the distribution difference. However, most domain adaptive methods extract the source domain and target domain features into a single space for feature alignment, which may leave out effective information and affect the accuracy of prediction. To address this problem, we propose a data-driven approach named long short-term memory network and multi-representation domain adaptation (LSTM-MRAN). We standardize and process the degraded sensor data with a sliding time window, use LSTM to extract features from the degraded data, and mine the time series information between the data. Then, we use multiple substructures in multi-representation domain adaptation to extract features of the source domain and target domain from different spaces and align features by minimizing conditional maximum mean difference (CMMD) loss functions. The effectiveness of the method is verified by the CMAPSS dataset. Compared with methods without domain adaptation and other transfer learning methods, the proposed method provides more reliable RUL prediction results under datasets with different operating conditions and failure modes.

In the absence of the labeled samples in a domain referred to as target domain, Domain Adaptation (DA) techniques come in handy. Generally, DA techniques assume there are available source domains that share similar predictive function with the target domain. Two core challenges of DA typically arise, variance that exists between source and target domains, and the inherent source hypothesis bias. In this paper, we first propose a Stability Transfer criterion for selecting relevant source domains with low variance. With this criterion, we introduce a TARget learning Assisted by Source Classifier Adaptation (TARASCA) method to address the two core challenges that have impeded the performances of DA techniques. To verify the robustness of TARASCA, extensive experimental studies are carried out with comparison to several state-of-the-art DA methods on the real-world Sentiment and Newsgroups datasets, where various settings for the class ratios of the source and target domains are considered.

Although Deep Convolutional Neural Network (DCNN) has been admittedly witnessed as an enormous success in a wide range of applications, most of them require sufficient annotations with time-consuming and labor-exhausting efforts. Existing domain adaptation (DA) approaches delve into designing an effective loss module to minimize the distribution gap between the source and target domains. However, few studies pay attention to improve the backbone or network architecture for DA issues. In this paper, we propose a new backbone for DA specially, i.e., Transferable ResNet (TransResNet). TransResNet remedies the residual block in ResNet, separating source and target input features and highlighting more transferable channels in each block. It can be easily applied to all kinds of DA methods, without adding any extra learning parameters. We conduct substantial experiments on two general DA datasets and embed TransResNet into two seminal DA methods, including DANN and CDAN. Experimental results demonstrate TransResNet improves the transferability of the architecture, indicating that it is a great substitute for ResNet as a network backbone in DA issues.

In this paper, we propose a new unsupervised domain adaptation method for document classification. We address the problem of domain adaptation for document classification where the source and target domains do not differ significantly and there is no labeled data in the target domain. In this case, we can use conventional semi-supervised learning. Thus, we use the naive Bayes-based expectation-maximization method (NBEM) which is very effective for document classification. However, NBEM does not utilize the difference between a source domain and a target domain. We combine NBEM with the feature weighted method for domain adaptation, referred to as “self-training feature weight” (STFW). Our proposed method alternately uses NBEM and STFW to gradually improve document classification precision for a target domain. This method significantly outperforms the conventional unsupervised methods for domain adaptation.

Great progress has been achieved in domain adaptation in decades. Existing works are always based on an ideal assumption that testing target domains are independent and identically distributed with training target domains. However, due to unpredictable corruptions (e.g., noise and blur) in real data, such as web images and real-world object detection, domain adaptation methods are increasingly required to be corruption robust on target domains. We investigate a new task, corruption-agnostic robust domain adaptation (CRDA), to be accurate on original data and robust against unavailable-for-training corruptions on target domains. This task is non-trivial due to the large domain discrepancy and unsupervised target domains. We observe that simple combinations of popular methods of domain adaptation and corruption robustness have suboptimal CRDA results. We propose a new approach based on two technical insights into CRDA, as follows: (1) an easy-to-plug module called domain discrepancy generator (DDG) that generates samples that enlarge domain discrepancy to mimic unpredictable corruptions; (2) a simple but effective teacher-student scheme with contrastive loss to enhance the constraints on target domains. Experiments verify that DDG maintains or even improves its performance on original data and achieves better corruption robustness than baselines. Our code is available at: https://github.com/YifanXu74/CRDA .

As one of the most prevalent branches of transfer learning, domain adaptation is dedicated to generalizing the knowledge of a source domain to a target domain to perform machine learning tasks. In domain adaptation, the key strategy is to overcome the shift between different domains and learn shared features with domain invariance. However, most existing methods focus on extracting the common features of the source and target domains, and do not consider the shift problem of class center in the target domain caused by this process. Specifically, when we align the domain distributions, we often ignore the inherent feature attributes of the data, or under the guidance of false pseudo-labels, cause the target domain data to be far away from the class center after projection. This is not conducive to classification task. To address these problems, in this study, we propose a novel domain adaptation method, referred to as discriminative invariant alignment (DIA), for image representation. DIA enriches the knowledge matrix by combining the class discriminative information of the source domain and local data structure information of the target domain into a new framework. By introducing the maximum margin criterion of the source domain, the classification boundaries are expanded. To verify the performance of the proposed method, we compared DIA with several state-of-the-art methods on five benchmark databases. The experimental results show that DIA is superior to the state-of-the-art methods.

In this paper, we propose a new framework called domain adaptation machine (DAM) for the multiple source domain adaption problem. Under this framework, we learn a robust decision function (referred to as target classifier) for label prediction of instances from the target domain by leveraging a set of base classifiers which are prelearned by using labeled instances either from the source domains or from the source domains and the target domain. With the base classifiers, we propose a new domain-dependent regularizer based on smoothness assumption, which enforces that the target classifier shares similar decision values with the relevant base classifiers on the unlabeled instances from the target domain. This newly proposed regularizer can be readily incorporated into many kernel methods (e.g., support vector machines (SVM), support vector regression, and least-squares SVM (LS-SVM)). For domain adaptation, we also develop two new domain adaptation methods referred to as FastDAM and UniverDAM. In FastDAM, we introduce our proposed domain-dependent regularizer into LS-SVM as well as employ a sparsity regularizer to learn a sparse target classifier with the support vectors only from the target domain, which thus makes the label prediction on any test instance very fast. In UniverDAM, we additionally make use of the instances from the source domains as Universum to further enhance the generalization ability of the target classifier. We evaluate our two methods on the challenging TRECIVD 2005 dataset for the large-scale video concept detection task as well as on the 20 newsgroups and email spam datasets for document retrieval. Comprehensive experiments demonstrate that FastDAM and UniverDAM outperform the existing multiple source domain adaptation methods for the two applications.

Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this topic and evaluate their proposed methods under the same standard. Second, we conduct benchmark experiments by using NINE state-of-the-art domain adaptation (DA) methods and SIX popular spatiotemporal descriptors for investigating CDMER problem from two different perspectives. Third, we propose a novel DA method called region selective transfer regression (RSTR) to deal with the CDMER task. The overall superior performance of RSTR over the state-of-the-art DA methods demonstrates that taking into consideration the facial local region information used in RSTR contributes to developing effective DA methods for dealing with CDMER problem.