Latent space domain transfer between high dimensional overlapping distributions

Previous chapter Next chapter Full AccessProceedings Proceedings of the 2012 SIAM International Conference on Data Mining (SDM)Dual Transfer LearningMingsheng Long, Jianmin Wang, Guiguang Ding, Wei Cheng, Xiang Zhang, and Wei WangMingsheng Long, Jianmin Wang, Guiguang Ding, Wei Cheng, Xiang Zhang, and Wei Wangpp.540 - 551Chapter DOI:https://doi.org/10.1137/1.9781611972825.47PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutAbstract Transfer learning aims to leverage the knowledge in the source domain to facilitate the learning tasks in the target domain. It has attracted extensive research interests recently due to its effectiveness in a wide range of applications. The general idea of the existing methods is to utilize the common latent structure shared across domains as the bridge for knowledge transfer. These methods usually model the common latent structure by using either the marginal distribution or the conditional distribution. However, without exploring the duality between these two distributions, these single bridge methods may not achieve optimal capability of knowledge transfer. In this paper, we propose a novel approach, Dual Transfer Learning (DTL), which simultaneously learns the marginal and conditional distributions, and exploits the duality between them in a principled way. The key idea behind DTL is that learning one distribution can help to learn the other. This duality property leads to mutual reinforcement when adapting both distributions across domains to transfer knowledge. The proposed method is formulated as an optimization problem based on joint nonnegative matrix trifactorizations (NMTF). The two distributions are learned from the decomposed latent factors that exhibit the duality property. An efficient alternating minimization algorithm is developed to solve the optimization problem with convergence guarantee. Extensive experimental results demonstrate that DTL is more effective than alternative transfer learning methods. Previous chapter Next chapter RelatedDetails Published:2012ISBN:978-1-61197-232-0eISBN:978-1-61197-282-5 https://doi.org/10.1137/1.9781611972825Book Series Name:ProceedingsBook Code:PRDT12Book Pages:1-1150Key words:Keywords Transfer Learning, Cross-Domain Classification, Dual Transfer Learning, Nonnegative Matrix Tri-Factorization

A multi-constrained domain adaptation network for remaining useful life prediction of bearings

For conditional tail inferences from multivariate distributions, we desire models with positive dependence for which conditional distributions of one variable given others have extreme value indices that can be functions of the values of conditioning variables. That is, the tails of the conditional distributions behave like Pareto distributions with varying tail parameter. It is shown in Arnold (Stat Probab Lett 5:263–266, 1987) and Arnold et al. (Stat Probab Lett 17:361–368, 1993) in the bivariate case that if all conditional distributions are (generalized) Pareto and the tail parameters are non-constant, then tractable solutions have limited range of dependence. To obtain models with our desired properties, we specify one set of conditional Pareto distributions and one marginal distribution: for example, (a) FY |X(⋅|x) is Pareto with tail parameter decreasing in x and/or scale parameter is increasing in x, (b) no specification is made for FX|Y(⋅|x), and (c) X has a distribution with regularly varying upper tail. Based on this construction, we study the following properties: (1) relation of concordance ordering to the tail parameter function, (2) relations of conditional extreme value indices to marginal extreme value indices, and (3) tail dependence.

Electroencephalograms (EEG) have shown to be a useful approach to measure the cognitive load in tasks where mental effort is involved. However, EEG signals present a high variability among subjects as well as a non-stationary behavior, so that distributions among samples of different subjects are mismatched. Methods based on Unsupervised Domain Adaptation (UDA) have been used as an effective solution to reduce such discrepancy, while the ones leveraged by deep learning (D-UDA) have improved the classification results over shallow approaches. However, most D-UDA methods assume that even though there are differences in marginal distributions between source and target domains, their conditional distributions remain fixed, which does not hold in many EEG databases. To address this problem, we propose a new D-UDA method, named Custom Domain Adaptation (CDA), which integrates Adaptive Batch Normalization (AdaBN) and Maximum Mean Discrepancy (MMD) into two independent deep neural networks in order to reduce the marginal and conditional distribution differences. CDA was compared with six popular D-UDA methods using a free-available dataset of cognitive loads and obtained an accuracy of $98.2\pm 2.67\%$ , which outperformed these state-of-the-art methods.

Adversarial domain adaptation is a powerful approach to transfer the knowledge of the label-rich source domain to the label-scarce target domain by mitigating domain shifts across distributions. Existing domain adaptation methods align either the marginal distribution with a single-domain discriminator or conditional distributions with multiple-domain discriminators. However, aligning both marginal (global) and conditional (local) distributions should be considered for domain adaptation. This paper proposes a novel adversarial distribution adaptation network (ADAN) to jointly reduce both the global and local distribution discrepancies between different domains for learning domain-invariant representations. ADAN utilizes a single-domain discriminator to adapt the global distribution between two domains, and source decision boundaries to align the local distributions between sub-domains. Furthermore, we extend our ADAN as improved ADAN (iADAN), in which we utilize a feature norm term to regularize the task-specific features to improve model generalization. Extensive experimental results show that our method outperforms other state-of-the-art domain adaptation methods on Office-Home and ImageCLEF-DA datasets.

Models are often defined through conditional rather than joint distributions, but it can be difficult to check whether the conditional distributions are compatible,i.e.whether there exists a joint probability distribution which generates them. When they are compatible, a Gibbs sampler can be used to sample from this joint distribution. When they are not, the Gibbs sampling algorithm may still be applied, resulting in a “pseudo-Gibbs sampler”. We show its stationary probability distribution to be the optimal compromise between the conditional distributions, in the sense that it minimizes a mean squared misfit between them and its own conditional distributions. This allows us to perform Objective Bayesian analysis of correlation parameters in Kriging models by using univariate conditional Jeffreys-rule posterior distributions instead of the widely used multivariate Jeffreys-rule posterior. This strategy makes the full-Bayesian procedure tractable. Numerical examples show it has near-optimal frequentist performance in terms of prediction interval coverage.

Asymptotic analysis of a storage allocation model with finite capacity: Marginal and conditional distributions

Ancestral graph models, introduced by Richardson and Spirtes (2002), generalize both Markov random fields and Bayesian networks to a class of graphs with a global Markov property that is closed under conditioning and marginalization. By design, ancestral graphs encode precisely the conditional independence structures that can arise from Bayesian networks with selection and unobserved (hidden/latent) variables. Thus, ancestral graph models provide a potentially very useful framework for exploratory model selection when unobserved variables might be involved in the data-generating process but no particular hidden structure can be specified. In this paper, we present the Iterative Conditional Fitting (ICF) algorithm for maximum likelihood estimation in Gaussian ancestral graph models. The name reflects that in each step of the procedure a conditional distribution is estimated, subject to constraints, while a marginal distribution is held fixed. This approach is in duality to the well-known Iterative Proportional Fitting algorithm, in which marginal distributions are fitted while conditional distributions are held fixed.

Joint probability distribution of winds and waves from wave simulation of 20 years (1989-2008) in Bohai Bay

SummaryConsider a pair of random variables, both subject to random right censoring. New estimators for the bivariate and marginal distributions of these variables are proposed. The estimators of the marginal distributions are not the marginals of the corresponding estimator of the bivariate distribution. Both estimators require estimation of the conditional distribution when the conditioning variable is subject to censoring. Such a method of estimation is proposed. The weak convergence of the estimators proposed is obtained. A small simulation study suggests that the estimators of the marginal and bivariate distributions perform well relatively to respectively the Kaplan–Meier estimator for the marginal distribution and the estimators of Pruitt and van der Laan for the bivariate distribution. The use of the estimators in practice is illustrated by the analysis of a data set.

Modeling Hong Kong’s stock index with the Student t-mixture autoregressive model

Existing surface reconstruction methods only work for low-dimensional, mostly 2D or 3D, data. In real applications, high-dimensional data is difficult to interpret as it requires more dimensions to represent. As a dimension reduction method, manifold learning provides an explicit representation for the useful implicit information hidden in the original feature space. But the internal topological and differential structure has disappeared in the dimensionality reduction process. In order to investigate the topological and differential structure of a data set, we introduce boundary compactness, which is used to study the shape of the data set in the original high-dimensional feature space. Beginning with the Delaunay graph (for 3D data) or the complete graph (for data of higher dimensions), we select the edges based on the boundary compactness to make the relevance graph. The relevance graph is then used to reconstruct the surface of the data. Experiments show that the introduced technique based on the boundary compactness works well for data of 3D or higher dimensions.

Missing data are nearly always a problem in research, and missing values represent a serious threat to the validity of inferences drawn from findings. Increasingly, social science researchers are turning to multiple imputation to handle missing data. Multiple imputation, in which missing values are replaced by values repeatedly drawn from conditional probability distributions, is an appropriate method for handling missing data when values are not missing completely at random. However, use of this method requires developing an imputation model from the observed data. This is typically a rigorous and time-consuming process. To encourage wider adoption of multiple imputation in social work research, a simple framework for designing imputation models is presented. The framework and its ability to generate unbiased estimates are demonstrated in a simulation study. KEY WORDS: missing data; multiple imputation; nonresponse ********** Missing data are ubiquitous in social research, and missingness or nonresponse can represent a threat to the validity of inferences because of undue effects on efficiency, power, and parameter bias (Shadish, Cook, & Campbell, 2002). Social work researchers are now addressing missing data in a more rigorous manner. Recently, Saunders et al. (2006) and Choi, Golder, Gillmore, and Morrison (2005) described important data imputation methods and dispelled misunderstandings regarding popular imputation methods, such as mean substitution. Recent advances in analytic methods, such as multiple imputation (MI), are taking hold in social work research. With MI, missing values are replaced with values repeatedly drawn from simulated conditional probability distributions (Schafer, 1997), thus creating multiple versions of the data set. Each version of the data set is analyzed according to the data analysis model, and the multiple results are combined into point estimates (Rubin, 1996). A critical task in MI is to devise an imputation model (Allison, 2002) or missing data model (Graham, Olchowski, & Gilreath, 2007), which involves specifying the measures that are putatively associated with the missing values. Although this process adds additional steps, the specification of an imputation model and the creation of multiple data sets can produce less-biased estimates in the presence of missing data across a wide variety of data analysis techniques (Schafer, 1997). Besides MI, there are many other methods for addressing missing data (Schafer, 1999; Schafer & Graham, 2002). An equally rigorous method known as direct or full information maximum likelihood (FIML) estimation can produce unbiased estimates and correct standard errors in the presence of missing data. When the number of imputations is sufficiently large, identical missing data models will produce the same estimates under MI and FIML (Graham et al., 2007). Unlike MI, FIML is limited to maximum likelihood analytic techniques and the missing data model must be included in the analysis model. Although we focus on MI, the steps we describe for developing an imputation model are equally appropriate for use in the missing data model for FIML. On the basis of the MI literature, this article describes a framework for developing an imputation model for use with any free or commercial software package that performs MI. BRIEF REVIEW OF MISSING DATA CONCEPTS Generally, both MI and a broad range of missing data issues have received ample attention in the applied literature (Graham et al., 2007). We briefly discuss the three types of distributions that describe the randomness of nonresponse given that this property has consequences for the development of an imputation model. For a discussion of general missing data concepts that are not critical to understanding our discussion of imputation model development, we refer readers to Schafer and Graham (2002). Distribution of Nonresponse The probability distribution of nonresponse--more frequently referred to as the missing data or nonresponse mechanism (Rubin, 1976)--is both an important factor in the decision to impute with MI and a context for the development of an imputation model. …

Infrared thermography provides abundant spatiotemporal degradation information, facilitating non-contact condition monitoring. Reducing domain shift between simulated and industrial infrared images is significantly desired for leveraging labeled simulated data to tackle practical insufficiency of run-to-failure samples. Recently, adversarial-based domain adaptation (DA) techniques have aroused broad concern in solving domain shifts. However, simultaneously aligning marginal and conditional distributions in cross-domain remaining useful life (RUL) prediction is rarely researched in adversarial-based DA. In this article, an adversarial regressive domain adaptation (ARDA) approach is, thus, put forward to address this challenge. First, a regressive disparity discrepancy is designed to describe the dissimilarity between distributions and derive the generalization bound for cross-domain prognostics. Guided by this bound, the ARDA effectively aligns marginal and conditional distributions by learning indistinguishable features and considering the relationship between samples and prediction tasks. Simulated and experimental infrared degradation image datasets are used to demonstrate the effectiveness and superiority of the proposed approach over existing methods for cross-domain RUL prediction.

In machinery fault diagnosis, labeled data are always difficult or even impossible to obtain. Transfer learning can leverage related fault diagnosis knowledge from fully labeled source domain to enhance the fault diagnosis performance in sparsely labeled or unlabeled target domain, which has been widely used for cross domain fault diagnosis. However, existing methods focus on either marginal distribution adaptation (MDA) or conditional distribution adaptation (CDA). In practice, marginal and conditional distributions discrepancies both have significant but different influences on the domain divergence. In this paper, a dynamic distribution adaptation based transfer network (DDATN) is proposed for cross domain bearing fault diagnosis. DDATN utilizes the proposed instance-weighted dynamic maximum mean discrepancy (IDMMD) for dynamic distribution adaptation (DDA), which can dynamically estimate the influences of marginal and conditional distribution and adapt target domain with source domain. The experimental evaluation on cross domain bearing fault diagnosis demonstrates that DDATN can outperformance the state-of-the-art cross domain fault diagnosis methods.