Big Data-Driven Control of Nonlinear Processes Through Dynamic Latent Variables Using an Autoencoder.

Dimension reduction on neural activity paves a way for unsupervised neural decoding by dissociating the measurement of internal neural state repetition from the measurement of external variable tuning. With assumptions only on the smoothness of latent dynamics and of internal tuning curves, the Poisson Gaussian-process latent variable model (P-GPLVM) (Wu et al., 2017) is a powerful tool to discover the low-dimensional latent structure for high-dimensional spike trains. However, when given novel neural data, the original model lacks a method to infer their latent trajectories in the learned latent space, limiting its ability for estimating the internal state repetition. Here, we extend the P-GPLVM to enable the latent variable inference of new data constrained by previously learned smoothness and mapping information. We also describe a principled approach for the constrained latent variable inference for temporally-compressed patterns of activity, such as those found in population burst events (PBEs) during hippocampal sharp-wave ripples, as well as metrics for assessing whether the inferred new latent variables are congruent with a previously learned manifold in the latent space. Applying these approaches to hippocampal ensemble recordings during active maze exploration, we replicate the result that P-GPLVM learns a latent space encoding the animal’s position. We further demonstrate that this latent space can differentiate one maze context from another. By inferring the latent variables of new neural data during running, certain internal neural states are observed to repeat, which is in accordance with the similarity of experiences encoded by its nearby neural trajectories in the training data manifold. Finally, repetition of internal neural states can be estimated for neural activity during PBEs as well, allowing the identification for replay events of versatile behaviors and more general experiences. Thus, our extension of the P-GPLVM framework for unsupervised analysis of neural activity can be used to answer critical questions related to scientific discovery.

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

A new redundant variable pruning approach—minor latent variable perturbation–PLS used for QSAR studies on anti-HIV drugs

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this article, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian networks (BNs) due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods, often through an auxiliary network, to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of maxsum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent (AugCA) are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark data sets of different models are given for both data representation and feature-learning tasks.

Many failure root cause analysis (RCA) algorithms for microservices have been proposed with the widespread adoption of microservices systems. Existing algorithms generally focus on RCA with ranking single-level (e.g. metric-level or service-level) root cause candidates (RCCs) with comprehensive monitoring metrics. However, many heterogeneous RCCs exist with limited observability in real-world microservices systems. Further, we find that the limited observability may result in inaccurate RCA through real-world failures in eBay. In this paper, for the first time, we propose to "model RCCs as latent variables". The core idea is to infer the status of RCCs as latent variables with related monitoring metrics instead of directly extracting features from only the observable metrics. Based on this, we propose LatentScope, an unsupervised RCA framework with heterogeneous RCCs under limited observability. A dual-space graph is proposed to model both observable and unobservable variables, with many-to-many relationships between spaces. To achieve fast inference of latent variables and RCA, we propose the LatentRegressor algorithm, which includes Regression-based Latent-space Intervention Recognition (RLIR) to achieve intervention recognition-based RCA in latent space. LatentScope has been deployed in eBay's production environment and evaluated on both eBay's real-world failures and a testbed dataset. The evaluation results show that, compared with baseline algorithms, our model significantly improves the Top-1 recall by 9.7%-57.9%. The source code of LatentScope and the dataset are available at https://github.com/NetManAIOps/LatentScope.

Noisy, high-dimensional time series observations can often be described by a set of low-dimensional latent variables. Commonly used methods to extract these latent variables typically assume instantaneous relationships between the latent and observed variables. In many physical systems, changes in the latent variables manifest as changes in the observed variables after time delays. Techniques that do not account for these delays can recover a larger number of latent variables than are present in the system, thereby making the latent representation more difficult to interpret. In this work, we introduce a novel probabilistic technique, time-delay gaussian-process factor analysis (TD-GPFA), that performs dimensionality reduction in the presence of a different time delay between each pair of latent and observed variables. We demonstrate how using a gaussian process to model the evolution of each latent variable allows us to tractably learn these delays over a continuous domain. Additionally, we show how TD-GPFA combines temporal smoothing and dimensionality reduction into a common probabilistic framework. We present an expectation/conditional maximization either (ECME) algorithm to learn the model parameters. Our simulations demonstrate that when time delays are present, TD-GPFA is able to correctly identify these delays and recover the latent space. We then applied TD-GPFA to the activity of tens of neurons recorded simultaneously in the macaque motor cortex during a reaching task. TD-GPFA is able to better describe the neural activity using a more parsimonious latent space than GPFA, a method that has been used to interpret motor cortex data but does not account for time delays. More broadly, TD-GPFA can help to unravel the mechanisms underlying high-dimensional time series data by taking into account physical delays in the system.

Modeling in the observable or latent space? A comparison of dynamic latent variable based monitoring methods for sensor fault detection

In this paper, a robust fault detection scheme based on the behavioral system theory and concept of dissipativity is proposed for linear time-invariant systems. Process model uncertainty is described using a dissipativity condition on the system from process input/output to the residue. Faults are detected in real time when such dissipativity condition (dissipation inequality) is violated. The proposed fault detection approach does not require models of faults. It can also be integrated with a robust control system. The connection and difference between the proposed approach and some existing fault detection methods are also discussed.

Nonlinear process monitoring based on load weighted denoising autoencoder

This article provides a unified framework for studying the identification of counterfactual parameters in a general class of discrete outcome models, allowing for endogenous regressors and multidimensional latent variables, all without parametric distributional assumptions. Our main theoretical result is that, when the covariates are discrete, the infinite-dimensional latent variable distribution can be replaced with a finite-dimensional version that is equivalent from an identification perspective. The finite-dimensional latent variable distribution is constructed in practice by enumerating regions of the latent variable space with a new and efficient cell enumeration algorithm for hyperplane arrangements. We then show that bounds on a certain class of counterfactual parameters can be computed by solving a sequence of linear programming problems, and show how the researcher can introduce additional assumptions as constraints in the linear programmes. Finally, we apply the method to a mobile phone choice example with heterogeneous choice sets, and to an airline entry game example.

In recent years, advancements in artificial intelligence, especially in natural language processing (NLP) models, have progressed rapidly. These models demonstrate remarkable results through training on large datasets and extensive architectures. However, the output process is often a black box, and the decision-making process remains unclear. Our research focuses on the internal representations, specifically the latent variables, generated by NLP models. In earlier work, we explored the latent variables and spaces produced by Sentence-BERT using image generation models. This approach aimed to visualize these spaces by converting discrete textual embeddings into images, introducing continuity and revealing novel relationships. This paper presents the development of an image generation model that uses a common decoder for both GPT-2 and Sentence-BERT, aimed at examining how differences in model architecture affect their latent spaces. We also investigate the impact of training dataset differences by comparing models trained in English and Japanese. Our findings indicate that while the models often generate similar outputs, significant differences emerge in sentences containing multiple elements, attributable to the differing focuses and objectives of the models. Our goal is to understand these latent spaces and contribute to the development of explainable AI.

Many real-world multivariable systems need to be modeled to capture the interconnected behavior of their physical variables and to understand how uncertainty in actuators and sensors affects the system dynamics. In system identification, some estimation algorithms are formulated as multivariate data problems by assuming symmetric noise distributions, yielding deterministic system models. Nevertheless, modern multivariable systems must incorporate the uncertainty behavior as a part of the system model structure, taking advantage of asymmetric distributions to model the uncertainty. This paper addresses the uncertainty modeling and identification of a class of multivariable linear dynamic systems, adopting a Stochastic Embedding approach. We consider a nominal system model and a Gaussian mixture distributed error-model driven by an exogenous input signal. The error-model parameters are treated as latent variables and a Maximum Likelihood algorithm that functions by marginalizing the latent variables is obtained. An Expectation-Maximization algorithm that jointly uses the measurements from multiple independent experiments is developed, yielding closed-form expressions for the Gaussian mixture estimators and the noise variance. Numerical simulations demonstrate that our approach yields accurate estimates of both the multivariable nominal system model parameters and the noise variance, even when the error-model non-Gaussian distribution does not correspond to a Gaussian mixture model.

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

Explaining and predicting the choices of consumers is perhaps the most enduring researchnproblem faced by the behavioural sciences. Of the range of methods used to gather information about consumers preferences, choice based approaches are common place in bothnacademia and industry. For the most part, consumers preferences from this type of datanare typically evaluated assuming preferences for the attributes of alternatives in choice arenindependent as a necessity due to the limitations of the statistical model. Recent advancesnin choice modelling however present an opportunity to consider that there may be a latentnstructure to decision makers’ taste sensitivities (preference parameters) and further, behavioural decision theory may be useful in helping the researcher specify that structure. Thisnthesis documents three applications of structural choice modelling that both test behaviouralndecision theory as well as contribute new interpretations about how behavioural decision theory may manifest as structures within the latent dimensions of consumers utility function.nEach application considers the structure of the taste variation (preference heterogeneity)nin decision makers’ taste sensitivities towards the attributes of transport services. In eachncase, behavioural decision theory is drawn upon to first predict what structures might benexpected, and subsequently structural choice models are specified to represent theory. Thisnunique modelling approach allows the following types of research questions to be addressed:• How does the nature of the choice environment affect the taste sensitivities in decisionnmakers’ preferences for particular product/service attributes?• Are latent sources of variation in consumers taste sensitivities stable or dynamic?n• What are the drivers of commonalities between the decision makers’ taste sensitivitiesnfor the attributes of alternatives in choice?• Is there a relationship between the way consumers think about their priorities and thenway consumers make choices?These questions are explored using so-called structural choice models. Structural choicenmodels are type of factor-analyticn choice model (a very general form of mixed logit). Thenmodel allows for a parsimonious representation of the ways in which consumers vary in theirntastes and preferences. Representing the unobserved sources of taste variation as latent variables and specifying a structure to the latent variables offer great flexibility, including datanfrom the same consumers from different choice tasks. This allows the researcher, for example,nto specify latent variables general to several data generation processes and others unique tonone particular process. Thus, the extent to which attributes under different scenarios arenindeed treated by decision makers’ as the same can be assessed. From a policy perspective, the design of more nuanced policy responses may be possible given the insight into thenstructure of the heterogeneity in decision makers’ preferences. The thesis is presented in thenfollowing way. Firstly, the introduction provides an overview of behavioural decision theory,nchoice based methods, structural choice modelling and its general econometric specifications.nNext, the empirical results of three separate but related studies are presented. The studiesndocument applications of the structural choice model to representing testing latent variablenrepresentations of taste variation. Finally, the conclusion provides a summary of all thenresults as well as discusses the future of research in this area.n

This paper presents a method of interest level estimation via multimodal Gaussian process latent variable factorization (mGPLVF). The proposed method estimates user interest levels for contents with high accuracy by using multi-modal features such as contents and users’ behavior. Generally, users’ behavior includes some noise, and it is difficult to prepare a large amount of data. For dealing with the problem, the proposed method newly derives mGPLVF calculating appropriate latent variables that do not overfit a small amount of training data including noise based on a probabilistic generative model. Furthermore, mGPLVF simultaneously performs not only construction of the robust latent space but also estimation of user interest levels via the latent variables based on an idea inspired by a factorization machine. The consistent framework of latent space construction and interest level estimation leads to the improvement of the final estimation accuracy. Experimental results show the effectiveness of the proposed method.