Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

To address the significant temporal characteristics and the scarcity of labelled data in complex industrial processes, specifically targeting the latent temporal dynamics that are difficult to observe directly during process evolution, this paper proposes a latent temporal feature dual‐head coordination network (LTFDHCN). First, a posterior generation (PG) module is constructed, utilizing a bi‐directional long short‐term memory (Bi‐LSTM) network to extract bidirectional temporal context and infer the posterior distribution in the latent space. Second, a prior prediction (PP) module is designed to establish a dynamic temporal model based on the latent variable of the previous time step. This mechanism predicts the prior distribution for the current moment, breaking the traditional assumption in variational autoencoders (VAE) that latent variables follow a static prior, thereby capturing deeper latent temporal dynamics. The Kullback–Leibler (KL) divergence is employed to minimize the discrepancy between the posterior and dynamic prior distributions, enforcing the latent space to encode temporal dependencies. Furthermore, a dual‐head coordination (DHC) strategy is proposed. Leveraging a shared latent space and decoder backbone, the model simultaneously executes unsupervised data reconstruction and supervised quality prediction. This strategy effectively overcomes the label scarcity issue by explicitly mining the latent information embedded in unlabelled data. The model is optimized via a weighted objective function that integrates KL divergence, reconstruction and prediction loss. Simulation experiments on a debutanizer column and a thermal power generation process demonstrate that the proposed method outperforms baseline methods, including VAE, VAE‐LSTM, VRNN, and temporal convolutional networks (TCN)‐VAE, verifying its effectiveness and superiority in handling complex industrial data.

Comparison of Bayesian survival analysis and Cox regression analysis in simulated and breast cancer data sets

The latent variable prior of the variational autoencoder (VAE) often utilizes a standard Gaussian distribution because of the convenience in calculation, but has an underfitting problem. This paper proposes a variational autoencoder with optimizing Gaussian mixture model priors. This method utilizes a Gaussian mixture model to construct prior distribution, and utilizes the Kullback-Leibler (KL) distance between posterior and prior distribution to implement an iterative optimization of the prior distribution based on the data. The greedy algorithm is used to solve the KL distance for defining the approximate variational lower bound solution of the loss function, and for realizing the VAE with optimizing Gaussian mixture model priors. Compared with the standard VAE method, the proposed method obtains state-of-the-art results on MNIST, Omniglot, and Frey Face datasets, which shows that the VAE with optimizing Gaussian mixture model priors can learn a better model.

Strange bayes indeed: uniform topological priors imply non-uniform clade priors

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

This research attempts to construct a network that can convert online and offline handwritten characters to each other. The proposed network consists of two Variational Auto-Encoders (VAEs) with a shared latent space. The VAEs are trained to generate online and offline handwritten Latin characters simultaneously. In this way, we create a cross-modal VAE (Cross-VAE). During training, the proposed Cross-VAE is trained to minimize the reconstruction loss of the two modalities, the distribution loss of the two VAEs, and a novel third loss called the space sharing loss. This third, space sharing loss is used to encourage the modalities to share the same latent space by calculating the distance between the latent variables. Through the proposed method mutual conversion of online and offline handwritten characters is possible. In this paper, we demonstrate the performance of the Cross-VAE through qualitative and quantitative analysis.

Bayesian methods have been proposed as a way of synthesizing all available evidence to inform decision making. However, few practical applications of the use of Bayesian methods for combining patient-level data (i.e., trial) with additional evidence (e.g., literature) exist in the cost-effectiveness literature. The objective of this study was to compare a Bayesian cost-effectiveness analysis using informative priors to a standard non-Bayesian nonparametric method to assess the impact of incorporating additional information into a cost-effectiveness analysis. Patient-level data from a previously published nonrandomized study were analyzed using traditional nonparametric bootstrap techniques and bivariate normal Bayesian models with vague and informative priors. Two different types of informative priors were considered to reflect different valuations of the additional evidence relative to the patient-level data (i.e., "face value" and "skeptical"). The impact of using different distributions and valuations was assessed in a sensitivity analysis. Models were compared in terms of incremental net monetary benefit (INMB) and cost-effectiveness acceptability frontiers (CEAFs). The bootstrapping and Bayesian analyses using vague priors provided similar results. The most pronounced impact of incorporating the informative priors was the increase in estimated life years in the control arm relative to what was observed in the patient-level data alone. Consequently, the incremental difference in life years originally observed in the patient-level data was reduced, and the INMB and CEAF changed accordingly. The results of this study demonstrate the potential impact and importance of incorporating additional information into an analysis of patient-level data, suggesting this could alter decisions as to whether a treatment should be adopted and whether more information should be acquired.

Variational autoencoders (VAE) have ushered in an new era of unsupervised learning methods for complex distributions. Although these techniques are elegant in their approach, they are typically not useful for representation learning. In this work, we propose a simple yet powerful class of VAEs that simultaneously result in meaningful learned representations. Our solution is to combine traditional VAEs with mutual information maximization, with the goal to enhance amortized inference in VAEs using Information Theoretic techniques. We call this approach InfoMax-VAE, and such an approach can significantly boost the quality of learned high-level representations. We realize this through explicit maximization of information measures associated with the representation. Using extensive experiments on varied datasets and setups, we show that InfoMax-VAE outperforms contemporary popular approaches, including Info- VAE and β-VAE.

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

Over the past 20years, there has been increasing interest in the use of Bayesian statistical methods for the analysis of clinical trials used to support regulatory decisions. Bayesian methods for the analysis of clinical trials are an attractive option when good prior information is available. Yet, in many cases, prior information is scarce and only tentative or proprietary prior information exists. In these situations, it is necessary to use noninformative type or skeptical-type priors. We undertook a systematic study of Bayesian methods and applied them to 13 randomized clinical trials in metastatic breast cancer submitted to the U.S. Food and Drug Administration for registrational purposes. Across all 13 studies, there were a total of 10,521 patients using 10 experimental agents. Our results demonstrated that Bayesian analyses with noninformative priors provided similar results to more traditional analyses. We also found that early interim looks at the study results can vary widely based upon the type of prior used. Finally, we found that pre-defined threshold stopping rules need to be relatively strong to prevent trials from stopping very early. Our results suggest that, when prior information is limited and a noninformative prior is used, there is little numerical difference between Bayesian methods and more traditional analysis methods. Bayesian methods, however, may provide additional summaries of the data that are more easily interpretable than means and confidence intervals.

Inverse problems often involve matching observational data using a physical model that takes a large number of parameters as input. These problems tend to be under-constrained and require regularization to impose additional structure on the solution in parameter space. A central difficulty in regularization is turning a complex conceptual model of this additional structure into a functional mathematical form to be used in the inverse analysis. In this work we propose a method of regularization involving a machine learning technique known as a variational autoencoder (VAE). The VAE is trained to map a low-dimensional set of latent variables with a simple structure to the high-dimensional parameter space that has a complex structure. We train a VAE on unconditioned realizations of the parameters for a hydrological inverse problem. These unconditioned realizations neither rely on the observational data used to perform the inverse analysis nor require any forward runs of the physical model, thus making the computational cost of generating the training data minimal. The central benefit of this approach is that regularization is then performed on the latent variables from the VAE, which can be regularized simply. A second benefit of this approach is that the VAE reduces the number of variables in the optimization problem, thus making gradient-based optimization more computationally efficient when adjoint methods are unavailable. After performing regularization and optimization on the latent variables, the VAE then decodes the problem back to the original parameter space. Our approach constitutes a novel framework for regularization and optimization, readily applicable to a wide range of inverse problems. We call the approach RegAE.

Previous research showed that employing results from meta-analyses of relevant previous fMRI studies can improve the performance of voxelwise Bayesian second-level fMRI analysis. In this process, prior distributions for Bayesian analysis can be determined by information acquired from the meta-analyses. However, only image-based meta-analysis, which is not widely accessible to fMRI researchers due to the lack of shared statistical images, was tested in the previous study, so the applicability of the prior determination method proposed by the previous study might be limited. In the present study, whether determining prior distributions based on coordinate-based meta-analysis, which is widely accessible to researchers, can also improve the performance of Bayesian analysis, was examined. Three different types of coordinate-based meta-analyses, BrainMap and Ginger ALE, and NeuroQuery, were tested as information sources for prior determination. Five different datasets addressing three task conditions, i.e., working memory, speech, and face processing, were analyzed via Bayesian analysis with a meta-analysis informed prior distribution, Bayesian analysis with a default Cauchy prior adjusted for multiple comparisons, and frequentist analysis with familywise error correction. The findings from the aforementioned analyses suggest that use of coordinate-based meta-analysis also significantly enhanced performance of Bayesian analysis as did image-based meta-analysis.

Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex

The aging of large concrete structures such as Nuclear Containment Buildings (NCB) or bridges involves a continuous strain evolution in time, which may affect their durability, safety and the safety of their environment. Then, the evaluation of structural integrity requires an accurate assessment of the long-term strain level. When considering probabilistic analysis of the delayed mechanical behavior of large concrete structures, the prediction results may present large uncertainties, which do not provide clear indicators aiming at supporting decisions related to structures’ maintenance. In this context, Bayesian updating enables to reduce uncertainties, by combining a prior state of knowledge with noisy monitoring data of the structure response. It requires the definition of a prior probability distribution, which summarizes all available information before collecting monitoring data. In former work concerning Bayesian approaches applied to the analysis of delayed strains, the prior distribution is usually defined through expert judgement, which constitutes a quite subjective process which may have a significant influence on Bayesian updating results. Moreover, the previously cited work involved strong hypotheses related to observation noise, which is usually assumed to be perfectly known. The present contribution aims at evaluating the influence of prior distributions defined by expert judgement on Bayesian updating results, through an illustrative case study of a well instrumented 1:3 scale NCB. The present work proposes also a Bayesian framework suitable for cases where the observation noise of data is unknown. The influence of the amount of monitoring data on the uncertainty reduction provided by Bayesian updating is also studied. Results underline that the modeling choices of the analyst are of paramount importance in the framework of long-term strains predictions, regardless the quantity of available data. Furthermore, results also suggest that Bayesian updating is well suitable for providing significant uncertainty reduction, even in the case of structures which dispose of a limited amount of monitoring data.