Variational Autoencoder With Optimizing Gaussian Mixture Model Priors

To address the significant temporal characteristics and the scarcity of labelled data in complex industrial processes, specifically targeting the latent temporal dynamics that are difficult to observe directly during process evolution, this paper proposes a latent temporal feature dual‐head coordination network (LTFDHCN). First, a posterior generation (PG) module is constructed, utilizing a bi‐directional long short‐term memory (Bi‐LSTM) network to extract bidirectional temporal context and infer the posterior distribution in the latent space. Second, a prior prediction (PP) module is designed to establish a dynamic temporal model based on the latent variable of the previous time step. This mechanism predicts the prior distribution for the current moment, breaking the traditional assumption in variational autoencoders (VAE) that latent variables follow a static prior, thereby capturing deeper latent temporal dynamics. The Kullback–Leibler (KL) divergence is employed to minimize the discrepancy between the posterior and dynamic prior distributions, enforcing the latent space to encode temporal dependencies. Furthermore, a dual‐head coordination (DHC) strategy is proposed. Leveraging a shared latent space and decoder backbone, the model simultaneously executes unsupervised data reconstruction and supervised quality prediction. This strategy effectively overcomes the label scarcity issue by explicitly mining the latent information embedded in unlabelled data. The model is optimized via a weighted objective function that integrates KL divergence, reconstruction and prediction loss. Simulation experiments on a debutanizer column and a thermal power generation process demonstrate that the proposed method outperforms baseline methods, including VAE, VAE‐LSTM, VRNN, and temporal convolutional networks (TCN)‐VAE, verifying its effectiveness and superiority in handling complex industrial data.

In this dissertation, a novel approach of integrating template matching with statistical modeling is proposed to improve continuous speech recognition. Hidden Markov Modeling (HMMs) has been the dominant approach in statistical speech recognition since it provides a principled way of jointly modeling speech spectral variations and time dynamics. However, HMMs have the shortcoming of assuming the observations being independent within each state, which makes it ineffective in modeling the details of speech temporal evolutions that are important for characterizing nonstationary speech sounds. Template-based methods make comparisons between a test pattern and the templates derived from training data, and therefore they are able to capture speech dynamics and time correlation of speech frames better than HMM based methods. However, template matching requires large memory space and computational time since feature vectors of training data need to be stored in computer memory for access at the recognition stage, which is difficult in large vocabulary continuous speech recognition (LVCSR). Our proposed approach takes advantages of both statistical modeling and template matching, which overcomes the weakness of conventional template-based method and is feasible for LVCSR. We use multiple Gaussian Mixture Model (GMM) indices to represent each frame of speech templates, and define the template unit to be context-dependent phone segments (triphone context). We also use phonetic decision trees borrowed from those commonly used in HMMs to tie triphone templates and predict triphones unseen in training data. Two local distances, log likelihood ratio (LLR) and Kullback-Leibler (KL) divergence, are proposed for dynamic time warping (DTW) based template matching. In order to reduce computational complexity and storage space, we propose methods of minimum distance template selection (MDTS) and maximum log-likelihood template selection (MLTS), and investigate a template compression method on top of template selection to further improve recognition performance. The template based methods were used to rescore lattices generated by baseline HMMs on the tasks of TIMIT continuous phone recognition and teleheath LVCSR and experimental results demonstrated that the proposed approach of integrating template matching with statistical modeling significantly improved recognition performances over the HMM baselines. The template selection methods also provided significant recognition accuracy improvements over the HMM baseline while largely reducing the computation and storage complexities. When all templates or MDTS were used, using the LLR local distance obtained better recognition performance than the KL divergence local distance. For MLTS and template compression, KL divergence local distance provided better performance than the LLR local distance, and the template compression method made further improvements over KL based MLTS. Since the templates were constructed based on the GMM indices extracted from HMM baselines, we also validated the effectiveness of the proposed template methods based on enhanced HMM baselines. Experimental results showed that LLR based all template method was able to consistently improve TIMIT phone recognition accuracies based on four enhanced HMM baselines. Prosodic features such as duration, energy, and pitch can reflect longer span information of speech than conventional single frame vectors but they have commonly been ignored by HMMs. Template based methods provide possibilities to conveniently integrate prosodic features into speech recognition, which has not been well studied in the past. In this dissertation, we investigate combining template based methods with the speech prosodic features of duration, energy and pitch to further improve speech recognition accuracy. The scores of prosodic information were computed by a GMM based method and a non-parametric method, and the prosodic scores were combined with the acoustic scores in triphone template matching. Experimental results obtained on the telehealth task showed that prosodic information had positive effects on vowel sound recognition.

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

At the start of new operation, a production process is commonly unstable. Then, a process condition is into being stable gradually over time. For the purpose of shifting to mass production, the stability in a process should be evaluated. This paper proposes a method of evaluating the process stability based on Bayesian statistics and information theory. In Bayesian statistics, the knowledge for a process is renewed by deriving posterior distribution based on current prior distribution and observations. Here, equivalency between prior and posterior distributions could be considered to be a criterion for the stability in a process. It is needed to define difference between prior and posterior distributions in order to evaluate the equivalency between prior and posterior distributions. We first formulate the relation between the prior distribution and the posterior distribution for process mean using Bayesian statistics. Secondly, we evaluate the difference between the both distributions based on Kullback‐Leibler (K‐L) divergence in information theory. Finally, some numerical examples in the method of evaluating the stability in process mean are illustrated. Also, we discuss about a decision rule for the stability in process mean through the numerical investigation.

Variational Autoencoder (VAE) is widely used as a generative model to approximate a model's posterior on latent variables by combining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE often converges to a degenerated local optimum known as "posterior collapse". Previous approaches consider the Kullback–Leibler divergence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL's distribution positive. Then we propose Batch Normalized-VAE (BN-VAE), a simple but effective approach to set a lower bound of the expectation by regularizing the distribution of the approximate posterior's parameters. Without introducing any new model component or modifying the objective, our approach can avoid the posterior collapse effectively and efficiently. We further show that the proposed BN-VAE can be extended to conditional VAE (CVAE). Empirically, our approach surpasses strong autoregressive baselines on language modeling, text classification and dialogue generation, and rivals more complex approaches while keeping almost the same training time as VAE.

Key performance indicator (KPI) anomaly detection is the underlying core technology in Artificial Intelligence for IT operations (AIOps). It has an important impact on subsequent anomaly location and root cause analysis. Variational auto-encoder (VAE) is a symmetry network structure composed of encoder and decoder, which has attracted extensive attention because of its ability to capture complex KPI data features and better detection results. However, VAE is not well applied to the modeling of KPI time series data and it is often necessary to set the threshold to obtain more accurate results. In response to these problems, this paper proposes a novel hybrid method for KPI anomaly detection based on VAE and support vector data description (SVDD). This method consists of two modules: a VAE reconstructor and SVDD anomaly detector. In the VAE reconstruction module, firstly, bi-directional long short-term memory (BiLSTM) is used to replace the traditional feedforward neural network in VAE to capture the time correlation of sequences; then, batch normalization is used at the output of the encoder to prevent the disappearance of KL (Kullback–Leibler) divergence, which prevents ignoring latent variables to reconstruct data directly. Finally, exponentially weighted moving average (EWMA) is used to smooth the reconstruction error, which reduces false positives and false negatives during the detection process. In the SVDD anomaly detection module, smoothed reconstruction errors are introduced into the SVDD for training to determine the threshold of adaptively anomaly detection. Experimental results on the public dataset show that this method has a better detection effect than baseline methods.

We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution, and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback—Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show that our procedure is related to other tools commonly used for model diagnostics, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature t...

Interpreting clinical latent representations using autoencoders and probabilistic models

ABSTRACTAs a probabilistic distance between two probability density functions, Kullback–Leibler divergence is widely used in many applications, such as image retrieval and change detection. Unfortunately, for some models, e.g., Gaussian Mixture Models (GMMs), Kullback–Leibler divergence is not analytically tractable. One has to resort to approximation methods. A number of methods have been proposed to address this issue. In this article, we compare seven methods, namely Monte Carlo method, matched bound approximation, product of Gaussians, variational method, unscented transformation, Gaussian approximation and min-Gaussian approximation, for approximating the Kullback–Leibler divergence between two Gaussian mixture models for satellite image retrieval. Two experiments using two public data sets have been performed. The comparison is carried out in terms of retrieval accuracy and computational time.

Utility scores associated with preference-based health-related quality-of-life instruments such as the EQ-5D-3L are reported as point estimates. In this study, we develop methods for capturing the uncertainty associated with the valuation study of the UK EQ-5D-3L that arises from the variability inherent in the underlying data, which is tacitly ignored by point estimates. We derive a new tariff that properly accounts for this and assigns a specific closed-form distribution to the utility of each of the 243 health states of the EQ-5D-3L. Using the UK EQ-5D-3L valuation study, we used a Bayesian approach to obtain the posterior distributions of the derived utility scores. We constructed a hierarchical model that accounts for model misspecification and the responses of the survey participants to obtain Markov chain Monte Carlo (MCMC) samples from the posteriors. The posterior distributions were approximated by mixtures of normal distributions under the Kullback-Leibler (KL) divergence as the criterion for the assessment of the approximation. We considered the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm to estimate the parameters of the mixture distributions. We derived an MCMC sample of total size 4,000 × 243. No evidence of nonconvergence was found. Our model was robust to changes in priors and starting values. The posterior utility distributions of the EQ-5D-3L states were summarized as 3-component mixtures of normal distributions, and the corresponding KL divergence values were low. Our method accounts for layers of uncertainty in valuation studies, which are otherwise ignored. Our techniques can be applied to other instruments and countries' populations. Guidelines for health technology assessments typically require that uncertainty be accounted for in economic evaluations, but the parameter uncertainty of the regression model used in the valuation study of the health instrument is often tacitly ignored.We consider the UK valuation study of the EQ-5D-3L and construct a Bayesian model that accounts for layers of uncertainty that would otherwise be disregarded, and we derive closed-form utility distributions.The derived tariff can be used by researchers in economic evaluations, as it allows analysts to directly sample a utility value from its corresponding distribution, which reflects the associated uncertainty of the utility score.

Knowledge distillation (KD), transferring knowledge from a cumbersome teacher model to a lightweight student model, has been investigated to design efficient neural architectures. Generally, the objective function of KD is the Kullback-Leibler (KL) divergence loss between the softened probability distributions of the teacher model and the student model with the temperature scaling hyperparameter τ. Despite its widespread use, few studies have discussed how such softening influences generalization. Here, we theoretically show that the KL divergence loss focuses on the logit matching when τ increases and the label matching when τ goes to 0 and empirically show that the logit matching is positively correlated to performance improvement in general. From this observation, we consider an intuitive KD loss function, the mean squared error (MSE) between the logit vectors, so that the student model can directly learn the logit of the teacher model. The MSE loss outperforms the KL divergence loss, explained by the penultimate layer representations difference between the two losses. Furthermore, we show that sequential distillation can improve performance and that KD, using the KL divergence loss with small τ particularly, mitigates the label noise. The code to reproduce the experiments is publicly available online at https://github.com/jhoon-oh/kd_data/.

The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple prior over the latent variables such as standard normal distribution, thereby restricting its applications to relatively simple phenomena. In this work, we propose hierarchical non-parametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space. Both the neural parameters and Bayesian priors are learned jointly using tailored variational inference. The resulting model induces a hierarchical structure of latent semantic concepts underlying the data corpus, and infers accurate representations of data instances. We apply our model in video representation learning. Our method is able to discover highly interpretable activity hierarchies, and obtain improved clustering accuracy and generalization capacity based on the learned rich representations.

Voice Conversion (VC) requires an alignment of the spectral features before learning the mapping function, due to the speaking rate variations across the source and target speakers. To address this issue, the idea of training two parallel networks with the use of speaker-independent representation was proposed. In this paper, we explore the unsupervised Gaussian Mixture Model (GMM) posteriorgram as a speaker-independent representation. However, in the GMM posteriorgram, the same phonetic information gets spread across more than one component due to the speaking style variations across the speakers. In particular, this spread is limited to a group of neighboring components for a given phone. We propose to share the posterior probability of each component with the limited number of neighboring components that are sorted based on the Kullback-Leibler (KL) divergence. We propose to employ a Deep Neural Network (DNN) and a Generative Adversarial Network (GAN)-based framework to measure the effectiveness of the proposed Inter Mixture Weighted GMM (IMW GMM) posteriorgram on the Voice Conversion Challenge (VCC) 2016 database. The relative improvement of 13.73%, and 5.25% is obtained with the proposed IMW GMM posteriorgram w.r.t. the GMM posteriorgram for the speech quality and the speaker similarity of the converted voices, respectively.

Generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs) have been recently applied to various fields. However, the VAE and GAN models have blur and mode collapse problems, respectively. Here, the authors propose a novel generative model, self‐converging generative network (SCGN), to address the issues. Self‐converging means the convergence of latent vectors into themselves through being trained in pairs with training data, by which the SCGN can reconstruct all training data. In the authors’ model, the latent vectors and weights of the generator are alternately trained. Specifically, the latent vectors are trained to follow a normal distribution, using a loss function derived from the Kullback–Leibler divergence and a pixel‐wise loss. The weights of the generator are adjusted for the generator to produce training data by means of a pixel‐wise loss. As a result, their SCGN did not fall into the mode collapse, which occurs in GANs, and made clearer images than VAEs thanks to no use of sampling. Moreover, the SCGN successfully learned the manifold of the dataset in the extensive experiments with CelebA.

In a content-based image retrieval (CBIR) system, indexing feature vectors and the similarity measure between feature vectors are two key factors for retrieval performance. We present a new CBIR system with statistical-model based image feature extraction in the wavelet domain and a Kullback divergence based similarity measure. A two component Gaussian mixture model (GMM) in the wavelet domain is employed and the model parameters are used to form features for image indexing. A new Kullback divergence based similarity measure is then presented for image retrieval. The experimental results demonstrate that the similarity measure based on the Kullback divergence is more effective than conventional similarity measures, such as the city-block distance and the Euclidean distance. It is shown that the new CBIR system, with the combination of the GMM and the new Kullback divergence based similarity measure, outperforms most other methods in retrieval performance for texture images, while keeping a comparable level of computational complexity.