Microservice Root Cause Analysis With Limited Observability Through Intervention Recognition in the Latent Space

Modern software applications increasingly rely on microservice architectures for scalability, flexibility, and rapid deployment. However, this architectural paradigm introduces new complexities in monitoring system behavior, identifying anomalies, and determining their root causes across distributed services. Existing solutions often address anomaly detection and root cause analysis (RCA) in isolation, leading to fragmented insights and delayed resolution. This paper proposes a unified framework that integrates real-time anomaly detection with automated RCA using machine learning and graph-based dependency modeling. The framework continuously monitors telemetry data—including metrics, logs, and traces—and applies an ensemble of statistical and deep learning models for multivariate anomaly detection. Detected anomalies are then contextualized through a service dependency graph and analyzed using causal inference techniques to identify the most probable root causes. We evaluate the framework on both synthetic benchmarks and real-world microservice deployments. Experimental results show that it achieves high precision and recall in anomaly detection while significantly reducing RCA latency compared to baseline methods. By combining anomaly detection and RCA in a cohesive pipeline, the proposed framework enhances system observability and reduces mean time to recovery (MTTR), thus improving operational resilience in complex microservice environments.

In recent years, advancements in artificial intelligence, especially in natural language processing (NLP) models, have progressed rapidly. These models demonstrate remarkable results through training on large datasets and extensive architectures. However, the output process is often a black box, and the decision-making process remains unclear. Our research focuses on the internal representations, specifically the latent variables, generated by NLP models. In earlier work, we explored the latent variables and spaces produced by Sentence-BERT using image generation models. This approach aimed to visualize these spaces by converting discrete textual embeddings into images, introducing continuity and revealing novel relationships. This paper presents the development of an image generation model that uses a common decoder for both GPT-2 and Sentence-BERT, aimed at examining how differences in model architecture affect their latent spaces. We also investigate the impact of training dataset differences by comparing models trained in English and Japanese. Our findings indicate that while the models often generate similar outputs, significant differences emerge in sentences containing multiple elements, attributable to the differing focuses and objectives of the models. Our goal is to understand these latent spaces and contribute to the development of explainable AI.

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

Resource-intensive applications such as scientific applications require the architecture or system on which they execute to display a very high level of dependability to reduce the impact of faults. Typically, the state of the underlying system is captured through messages that are recorded in a log file, which has been proven useful to system administrators in understanding the root-causes of system failures (and for their subsequent debugging). However, the time window between when the first error message is detected in the log file and time of the ensuing failure may not be large enough to allow the administrators to save the state of the running application, which will result in lost execution time. We thus address this fundamental question: Is it possible to extend this time window? The answer is positive: We show that, by using (i) resource usage logs to track anomalous resource usage and (ii) error logs to identify root-causes of system failures, it is possible to increase the time window, on average, by 50 minutes. These files were those obtained for the Ranger Supercomputer from TACC. We achieve this by applying anomaly detection techniques on resource usage data and conducting a root-cause analysis on error log files.

This article presents a novel data-driven approach to nonlinear system control using a behavioral systems framework. A dynamic latent variable autoencoder (DLVAE) is proposed to project the nonlinear physical variable space onto a linear latent variable space. A data-predictive control approach is developed to control the physical process variables through the latent variables. Based on the behavioral systems theory, the proposed data-driven control framework does not require knowledge of the causality of the latent variables. The stability of the controlled system is ensured by utilizing the concept of trajectory-based dissipativity. The robustness of this control approach is achieved by incorporating the Lipschitz bounds between the latent and physical variables under dissipativity conditions.

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

Dimension reduction on neural activity paves a way for unsupervised neural decoding by dissociating the measurement of internal neural state repetition from the measurement of external variable tuning. With assumptions only on the smoothness of latent dynamics and of internal tuning curves, the Poisson Gaussian-process latent variable model (P-GPLVM) (Wu et al., 2017) is a powerful tool to discover the low-dimensional latent structure for high-dimensional spike trains. However, when given novel neural data, the original model lacks a method to infer their latent trajectories in the learned latent space, limiting its ability for estimating the internal state repetition. Here, we extend the P-GPLVM to enable the latent variable inference of new data constrained by previously learned smoothness and mapping information. We also describe a principled approach for the constrained latent variable inference for temporally-compressed patterns of activity, such as those found in population burst events (PBEs) during hippocampal sharp-wave ripples, as well as metrics for assessing whether the inferred new latent variables are congruent with a previously learned manifold in the latent space. Applying these approaches to hippocampal ensemble recordings during active maze exploration, we replicate the result that P-GPLVM learns a latent space encoding the animal’s position. We further demonstrate that this latent space can differentiate one maze context from another. By inferring the latent variables of new neural data during running, certain internal neural states are observed to repeat, which is in accordance with the similarity of experiences encoded by its nearby neural trajectories in the training data manifold. Finally, repetition of internal neural states can be estimated for neural activity during PBEs as well, allowing the identification for replay events of versatile behaviors and more general experiences. Thus, our extension of the P-GPLVM framework for unsupervised analysis of neural activity can be used to answer critical questions related to scientific discovery.

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this article, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian networks (BNs) due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods, often through an auxiliary network, to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of maxsum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent (AugCA) are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark data sets of different models are given for both data representation and feature-learning tasks.

Estimates of longitudinal change in the parameters of latent (i.e. unobserved) variables, including means, are affected by non-response on the items or indicators of the latent variable. This study used Monte Carlo simulation and a numeric example to compare four ordinal item non-response methods: non-negative matrix factorization (NNMF), multiple imputation with conditional proportional odds model (POM), full information maximum likelihood (FIML) and complete-case analysis, when estimating the longitudinal change in latent variable means. The mean squared error for the NNMF method was more than 40% lower than for the FIML and POM methods when the latent variable correlations over time were strong, percentage of missing data was 25% or more, and sample size was large. The NNMF method is a promising method to address item non-response. It is relatively efficient when sample size is large, and the percentage of missing data is high but has limitations under other data-analytic conditions.

Effective health indicator (HI) construction can help equipment managers detect the abnormal state of rotating machinery quickly. However, although the current deep learning-based HI construction methods have good life prediction value, most of them lose the ability to detect device anomalies and little work has been done on model interpretability. Therefore, an interpretable HI construction method based on semi-supervised autoencoder (AE) latent space variance maximization (SSALSVM) was proposed to monitor the health status of bearings. In order to fully excavate degradation features inside the device and make the model focus on the encoding process, a deep convolutional neural network (DCNN) is used as the encoding layer, while only a layer of fully-connected layer is used as the decoding layer. In addition, to enable the latent space to capture the device early degradation point (EDP) successfully, an auxiliary layer is added to the output of the encoder layer. Simultaneously, for improving the sensitivity of the indicator to capture equipment abnormal state and highlight the difference between equipment health state and degradation state, the constraint of variance maximization is added into the latent space. The model optimizing process was presented by observing the projected variance of the test set in latent space of each epoch model. The validity of the proposed HI was verified by comparison experiments on two datasets.

PurposeTo investigate the feasibility of extracting a low-dimensional latent structure of anterior segment optical coherence tomography (AS-OCT) images by use of a β-variational autoencoder (β-VAE).MethodsWe retrospectively collected 2111 AS-OCT images from 2111 eyes of 1261 participants from the ongoing Asan Glaucoma Progression Study. After hyperparameter optimization, the images were analyzed with β-VAE.ResultsThe mean participant age was 64.4 years, with mean values of visual field index and mean deviation of 86.4% and −5.33 dB, respectively. After experiments, a latent space size of 6 and β value of 53 were selected for latent space analysis with β-VAE. Latent variables were successfully disentangled, showing readily interpretable distinct characteristics, such as the overall depth and area of the anterior chamber (η1), pupil diameter (η2), iris profile (η3 and η4), and corneal curvature (η5).Conclusionsβ-VAE can successfully be applied for disentangled latent space representation of AS-OCT images, revealing the high possibility of applying unsupervised learning in the medical image analysis.Translational RelevanceThis study demonstrates that a deep learning–based latent space model can be applied for the analysis of AS-OCT images.

Noisy, high-dimensional time series observations can often be described by a set of low-dimensional latent variables. Commonly used methods to extract these latent variables typically assume instantaneous relationships between the latent and observed variables. In many physical systems, changes in the latent variables manifest as changes in the observed variables after time delays. Techniques that do not account for these delays can recover a larger number of latent variables than are present in the system, thereby making the latent representation more difficult to interpret. In this work, we introduce a novel probabilistic technique, time-delay gaussian-process factor analysis (TD-GPFA), that performs dimensionality reduction in the presence of a different time delay between each pair of latent and observed variables. We demonstrate how using a gaussian process to model the evolution of each latent variable allows us to tractably learn these delays over a continuous domain. Additionally, we show how TD-GPFA combines temporal smoothing and dimensionality reduction into a common probabilistic framework. We present an expectation/conditional maximization either (ECME) algorithm to learn the model parameters. Our simulations demonstrate that when time delays are present, TD-GPFA is able to correctly identify these delays and recover the latent space. We then applied TD-GPFA to the activity of tens of neurons recorded simultaneously in the macaque motor cortex during a reaching task. TD-GPFA is able to better describe the neural activity using a more parsimonious latent space than GPFA, a method that has been used to interpret motor cortex data but does not account for time delays. More broadly, TD-GPFA can help to unravel the mechanisms underlying high-dimensional time series data by taking into account physical delays in the system.

Modeling in the observable or latent space? A comparison of dynamic latent variable based monitoring methods for sensor fault detection

Ranking evaluation of institutions based on a Bayesian network having a latent variable

In the past, the way that the discrete choice models treated the unobservable variables such as perceived and attitude variables caused some problems. For example, the estimated parameters were not consistent and efficient, the index could not be used directly to affect the policies, and the index may not be suitable in the prediction, and the unobservable variables are unchangeable as individuals vary. It results in some disadvantages for policies analysis and applications. Now the most widely used models is Maximum Simulated Likelihood to overcome the model with multi dimension parameter integration problems. There exists unstable situation in parameter calibrating as the number of simulated draws is not enough. The earlier reference seldom explored the influence of the model calibrating by simulating the number of draws. The number of draws is quite important. The purpose of this research is to use the simultaneous equations concepts for integrating latent variables and individual choice models, and decide the best number of the random draws. The parameters estimation will be more consistent, efficient, and stable. In addition, this study also discusses the heterogeneity, correlation, and hierarchy cause-and-effect of service quality perceived variables to increase the explanatory power in the models and to know the travelers’ riding behaviors. It will be implementing to policy analysis, to understand every policy variables on direct and indirect effect of choice behaviors, and to solve the problem which came about in the previous researches. The scope of this research was the intercity bus travelers of Taipei-Kaohsiung route in Taiwan. For the part of dealing with the latent variable choice models, the evidence shows that as the service satisfaction factors add in the model, the explanatory power will be improved. It also presented the methodology is efficient. For the part of the number of random draws, simulating the log-likelihood function of different latent variable choice models will boost as the simulating number increase. While the Halton random number reached to 4000, the improvement of the simulation to simulating the log-likelihood function was getting smooth. So this research picks 4,000 as the number of random draws. In this paper, if the latent variables are assumed as exogenous variables, the I II - - integrating latent variables heterogeneity and correlation of the choice model has better explanatory power. It presents that there are individual heterogeneity and correlation among the three service quality variables-convenience, commoditization and entertainment, the passengers’ interaction. In addition, there is positive influence on the travelers’ choices to the intercity bus companies by these three service quality satisfaction variables. The convenience and commoditization and entertainment satisfaction of the travelers to the bus companies will influence positively on travelers’ choice behaviors, and the convenience satisfaction is stronger than commoditization and entertainment satisfaction. Moreover, the travelers’ interaction satisfaction influence indirectly on the travelers’ choice behaviors to the bus companies through the convenience and commoditization and entertainment satisfaction of the travelers.