Central limit theorem for bayesian neural network trained with variational inference

Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem. This approach has been successfully applied to various models and large-scale applications. In this review, we give an overview of recent trends in variational inference. We first introduce standard mean field variational inference, then review recent advances focusing on the following aspects: (a) scalable VI, which includes stochastic approximations, (b) generic VI, which extends the applicability of VI to a large class of otherwise intractable models, such as non-conjugate models, (c) accurate VI, which includes variational models beyond the mean field approximation or with atypical divergences, and (d) amortized VI, which implements the inference over local latent variables with inference networks. Finally, we provide a summary of promising future research directions.

Stochastic variational inference for Bayesian deep neural networks (DNNs) requires specifying priors and approximate posterior distributions for neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dimensional weight space. Based on empirical Bayes approach, we propose Bayesian MOdel Priors Extracted from Deterministic DNN (MOPED) method to choose meaningful prior distributions over weight space using deterministic weights derived from the pretrained DNNs of equivalent architecture. We empirically evaluate the proposed approach on real-world applications including image classification, video activity recognition and audio classification tasks with varying complex neural network architectures. The proposed method enables scalable variational inference with faster training convergence and provides reliable uncertainty quantification.

This project evaluated the use of emerging spintronic memory devices for robust and efficient variational inference schemes. Variational inference (VI) schemes, which constrain the distribution for each weight to be a Gaussian distribution with a mean and standard deviation, are a tractable method for calculating posterior distributions of weights in a Bayesian neural network such that this neural network can also be trained using the powerful backpropagation algorithm. Our project focuses on domain-wall magnetic tunnel junctions (DW-MTJs), a powerful multi-functional spintronic synapse design that can achieve low power switching while also opening the pathway towards repeatable, analog operation using fabricated notches. Our initial efforts to employ DW-MTJs as an all-in-one stochastic synapse with both a mean and standard deviation didn’t end up meeting the quality metrics for hardware-friendly VI. In the future, new device stacks and methods for expressive anisotropy modification may make this idea still possible. However, as a fall back that immediately satisfies our requirements, we invented and detailed how the combination of a DW-MTJ synapse encoding the mean and a probabilistic Bayes-MTJ device, programmed via a ferroelectric or ionically modifiable layer, can robustly and expressively implement VI. This design includes a physics-informed small circuit model, that was scaled up to perform and demonstrate rigorous uncertainty quantification applications, up to and including small convolutional networks on a grayscale image classification task, and larger (Residual) networks implementing multi-channel image classification. Lastly, as these results and ideas all depend upon the idea of an inference application where weights (spintronic memory states) remain non-volatile, the retention of these synapses for the notched case was further interrogated. These investigations revealed and emphasized the importance of both notch geometry and anisotropy modification in order to further enhance the endurance of written spintronic states. In the near future, these results will be mapped to effective predictions for room temperature and elevated operation DW-MTJ memory retention, and experimentally verified when devices become available.

Context.All-sky observations show both Galactic and non-Galactic diffuse emission, for example from interstellar matter or the cosmic microwave background (CMB). The decomposition of the emission into different underlying radiative components is an important signal reconstruction problem.Aims.We aim to reconstruct radiative all-sky components using spectral data, without incorporating knowledge about physical or spatial correlations.Methods.We built a self-instructing algorithm based on variational autoencoders following three steps: (1)We stated a forward model describing how the data set was generated from a smaller set of features, (2) we used Bayes’ theorem to derive a posterior probability distribution, and (3) we used variational inference and statistical independence of the features to approximate the posterior. From this, we derived a loss function and optimized it with neural networks. The resulting algorithm contains a quadratic error norm with a self-adaptive variance estimate to minimize the number of hyperparameters. We trained our algorithm on independent pixel vectors, each vector representing the spectral information of the same pixel in 35 Galactic all-sky maps ranging from the radio to theγ-ray regime.Results.The algorithm calculates a compressed representation of the input data. We find the feature maps derived in the algorithm’s latent space show spatial structures that can be associated with all-sky representations of known astrophysical components. Our resulting feature maps encode (1) the dense interstellar medium (ISM), (2) the hot and dilute regions of the ISM, and (3) the CMB, without being informed about these components a priori.Conclusions.We conclude that Bayesian signal reconstruction with independent Gaussian latent space statistics is sufficient to reconstruct the dense and the dilute ISM, as well as the CMB, from spectral correlations only. The computational approximation of the posterior can be performed efficiently using variational inference and neural networks, making them a suitable approach to probabilistic data analysis.

Bayesian inference is now widely established as one of the principal foundations for machine learning. In practice, exact inference is rarely possible, and so a variety of approximation techniques have been developed, one of the most widely used being a deterministic framework called variational inference. In this paper we introduce Variational Message Passing (VMP), a general purpose algorithm for applying variational inference to Bayesian Networks. Like belief propagation, VMP proceeds by sending messages between nodes in the network and updating posterior beliefs using local operations at each node. Each such update increases a lower bound on the log evidence (unless already at a local maximum). In contrast to belief propagation, VMP can be applied to a very general class of conjugate-exponential models because it uses a factorised variational approximation. Furthermore, by introducing additional variational parameters, VMP can be applied to models containing non-conjugate distributions. The VMP framework also allows the lower bound to be evaluated, and this can be used both for model comparison and for detection of convergence. Variational message passing has been implemented in the form of a general purpose inference engine called VIBES ('Variational Inference for BayEsian networkS') which allows models to be specified graphically and then solved variationally without recourse to coding.

Bayesian Neural Networks with Maximum Mean Discrepancy regularization

Bayesian neural networks (BNNs) have received more and more attention because they are capable of modeling epistemic uncertainty which is hard for conventional neural networks. Markov chain Monte Carlo (MCMC) methods and variational inference (VI) are two mainstream methods for Bayesian deep learning. The former is effective but its storage cost is prohibitive since it has to save many samples of neural network parameters. The latter method is more time and space efficient, however the approximate variational posterior limits its performance. In this paper, we aim to combine the advantages of above two methods by distilling MCMC samples into an approximate variational posterior. On the basis of an existing distillation technique we first propose variational Bayesian dark knowledge method. Moreover, we propose Bayesian dark prior knowledge, a novel distillation method which considers MCMC posterior as the prior of a variational BNN. Two proposed methods both not only can reduce the space overhead of the teacher model so that are scalable, but also maintain a distilled posterior distribution capable of modeling epistemic uncertainty. Experimental results manifest our methods outperform existing distillation method in terms of predictive accuracy and uncertainty modeling.

Remaining useful life (RUL) prognostics is critical to maintenance decision making and reliability assessment. A significant amount of neural networks research has been reported to develop prognostics models that can provide high prediction accuracy. These models use advanced neural networks to improve prediction performance and provide sole point estimation for RUL. However, accurate uncertainty quantification of the RUL is essential to understand the uncertainty of the degradation process and perform reliable risk analysis and maintenance decisions. This paper proposes a new hybrid Bayesian deep learning (HBDL)-based prognostics approach. It uses long short term memory autoencoder (LSTM-AE) to extract features that include degradation information, and then uses the Bayesian neural network (BNN) to model and predict the equipment degeneration process. It learns network weights through variational inference (VI) and provides RUL prognostics results while obtaining interval estimation. Finally, a general aircraft engine data set is used to verify the proposed model. The experimental results show that this method can achieve satisfying prediction accuracy and uncertainty quantification capability.

With the analysis of the characteristic and function of circular RNAs (circRNAs), people have realized that they play a critical role in the diseases. Exploring the relationship between circRNAs and diseases is of far-reaching significance for searching the etiopathogenesis and treatment of diseases. Nevertheless, it is inefficient to learn new associations only through biotechnology. Consequently, we present a computational method, GMNN2CD, which employs a graph Markov neural network (GMNN) algorithm to predict unknown circRNA-disease associations. First, used verified associations, we calculate semantic similarity and Gaussian interactive profile kernel similarity (GIPs) of the disease and the GIPs of circRNA and then merge them to form a unified descriptor. After that, GMNN2CD uses a fusion feature variational map autoencoder to learn deep features and uses a label propagation map autoencoder to propagate tags based on known associations. Based on variational inference, GMNN alternate training enhances the ability of GMNN2CD to obtain high-efficiency high-dimensional features from low-dimensional representations. Finally, 5-fold cross-validation of five benchmark datasets shows that GMNN2CD is superior to the state-of-the-art methods. Furthermore, case studies have shown that GMNN2CD can detect potential associations. The source code and data are available at https://github.com/nmt315320/GMNN2CD.git.

Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). One of the approaches to encourage more systematic and efficient exploration relies on surprise as an intrinsic reward for the agent. We introduce a new definition of surprise and its RL implementation named variational assorted surprise exploration (VASE). VASE uses a Bayesian neural network as a model of the environment dynamics and is trained using variational inference, alternately updating the accuracy of the agent's model and policy. Our experiments show that in continuous control sparse reward environments, VASE outperforms other surprise-based exploration techniques.

With the fast scaling-down and evolution of integrated circuit (IC) manufacturing technology, the fabrication process becomes highly complex, and the experimental cost of the processes is significantly elevated. Therefore, in many cases, it is very costly to obtain a sufficient amount of experimental data. To develop an efficient method to predict the results of semiconductor experiments with a small amount of known data, we use a novel method based on Bayesian framework with the prior distribution constructed by technology computer-aided-design (TCAD) physical models. This method combines the advantages of statistical models and physical models in the aspect that TCAD can provide visionary guidance on an experiment when a limited amount of experimental data is available, and a machine learning model can account for subtle anomalous effects. Specifically, we use aspect ratio dependent etching (ARDE) phenomenon as an example and use variational inference with Kullback-Leibler divergence minimization to achieve the approximation to the posterior distribution. The relation between etching process input parameters and etching depth is learned using the Bayesian neural network with TCAD priors. Using this method with 35 neurons per hidden layer, mean square error (MSE) in the test set is reduced from 0.2896 to 0.0175, 0.058 to 0.0183, 0.0563 to 0.0188, 0.058 to 0.019 for partition=10, 20, 30, 40, respectively, reference to the baseline BNN where a regular normal distribution prior with zero mean and unity standard deviation N(0,1) is used.

Adolescent depression is a significant public health concern in low- and middle-income countries, including Kenya, where limited screening capacity contributes to under diagnosis. This study developed and validated a Bayesian Neural Network for predicting depression risk among adolescents, leveraging probabilistic inference to capture predictive uncertainty and complex nonlinear relationships. Secondary data for 2,192 adolescents aged 12-18 years were obtained from the Open Science Framework, incorporating psychosocial, demographic, and mental health measures. Significant predictors were identified using chi-square tests and point-biserial correlations, followed by forward feature selection. The proposed Bayesian Neural Network employed a three-hidden-layer feedforward architecture with ReLU and sigmoid activations, Gaussian priors on weights, and Bayes-by-Backprop Variational inference. Model performance was benchmarked against a Random Forest classifier using cross-validation. Results indicate that anxiety, loneliness, perceived social support, gratitude, positive youth development, academic self-perception, gender, academic form, financial status, parental education, and age are significant predictors of depression. The model achieved superior performance, with an accuracy of 80.13%, F1 score of 0.876, recall of 0.965, ROC-AUC of 0.813, and PR-AUC of 0.913, outperforming the Random Forest in most metrics except precision. Calibration analysis yielded a low Brier score (0.0693), indicating well-calibrated probabilistic predictions. These findings demonstrate the suitability of Bayesian Neural Networks for adolescent depression risk screening in resource-constrained settings, where uncertainty-aware decision support is critical.

The free energy principle, an influential framework in computational neuroscience and theoretical neurobiology, starts from the assumption that living systems ensure adaptive exchanges with their environment by minimizing the objective function of variational free energy. Following this premise, it claims to deliver a promising integration of the life sciences. In recent work, Markov blankets, one of the central constructs of the free energy principle, have been applied to resolve debates central to philosophy (such as demarcating the boundaries of the mind). The aim of this paper is twofold. First, we trace the development of Markov blankets starting from their standard application in Bayesian networks, via variational inference, to their use in the literature on active inference. We then identify a persistent confusion in the literature between the formal use of Markov blankets as an epistemic tool for Bayesian inference, and their novel metaphysical use in the free energy framework to demarcate the physical boundary between an agent and its environment. Consequently, we propose to distinguish between "Pearl blankets" to refer to the original epistemic use of Markov blankets and "Friston blankets" to refer to the new metaphysical construct. Second, we use this distinction to critically assess claims resting on the application of Markov blankets to philosophical problems. We suggest that this literature would do well in differentiating between two different research programmes: "inference with a model" and "inference within a model." Only the latter is capable of doing metaphysical work with Markov blankets, but requires additional philosophical premises and cannot be justified by an appeal to the success of the mathematical framework alone.

One of the most important foundational challenge of Statistical relational learning is the development of a uniform framework in which learning and logical reasoning are seamlessly integrated. State of the art approaches propose to modify well known machine learning methods based on parameter optimization (e.g., neural networks and graphical models) in order to take into account structural knowledge expressed by logical constraints. In this paper, we follow an alternative direction, considering the Bayesian approach to machine learning. In particular, given a partial knowledge in hybrid domains (i.e., domains that contains relational structure and continuous features) as a set \(\mathcal {T}\) of axioms and a stochastic (in)dependence hypothesis \(\mathscr {F}\) encoded in a first order language \(\mathcal {L}\), we propose to model it by a probability distribution function (PDF) \(p(\mathbf {x}\mid \mathcal {T},\mathscr {F})\) over the \(\mathcal {L}\)-interpretations \(\mathbf {x}\). The stochastic (in)dependence \(\mathscr {F}\) is represented as a Bayesian Markov Logic Network w.r.t. a parametric undirected graph, interpreted as the PDF. We propose to approximate \(P(\mathbf {x}\mid \mathcal {T},\mathscr {F})\) by variational inference and show that such approximation is possible if and only if \(\mathscr {F}\) satisfies a property called orthogonality. This property can be achieved also by extending \(\mathcal {L}\), and adjusting \(\mathcal {T}\) and \(\mathscr {F}\).

Modern learning tasks relies on flexible, expressive models, often via deep neural networks, to capture complex data patterns. While these models expand modeling capability, they make exact inference computationally intractable. Variational Inference (VI) offers a scalable framework that turns approximate inference into an optimization problem, enabling efficient computation of quantities such as likelihoods, distances, and information measures. This framework serves as the theoretical backbone of modern deep generative models, with variational autoencoders (VAEs) not only standing as a core example but also forming essential components in more advanced architectures, such as diffusion-based models for image generation and emerging diffusion large language models.However, VI introduces its own challenges. Since its formal introduction, VI remains relatively new as a statistical notion, and the increasing diversity of its applications and algorithmic variants has further intensified open questions about its consistency, convergence, and approximation fidelity.In this dissertation, I demonstrate the methodological shortcomings of existing algorithms and applications, and offer statistically grounded perspectives and solutions to enhance the flexibility and reliability of VI. The goal is to bridge the gap between the growing complexity of deep learning models and the necessity of statistical rigor in learning principles. The first part of our research investigates the weighted ELBO in the $\beta$-VAE framework and interprets it through the lens of a learned decoder variance, thus it outperforms in conditional density estimation due to the expressiveness of Gaussian VAEs. Building on this insight, an efficient annealing optimization scheme is proposed for robust inference of model parameters, preventing posterior collapse. The second work studies the application of VI in the context of survival analysis, where a censor-dependent variational framework is introduced to address challenges of the censored response variable and to provide more reliable predictive performance in practice. In the final part of our research, I generalize variational inference beyond its conventional settings to encompass diverse learning tasks, including model compression and coreset construction.