Time-efficient Bayesian Inference for a (Skewed) Von Mises Distribution on the Torus in a Deep Probabilistic Programming Language

Efficient Bayesian inference with latent Hamiltonian neural networks in No-U-Turn Sampling

Hybrid Monte Carlo (HMC) has been widely applied to numerous posterior inference problems in machine learning and statistics. HMC has two main practical issues, the first is the deterioration in acceptance rates as the system size increases and the second is its sensitivity to two user-specified parameters: the step size and trajectory length. The former issue is addressed by sampling from an integrator-dependent modified or shadow density and compensating for the induced bias via importance sampling. The latter issue is addressed by adaptively setting the HMC parameters, with the state-of-the-art method being the No-U-Turn Sampler (NUTS). We combine the benefits of NUTS with those attained by sampling from the shadow density, by adaptively setting the trajectory length and step size of Separable Shadow Hamiltonian Hybrid Monte Carlo (S2HMC). This leads to a new algorithm, adaptive S2HMC (A-S2HMC), that shows improved performance over S2HMC and NUTS across various targets and leaves the target density invariant.

Probabilistic Logic Programming extends Logic Programming by enabling the representation of uncertain information by means of probability theory. Probabilistic Logic Programming is at the intersection of two wider research fields: the integration of logic and probability and Probabilistic Programming. Logic enables the representation of complex relations among entities while probability theory is useful for model uncertainty over attributes and relations. Combining the two is a very active field of study. Probabilistic Programming extends programming languages with probabilistic primitives that can be used to write complex probabilistic models. Algorithms for the inference and learning tasks are then provided automatically by the system. Probabilistic Logic programming is at the same time a logic language, with its knowledge representation capabilities, and a Turing complete language, with its computation capabilities, thus providing the best of both worlds. Since its birth, the field of Probabilistic Logic Programming has seen a steady increase of activity, with many proposals for languages and algorithms for inference and learning. Foundations of Probabilistic Logic Programming aims at providing an overview of the field with a special emphasis on languages under the Distribution Semantics, one of the most influential approaches. The book presents the main ideas for semantics, inference, and learning and highlights connections between the methods. Many examples of the book include a link to a page of the web application http://cplint.eu where the code can be run online.

The basics of logic programming, as embodied by Prolog, are generally well-known in the programming language community. However, more advanced techniques, such as tabling, answer subsumption and probabilistic logic programming fail to attract the attention of a larger audience. The cause for the community's seemingly limited interest lies with the presentation of these features: the literature frequently focuses on implementations and examples that do little to aid the understanding of non-experts in the field. The key point is that many of these advanced logic programming features can be characterised in more generally known, more accessible terms. In my research I try to reconcile these advanced concepts from logic programming (Tabling, Answer subsumption and probabilistic programming) with concepts from functional programming (effects, monads and applicative functors).

Probabilistic programming refers to the idea of using standard programming constructs for specifying probabilistic models from machine learning and statistics, and employing generic inference algorithms for answering various queries on these models, such as posterior inference and estimation of model evidence. Although this idea itself is not new and was, in fact, explored by several programming-language and statistics researchers in the early 2000, it is only in the last few years that probabilistic programming has gained a large amount of attention among researchers in machine learning and programming languages, and that expressive and efficient probabilistic programming systems (such as Anglican, Church, Figaro, Infer.net, PSI, PyMC, Stan, and Venture) started to appear and have been taken up by a nontrivial number of users. The primary goal of my talk is to introduce probabilistic programming to the CONCUR/QUEST/FORMATS audience. At the end of my talk, I want the audience to understand basic results and techniques in probabilistic programming and to feel that these results and techniques are relevant or at least related to what she or he studies, although they typically come from foreign research areas, such as machine learning and statistics. My talk will contain both technical materials and lessons that I learnt from my machine-learning colleagues in Oxford, who are developing a highly-expressive higher-order probabilistic programming language, called Anglican. It will also include my work on the denotational semantics of higher-order probabilistic programming languages and their inference algorithms, which are jointly pursued with colleagues in Cambridge, Edinburgh, Oxford and Tubingen.

We explore a general framework in Markov chain Monte Carlo (MCMC) sampling where sequential proposals are tried as a candidate for the next state of the Markov chain. This sequential-proposal framework can be applied to various existing MCMC methods, including Metropolis–Hastings algorithms using random proposals and methods that use deterministic proposals such as Hamiltonian Monte Carlo (HMC) or the bouncy particle sampler. Sequential-proposal MCMC methods construct the same Markov chains as those constructed by the delayed rejection method under certain circumstances. In the context of HMC, the sequential-proposal approach has been proposed as extra chance generalized hybrid Monte Carlo (XCGHMC). We develop two novel methods in which the trajectories leading to proposals in HMC are automatically tuned to avoid doubling back, as in the No-U-Turn sampler (NUTS). The numerical efficiency of these new methods compare favorably to the NUTS. We additionally show that the sequential-proposal bouncy particle sampler enables the constructed Markov chain to pass through regions of low target density and thus facilitates better mixing of the chain when the target density is multimodal.

Bayesian approaches to Neural Architecture Search (NAS) offer several benefits when compared to typical grid-search methods. Marginalization of a joint posterior distribution of network parameters as well as network architecture specification allows for a principled analysis of predictive and model uncertainty. In this paper, we propose a method of applying Reversible Jump Markov Chain Monte Carlo (RJMCMC) as inference over a distribution of neural network depths. We refer to these models as Reversible Jump Bayesian Neural Networks (RJBNN), and show that a tenable acceptance probability for across-dimension proposals can be achieved through the use of a composite sampler featuring the Hamiltonian Monte Carlo (HMC) based No U-Turn Sampler (NUTS) algorithm. We compare the results of an RJBNN model with standard Bayesian Neural Networks (BNN) of comparable architecture sizes on a regression task for the Boston Housing dataset.

We propose design guidelines for a probabilistic programming facility suitable for deployment as a part of a production software system. As a reference implementation, we introduce Infergo, a probabilistic programming facility for Go, a modern programming language of choice for server-side software development. We argue that a similar probabilistic programming facility can be added to most modern general-purpose programming languages. Probabilistic programming enables automatic tuning of program parameters and algorithmic decision making through probabilistic inference based on the data. To facilitate addition of probabilistic programming capabilities to other programming languages, we share implementation choices and techniques employed in development of Infergo. We illustrate applicability of Infergo to various use cases on case studies, and evaluate Infergo's performance on several benchmarks, comparing Infergo to dedicated inference-centric probabilistic programming frameworks.

In the contribution presented we deal with one method of stochastic programming - probabilistic programming. Probabilistic programming is a method of stochastic programming in which the probabilities of the values of the variables are of interest. Our approach is that scientific problem solving we can construct in logical reasoning over some mathematical theories. In this approach we deal with category theory for construction of type theory and of the logical system. Then we formulate the logical theory to enclose the solution of the problem. These steps we show at well-known examples: the random number generator, the Blackjack game and the Monty Hall problem.

Probabilistic or stochastic programming is concerned with optimization problem in which some or all parameters are treated as random variable. A general approach to deal with uncertainty is to assign a probability distribution to the unknown parameter. In this paper a multi-objective probabilistic programming problem has been considered when right hand side vector follows Weibull distribution. The probabilistic problem is converted into an equivalent deterministic model, and is solved using LINGO software. A numerical is presented to illustrate the methodology.

What does a probabilistic program actually compute? How can one formally reason about such probabilistic programs? This valuable guide covers such elementary questions and more. It provides a state-of-the-art overview of the theoretical underpinnings of modern probabilistic programming and their applications in machine learning, security, and other domains, at a level suitable for graduate students and non-experts in the field. In addition, the book treats the connection between probabilistic programs and mathematical logic, security (what is the probability that software leaks confidential information?), and presents three programming languages for different applications: Excel tables, program testing, and approximate computing. This title is also available as Open Access on Cambridge Core.

The control and data flow of a program can be represented using continuations, a concept from denotational semantics that has practical application in real compilers. This book shows how continuation-passing style is used as an intermediate representation on which to perform optimisations and program transformations. Continuations can be used to compile most programming languages. The method is illustrated in a compiler for the programming language Standard ML. However, prior knowledge of ML is not necessary, as the author carefully explains each concept as it arises. This is the first book to show how concepts from the theory of programming languages can be applied to the producton of practical optimising compilers for modern languages like ML. This book will be essential reading for compiler writers in both industry and academe, as well as for students and researchers in programming language theory.

Probabilistic programming languages (PPLs) are an expressive means for creating and reasoning about probabilistic models. Unfortunately hybrid probabilistic programs that involve both continuous and discrete structures are not well supported by today’s PPLs. In this paper we develop a new approximate inference algorithm for hybrid probabilistic programs that first discretizes the continuous distributions and then performs discrete inference on the resulting program. The key novelty is a form of discretization that we call bit blasting , which uses a binary representation of numbers such that a domain of 2 b discretized points can be succinctly represented as a discrete probabilistic program over poly b Boolean random variables. Surprisingly, we prove that many common continuous distributions can be bit blasted in a manner that incurs no loss of accuracy over an explicit discretization and supports efficient probabilistic inference. We have built a probabilistic programming system for hybrid programs called HyBit , which employs bit blasting followed by discrete probabilistic inference. We empirically demonstrate the benefits of our approach over existing sampling-based and symbolic inference approaches

This collection of original essays reflects the breadth of current research in computer science. This collection of original essays reflects the breadth of current research in computer science. Robin Milner, a major figure in the field, has made many fundamental contributions, particularly in theoretical computer science, the theory of programming languages, and functional programming languages. Following a brief biography of Milner, the book contains five sections: Semantic Foundations, Programming Logic, Programming Languages, Concurrency, and Mobility. Together the pieces convey a seamless whole, ranging from highly abstract concepts to systems of great utility. Contributors Samson Abramsky, J. C. M. Baeten, Sergey Berezin, J. A. Bergstra, Gérard Berry, Lars Birkedal, Gérard Boudol, Edmund Clarke, Pierre Collette, Robert L. Constable, Pierre-Louis Curien, Jaco de Bakker, Uffe H. Engberg, William Ferreira, Fabio Gadducci, Mike Gordon, Robert Harper, Matthew Hennessy, Yoram Hirshfeld, C. A. R. Hoare, Gérard Huet, Paul B. Jackson, Alan S. A. Jeffrey, Somesh Jha, He Jifeng, Cliff B. Jones, Cosimo Laneve, Xinxin Liu, Will Marrero, Faron Moller, Ugo Montanari, Pavel Naumov, Mogens Nielsen, Joachim Parrow, Lawrence C. Paulson, Benjamin C. Pierce, Gordon Plotkin, M. A. Reniers, Amokrane Saïbi, Augusto Sampaio, Davide Sangiorgi, Scott A. Smolka, Eugene W. Stark, Christopher Stone, Mads Tofte, David N. Turner, Juan Uribe, Franck van Breugel, David Walker, Glynn Winskel

Satellite data provides essential insights into the spatiotemporal distribution of CO2 concentrations. However, many atmospheric inverse models fail to adequately incorporate the spatial and temporal correlations inherent in satellite observations and often lack rigorous methods for estimating parameters like spatial length scales. We introduce an inference model that processes the spatiotemporal covariance in satellite data and estimates hyperparameters such as covariance length scales. Our approach uses the Gaussian process (GP) machine learning (ML) and modern probabilistic programming languages (PPLs) to perform atmospheric inversions of emissions from satellite data. We develop a GP ML inversion system based on modern PPLs and the GEOS-Chem chemical transport model, simulating atmospheric CO2 concentrations corresponding to the Orbiting Carbon Observatory-2/3 (OCO-2/3) data for July 2020. In our supervised learning framework, we treat the GEOS-Chem simulated data set as the target, with predictors derived by scaling the target with sector-specific factors hidden from the GP machine. Our results show that the GP model, combined with GPU-enabled PPLs, effectively retrieves true emission scaling factors and infers noise levels concealed within the data. This suggests that our method could be applied over larger areas with more complex covariance structures, enabling comprehensive analysis of the spatiotemporal patterns observed in OCO-2/3 and similar satellite data sets.