Likelihood-Free Inference in High-Dimensional Models.

The distribution of fitness effects (DFE) for new mutations is fundamental for many aspects of population and quantitative genetics. In this study, we have inferred the DFE in the single-celled alga Chlamydomonas reinhardtii by estimating changes in the frequencies of 254 spontaneous mutations under experimental evolution and equating the frequency changes of linked mutations with their selection coefficients. We generated seven populations of recombinant haplotypes by crossing seven independently derived mutation accumulation lines carrying an average of 36 mutations in the haploid state to a mutation-free strain of the same genotype. We then allowed the populations to evolve under natural selection in the laboratory by serial transfer in liquid culture. We observed substantial and repeatable changes in the frequencies of many groups of linked mutations, and, surprisingly, as many mutations were observed to increase as decrease in frequency. Mutation frequencies were highly repeatable among replicates, suggesting that selection was the cause of the observed allele frequency changes. We developed a Bayesian Monte Carlo Markov Chain method to infer the DFE. This computes the likelihood of the observed distribution of changes of frequency, and obtains the posterior distribution of the selective effects of individual mutations, while assuming a two-sided gamma distribution of effects. We infer that the DFE is a highly leptokurtic distribution, and that approximately equal proportions of mutations have positive and negative effects on fitness. This result is consistent with what we have observed in previous work on a different C. reinhardtii strain, and suggests that a high fraction of new spontaneously arisen mutations are advantageous in a simple laboratory environment.

Many statistical models in cosmology can be simulated forwards but have intractable likelihood functions. Likelihood-free inference methods allow us to perform Bayesian inference from these models using only forward simulations, free from any likelihood assumptions or approximations. Likelihood-free inference generically involves simulating mock data and comparing to the observed data; this comparison in data-space suffers from the curse of dimensionality and requires compression of the data to a small number of summary statistics to be tractable. In this paper we use massive asymptotically-optimal data compression to reduce the dimensionality of the data-space to just one number per parameter, providing a natural and optimal framework for summary statistic choice for likelihood-free inference. Secondly, we present the first cosmological application of Density Estimation Likelihood-Free Inference (\textsc{delfi}), which learns a parameterized model for joint distribution of data and parameters, yielding both the parameter posterior and the model evidence. This approach is conceptually simple, requires less tuning than traditional Approximate Bayesian Computation approaches to likelihood-free inference and can give high-fidelity posteriors from orders of magnitude fewer forward simulations. As an additional bonus, it enables parameter inference and Bayesian model comparison simultaneously. We demonstrate Density Estimation Likelihood-Free Inference with massive data compression on an analysis of the joint light-curve analysis supernova data, as a simple validation case study. We show that high-fidelity posterior inference is possible for full-scale cosmological data analyses with as few as $\sim 10^4$ simulations, with substantial scope for further improvement, demonstrating the scalability of likelihood-free inference to large and complex cosmological datasets.

Variational Bayes approximation of factor stochastic volatility models

The Bayesian framework is commonly used to quantify uncertainty in seismic inversion. To perform Bayesian inference, Markov chain Monte Carlo (MCMC) algorithms are regarded as the gold standard technique for sampling from the posterior probability distribution. Consistent MCMC methods have trouble for complex, high-dimensional models, and most methods scale poorly to large datasets, such as those arising in seismic inversion. As an alternative, approximate MCMC methods based on unadjusted Langevin dynamics offer scalability and more rapid sampling at the cost of biased inference. However, when assessing the quality of approximate MCMC samples for characterizing the posterior distribution, most diagnostics fail to account for these biases. In this work, we introduce the kernel Stein discrepancy (KSD) as a diagnostic tool to determine the convergence of MCMC samples for Bayesian seismic inversion. We demonstrate the use of the KSD for measuring sample quality and selecting the optimal Langevin MCMC algorithm for two Gaussian Bayesian inference problems.

We present a new Bayesian method for estimating demographic and phylogenetic history using population genomic data. Several key innovations are introduced that allow the study of diverse models within an Isolation-with-Migration framework. The new method implements a 2-step analysis, with an initial Markov chain Monte Carlo (MCMC) phase that samples simple coalescent trees, followed by the calculation of the joint posterior density for the parameters of a demographic model. In step 1, the MCMC sampling phase, the method uses a reduced state space, consisting of coalescent trees without migration paths, and a simple importance sampling distribution without the demography of interest. Once obtained, a single sample of trees can be used in step 2 to calculate the joint posterior density for model parameters under multiple diverse demographic models, without having to repeat MCMC runs. Because migration paths are not included in the state space of the MCMC phase, but rather are handled by analytic integration in step 2 of the analysis, the method is scalable to a large number of loci with excellent MCMC mixing properties. With an implementation of the new method in the computer program MIST, we demonstrate the method's accuracy, scalability, and other advantages using simulated data and DNA sequences of two common chimpanzee subspecies: Pan troglodytes (P. t.) troglodytes and P. t. verus.

Identification of contaminant sources in rivers is crucial for river protection and emergency response. This study presents an innovative approach for identifying river pollution sources by using Bayesian inference and cellular automata (CA) modeling. A general Bayesian framework is proposed that combines the CA model with observed data to identify unknown sources of river pollution. To reduce the computational burden of the Bayesian inference, a CA contaminant transport model is developed to efficiently simulate pollutant concentration values in the river. These simulated concentration values are then used to calculate the likelihood function of available measurements. The Markov chain Monte Carlo (MCMC) method is used to produce the posterior distribution of contaminant source parameters, which is a sampling-based method that enables the estimation of complex posterior distributions. The suggested methodology is applied to a real case study of the Fen River in Yuncheng City, Shanxi Province, Northern China, and it estimates the release time, release mass, and source location with relative errors below 19%. The research indicates that the proposed methodology is an effective and flexible way to identify the location and concentrations of river contaminant sources.

Bayesian model updating provides a rigorous framework to account for uncertainty induced by lack of knowledge about engineering systems in their respective mathematical models through updates of the joint probability density function (PDF), the so-called posterior PDF, of the unknown model parameters. The Markov chain Monte Carlo (MCMC) methods are currently the most popular approaches for generating samples from the posterior PDF. However, these methods often found wanting when sampling from difficult distributions (e.g., high-dimensional PDFs, PDFs with flat manifolds, multimodal PDFs, and very peaked PDFs). This paper introduces a new multi-level sampling approach for Bayesian model updating, called Sequential Gauss-Newton algorithm, which is inspired by the Transitional Markov chain Monte Carlo (TMCMC) algorithm. The Sequential Gauss-Newton algorithm improves two aspects of TMCMC to make an efficient and effective MCMC algorithm for drawing samples from difficult posterior PDFs. First, the statistical efficiency of the algorithm is enhanced by use of the systematic resampling scheme. Second, a new MCMC algorithm, called Gauss-Newton MCMC algorithm, is proposed which is essentially an M-H algorithm with a Gaussian proposal PDF tailored to the posterior PDF using the gradient and Hessian information of the negative log posterior. The effectiveness of the proposed algorithm for solving the Bayesian model updating problem is illustrated using three examples with irregularly shaped posterior PDFs.

Density-estimation likelihood-free inference (DELFI) has recently been\nproposed as an efficient method for simulation-based cosmological parameter\ninference. Compared to the standard likelihood-based Markov Chain Monte Carlo\n(MCMC) approach, DELFI has several advantages: it is highly parallelizable,\nthere is no need to assume a possibly incorrect functional form for the\nlikelihood and complicated effects (e.g the mask and detector systematics) are\neasier to handle with forward models. In light of this, we present two DELFI\npipelines to perform weak lensing parameter inference with lognormal\nrealizations of the tomographic shear field -- using the C_l summary statistic.\nThe first pipeline accounts for the non-Gaussianities of the shear field,\nintrinsic alignments and photometric-redshift error. We validate that it is\naccurate enough for Stage III experiments and estimate that O(1000) simulations\nare needed to perform inference on Stage IV data. By comparing the second DELFI\npipeline, which makes no assumption about the functional form of the\nlikelihood, with the standard MCMC approach, which assumes a Gaussian\nlikelihood, we test the impact of the Gaussian likelihood approximation in the\nMCMC analysis. We find it has a negligible impact on Stage IV parameter\nconstraints. Our pipeline is a step towards seamlessly propagating all\ndata-processing, instrumental, theoretical and astrophysical systematics\nthrough to the final parameter constraints.\n

Sampling technique is one of the asymptotically unbiased estimation approaches for inference in Bayesian probabilistic models. Markov chain Monte Carlo (MCMC) is a kind of sampling methods, which is widely used in the inference of complex probabilistic models. However, current MCMC methods can incur high autocorrelation of samples, which means that the samples generated by MCMC samplers are far from independent. In this paper, we propose the neural networks Langevin Monte Carlo (NNLMC) which makes full use of the flexibility of neural networks and the high efficiency of the Langevin dynamics sampling to construct a new MCMC sampling method. We propose the new update function to generate samples and employ appropriate loss functions to improve the performance of NNLMC during the process of sampling. We evaluate our method on a large diversity of challenging distributions and real datasets. Our results show that NNLMC is able to sample from the target distribution with low autocorrelation and rapid convergence, and outperforms the state-of-the-art MCMC samplers.

Markov Chain Monte Carlo (MCMC) based methods have been the main tool for Bayesian Inference for some years now, and recently they find increasing applications in modern statistics and machine learning. Nevertheless, with the availability of large datasets and increasing complexity of Bayesian models, MCMC methods are becoming prohibitively expensive for real-world problems. At the heart of these methods, lies the computation of likelihood functions that requires access to all input data points in each iteration of the method. Current approaches, based on data subsampling, aim to accelerate these algorithms by reducing the number of the data points for likelihood evaluations at each MCMC iteration. However the existing work doesn't consider the properties of modern memory hierarchies, but treats the memory as one monolithic storage space. This paper proposes a communication-aware MCMC framework that takes into account the underlying performance of the memory subsystem. The framework is based on a novel subsampling algorithm that utilises an unbiased likelihood estimator based on Probability Proportional-to-Size (PPS) sampling, allowing information on the performance of the memory system to be taken into account during the sampling stage. The proposed MCMC sampler is mapped to an FPGA device and its performance is evaluated using the Bayesian logistic regression model on MNIST dataset. The proposed system achieves a 3.37x speed up over a highly optimised traditional FPGA design, therefore the risk in the estimates based on the generated samples is largely decreased.

This article reviews the Bayesian inference with the Monte Carlo Markov Chain (MCMC) and the Hamiltonian Monte Carlo (HMC) samplers as a competitor of the classical likelihood statistical inference for pharmacometricians. The MCMC and the HMC samplers have greatly contributed to realization of the Bayesian methods with minimal requirement of mathematical theory. They do not require any closed form of the posterior density nor linear approximation of complex nonlinear models in high dimension even with non-conjugate priors. The HMC even weakens the dependency of the chain and improves computational efficiency. Pharmacometrics is one of great beneficiaries since they use complex multivariate multilevel nonlinear mixed effects models based on the restricted maximum likelihood estimation. Comprehension of the Bayesian approach will help pharmacometricians to access the data analysis more conveniently.

Gaussian process (GP) regression or kriging has been extensively applied in the engineering literature for the purposes of building a cheap-to-evaluate surrogate, within the contexts of multi-fidelity modeling, model calibration, and design optimization. With the ongoing automation of manufacturing and industrial practices as a part of Industry 4.0, there has been a greater need for advancing GP regression techniques to handle challenges such as high input dimensionality, data paucity or big data problems, these consist primarily of proposing efficient design of experiments, optimal data acquisition strategies, sparsifying covariance kernels, and other mathematical tricks. In this work, our attention is focused on the challenges of efficiently training a GP model, which, to the authors opinion, has attracted very little attention and is to-date poorly addressed. The performance of widely used training approaches such as maximum likelihood estimation and Markov Chain Monte Carlo (MCMC) sampling can deteriorate significantly in high-dimensional and big data problems and can lead to cost deficient implementations of critical importance to many industrial applications. Here, we compare an Adaptive Sequential Monte Carlo (ASMC) sampling algorithm to classic MCMC sampling strategies and we demonstrate the effectiveness of our implementation on several mathematical problems and challenging industry applications of varying complexity. The computational time savings of the ASMC approach manifest in large-scale problems helping us to push the boundaries of applicability and scalability of GPs for model calibration in various domains of the industry, including but not limited to design automation, design engineering, smart manufacturing, predictive maintenance, and supply chain manufacturing.

Bayesian model calibration for block copolymer self-assembly: Likelihood-free inference and expected information gain computation via measure transport

저수분석(low flow analysis)은 수자원공학에서 중요한 분야 중 하나이며, 특히 저수량 빈도분석(low flow frequency analysis)의 결과는 저수(貯水)용량의 설계, 물 수급계획, 오염원의 배치 및 관개와 생태계의 보존을 위한 수량과 수질의 관리에 중요하게 사용된다. 그러므로 본 연구에서는 저수량 빈도분석을 위한 점 빈도분석을 수행하였으며, 특히 빈도분석에 있어서의 불확실성을 탐색하기 위하여 Bayesian 방법을 적용하고 그 결과를 기존에 사용되던 불확실성 탐색방법과 비교하였다. 본 논문의Ⅰ편에서는 Bayesian 방법 중 사전분포(prior distribution)와 우도함수(likelihood function)의 복잡성에 상관없이 계산이 가능한 Bayesian MCMC(Bayesian Markov Chain Monte Carlo) 방법과 Metropolis-Hastings 알고리즘을 사용하기 위한 여러 과정의 이론적 배경과 Bayesian 방법에서 가장 중요한 요소인 사전분포를 구축하고 이를 비교 및 평가하였다. 고려된 사전분포는 자료에 기반하지 않은 사전분포와 자료에 기반한 사전분포로써 두 사전분포를 이용하여 Metropolis-Hastings 알고리즘을 수행하고 그 결과를 비교하여 저수량 빈도분석에 합리적인 사전분포를 선정하였다. 또한 알고리즘의 수행과정에서 필요한 제안분포(proposal distribution)를 적용하여 그에 따른 알고리즘의 효율성을 채택률(acceptance rate)을 산정하여 검증해 보았다. 사전분포의 분석 결과, 자료에 기반한 사전분포가 자료에 기반하지 않은 사전분포보다 정확성 및 불확실성의 표현에 있어서 우수한 결과를 제시하는 것을 확인할 수 있었고, 채택률을 이용한 알고리즘의 효용성 역시 기존 연구자들이 제시하였던 만족스러운 범위를 가지는 것을 알 수 있었다. 최종적으로 선정된 사전분포는 본 연구의 II편에서 Bayesian MCMC방법의 사전분포로 이용되었으며, 그 결과를 기존 불확실성의 추정방법의 하나인 2차 근사식을 이용한 최우추정(maximum likelihood estimation)방법의 결과와 비교하였다. The low flow analysis is an important part in water resources engineering. Also, the results of low flow frequency analysis can be used for design of reservoir storage, water supply planning and design, waste-load allocation, and maintenance of quantity and quality of water for irrigation and wild life conservation. Especially, for identification of the uncertainty in frequency analysis, the Bayesian approach is applied and compared with conventional methodologies in at-site low flow frequency analysis. In the first manuscript, the theoretical background for the Bayesian MCMC (Bayesian Markov Chain Monte Carlo) method and Metropolis-Hasting algorithm are studied. Two types of the prior distribution, a non-data- based and a data-based prior distributions are developed and compared to perform the Bayesian MCMC method. It can be suggested that the results of a data-based prior distribution is more effective than those of a non-data-based prior distribution. The acceptance rate of the algorithm is computed to assess the effectiveness of the developed algorithm. In the second manuscript, the Bayesian MCMC method using a data-based prior distribution and MLE(Maximum Likelihood Estimation) using a quadratic approximation are performed for the at-site low flow frequency analysis.

Spiking neural networks (SNNs) have been proposed both as models of cortical computation and as candidates for solving problems in machine learning. While increasing recent works have improved their performances in benchmark discriminative tasks, most of them learn by surrogates of backpropagation where biological features such as spikes are regarded more as defects than merits. In this thesis, we explore the enerative abilities of SNNs with built-in biological mechanisms. When sampling from high-dimensional multimodal distributions, models based on general Markov chain Monte Carlo methods often have the mixing problem that the sampler is easy to get trapped in local minima. Inspired from traditional annealing or tempering approaches, we demonstrate that increasing the rate of background Poisson noise in an SNN can flatten the energy landscape and facilitate mixing of the system. In addition, we show that with synaptic short-term plasticity (STP) the SNN can achieve more efficient mixing by local modulation of active attractors and eventually outperforming traditional benchmark models. We reveal diverse sampling statistics of SNNs induced by STP and finally study its implementation on conventional machine learning methods. Our work thereby highlights important computational consequences of biological features that might otherwise appear as artifacts of evolution.