Chapter 13 - Bayesian Learning: Approximate Inference and Nonparametric Models

In applied Bayesian inference scenarios, users may have access to a large number of pre-existing model evaluations, for example from maximum-a-posteriori (MAP) optimization runs. However, traditional approximate inference techniques make little to no use of this available information. We propose the framework of post-process Bayesian inference as a means to obtain a quick posterior approximation from existing target density evaluations, with no further model calls. Within this framework, we introduce Variational Sparse Bayesian Quadrature (vsbq), a method for post-process approximate inference for models with black-box and potentially noisy likelihoods. vsbq reuses existing target density evaluations to build a sparse Gaussian process (GP) surrogate model of the log posterior density function. Subsequently, we leverage sparse-GP Bayesian quadrature combined with variational inference to achieve fast approximate posterior inference over the surrogate. We validate our method on challenging synthetic scenarios and real-world applications from computational neuroscience. The experiments show that vsbq builds high-quality posterior approximations by post-processing existing optimization traces, with no further model evaluations.

Deep Gaussian processes (DGPs) are the natural extension of Gaussian processes (GPs) to a multi-layer architecture. DGPs are powerful probabilistic models that have shown better results than standard GPs in terms of generalization performance and prediction uncertainty estimation. Nevertheless, exact inference in DGPs is intractable, making these models hard to train. For this task, current approaches in the literature use approximate inference techniques such as variational inference or approximate expectation propagation. In this work, we present a new method for inference in DGPs using an approximate inference technique based on Monte Carlo methods and the expectation propagation algorithm. Our experiments show that our method scales well to large datasets and that its performance is comparable or better than other state of the art methods. Furthermore, our training method leads to interesting properties in the predictive distribution of the DGP. More precisely, it is able to capture output noise that is dependent on the input and it can generate multimodal predictive distributions. These two properties, which are not shared by other state-of-the-art approximate inference methods for DGPs, are analyzed in detail in our experiments.

There have been few investigations of cancer prognosis models based on Bayesian hierarchical models. In this study, we used a novel Bayesian method to screen mRNAs and estimate the effects of mRNAs on the prognosis of patients with lung adenocarcinoma. Based on the identified mRNAs, we can build a prognostic model combining mRNAs and clinical features, allowing us to explore new molecules with the potential to predict the prognosis of lung adenocarcinoma. The mRNA data (n = 594) and clinical data (n = 470) for lung adenocarcinoma were obtained from the TCGA database. Gene set enrichment analysis (GSEA), univariate Cox proportional hazards regression, and the Bayesian hierarchical Cox proportional hazards model were used to explore the mRNAs related to the prognosis of lung adenocarcinoma. Multivariate Cox proportional hazard regression was used to identify independent markers. The prediction performance of the prognostic model was evaluated not only by the internal cross-validation but also by the external validation based on the GEO dataset (n = 437). With the Bayesian hierarchical Cox proportional hazards model, a 14-gene signature that included CPS1, CTPS2, DARS2, IGFBP3, MCM5, MCM7, NME4, NT5E, PLK1, POLR3G, PTTG1, SERPINB5, TXNRD1, and TYMS was established to predict overall survival in lung adenocarcinoma. Multivariate analysis demonstrated that the 14-gene signature (HR 3.960, 95% CI 2.710–5.786), T classification (T1, reference; T3, HR 1.925, 95% CI 1.104–3.355) and N classification (N0, reference; N1, HR 2.212, 95% CI 1.520–3.220; N2, HR 2.260, 95% CI 1.499–3.409) were independent predictors. The C-index of the model was 0.733 and 0.735, respectively, after performing cross-validation and external validation, a nomogram was provided for better prediction in clinical application. Bayesian hierarchical Cox proportional hazards models can be used to integrate high-dimensional omics information into a prediction model for lung adenocarcinoma to improve the prognostic prediction and discover potential targets. This approach may be a powerful predictive tool for clinicians treating malignant tumours.

A Bayesian approach to model the trends and variability in urban stormwater quality associated with catchment and hydrologic parameters

In this paper, we propose an improved scene clustering algorithm based on Bayesian model. We present a Bayesian hierarchical framework model for clustering objects in multimedia scene. The Multimedia scene can switch between different shots, the unknown objects can leave or enter the scene at multiple times, and the background can be clustered. The proposed framework model consists of annotation part and Bayesian hierarchical clustering part. This algorithm has several advantages over traditional distance-based agglomerative clustering algorithms. Bayesian hierarchical hypothesis testing is used to decide which merges are advantageous and to output the recommended depth of the tree. The framework model can be interpreted as a novel fast bottom-up approximate inference method for a process mixture model. We describe procedures for clustering the model hyperparameters, computing the predictive distribution, and extensions to the framework model. Experimental results on virtual reality multimedia scene data sets demonstrate useful properties of the framework model.

BackgroundAmyotrophic Lateral Sclerosis (ALS), also known as Lou Gehrig’s disease, is a rare disease with extreme between-subject variability, especially with respect to rate of disease progression. This makes modelling a subject’s disease progression, which is measured by the ALS Functional Rating Scale (ALSFRS), very difficult. Consider the problem of predicting a subject’s ALSFRS score at 9 or 12 months after a given time-point.MethodsWe obtained ALS subject data from the Pooled Resource Open-Access ALS Clinical Trials Database, a collection of data from various ALS clinical trials. Due to the typical linearity of the ALSFRS, we consider several Bayesian hierarchical linear models. These include a mixture model (to account for the two potential classes of “fast” and “slow” ALS progressors) as well as an onset-anchored model, in which an additional artificial data-point, using time of disease onset, is utilized to improve predictive performance.ResultsThe onset-anchored model had a drastically reduced posterior predictive mean-square-error distributions, when compared to the Bayesian hierarchical linear model or the mixture model under a cross-validation approach. No covariates, other than time of disease onset, consistently improved predictive performance in either the Bayesian hierarchical linear model or the onset-anchored model.ConclusionsAugmenting patient data with an additional artificial data-point, or onset anchor, can drastically improve predictive modelling in ALS by reducing the variability of estimated parameters at the cost of a slight increase in bias. This onset-anchored model is extremely useful if predictions are desired directly after a single baseline measure (such as at the first day of a clinical trial), a feat that would be very difficult without the onset-anchor. This approach could be useful in modelling other diseases that have bounded progression scales (e.g. Parkinson’s disease, Huntington’s disease, or inclusion-body myositis). It is our hope that this model can be used by clinicians and statisticians to improve the efficacy of clinical trials and aid in finding treatments for ALS.

My dissertation is motivated by the need for improved statistical approaches for understanding differences in cancer risk in rural vs. urban areas. To accomplish this goal, we develop three statistical methods that can be used to model and explain differences in age-adjusted rates between small areas with differing characteristics. These methods allow researchers to draw meaningful inferences from sparse spatial or spatio-temporal datasets at the ZIP code or county levels. Furthermore, they can be used to identify regions that might require public health interventions and to motivate costly data collection efforts at the individual level. The statistical methods we propose in this dissertation can be classified into two categories: estimation and mediation. Chapter 2 focuses on precisely estimating age-adjusted cancer mortality rates for small areas from spatio-temporal datasets with an excessive number of zeros. A large proportion of zeros are often present in datasets involving low-prevalence diseases and in datasets that include rural regions with small population sizes. When stratifying counts for each region and year by age group in the process of estimating age-adjusted rates, the proportion of zeros in the dataset is further inflated. We propose a novel Bayesian hierarchical hurdle model with spatial and temporal random effects for estimating age-adjusted rates in these settings. Via a simulation study and two data examples, we study the performance of this model and make recommendations as to when it should be used over a Bayesian hierarchical Poisson regression model for age-adjusted rates. Chapters 3 and 4 address the goal of mediation. Specifically, Chapter 3 focuses on a method for performing a mediation analysis and Chapter 4 introduces a method for performing a decomposition analysis. Both chapters aim to explain a difference in age-adjusted rates at the ZIP code level based on either a ZIP code-level exposure (mediation) or a fixed characteristic (decomposition). We develop flexible statistical methods that use Bayesian hierarchical models with spatial random effects and a Bayesian version of the g-computation technique from causal inference. Both methods were designed to create counterfactual small area estimates of age-adjusted rates within the analysis and consequently stable estimates of the effects of interest. Through a simulation study, we illustrate a high level of precision and minimal bias in the total, direct, and indirect effects when using our proposed mediation analysis method. We also compare our proposed mediation method to two methods used in the medical literature and show that our method exhibits the best trade-off of bias and variance. To demonstrate the performance of our decomposition method, we perform a data analysis to understand whether park access explains the disparity in age-adjusted colorectal cancer incidence rates in rural vs. urban ZIP codes in Iowa. We illustrate that precise inferences can be made from sparse ZIP code-level datasets when incorporating small area estimation techniques into each analysis.

Appropriate inference for stocks or species with low-quality data (poor data) or limited data (data poor) is extremely important. Hierarchical Bayesian methods are especially applicable to small-area, small-sample-size estimation problems because they allow poor-data species to borrow strength from species with good-quality data. We used a hammerhead shark complex as an example to investigate the advantages of using hierarchical Bayesian models in assessing the status of poor-data and data-poor exploited species. The hammerhead shark complex (Sphyrna spp.) along the Atlantic and Gulf of Mexico coasts of the United States is composed of three species: the scalloped hammerhead (S. lewini), the great hammerhead (S. mokarran), and the smooth hammerhead (S. zygaena) sharks. The scalloped hammerhead comprises 70-80% of the catch and has catch and relative abundance data of good quality, whereas great and smooth hammerheads have relative abundance indices that are both limited and of low quality presumably because of low stock density and limited sampling. Four hierarchical Bayesian state-space surplus production models were developed to simulate variability in population growth rates, carrying capacity, and catchability of the species. The results from the hierarchical Bayesian models were considerably more robust than those of the nonhierarchical models. The hierarchical Bayesian approach represents an intermediate strategy between traditional models that assume different population parameters for each species and those that assume all species share identical parameters. Use of the hierarchical Bayesian approach is suggested for future hammerhead shark stock assessments and for modeling fish complexes with species-specific data, because the poor-data species can borrow strength from the species with good data, making the estimation more stable and robust.

Meta-analyses are frequently performed on adverse event data and are primarily used for improving statistical power to detect safety signals. However, in the evaluation of drug safety for New Drug Applications, simple pooling of adverse event data from multiple clinical trials is still commonly used. We sought to propose a new Bayesian hierarchical meta-analytic approach based on consideration of a hierarchical structure of reported individual adverse event data from multiple randomized clinical trials. To develop our meta-analysis model, we extended an existing three-stage Bayesian hierarchical model by including an additional stage of the clinical trial level in the hierarchical model; this generated a four-stage Bayesian hierarchical model. We applied the proposed Bayesian meta-analysis models to published adverse event data from three premarketing randomized clinical trials of tadalafil and to a simulation study motivated by the case example to evaluate the characteristics of three alternative models. Comparison of the results from the Bayesian meta-analysis model with those from Fisher's exact test after simple pooling showed that 6 out of 10 adverse events were the same within a top 10 ranking of individual adverse events with regard to association with treatment. However, more individual adverse events were detected in the Bayesian meta-analysis model than in Fisher's exact test under the body system "Musculoskeletal and connective tissue disorders." Moreover, comparison of the overall trend of estimates between the Bayesian model and the standard approach (odds ratios after simple pooling methods) revealed that the posterior median odds ratios for the Bayesian model for most adverse events shrank toward values for no association. Based on the simulation results, the Bayesian meta-analysis model could balance the false detection rate and power to a better extent than Fisher's exact test. For example, when the threshold value of the posterior probability for signal detection was set to 0.8, the false detection rate was 41% and power was 88% in the Bayesian meta-analysis model, whereas the false detection rate was 56% and power was 86% in Fisher's exact test. Adverse events under the same body system were not necessarily positively related when we used "system organ class" and "preferred term" in the Medical Dictionary for Regulatory Activities as a hierarchical structure of adverse events. For the Bayesian meta-analysis models to be effective, the validity of the hierarchical structure of adverse events and the grouping of adverse events are critical. Our proposed meta-analysis models considered trial effects to avoid confounding by trial and borrowed strength from both within and across body systems to obtain reasonable and stable estimates of an effect measure by considering a hierarchical structure of adverse events.

Approximations of loopy belief propagation, including expectation propagation and approximate message passing, have attracted considerable attention for probabilistic inference problems. This paper proposes and analyzes a generalization of Opper and Winther's expectation consistent (EC) approximate inference method. The proposed method, called Generalized Expectation Consistency (GEC), can be applied to both maximum a posteriori (MAP) and minimum mean squared error (MMSE) estimation. Here we characterize its fixed points, convergence, and performance relative to the replica prediction of optimality.

We propose a Bayesian framework for regression modeling that uses Pearson Type VII Processes (P7Ps) as an adaptable generalization of Gaussian Processes (GPs). The P7P is a scale mixture of Normal and Gamma distributions. It has flexible heavy-tailed properties that enhance its robustness against outliers and render it suitable for modelling heavytailed data. The P7P framework outperforms GP functions by providing explicit control over tail behavior and variance modulation via the shape and scale parameters. These factors dictate the weight of the tails and the overall dispersion of the process, facilitating a seamless transition between Gaussian and heavy-tailed regimes, thus improving modelling flexibility and robustness. We derive the predictive distribution for new data points and use the Laplace approximation for efficient posterior inference under the Pearson Type VII (P7) prior, which ensures scalability while maintaining analytical tractability. The Pearson Type VII Process for regression (P7PR) outperforms Gaussian Process for regression (GPR) in terms of resilience, adaptability and predictive accuracy, especially in challenging scenarios with heavy-tailed noise and outliers, as evidenced using comparative analysis based on Mean Square Error (MSE), Mean Absolute Error (MAE) and predictive log-likelihood (PLL), using both simulated and real-world datasets. These results validate the benefits of heavy-tailed priors in modelling non-Gaussian distributions and illustrate the resilience of P7PR as a Bayesian substitute for conventional GPR. Although the proposed P7PR framework exhibits robustness and strong predictive accuracy, its computational cost increases remarkably as the size of the datasets, the complexity of the model, and the posterior space expand. Consequently, in order to preserve scalability without compromising performance, practical implementations may necessitate moderate model sizes or approximate inference methods.

Generalizing expectation propagation with mixtures of exponential family distributions and an application to Bayesian logistic regression

This paper discusses about a sparsity based algorithm used for Hyperspectral Image (HSI) classification where the test pixel vectors are sparsely represented as the linear combination of a few number of training samples from a well-organised dictionary matrix. The sparse vector is obtained using Basis Pursuit (BP) which is a constrained l4 minimization problem. This problem is solved by using a simple and powerful iterative algorithm known as Alternating Direction Method of Multipliers (ADMM) which significantly reduces the computational complexity of the problem and thereby speeds up the convergence. The classification accuracy is considerably improved by including efficient preprocessing techniques to remove the unwanted information (noise) present in Hyperspectral images. This paper uses a fast and reliable denoising technique based on Legendre Fenchel Transformation (LFT) to effectively denoise each band of HSI prior to ADMM based classification (proposed method). A comparison of proposed technique with one of the convex optimization tools namely, CVX is given to exhibit the fast convergence of the former method. The experiment is performed on standard Indian Pines dataset captured using AVIRIS sensor. The potential of the proposed method is illustrated by analyzing the classification indices obtained with and without applying any preprocessing methods. With only 10% training set, an overall accuracy of 96.76% is obtained for the proposed method at a much faster rate compared to computation time taken by CVX solver.

Structured Bayesian compressive sensing exploiting dirichlet process priors

Protein X-ray crystallography -- the most popular method for determining protein structures -- remains a laborious process requiring a great deal of manual crystallographer effort to interpret low-quality protein images. Automating this process is critical in creating a high-throughput protein-structure determination pipeline. Previously, our group developed ACMI, a probabilistic framework for producing protein-structure models from electron-density maps produced via X-ray crystallography. ACMI uses a Markov Random Field to model the 3D location of each non-hydrogen atom in a protein. Calculating the best structure in this model is intractable, so ACMI uses approximate inference methods to estimate the optimal structure. While previous results have shown ACMI to be the state-of-the-art method on this task, its approximate inference algorithm remains computationally expensive and susceptible to errors. In this work, we develop Probabilistic Ensembles in ACMI (PEA), a framework for leveraging multiple, independent runs of approximate inference to produce estimates of protein structures. Our results show statistically significant improvements in the accuracy of inference resulting in more complete and accurate protein structures. In addition, PEA provides a general framework for advanced approximate inference methods in complex problem domains.