Model selection in cosmology

Bayesian inference is usually divided into two categories: parameter estimation and model selection. Parameter estimation is mostly performed using Markov Chain Monte Carlo (MCMC) sampling methods, most often based on the standard Metropolis-Hastings algorithm or its variants, such as slice, Gibbs' and Hamiltonian sampling. These methods can be highly inefficient in exploring multi-modal distributions or ones exhibiting degeneracies between parameters. Moreover, in order to perform Bayesian model selection, estimation of Bayesian evidence is needed which is by definition, a (possibly high dimensional) integration over the entire parameter space. Unfortunately, the computational expense involved in the evaluation of Bayesian evidence is typically an order of magnitude higher than doing the parameter estimation. In this paper, we discuss some techniques, built around the nested sampling framework, that have been developed to efficiently calculate the Bayesian evidence. We also discuss applications of Bayesian evidence in astrophysics and particle physics, in problems involving model selection, source detection, consistency checks between different data-sets, multi-model inference and in determining the constraining power of different observables.

<p>Dispersion curve inversion is a key step of analyzing data from multichannel surface wave method (MASW) for investigating the shear wave velocity-depth (<em>v<sub>s</sub></em>-<em>h</em>) profile. The profile is usually simplified to be a stratification model consisting of horizontal and homogenous layers. Model parameters include the number (<em>N</em>) and thicknesses (<strong><em>h</em></strong>) of layers and shear wave velocity (<em>v<sub>s</sub></em>) in each layer. The <em>N</em> represents model complexity. The larger <em>N</em> value is, the more complex the model is. A model that is too complex is prone to overfitting. The opposite is true for too simple models that underfit. Mathematically, the dispersion curve inversion problem is ill-posed, that is, there are a number of stratification models of the <em>v<sub>s</sub></em>-<em>h</em> profile with different <em>N</em>, <strong><em>h</em></strong>, and <em>v<sub>s</sub></em> values resulting in identical, or at least, similar dispersion curves. Because the <em>N</em> value is usually unknown during the inversion, the model selection is necessary in which a model fitting well with data and proper model complexity is identified in a pool of competitive models.</p><p>Nonetheless, research is rare that addresses the model selection issue in dispersion curve inversion problems. Bayesian framework can be used in model selection by quantifying the uncertainty in the stratification model. In this study, critical issues of Bayesian frameworks for quantifying the uncertainty in stratification model selection are discussed, including Bayesian inversion with a variable or a fixed number of layers in stratification models. Then, the major difficulties in computing the posterior distribution and Bayesian model evidence for determining <em>N</em> are demonstrated and discussed, which are mainly caused by the high-dimensional and highly nonlinear likelihood function. Finally, the sensitivity of model complexity to the error associated with the dispersion curve is explored preliminarily.</p>

A biologically plausible model of myelin plasticity demonstrates how oligodendrocytes can selectively adjust conduction delays in axon bundles to allow robust communication between brain regions through long-range synchronization.

The dynamic nature of the ocean environment presents numerous challenges to parameter estimation in ocean acoustics. We present a variational Bayesian method which selects an optimal model for parameter estimation in ocean acoustics using Gaussian process (GP) regression. GP model selection is based on Bayesian model selection, which treats the parameters and kernel selection of the model as hyperparameters and uses Bayesian inference to compute a model’s likelihood with respect to its hyperparameters. However, in Bayesian model selection, the integrals over parameter space are often intractable and require approximation or sampling methods such as Markov chain Monte Carlo. GP renders these integrals tractable by treating hyperparameters as multivariate Gaussian distributions, which greatly speeds up the optimization. Furthermore, model selection using GPs allows for straightforward evaluation of uncertainty in the selected model’s likelihood. In this study, we compare results of acoustic parameter estimation from Bayesian and GP model selection using an ocean acoustic propagation model.

We apply Bayesian statistics to perform model selection on different reionisation scenarios via the Multinest algorithm. Initially, we recover the results shown by 21CMMC for the parameter estimation of 21cmFAST models. We proceed to test several toy models of the Epoch of Reionisation (EoR) defined in contrasting morphology and scale. We find that LOFAR observations are unlikely to allow model selection even with long integration times. HERA would require 61 dipoles to perform the same analysis in 1080 hours, and becomes comparable to the SKA with 217 dipoles. We find the SKA requires only 324 hours of observation to conclusively distinguish between our models. Once model selection is achievable, an analysis of observational priors is performed finding that neutral fraction checks at specific redshifts add little to no inference. We show the difficulties in model selection at the level of distinguishing fiducial parameters within a model or distinguishing galaxies with a constant versus power law mass-to-light ratio. Finally, we explore the use of the Savage-Dickey density ratio to show the redundancy of the parameter Rmfp within 21cmFAST.

We use a newly released version of the SuperBayeS code to analyze the impact of the choice of priors and the influence of various constraints on the statistical conclusions for the preferred values of the parameters of the Constrained MSSM. We assess the effect in a Bayesian framework and compare it with an alternative likelihood-based measure of a profile likelihood. We employ a new scanning algorithm (MultiNest) which increases the computational efficiency by a factor ~200 with respect to previously used techniques. We demonstrate that the currently available data are not yet sufficiently constraining to allow one to determine the preferred values of CMSSM parameters in a way that is completely independent of the choice of priors and statistical measures. While b->s gamma generally favors large m_0, this is in some contrast with the preference for low values of m_0 and m_1/2 that is almost entirely a consequence of a combination of prior effects and a single constraint coming from the anomalous magnetic moment of the muon, which remains somewhat controversial. Using an information-theoretical measure, we find that the cosmological dark matter abundance determination provides at least 80% of the total constraining power of all available observables. Despite the remaining uncertainties, prospects for direct detection in the CMSSM remain excellent, with the spin-independent neutralino-proton cross section almost guaranteed above sigma_SI ~ 10^{-10} pb, independently of the choice of priors or statistics. Likewise, gluino and lightest Higgs discovery at the LHC remain highly encouraging. While in this work we have used the CMSSM as particle physics model, our formalism and scanning technique can be readily applied to a wider class of models with several free parameters.

Assessing the identifiability of model selection frameworks for the prediction of patient outcomes in the clinical breast cancer setting

Feature selection, reproducibility, and model selection are of fundamental importance in contemporary statistics. Feature selection methods are required in a wide range of applications in order to evaluate the significance of covariates. Meanwhile, reproducibility of selected features is needed to claim that findings are meaningful and interpretable. Finally, model selection is employed for pinpointing the best set of covariates among a sequence of candidate models produced by feature selection methods. ❧ We show that p-values, a common tool for feature selection, behave differently in nonlinear models and p-values in nonlinear models can break down earlier than their linear counterparts. Next, we provide important theoretical foundations of model-X knockoffs which is a recent state-of-the-art method for reproducibility. We establish the power and robustness results for model-X knockoffs. Finally, we tackle large-scale model selection problem for misspecified models. We propose a novel information criterion which is tailored for both model misspecification and high dimensionality.

기존의 frequentist 추론에 비해 Bayesian 추론에서의 가설 검정 및 모형 선택 문제는 학자들 간에 일치된 견해를 보이지 못하고 있으며 아직도 논란이 되는 것들이 많다. Bayesian 추론에서 가설 검정 및 모형 선택의 기준으로 널리 쓰이는 Bayes factor는 이해하기 쉬우나 여러 경우에 구하기 어려운 단점이 존재한다. 그 외에 다른 기준으로 Spiegelhalter 등 (2002)가 제시한 DIC(Deviance Information Criterion)과 frequentist 추론에서의 P-value에 대비되는 Bayesian P-value가 있다. 본 논문에서는 Swiss banknote 자료를 Bayesian 로지스틱 회귀모형으로 분석하고 관련 기준들을 구하여 각 기준들이 일관성 있는 결론을 보이는지 확인하고자 한다. Model selection and hypothesis testing problems in Bayesian inference are still debated between scholars. Bayesian factors traditionally used as a criterion in Bayesian hypothesis testing and model selection, are easy to understand but sometimes hard to compute. In addition, there are other model selection criterions such as DIC(Deviance Information Criterion) by Spiegelhalter et al. (2002) and Bayesian P-values for testing. In this paper, we briefly introduce the Bayesian hypothesis testing and model selection procedure. In addition we have applied a Bayesian inference to Swiss banknote data by a fitting logistic regression model and computing several test statistics to see if they provide consistent results.

Bayesian model selection (BMS) and Bayesian model justifiability analysis (BMJ) provide a statistically rigorous framework for comparing competing models through the use of Bayesian model evidence (BME). However, a BME‐based analysis has two main limitations: (a) it does not account for a model's posterior predictive performance after using the data for calibration and (b) it leads to biased results when comparing models that use different subsets of the observations for calibration. To address these limitations, we propose augmenting BMS and BMJ analyses with additional information‐theoretic measures: expected log‐predictive density (ELPD), relative entropy (RE) and information entropy (IE). Exploring the connection between Bayesian inference and information theory, we explicitly link BME and ELPD together with RE and IE to highlight the information flow in BMS and BMJ analyses. We show how to compute and interpret these scores alongside BME, and apply the framework to a controlled 2D groundwater setup featuring five models, one of which uses a subset of the data for calibration. Our results show how the information‐theoretic scores complement BME by providing a more complete picture concerning the Bayesian updating process. Additionally, we demonstrate how both RE and IE can be used to objectively compare models that feature different data sets for calibration. Overall, the introduced Bayesian information‐theoretic framework can lead to a better‐informed decision by incorporating a model's post‐calibration predictive performance, by allowing to work with different subsets of the data and by considering the usefulness of the data in the Bayesian updating process.

Model selection refers to a data-based choice among competing statistical models, for example choosing between a linear or a quadratic regression function. The most popular model selection techniques are based on interpretations of p-values, using a scale originally suggested by Fisher: .05 is moderate evidence against the smaller model, .01 is strong evidence, etc. Recent Bayesian literature, going back to work by Jeffreys, suggests a quite different answer to the model selection problem. Jeffreys provided an interpretive scale for Bayes factors, a scale which can be implemented in practice by use of the BIC (Bayesian Information Criterion.) The Jeffreys scale often produces much more conservative results, especially in large samples, so for instance a .01 p-value may correspond to barely any evidence at all against the smaller model. This paper tries to reconcile the two theories by giving an interpretation of Fisher's scale in terms of Bayes factors. A general interpretation is given which works fine when checked for the onedimensional Gaussian problem, where standard hypothesis testing is seen to coincide with a Bayesian analysis that assumes stronger (more informative) priors than those used by the BIC. This argument fails in higher dimensions, where Fisher's scale must be made more conservative in order to get a proper Bayes justification.

SummaryThe classical approach to statistical analysis is usually based upon finding values for model parameters that maximize the likelihood function. Model choice in this context is often also based on the likelihood function, but with the addition of a penalty term for the number of parameters. Though models may be compared pairwise by using likelihood ratio tests for example, various criteria such as the Akaike information criterion have been proposed as alternatives when multiple models need to be compared. In practical terms, the classical approach to model selection usually involves maximizing the likelihood function associated with each competing model and then calculating the corresponding criteria value(s). However, when large numbers of models are possible, this quickly becomes infeasible unless a method that simultaneously maximizes over both parameter and model space is available. We propose an extension to the traditional simulated annealing algorithm that allows for moves that not only change parameter values but also move between competing models. This transdimensional simulated annealing algorithm can therefore be used to locate models and parameters that minimize criteria such as the Akaike information criterion, but within a single algorithm, removing the need for large numbers of simulations to be run. We discuss the implementation of the transdimensional simulated annealing algorithm and use simulation studies to examine its performance in realistically complex modelling situations. We illustrate our ideas with a pedagogic example based on the analysis of an autoregressive time series and two more detailed examples: one on variable selection for logistic regression and the other on model selection for the analysis of integrated recapture–recovery data.

Event Abstract Back to Event Statistical uncertainty and sensitivity analysis of intracellular signaling models - through approximate Bayesian computation and variance based global sensitivity analysis Olivia Eriksson1, 2*, Alexandra Jauhiainen1, 3, Sara Maad Sasane4, Anu G Nair5, Carolina Sartorius4 and Jeanette Hellgren Kotaleski2, 5, 6 1 Contributed equally to this work, Sweden 2 Stockholm University, Department of Numerical Analysis and Computer Science, Sweden 3 AstraZeneca AB R&D, Early Clinical Biometrics, Sweden 4 Lund University, Centre for Mathematical Sciences, Sweden 5 KTH Royal Institute of Technology, School of Computer Science and Communication, Sweden 6 Karolinska Institute, Department of Neuroscience, Sweden Computational dynamical models describing intracellular signaling pathways of nerve cells are increasing in size and complexity as more information is obtained from experiments. Models describing protein interactions within neurons can contain over hundred protein reactions [1, 2]. These models are built from knowledge about the interaction topology, inferred from e.g. gene knock-out experiments, as well as from experimental quantitative data describing the input-output relationship of the observed system. The quantitative data are often sparse as compared to the size of the system, and translating the experimental information into dynamical models often results in large uncertainties in the parameter (e.g reaction rates) values. This uncertainty is partly due to the sparseness in experimental time series data (practical unidentifiability), but it is also an intrinsic feature of the modeled system (structural unidentifiability) and could reflect the possibility for biological variation [3]; in many biological systems the same function can be achieved in partly different ways due to degeneracy and redundancy. Here we investigate the extent of this parameter uncertainty, in a model describing calcium (Ca)-dependent activation of Calmodulin (CaM), Protein phosphatase 2B (PP2B) and Ca/CaM-dependent protein kinase II (CamKII) [4], which is an intracellular pathway of importance to synaptic plasticity. We presume that the structure of the model is correct and then characterize the uncertainty in the parameters based on data. This is done by sampling the subspace of parameter values which has an equally good fit to data (corresponding to the “viable space” in [5]), through a statistical approach known as Approximate Bayesian Computation (ABC) with Markov Chain Monte Carlo (MCMC) [6]. ABC has been proposed for parameter estimation in system biology models previously, albeit in slightly different settings, e.g. [7, 8], and can be applied to a wide range of problems. In order to characterize the viable space efficiently with several experimental datasets, we employ multivariate probability distributions called copulas as a part of the ABC sampling. The result is a multivariate posterior distribution for the parameters, which represents how much they are constrained by the data (i.e. the uncertainty) as illustrated in Figure 1 (pairwise plots of viable space for four parameters indicating constraints and dependencies before and after calibration to data). The next step is to look into how the parameter uncertainty is translated to uncertainty of the predictions made from the model. We also investigate the role and importance of different parameters, with respect to different model outputs including the predictions, by means of global sensitivity analysis [9]. Figure 2 illustrates the sensitivity of one of the outputs calculated by a method based on the decomposition of the variance [10, 11, 12], showing different types of sensitivity measures. Finally, we explore how the sensitivity of different subsets of parameters change as different parts of the viable space are investigated, which is shown in Figure 3, where sensitivity profiles are clustered into groups with different sensitivity characteristics. Figure 1 Figure 2 Figure 3 Acknowledgements This research was supported by funding from the Swedish e-Science Research Centre and the Human Brain Project References [1] Gutierrez-Arenas O, Eriksson O, Hellgren Kotaleski J (2014) Segregation and Crosstalk of D1 Receptor-Mediated Activation of ERK in Striatal Medium Spiny Neurons upon Acute Administration of Psychostimulants. PLoS Comput Biol 10, e1003445. [2] Nair AG, Gutierrez-Arenas O, Eriksson O, Vincent P, and Hellgren Kotaleski J (2015) Sensing Positive versus Negative Reward Signals through Adenylyl Cyclase-Coupled GPCRs in Direct and Indirect Pathway Striatal Medium Spiny Neurons. J. Neuroscience 35, 14017-14030. [3] Marder E, Goeritz ML, Otopalik AG (2015) Robust circuit rhythms in small circuits arise from variable circuit components and mechanisms. Curr Opin Neurobiol 31, 156-163. [4] Nair AG, Gutierrez-Arenas O, Eriksson O, Jauhiainen A, Blackwell KT and Kotaleski, JH (2014). Modeling intracellular signaling underlying striatal function in health and disease. Progress in molecular biology and translational science, 123, 277 [5] Zamora-Sillero E, Hafner M, Ibig A, Stelling J, and Wagner, A. (2011). Efficient characterization of high-dimensional parameter spaces for systems biology. BMC systems biology 5, 142. [6] Marjoram P, Molitor J, Plagnol V, and Tavare S (2003) Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. U.S.A. 100, 15324–15328. [7] Toni T, Welch D, Strelkowa N, Ipsen A, and Stumpf MP. Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. J R Soc Interface, 6, 187–202. [8] Liepe J, Kirk P, Filippi S, Toni T, Barnes CP, and Stumpf MP. A framework for parameter estimation and model selection from experimental data in systems biology using approximate Bayesian computation. Nat Protoc, 9, 439–456. [9] Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S Global sensitivity analysis: the primer. In Wiley 2008 New York, NY:Wiley. [10] Sobol IM (2001) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates, Math. Comput. Simul. 55, 271–280. [11] Saltelli A (2002) Making best use of model evaluations to compute sensitivity indices. Comp Phys Comm 145, 280 – 297. [12] Halnes G, Ulfhielm E Ljunggren EE, Kotaleski JH and Rospars JP (2009) Modelling and sensitivity analysis of the reactions involving receptor, G-protein and effector in vertebrate olfactory receptor neurons. Journal of computational neuroscience 27, 471-491. Keywords: sensitivity analysis, Approximate Bayesian Computation, copulas, intracellular signalling, Dynamical Modeling, global sensitivity analysis, Markov chain Monte Carlo, variance decomposition Conference: Neuroinformatics 2016, Reading, United Kingdom, 3 Sep - 4 Sep, 2016. Presentation Type: Investigator presentations Topic: Computational neuroscience Citation: Eriksson O, Jauhiainen A, Maad Sasane S, Nair A, Sartorius C and Hellgren Kotaleski J (2016). Statistical uncertainty and sensitivity analysis of intracellular signaling models - through approximate Bayesian computation and variance based global sensitivity analysis. Front. Neuroinform. Conference Abstract: Neuroinformatics 2016. doi: 10.3389/conf.fninf.2016.20.00014 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 31 May 2016; Published Online: 18 Jul 2016. * Correspondence: PhD. Olivia Eriksson, Contributed equally to this work, Stockholm, Sweden, olivia@kth.se Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Olivia Eriksson Alexandra Jauhiainen Sara Maad Sasane Anu G Nair Carolina Sartorius Jeanette Hellgren Kotaleski Google Olivia Eriksson Alexandra Jauhiainen Sara Maad Sasane Anu G Nair Carolina Sartorius Jeanette Hellgren Kotaleski Google Scholar Olivia Eriksson Alexandra Jauhiainen Sara Maad Sasane Anu G Nair Carolina Sartorius Jeanette Hellgren Kotaleski PubMed Olivia Eriksson Alexandra Jauhiainen Sara Maad Sasane Anu G Nair Carolina Sartorius Jeanette Hellgren Kotaleski Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

This paper proposes a novel direct policy search (DPS) method with model selection for partially observed Markov decision processes (POMDPs). DPSs have been standard for learning POMDPs due to their computational efficiency and natural ability to maximize total rewards. An important open challenge for the best use of DPS methods is model selection, i.e., determination of the proper dimensionality of hidden states and complexity of policy functions, to mitigate overfitting in highly-flexible model representations of POMDPs. This paper bridges Bayesian inference and reward maximization and derives marginalized weighted log-likelihood~(MWL) for POMDPs which takes both advantages of Bayesian model selection and DPS. Then we propose factorized asymptotic Bayesian policy search (FABPS) to explore the model and the policy which maximizes MWL by expanding recently-developed factorized asymptotic Bayesian inference. Experimental results show that FABPS outperforms state-of-the-art model selection methods for POMDPs, with respect both to model selection and to expected total rewards.

Equality and inequality constrained multivariate linear models: Objective model selection using constrained posterior priors