Reliability analysis using bootstrap information criterion for small sample size response functions

Only unstructured single-path model selection techniques, i.e., Information Criteria, are used by Bounds test of cointegration for model selection. The aim of this paper was twofold; one was to evaluate the performance of these five routinely used information criteria {Akaike Information Criterion (AIC), Akaike Information Criterion Corrected (AICC), Schwarz/Bayesian Information Criterion (SIC/BIC), Schwarz/Bayesian Information Criterion Corrected (SICC/BICC), and Hannan and Quinn Information Criterion (HQC)} and three structured approaches (Forward Selection, Backward Elimination, and Stepwise) by assessing their size and power properties at different sample sizes based on Monte Carlo simulations, and second was the assessment of the same based on real economic data. The second aim was achieved by the evaluation of the long-run relationship between three pairs of macroeconomic variables, i.e., Energy Consumption and GDP, Oil Price and GDP, and Broad Money and GDP for BRICS (Brazil, Russia, India, China and South Africa) countries using Bounds cointegration test. It was found that information criteria and structured procedures have the same powers for a sample size of 50 or greater. However, BICC and Stepwise are better at small sample sizes. In the light of simulation and real data results, a modified Bounds test with Stepwise model selection procedure may be used as it is strongly theoretically supported and avoids noise in the model selection process.

There exist quite a few criteria for penalty-based model selection. Although they have various justifications for large sample problems, their performance under small or moderate sample size is unclear which hinders the development of model combination methods using the appropriate penalty term. In this paper, we assess the performance of seven model selection criteria based on linear regression models with unknown noise variance. We set the true data generation mechanism to be within the model set as well as outside the model set. In the latter case, soft model selection through multiple model fusion is proposed and its difference from Bayesian model averaging is highlighted. The penalty term used in each model selection criterion provides a natural link to estimate the model probability without assuming any prior knowledge of the unknown parameter. An important question is whether the estimated model probabilities are consistent when multiple models are fused for prediction or interpolation. We argue that strong consistency only holds under large sample regime while soft model selection can still be better than choosing a single model with small sample size. Our numerical results using different model selection criteria for polynomial fitting indicate that the conditional model estimator (CME) has the best performance in selecting the correct model order and fusing multiple models for prediction and interpolation. The minimum description length (MDL) based criteria are next to CME and outperform Bayesian information criterion (BIC) and Akaike information criterion (AIC) significantly.

Latent profile analysis with nonnormal mixtures: A Monte Carlo examination of model selection using fit indices

This study presents a comparative analysis of the Inverse Weibull and Inverse Lognormal distributions using both simulated and real-world data with Maximum Likelihood Estimation (MLE).Model selection criteria included Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), Anderson-Darling (AD), and Kolmogorov-Smirnov (KS) tests.Six simulated sample sizes (50, 100, 150, 200, 250, and 500) were used, with 1000 replications each.Results showed the Inverse Lognormal distribution consistently had lower AIC and BIC values at small to moderate sample sizes.Furthermore, real-world stock price data (sample size 100) from the Nigerian Stock Exchange was analyzed.Descriptive statistics and goodness-of-fit tests favored the Inverse Lognormal model.These findings support the utility of AIC, BIC, AD, and KS in lifetime data modeling and model selection.

This article discusses the use of association models to detect group differences and the choice of model selection criteria under different conditions. The performances of several commonly used model selection criteria in log-linear modeling are examined using Monte Carlo simulations. The results suggest that no single criterion can play the role of panacea in model selection. L2/ df, normed fit index (NFI) (or 1 - L12/L02), and Akaike's information criterion (AIC) are systematically biased toward models incorporating group differences. The log-likelihood ratio test (LRT), in conjunction with the nested chi-square difference test, is useful for samples with moderate sizes. Bayesian information criterion (BIC) is most reliable among the measures tested. It consistently provides correct information about group differences in association, especially when the sample size is large. Although BIC may give equivocal results under certain conditions, the ambiguities can be mitigated by an inclusion of competing models and a cautious interpretation of any small improvement in BIC.

<title>Abstract</title> Model selection is critical in spatial econometrics, particularly when specifying the degree of spatial dependence in regression models. Traditional criteria such as Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), and the Hannan-Quinn Criterion (HQC) are widely applied but do not adaptively account for the complexity introduced by spatial autocorrelation. This paper evaluates the performance of the Renormalized Maximum Likelihood (RNML) criterion, which incorporates a data-driven penalty derived from the Fisher Information Matrix to balance model fit and complexity. Through Monte Carlo simulations, we assess RNML’s capacity to accurately recover the true spatial dependence structure under varying degrees of spatial autocorrelation and sample sizes. We complement these simulations with three empirical applications: the spatial distribution of GDP per capita across European NUTS-2 regions, the spread of COVID-19 incidence across Italian provinces, and the share of foreign-born residents across Germany's NUTS-3 regions. These examples span diverse spatial scales and policy domains, economic performance, public health, and migration, providing a comprehensive evaluation of RNML in practice. Our findings consistently show that RNML tends to select higher spatial autoregressive coefficients compared to classical criteria, especially in settings with pronounced spatial dependence or higher spatial resolution. This adaptiveness enhances RNML’s reliability for robust inference in spatial modeling, particularly where spatial spillovers and feedback effects are substantively important.

The analysis of ecological data has changed in two important ways over the last 15years. The development and easy availability of Bayesian computational methods has allowed and encouraged the fitting of complex hierarchical models. At the same time, there has been increasing emphasis on acknowledging and accounting for model uncertainty. Unfortunately, the ability to fit complex models has outstripped the development of tools for model selection and model evaluation: familiar model selection tools such as Akaike's information criterion and the deviance information criterion are widely known to be inadequate for hierarchical models. In addition, little attention has been paid to the evaluation of model adequacy in context of hierarchical modeling, i.e., to the evaluation of fit for a single model. In this paper, we describe Bayesian cross-validation, which provides tools for model selection and evaluation. We describe the Bayesian predictive information criterion and a Bayesian approximation to the BPIC known as the Watanabe-Akaike information criterion. We illustrate the use of these tools for model selection, and the use of Bayesian cross-validation as a tool for model evaluation, using three large data sets from the North American Breeding Bird Survey.

Information criteria are used in many applications including statistical model selection and intelligent systems. The traditional information criteria such as the Akaike information criterion (AIC) do not always provide an adequate penalty on the number of model covariates. To address this issue, we propose a novel method for evaluating statistical models based on information criterion. The proposed method, called regularized information criterion (RIL), modifies the penalty term in AIC to reduce model overfitting. The results of numerical experiments show that RIL provides a better reflection of model predictive error than AIC. Thus, RIL can be a useful tool in model selection.

Considerations of sample size in medical research

The rapid development of modeling techniques has brought many opportunities for data‐driven discovery and prediction. However, this also leads to the challenge of selecting the most appropriate model for any particular data task. Information criteria, such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC), have been developed as a general class of model selection methods with profound connections with foundational thoughts in statistics and information theory. Many perspectives and theoretical justifications have been developed to understand when and how to use information criteria, which often depend on particular data circumstances. This review article will revisit information criteria by summarizing their key concepts, evaluation metrics, fundamental properties, interconnections, recent advancements, and common misconceptions to enrich the understanding of model selection in general.This article is categorized under:Data: Types and Structure &gt; Traditional Statistical DataStatistical Learning and Exploratory Methods of the Data Sciences &gt; Modeling MethodsStatistical and Graphical Methods of Data Analysis &gt; Information Theoretic MethodsStatistical Models &gt; Model Selection

The detection of multiple outliers can be interpreted as a model selection problem. Models that can be selected are the null model, which indicates an outlier free set of observations, or a class of alternative models, which contain a set of additional bias parameters. A common way to select the right model is by using a statistical hypothesis test. In geodesy data snooping is most popular. Another approach arises from information theory. Here, the Akaike information criterion (AIC) is used to select an appropriate model for a given set of observations. The AIC is based on the Kullback-Leibler divergence, which describes the discrepancy between the model candidates. Both approaches are discussed and applied to test problems: the fitting of a straight line and a geodetic network. Some relationships between data snooping and information criteria are discussed. When compared, it turns out that the information criteria approach is more simple and elegant. Along with AIC there are many alternative information criteria for selecting different outliers, and it is not clear which one is optimal.

A jackknife type approach to statistical model selection

This paper introduces a SAS/IML program to select among the multivariate model candidates based on a few well-known multivariate model selection criteria. Stepwise regression and all-possible-regression are considered. The program is user friendly and requires the user to paste or read the data at the beginning of the module, include the names of the dependent and independent variables (the y's and the x's), and then run the module. The program produces the multivariate candidate models based on the following criteria: Forward Selection, Forward Stepwise Regression, Backward Elimination, Mean Square Error, Coefficient of Multiple Determination, Adjusted Coefficient of Multiple Determination, Akaike's Information Criterion, the Corrected Form of Akaike's Information Criterion, Hannan and Quinn Information Criterion, the Corrected Form of Hannan and Quinn (HQc) Information Criterion, Schwarz's Criterion, and Mallow's PC. The output also constitutes detailed as well as summarized results.

Capture-recapture methods estimate the size of hidden populations by leveraging the proportion of overlap of the population on independent lists. Log-linear modeling relaxes the assumption of list independence, but best model selection criteria remain uncertain. Incorrect model selection can deliver incorrect and even implausible size estimates. We used simulations to model when capture-recapture methods deliver biased or unbiased estimates and compare model selection criteria. Simulations included five scenarios for list dependence among three incomplete lists of population of interest. We compared metrics of log-linear model selection, accuracy, and precision. We also compared log-linear model performance to the decomposable graph approach (a Bayesian model average), the sparse multiple systems estimation (SparseMSE) approach that accounts for zero or low cell counts, and the Sample Coverage approach. Log-linear models selected by Akaike's information criterion (AIC) calculated accurate population size estimates in all scenarios except for those with sparse or zero cell counts. In these scenarios, the decomposable graph and the Sample Coverage models produced more accurate size estimates. Conventional capture-recapture model selection fails with sparse cell counts. This naïve approach to model selection should be replaced with the implementation of multiple different models in order triangulate the truth in real-world applications.

The weighted average information criterion for multivariate regression model selection