How Many Genes are Needed for a Discriminant Microarray Data Analysis

Most model selection procedures use a fixed penalty penalizing an increase in the size of a model. These nonadaptive selection procedures perform well only in one type of situation. For instance, Bayesian information criterion (BIC) with a large penalty performs well for “small” models and poorly for “large” models, and Akaike's information criterion (AIC) does just the opposite. This article proposes an adaptive model selection procedure that uses a data-adaptive complexity penalty based on a concept of generalized degrees of freedom. The proposed procedure, combining the benefit of a class of nonadaptive procedures, approximates the best performance of this class of procedures across a variety of different situations. This class includes many well-known procedures, such as AIC, BIC, Mallows's Cp, and risk inflation criterion (RIC). The proposed procedure is applied to wavelet thresholding in nonparametric regression and variable selection in least squares regression. Simulation results and an asymptotic analysis support the effectiveness of the proposed procedure.

Latent profile analysis with nonnormal mixtures: A Monte Carlo examination of model selection using fit indices

This study presents a comparative analysis of the Inverse Weibull and Inverse Lognormal distributions using both simulated and real-world data with Maximum Likelihood Estimation (MLE).Model selection criteria included Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), Anderson-Darling (AD), and Kolmogorov-Smirnov (KS) tests.Six simulated sample sizes (50, 100, 150, 200, 250, and 500) were used, with 1000 replications each.Results showed the Inverse Lognormal distribution consistently had lower AIC and BIC values at small to moderate sample sizes.Furthermore, real-world stock price data (sample size 100) from the Nigerian Stock Exchange was analyzed.Descriptive statistics and goodness-of-fit tests favored the Inverse Lognormal model.These findings support the utility of AIC, BIC, AD, and KS in lifetime data modeling and model selection.

In this paper, we are interested in investigating the causal relationships among futures sugar prices in the Zhengzhou futures exchange market (ZF), the spot sugar prices in Zhengzhou (ZS) and the futures sugar prices in New York futures exchange market (NF). A useful tool called Bayesian network is introduced to analyze the problem. Since there are only three variables in our Bayesian network, the algorithm is straightforward: we display all the 25 possible network structures and adopt certain scoring metrics to evaluate them. We applied five different scoring metrics in total. Firstly, for each metric, we obtained 24 scores, each calculated from one of the 24 possible structures i.e. a Directed Acyclic Graph (DAG). Then we eliminated the network structure which represents the independence of the three variables according to our prior knowledge concerning the futures sugar market. After that, the optimal network structure which implies the causal relationships was selected according to the corresponding scoring metric. Finally, after comparing the results from different scoring metrics, we obtained the relatively affirmative conclusion that ZS causes ZF from both the Bayesian Dirichlet (BD) metric, Bayesian Dirichlet-Akaike Information Criterion (BD-AIC) metric, Bayesian Dirichlet-Bayesian Information Criterion (BD-BIC) metric and Bayesian Information Criterion (BIC) metric. The conclusions that NF causes ZF and ZF causes ZS from the Akaike Information Criterion (AIC) metric and ZF causes ZS from the BIC metric were useful and significant to our investigation.

Consider the simple normal linear regression model for estimation/prediction at a new design point. When the slope parameter is not obviously nonzero, hypothesis testing and information criteria can be used for identifying the right model. We compare the performances of such methods both theoretically and empirically from different perspectives for more insight. The testing approach at the conventional size of 0.05, in spite of being the “standard approach,” performs poorly in estimation. We also found that the frequently told story “the Bayesian information criterion (BIC) is good when the true model is finite-dimensional, and the Akaike information criterion (AIC) is good when the true model is infinite-dimensional” is far from being accurate. In addition, despite some successes in the effort to go beyond the debate between AIC and BIC by adaptive model selection, it turns out that it is not possible to share the pointwise adaptation property of BIC and the minimax-rate adaptation property of AIC by any model selection method. When model selection methods have difficulty in selection, model combining is a better alternative in terms of estimation accuracy.This work was completed when the author was on leave from Iowa State University and was a New Direction Visiting Professor at the Institute for Mathematics and its Applications (IMA) at the University of Minnesota. The fundings from both IMA and ISU are greatly appreciated. The work was also partly supported by NSF CAREER grant DMS0094323. The author thanks Xiaotong Shen and Hannes Leeb for very helpful discussions. The paper also benefited from the questions and comments from the participants at the statistics seminars the author gave at the University of Minnesota and Duke University. The author is very grateful to the anonymous reviewers and the co-editor Benedikt Potscher for carefully reading earlier versions of the paper, bringing my attention to several closely related previous and current results, and making many very valuable suggestions, which significantly improved the paper in both content and presentation.

The rapid development of modeling techniques has brought many opportunities for data‐driven discovery and prediction. However, this also leads to the challenge of selecting the most appropriate model for any particular data task. Information criteria, such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC), have been developed as a general class of model selection methods with profound connections with foundational thoughts in statistics and information theory. Many perspectives and theoretical justifications have been developed to understand when and how to use information criteria, which often depend on particular data circumstances. This review article will revisit information criteria by summarizing their key concepts, evaluation metrics, fundamental properties, interconnections, recent advancements, and common misconceptions to enrich the understanding of model selection in general.This article is categorized under:Data: Types and Structure > Traditional Statistical DataStatistical Learning and Exploratory Methods of the Data Sciences > Modeling MethodsStatistical and Graphical Methods of Data Analysis > Information Theoretic MethodsStatistical Models > Model Selection

This article discusses the use of association models to detect group differences and the choice of model selection criteria under different conditions. The performances of several commonly used model selection criteria in log-linear modeling are examined using Monte Carlo simulations. The results suggest that no single criterion can play the role of panacea in model selection. L2/ df, normed fit index (NFI) (or 1 - L12/L02), and Akaike's information criterion (AIC) are systematically biased toward models incorporating group differences. The log-likelihood ratio test (LRT), in conjunction with the nested chi-square difference test, is useful for samples with moderate sizes. Bayesian information criterion (BIC) is most reliable among the measures tested. It consistently provides correct information about group differences in association, especially when the sample size is large. Although BIC may give equivocal results under certain conditions, the ambiguities can be mitigated by an inclusion of competing models and a cautious interpretation of any small improvement in BIC.

This paper introduces a SAS/IML program to select among the multivariate model candidates based on a few well-known multivariate model selection criteria. Stepwise regression and all-possible-regression are considered. The program is user friendly and requires the user to paste or read the data at the beginning of the module, include the names of the dependent and independent variables (the y's and the x's), and then run the module. The program produces the multivariate candidate models based on the following criteria: Forward Selection, Forward Stepwise Regression, Backward Elimination, Mean Square Error, Coefficient of Multiple Determination, Adjusted Coefficient of Multiple Determination, Akaike's Information Criterion, the Corrected Form of Akaike's Information Criterion, Hannan and Quinn Information Criterion, the Corrected Form of Hannan and Quinn (HQc) Information Criterion, Schwarz's Criterion, and Mallow's PC. The output also constitutes detailed as well as summarized results.

We illustrate a recently proposed two-step bootstrap model averaging (bootstrap MA) approach to cope with model selection uncertainty. The predictive performance is investigated in an example and in a simulation study. Results are compared to those derived from other model selection methods. In the framework of the linear regression model we use the two-step bootstrap MA, which consists of a screening step to eliminate covariates thought to have no influence on the response, and a model-averaging step. We also apply the full model, variable selection using backward elimination based on Akaike's Information Criterion (AIC), the Bayes Information Criterion (BIC) and the bagging approach. The predictive performance is measured by the mean squared error (MSE) and the coverage of confidence intervals for the true response. We obtained similar results for all approaches in the example. In the simulation the MSE was reduced by all approaches in comparison to the full model. The smallest values are obtained for bootstrap MA. Only the bootstrap MA and the full model correctly estimated the nominal coverage. The backward elimination procedures led to substantial underestimation and bagging to an overestimation of the true coverage. The screening step of bootstrap MA eliminates most of the unimportant factors. The new bootstrap MA approach shows promising results for predictive performance. It increases practical usefulness by eliminating unimportant factors in the screening step.

Model selection is an omnipresent problem in signal processing applications. The Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are the most commonly used solutions to this problem. These criteria have been found to have satisfactory performance in many cases and had a dominant role in the model selection literature since their introduction several decades ago, despite numerous attempts to dethrone them. Model selection can be viewed as a multiple hypothesis testing problem. This simple observation makes it possible to use for model selection a number of powerful hypothesis testing procedures that control the false discovery rate (FDR) or the familywise error rate (FER). This is precisely what we do in this paper in which we follow the lead of the proposers of the said procedures and introduce two <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">general</i> rules for model selection based on FDR and FER, respectively. We show in a numerical performance study that the FDR and FER rules are serious competitors of AIC and BIC with significant performance gains in more demanding cases, essentially at the same computational effort.

A jackknife type approach to statistical model selection

The analysis of ecological data has changed in two important ways over the last 15years. The development and easy availability of Bayesian computational methods has allowed and encouraged the fitting of complex hierarchical models. At the same time, there has been increasing emphasis on acknowledging and accounting for model uncertainty. Unfortunately, the ability to fit complex models has outstripped the development of tools for model selection and model evaluation: familiar model selection tools such as Akaike's information criterion and the deviance information criterion are widely known to be inadequate for hierarchical models. In addition, little attention has been paid to the evaluation of model adequacy in context of hierarchical modeling, i.e., to the evaluation of fit for a single model. In this paper, we describe Bayesian cross-validation, which provides tools for model selection and evaluation. We describe the Bayesian predictive information criterion and a Bayesian approximation to the BPIC known as the Watanabe-Akaike information criterion. We illustrate the use of these tools for model selection, and the use of Bayesian cross-validation as a tool for model evaluation, using three large data sets from the North American Breeding Bird Survey.

Time series study is of paramount importance especially in developing countries such as India. The objective of this study is to forecast the number of dengue cases by applying time series models. Furthermore, this study also aims to uncover the correlation of ecological variables with dengue cases using a p value test. The ecological variables considered are the rainfall (with a 2-month lag period) and average relative humidity. An evaluation of host factors such as age and gender is done with the number of cases to determine the most affected strata. Monthly data on dengue cases from 2014 to 2017 are taken from the integrated disease surveillance programme, Government of India, and data on ecological variables, humidity and rainfall are taken from www.Indiastat.com . A 2-month lag period is considered for rainfall. These data were analysed by the autoregressive integrated moving average (ARIMA) and exponential smoothing model (ES). Predictive performance of these techniques was assessed by evaluation parameters such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), mean absolute error and mean absolute scaled error. Applying time series models was useful in the prediction. There were many more males than females according to the data for 2014–2017. The Akaike information criterion (AIC), Bayesian information criterion (BIC) and mean absolute scaled error for ARIMA are 48.6526, 448.333333, 449.30008, 453.08 and 0.507826, respectively. Based on model indices such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), mean absolute error and mean absolute scaled error, the ARIMA performed better than the ES model. Rainfall at the 2-month lag was correlated with the number of cases.

Multiple imputation and maximum likelihood estimation (via the expectation‐maximization algorithm) are two well‐known methods readily used for analyzing data with missing values. While these two methods are often considered as being distinct from one another, multiple imputation (when using improper imputation) is actually equivalent to a stochastic expectation‐maximization approximation to the likelihood. In this article, we exploit this key result to show that familiar likelihood‐based approaches to model selection, such as Akaike's information criterion (AIC) and the Bayesian information criterion (BIC), can be used to choose the imputation model that best fits the observed data. Poor choice of imputation model is known to bias inference, and while sensitivity analysis has often been used to explore the implications of different imputation models, we show that the data can be used to choose an appropriate imputation model via conventional model selection tools. We show that BIC can be consistent for selecting the correct imputation model in the presence of missing data. We verify these results empirically through simulation studies, and demonstrate their practicality on two classical missing data examples. An interesting result we saw in simulations was that not only can parameter estimates be biased by misspecifying the imputation model, but also by overfitting the imputation model. This emphasizes the importance of using model selection not just to choose the appropriate type of imputation model, but also to decide on the appropriate level of imputation model complexity.

In order to have confidence in model-based phylogenetic analysis, the model of nucleotide substitution adopted must be selected in a statistically rigorous manner. Several model-selection methods are applicable to maximum likelihood (ML) analysis, including the hierarchical likelihood-ratio test (hLRT), Akaike information criterion (AIC), Bayesian information criterion (BIC), and decision theory (DT), but their performance relative to empirical data has not been investigated thoroughly. In this study, we use 250 phylogenetic data sets obtained from TreeBASE to examine the effects that choice in model selection has on ML estimation of phylogeny, with an emphasis on optimal topology, bootstrap support, and hypothesis testing. We show that the use of different methods leads to the selection of two or more models for approximately 80% of the data sets and that the AIC typically selects more complex models than alternative approaches. Although ML estimation with different best-fit models results in incongruent tree topologies approximately 50% of the time, these differences are primarily attributable to alternative resolutions of poorly supported nodes. Furthermore, topologies and bootstrap values estimated with ML using alternative statistically supported models are more similar to each other than to topologies and bootstrap values estimated with ML under the Kimura two-parameter (K2P) model or maximum parsimony (MP). In addition, Swofford-Olsen-Waddell-Hillis (SOWH) tests indicate that ML trees estimated with alternative best-fit models are usually not significantly different from each other when evaluated with the same model. However, ML trees estimated with statistically supported models are often significantly suboptimal to ML trees made with the K2P model when both are evaluated with K2P, indicating that not all models perform in an equivalent manner. Nevertheless, the use of alternative statistically supported models generally does not affect tests of monophyletic relationships under either the Shimodaira-Hasegawa (S-H) or SOWH methods. Our results suggest that although choice in model selection has a strong impact on optimal tree topology, it rarely affects evolutionary inferences drawn from the data because differences are mainly confined to poorly supported nodes. Moreover, since ML with alternative best-fit models tends to produce more similar estimates of phylogeny than ML under the K2P model or MP, the use of any statistically based model-selection method is vastly preferable to forgoing the model-selection process altogether.