Model Selection Techniques and Merging Rules for Range Data Segmentation Algorithms

Model selection is an important part of any statistical analysis. Many tools are suggested for selecting the best model including frequentist and Bayesian perspectives. There is often a considerable uncertainty in the selection of a particular model to be the best approximating model. Model selection uncertainty arises when the data are used for both model selection and parameter estimation. Bias in estimators of model parameters often arise when data based selection has been done. Therefore, model averaging of the parameter estimators will be done to alleviate the bias in model selection in a set of candidate models, by combining the information from a set of candidate models. This paper is two-fold, new criteria of model selection are proposed based on different averages of AIC, BIC, AICc, and HQC. Also, model averaging is introduced to compare the parameter estimators in model averaging with the ones in model selection. Two Simulation studies are considered, the first is for model selection and showed that the new proposed criteria are lies between some of the known criteria such as AIC, BIC, AICc, and HQC, and so they can be used as new criteria of model selection. The second simulation study is for model averaging and showed that the parameter estimators have less bias and less predicted mean square error (PMSE) compared with the parameter estimators in model selection.

기존의 frequentist 추론에 비해 Bayesian 추론에서의 가설 검정 및 모형 선택 문제는 학자들 간에 일치된 견해를 보이지 못하고 있으며 아직도 논란이 되는 것들이 많다. Bayesian 추론에서 가설 검정 및 모형 선택의 기준으로 널리 쓰이는 Bayes factor는 이해하기 쉬우나 여러 경우에 구하기 어려운 단점이 존재한다. 그 외에 다른 기준으로 Spiegelhalter 등 (2002)가 제시한 DIC(Deviance Information Criterion)과 frequentist 추론에서의 P-value에 대비되는 Bayesian P-value가 있다. 본 논문에서는 Swiss banknote 자료를 Bayesian 로지스틱 회귀모형으로 분석하고 관련 기준들을 구하여 각 기준들이 일관성 있는 결론을 보이는지 확인하고자 한다. Model selection and hypothesis testing problems in Bayesian inference are still debated between scholars. Bayesian factors traditionally used as a criterion in Bayesian hypothesis testing and model selection, are easy to understand but sometimes hard to compute. In addition, there are other model selection criterions such as DIC(Deviance Information Criterion) by Spiegelhalter et al. (2002) and Bayesian P-values for testing. In this paper, we briefly introduce the Bayesian hypothesis testing and model selection procedure. In addition we have applied a Bayesian inference to Swiss banknote data by a fitting logistic regression model and computing several test statistics to see if they provide consistent results.

A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools

In quantitative analysis of tissue vascular permeability using Dynamic-Contrast-Enhanced (DCE) MR data analysis, one of the main challenges is to choose the best Pharmacokinetic (PK) model among competing models to describe the behavior of the time trace of contrast agent (CA) concentration. The F-Statistic (F-test), Akaike-Information-Criterion (AIC), Bayesian-Information-Criterion (BIC), and Log-Likelihood-Ratio (LLR) tests are the most widely used classical Model-Selection (MS) techniques. Our group has recently introduced an Adaptive-MS (AMS) method to perform PK-MS in DCE-MRI data. This study investigates the performance of different MS techniques (Classical and Adaptive) for selection of the PK model as well as the impact of the recruited technique on the estimation of PK parameters. In this study, three physiologically Nested Models were used to describe possible physiological conditions of underlying tissue pathology as Model 1, 2, and 3 respectively: The tissue-vascular-compartment is filled with CA with no outward-leakage, with outward-leakage but no evidence of back-flux, and with both outward-and backward-flux. Plasma volume vp, vp, and ktrans, vp, ktrans and kep, are the PK parameters for Models 1, 2 and 3 respectively. 21 Arterial Input Function curves were used to simulate the CA profile. By varying PK parameters (time-resolution = 5.03 sec), 9072 time traces for 12 levels of SNRs were generated. Then confusion matrices were computed for all the MS techniques over 9072 tested hypotheses. The performance of the AMS is superior to the other classical techniques. Other techniques show performances in the following order: LLR, BIC, AICc, AIC and F-Test. All techniques never miss any leaky tissue. Also the AMS generates significantly less-biased estimates of PK parameters compared to the classical techniques while the LLR and BIC methods outperform the other classical techniques. This study suggests that AMS, LLR, and BIC are the best candidates among the MS techniques for PK analysis of DCE-MRI data.

In this paper we apply model selection criteria in time series and regression analysis to the estimation of the number of signals in the MUSIC (multiple signal classification) method. We compare the following model selection (information) criteria: AIC, HQ, BIC, AICC, and the recently introduced WIC as the weighted average of AICC and BIC. The general form of the above information criteria consists of a log likelihood function expressed in terms of the eigenvalues of the covariance matrix and a unique penalty term. In our estimation procedure, the number of signals is obtained by minimizing each of the above criteria. A linear antenna array example is presented to compare the performance of the above model selection criteria in signal processing problem.

There exist quite a few criteria for penalty-based model selection. Although they have various justifications for large sample problems, their performance under small or moderate sample size is unclear which hinders the development of model combination methods using the appropriate penalty term. In this paper, we assess the performance of seven model selection criteria based on linear regression models with unknown noise variance. We set the true data generation mechanism to be within the model set as well as outside the model set. In the latter case, soft model selection through multiple model fusion is proposed and its difference from Bayesian model averaging is highlighted. The penalty term used in each model selection criterion provides a natural link to estimate the model probability without assuming any prior knowledge of the unknown parameter. An important question is whether the estimated model probabilities are consistent when multiple models are fused for prediction or interpolation. We argue that strong consistency only holds under large sample regime while soft model selection can still be better than choosing a single model with small sample size. Our numerical results using different model selection criteria for polynomial fitting indicate that the conditional model estimator (CME) has the best performance in selecting the correct model order and fusing multiple models for prediction and interpolation. The minimum description length (MDL) based criteria are next to CME and outperform Bayesian information criterion (BIC) and Akaike information criterion (AIC) significantly.

Over-fitting is a ubiquitous problem in machine learning, and a variety of techniques to avoid over-fitting the training sample have proven highly effective, including early stopping, regularization, and ensemble methods. However, while over-fitting in training is widely appreciated and its avoidance now a standard element of best practice, over-fitting can also occur in model selection. This form of over-fitting can significantly degrade generalization performance, but has thus far received little attention. For example the kernel and regularization parameters of a support vector machine are often tuned by optimizing a cross-validation based model selection criterion. However the cross-validation estimate of generalization performance will inevitably have a finite variance, such that its minimizer depends on the particular sample on which it is evaluated, and this will generally differ from the minimizer of the true generalization error. Therefore if the cross-validation error is aggressively minimized, generalization performance may be substantially degraded. In general, the smaller the amount of data available, the higher the variance of the model selection criterion, and hence the more likely over-fitting in model selection will be a significant problem. Similarly, the more hyper-parameters to be tuned in model selection, the more easily the variance of the model selection criterion can be exploited, which again increases the likelihood of over-fitting in model selection. Over-fitting in model selection is empirically demonstrated to pose a substantial pitfall in the application of kernel learning methods and Gaussian process classifiers. Furthermore, evaluation of machine learning methods can easily be significantly biased unless the evaluation protocol properly accounts for this type of over-fitting. Fortunately the common solutions to avoiding over-fitting in training also appear to be effective in avoiding over-fitting in model selection. Three examples are presented based on regularization of the model selection criterion, early stopping in model selection and minimizing the number of hyper-parameters to be tuned during model selection.

Model selection is an important and challenging problem in statistics. The model selection is inevitable in a large number of applications including life sciences, social sciences, business, or economics. In this article, we propose a resampling-based information criterion called paired bootstrap criterion (PBC) for model selection. The proposed criterion is based on minimizing the conditional expected prediction loss for selecting the best subset of variables. We estimate the conditional expected prediction loss by using the out-of-bag (OOB) bootstrap approach. Other classical criteria for model selection such as AIC, BIC are also presented for comparison purpose. We demonstrate that the proposed paired bootstrap model selection criterion is effective in selecting accurate models via real and simulated data examples. The results confirm the satisfactory behavior of the proposed model selection criterion to select parsimonious models that fit the data well. We apply the proposed methodology to a real data example.

Uncertainty assessment for economic valuation of a field generally involves developing multiple subsurface realizations combined with different development scenarios, thereby rendering the task computationally expensive and time consuming. In addition, the task of updating the models with new information is equally daunting leading to enormous and unreasonable simplifications to the economic valuation procedure. This paper presents a novel approach that renders the uncertainty assessment less cumbersome and more efficient/accurate with the help of a new uncertainty assessment approach. The proposed approach makes use of a model selection algorithm to reduce the number of subsurface realization necessary for uncertainty assessment. A model selection framework does the work of refining an initial suite of reservoir models to a final set of reservoir models that depict production characteristics close to the observed field history. Model selection algorithm attempts to group reservoir models based on the common connectivity characteristics exhibited by them and then retrieve the group exhibiting characteristics that are closest to the observed response for a reservoir. The model selection technique is thus an efficient way to reduce number of realizations and save computational time for doing uncertainty analysis, and yet be optimally constrained to the available data. An illustrative example with a dataset for a large field is used to show the proposed methodology for economic valuation of a large field. Well testing data from two wells is used as two sets of history data for the model selection procedure. Real option valuation (ROV), a probabilistic approach, is used to infer the economics of the reservoir. The case described will demonstrate that the proposed approach is systematic, reduces the time of computation, and allows the rigorous application of economic analysis without making any undue simplifications and assumptions.

Model selection is a necessary step in many practical regression analyses. But for methods based on estimating equations, such as the quasi-likelihood and generalized estimating equation (GEE) approaches, there seem to be few well-studied model selection techniques. In this article, we propose a new model selection criterion that minimizes the expected predictive bias (EPB) of estimating equations. A bootstrap smoothed cross-validation (BCV) estimate of EPB is presented and its performance is assessed via simulation for overdispersed generalized linear models. For illustration, the method is applied to a real data set taken from a study of the development of ewe embryos.

Capture-recapture methods estimate the size of hidden populations by leveraging the proportion of overlap of the population on independent lists. Log-linear modeling relaxes the assumption of list independence, but best model selection criteria remain uncertain. Incorrect model selection can deliver incorrect and even implausible size estimates. We used simulations to model when capture-recapture methods deliver biased or unbiased estimates and compare model selection criteria. Simulations included five scenarios for list dependence among three incomplete lists of population of interest. We compared metrics of log-linear model selection, accuracy, and precision. We also compared log-linear model performance to the decomposable graph approach (a Bayesian model average), the sparse multiple systems estimation (SparseMSE) approach that accounts for zero or low cell counts, and the Sample Coverage approach. Log-linear models selected by Akaike's information criterion (AIC) calculated accurate population size estimates in all scenarios except for those with sparse or zero cell counts. In these scenarios, the decomposable graph and the Sample Coverage models produced more accurate size estimates. Conventional capture-recapture model selection fails with sparse cell counts. This naïve approach to model selection should be replaced with the implementation of multiple different models in order triangulate the truth in real-world applications.

Multiple response surface methodology (MRSM) most often involves the analysis of small sample size datasets which have associated inherent statistical modeling problems. Firstly, classical model selection criteria in use are very inefficient with small sample size datasets. Secondly, classical model selection criteria have an acknowledged selection uncertainty problem. Finally, there is a credibility problem associated with modeling small sample sizes of the order of most MRSM datasets. This work focuses on determination of a solution to these identified problems. The small sample model selection uncertainty problem is analysed using sixteen model selection criteria and a typical two-input MRSM dataset. Selection of candidate models, for the responses in consideration, is done based on response surface conformity to expectation to deliberately avoid selection of models using the problematic classical model selection criteria. A set of permutations of combinations of response models with conforming response surfaces is determined. Each combination is optimised and results are obtained using overlaying of data matrices. The permutation of results is then averaged to obtain credible results. Thus, a transparent multiple model approach is used to obtain the solution which gives some credibility to the small sample size results of the typical MRSM dataset. The conclusion is that, for a two-input process MRSM problem, conformity of response surfaces can be effectively used to select candidate models and thus the use of the problematic model selection criteria is avoidable.

A user-friendly SAS macro application to perform all possible model selection of fixed effects including quadratic and cross products in the presence of random and repeated measures effects using SAS PROC MIXED is available. This macro application will complement the model selection option currently available in the SAS PROC REG for multiple linear regressions and the experimental SAS procedure GLMSELECT that focuses on the standard independently and identically distributed general linear model for univariate responses. Options are also included in this macro to select the best covariance structure associated with the user-specified fully saturated repeated measures model; to graphically explore and to detect statistical significance of user specified linear, quadratic, interaction terms for fixed effects; and to diagnose multicollinearity, via the VIF statistic for each continuous predictors involved in each model selection step. Two model selection criteria, AICC (corrected Akaike Information Criterion) and MDL (minimal description length) are used in all possible model selection and summaries of the best model selection are compared graphically. The differences in the degree of penalty factors associated with the model dimension between AICC and MDL are investigated. Complete mixed model analysis of final model including data exploration, influential diagnostics, and checking for model violations using the experimental ODS GRAPHICS option available in Version 9.13 is also implemented. Instructions for downloading and running this user-friendly macro application are included.

Longitudinal data often encounter missingness with monotone and/or intermittent missing patterns. Multiple imputation (MI) has been popularly employed for analysis of missing longitudinal data. In particular, the MI-GEE method has been proposed for inference of generalized estimating equations (GEE) when missing data are imputed via MI. However, little is known about how to perform model selection with multiply imputed longitudinal data. In this work, we extend the existing GEE model selection criteria, including the "quasi-likelihood under the independence model criterion" (QIC) and the "missing longitudinal information criterion" (MLIC), to accommodate multiple imputed datasets for selection of the MI-GEE mean model. According to real data analyses from a schizophrenia study and an AIDS study, as well as simulations under nonmonotone missingness with moderate proportion of missing observations, we conclude that: (i) more than a few imputed datasets are required for stable and reliable model selection in MI-GEE analysis; (ii) the MI-based GEE model selection methods with a suitable number of imputations generally perform well, while the naive application of existing model selection methods by simply ignoring missing observations may lead to very poor performance; (iii) the model selection criteria based on improper (frequentist) multiple imputation generally performs better than their analogies based on proper (Bayesian) multiple imputation.

With the growing popularity of spatial mixture models in cluster analysis, model selection criteria have become an established tool in the search for parsimony. However, the label-switching problem is often inherent in Bayesian implementation of mixture models and a variety of relabeling algorithms have been proposed. We use a space-time mixture of Poisson regression models with homogeneous covariate effects to illustrate that the best model selected by using model selection criteria does not always support the model that is chosen by the optimal relabeling algorithm. The results are illustrated for real and simulated datasets. The objective is to make the reader aware that if the purpose of statistical modeling is to identify clusters, applying a relabeling algorithm to the model with the best fit may not generate the optimal relabeling.