Model selection for Markov random fields on graphs under a mixing condition

This thesis introduces a novel approach for estimating the graph of conditional dependencies in a random vector based on finite sample data. We define this approach as a global model selection criterion, which means optimizing a function across the entire set of potential graphs, removing the need to estimate and combine individual neighborhoods as commonly proposed in the literature. Our results establishes the strong convergence of this graph estimator, provided that the multivariate stochastic process satisfies a mixing condition. To the best of our knowledge, these results represent a pioneering demonstration of the consistency of a model selection criterion for Markov random fields on graphs when dealing with non-independent data. Additionally, we propose efficient algorithms for graph estimation and complement our theoretical results with simulation studies. To illustrate the practical applicability of our approach, we present two real-world examples: a study of the dependence structure among water flow measurements gauges located in the course of the S\\\\~ao Francisco River in Brazil; and a daily stock market index performance analysis in order to identify the conditional dependence among the stock markets around the world.

Model selection is an important part of any statistical analysis. Many tools are suggested for selecting the best model including frequentist and Bayesian perspectives. There is often a considerable uncertainty in the selection of a particular model to be the best approximating model. Model selection uncertainty arises when the data are used for both model selection and parameter estimation. Bias in estimators of model parameters often arise when data based selection has been done. Therefore, model averaging of the parameter estimators will be done to alleviate the bias in model selection in a set of candidate models, by combining the information from a set of candidate models. This paper is two-fold, new criteria of model selection are proposed based on different averages of AIC, BIC, AICc, and HQC. Also, model averaging is introduced to compare the parameter estimators in model averaging with the ones in model selection. Two Simulation studies are considered, the first is for model selection and showed that the new proposed criteria are lies between some of the known criteria such as AIC, BIC, AICc, and HQC, and so they can be used as new criteria of model selection. The second simulation study is for model averaging and showed that the parameter estimators have less bias and less predicted mean square error (PMSE) compared with the parameter estimators in model selection.

In this paper we apply model selection criteria in time series and regression analysis to the estimation of the number of signals in the MUSIC (multiple signal classification) method. We compare the following model selection (information) criteria: AIC, HQ, BIC, AICC, and the recently introduced WIC as the weighted average of AICC and BIC. The general form of the above information criteria consists of a log likelihood function expressed in terms of the eigenvalues of the covariance matrix and a unique penalty term. In our estimation procedure, the number of signals is obtained by minimizing each of the above criteria. A linear antenna array example is presented to compare the performance of the above model selection criteria in signal processing problem.

There exist quite a few criteria for penalty-based model selection. Although they have various justifications for large sample problems, their performance under small or moderate sample size is unclear which hinders the development of model combination methods using the appropriate penalty term. In this paper, we assess the performance of seven model selection criteria based on linear regression models with unknown noise variance. We set the true data generation mechanism to be within the model set as well as outside the model set. In the latter case, soft model selection through multiple model fusion is proposed and its difference from Bayesian model averaging is highlighted. The penalty term used in each model selection criterion provides a natural link to estimate the model probability without assuming any prior knowledge of the unknown parameter. An important question is whether the estimated model probabilities are consistent when multiple models are fused for prediction or interpolation. We argue that strong consistency only holds under large sample regime while soft model selection can still be better than choosing a single model with small sample size. Our numerical results using different model selection criteria for polynomial fitting indicate that the conditional model estimator (CME) has the best performance in selecting the correct model order and fusing multiple models for prediction and interpolation. The minimum description length (MDL) based criteria are next to CME and outperform Bayesian information criterion (BIC) and Akaike information criterion (AIC) significantly.

Over-fitting is a ubiquitous problem in machine learning, and a variety of techniques to avoid over-fitting the training sample have proven highly effective, including early stopping, regularization, and ensemble methods. However, while over-fitting in training is widely appreciated and its avoidance now a standard element of best practice, over-fitting can also occur in model selection. This form of over-fitting can significantly degrade generalization performance, but has thus far received little attention. For example the kernel and regularization parameters of a support vector machine are often tuned by optimizing a cross-validation based model selection criterion. However the cross-validation estimate of generalization performance will inevitably have a finite variance, such that its minimizer depends on the particular sample on which it is evaluated, and this will generally differ from the minimizer of the true generalization error. Therefore if the cross-validation error is aggressively minimized, generalization performance may be substantially degraded. In general, the smaller the amount of data available, the higher the variance of the model selection criterion, and hence the more likely over-fitting in model selection will be a significant problem. Similarly, the more hyper-parameters to be tuned in model selection, the more easily the variance of the model selection criterion can be exploited, which again increases the likelihood of over-fitting in model selection. Over-fitting in model selection is empirically demonstrated to pose a substantial pitfall in the application of kernel learning methods and Gaussian process classifiers. Furthermore, evaluation of machine learning methods can easily be significantly biased unless the evaluation protocol properly accounts for this type of over-fitting. Fortunately the common solutions to avoiding over-fitting in training also appear to be effective in avoiding over-fitting in model selection. Three examples are presented based on regularization of the model selection criterion, early stopping in model selection and minimizing the number of hyper-parameters to be tuned during model selection.

Model Selection Techniques and Merging Rules for Range Data Segmentation Algorithms

Multiple response surface methodology (MRSM) most often involves the analysis of small sample size datasets which have associated inherent statistical modeling problems. Firstly, classical model selection criteria in use are very inefficient with small sample size datasets. Secondly, classical model selection criteria have an acknowledged selection uncertainty problem. Finally, there is a credibility problem associated with modeling small sample sizes of the order of most MRSM datasets. This work focuses on determination of a solution to these identified problems. The small sample model selection uncertainty problem is analysed using sixteen model selection criteria and a typical two-input MRSM dataset. Selection of candidate models, for the responses in consideration, is done based on response surface conformity to expectation to deliberately avoid selection of models using the problematic classical model selection criteria. A set of permutations of combinations of response models with conforming response surfaces is determined. Each combination is optimised and results are obtained using overlaying of data matrices. The permutation of results is then averaged to obtain credible results. Thus, a transparent multiple model approach is used to obtain the solution which gives some credibility to the small sample size results of the typical MRSM dataset. The conclusion is that, for a two-input process MRSM problem, conformity of response surfaces can be effectively used to select candidate models and thus the use of the problematic model selection criteria is avoidable.

Model selection is an important and challenging problem in statistics. The model selection is inevitable in a large number of applications including life sciences, social sciences, business, or economics. In this article, we propose a resampling-based information criterion called paired bootstrap criterion (PBC) for model selection. The proposed criterion is based on minimizing the conditional expected prediction loss for selecting the best subset of variables. We estimate the conditional expected prediction loss by using the out-of-bag (OOB) bootstrap approach. Other classical criteria for model selection such as AIC, BIC are also presented for comparison purpose. We demonstrate that the proposed paired bootstrap model selection criterion is effective in selecting accurate models via real and simulated data examples. The results confirm the satisfactory behavior of the proposed model selection criterion to select parsimonious models that fit the data well. We apply the proposed methodology to a real data example.

Capture-recapture methods estimate the size of hidden populations by leveraging the proportion of overlap of the population on independent lists. Log-linear modeling relaxes the assumption of list independence, but best model selection criteria remain uncertain. Incorrect model selection can deliver incorrect and even implausible size estimates. We used simulations to model when capture-recapture methods deliver biased or unbiased estimates and compare model selection criteria. Simulations included five scenarios for list dependence among three incomplete lists of population of interest. We compared metrics of log-linear model selection, accuracy, and precision. We also compared log-linear model performance to the decomposable graph approach (a Bayesian model average), the sparse multiple systems estimation (SparseMSE) approach that accounts for zero or low cell counts, and the Sample Coverage approach. Log-linear models selected by Akaike's information criterion (AIC) calculated accurate population size estimates in all scenarios except for those with sparse or zero cell counts. In these scenarios, the decomposable graph and the Sample Coverage models produced more accurate size estimates. Conventional capture-recapture model selection fails with sparse cell counts. This naïve approach to model selection should be replaced with the implementation of multiple different models in order triangulate the truth in real-world applications.

With the growing popularity of spatial mixture models in cluster analysis, model selection criteria have become an established tool in the search for parsimony. However, the label-switching problem is often inherent in Bayesian implementation of mixture models and a variety of relabeling algorithms have been proposed. We use a space-time mixture of Poisson regression models with homogeneous covariate effects to illustrate that the best model selected by using model selection criteria does not always support the model that is chosen by the optimal relabeling algorithm. The results are illustrated for real and simulated datasets. The objective is to make the reader aware that if the purpose of statistical modeling is to identify clusters, applying a relabeling algorithm to the model with the best fit may not generate the optimal relabeling.

A user-friendly SAS macro application to perform all possible model selection of fixed effects including quadratic and cross products in the presence of random and repeated measures effects using SAS PROC MIXED is available. This macro application will complement the model selection option currently available in the SAS PROC REG for multiple linear regressions and the experimental SAS procedure GLMSELECT that focuses on the standard independently and identically distributed general linear model for univariate responses. Options are also included in this macro to select the best covariance structure associated with the user-specified fully saturated repeated measures model; to graphically explore and to detect statistical significance of user specified linear, quadratic, interaction terms for fixed effects; and to diagnose multicollinearity, via the VIF statistic for each continuous predictors involved in each model selection step. Two model selection criteria, AICC (corrected Akaike Information Criterion) and MDL (minimal description length) are used in all possible model selection and summaries of the best model selection are compared graphically. The differences in the degree of penalty factors associated with the model dimension between AICC and MDL are investigated. Complete mixed model analysis of final model including data exploration, influential diagnostics, and checking for model violations using the experimental ODS GRAPHICS option available in Version 9.13 is also implemented. Instructions for downloading and running this user-friendly macro application are included.

Longitudinal data often encounter missingness with monotone and/or intermittent missing patterns. Multiple imputation (MI) has been popularly employed for analysis of missing longitudinal data. In particular, the MI-GEE method has been proposed for inference of generalized estimating equations (GEE) when missing data are imputed via MI. However, little is known about how to perform model selection with multiply imputed longitudinal data. In this work, we extend the existing GEE model selection criteria, including the "quasi-likelihood under the independence model criterion" (QIC) and the "missing longitudinal information criterion" (MLIC), to accommodate multiple imputed datasets for selection of the MI-GEE mean model. According to real data analyses from a schizophrenia study and an AIDS study, as well as simulations under nonmonotone missingness with moderate proportion of missing observations, we conclude that: (i) more than a few imputed datasets are required for stable and reliable model selection in MI-GEE analysis; (ii) the MI-based GEE model selection methods with a suitable number of imputations generally perform well, while the naive application of existing model selection methods by simply ignoring missing observations may lead to very poor performance; (iii) the model selection criteria based on improper (frequentist) multiple imputation generally performs better than their analogies based on proper (Bayesian) multiple imputation.

Model selection is an important problem for intelligent systems. We visit the well known criteria such as AIC, BIC and MDL, and show new remarkable facts which remain unexplored. When we compare two statistical models of which one is a submodel of the larger one, we show there is a better or more economic way of describing given data. This leads us a new criterion of model selection, which sits between MDL and AIC. We also point out that, for a hierarchical family of models such as neural networks or Gaussian mixtures, the classic theories of AIC, BIC and MDL do not hold. A new consideration is again necessary. We give some hints on this problem.

In empirical research, we often need to address the issue of what model to use given a collection of candidate models. Conventionally, we use model selection to choose one best model from the collection of candidate models based on some model selection criteria. Model averaging is a generalization of model selection in the sense that it assigns weights to candidate models and uses a weighted average to construct an aggregated model. Usually model averaging provides better performance than model selection which chooses a single candidate model based on AIC or BIC.

Mottled Enamel