A Study on Clustering Method by Self-Organizing Map and Information Criteria

This paper introduces a SAS/IML program to select among the multivariate model candidates based on a few well-known multivariate model selection criteria. Stepwise regression and all-possible-regression are considered. The program is user friendly and requires the user to paste or read the data at the beginning of the module, include the names of the dependent and independent variables (the y's and the x's), and then run the module. The program produces the multivariate candidate models based on the following criteria: Forward Selection, Forward Stepwise Regression, Backward Elimination, Mean Square Error, Coefficient of Multiple Determination, Adjusted Coefficient of Multiple Determination, Akaike's Information Criterion, the Corrected Form of Akaike's Information Criterion, Hannan and Quinn Information Criterion, the Corrected Form of Hannan and Quinn (HQc) Information Criterion, Schwarz's Criterion, and Mallow's PC. The output also constitutes detailed as well as summarized results.

Information criteria are used in many applications including statistical model selection and intelligent systems. The traditional information criteria such as the Akaike information criterion (AIC) do not always provide an adequate penalty on the number of model covariates. To address this issue, we propose a novel method for evaluating statistical models based on information criterion. The proposed method, called regularized information criterion (RIL), modifies the penalty term in AIC to reduce model overfitting. The results of numerical experiments show that RIL provides a better reflection of model predictive error than AIC. Thus, RIL can be a useful tool in model selection.

In this paper, we cluster and analyze data from the past embedded software development projects using self-organizing maps (SOMs)[9] that are a type of artificial neural networks that rely on unsupervised learning. The purpose of the clustering and analysis is to improve the accuracy of predicting the number of errors. A SOMproduces a low-dimensional, discretized representation of the input space of training samples; these representations are called maps. SOMs are useful for visualizing low-dimensional views of high-dimensional data, a multidimensional scaling technique. The advantages of SOMs for statistical applications are as follows: (1) data visualization, (2) information processing on association and recollection, (3) summarizing large-scale data, and (4) creating nonlinear models. To verify our approach, we perform an evaluation experiment that compares SOM classification to product type classification using Welch’s t-test for Akaike’s Information Criterion (AIC). The results indicate that the SOM classification method is more contributive than product type classification in creating estimation models, because the mean AIC of SOM classification is statistically significantly lower.KeywordsSelf-organizing mapsclusteringembedded software development

Latent profile analysis with nonnormal mixtures: A Monte Carlo examination of model selection using fit indices

Background: Metastatic regional lymph nodes (LN) is a strong predictor of worse long-term outcome after resection of breast cancer. This study aimed to compare the prognostic performance of American Joint Committee on Cancer (AJCC) N stage relative to lymph node ratio (LNR), log odds of metastatic lymph nodes (LODDS), number of removed lymph nodes (NRLNs), and number of negative lymph nodes (NNLNs) in breast cancer patients. Methods: All of the breast cancer patients who underwent surgery between 2004 and 2012 were identified from the Surveillance, Epidemiology, and End Results database. Restricted cubic spline functions were used to characterize the association between continuous variables and the risk of death and determine the optimal cut-off points. The Cox proportional hazards models were constructed, and the relative discriminative abilities of the different LN staging systems were assessed using the Akaike's Information Criterion (AIC) and the Harrell's concordance index (C-index). Results: A total of 264,096 breast cancer patients were enrolled, and 177,598 (67.2%) had no lymph node metastasis, whereas 86,498 (32.8%) had lymph node metastasis. 187,785 (71.1%) patients had a limited number of LNs harvested (NRLN <10). The median follow-up time was 73 months, and the 8-year overall survival (OS) and breast cancer-specific survival (BCSS) were 82.6% and 90.4%, respectively. LNR, LODDS, NRLNs, and NNLNs were all nonlinearly associated with OS and BCSS. Patients with metastatic LN had an increased risk of OS (hazards ratio: 2.32, 95% confidence interval: 2.27–2.37; P < 0.001) and BCSS (hazards ratio: 4.53, 95% confidence interval: 4.40–4.66; P < 0.001). When LNR was equal to 0 or 1, there was a heterogeneity of outcomes, and LODDS still yielded informative values compared to LNR. Among the entire cohort, LNR modeled as a continuous variable had a somewhat better prognostic performance (AIC: 923231.4 and C-index: 0.722 for OS; AIC: 482962.3 and C-index: 0.817 for BCSS) than any of other LN staging systems. However, a model with AJCC N stage showed the best fit in patients with a limited number of LNs harvested (AIC: 501321.8 and C-index: 0.699 for OS; AIC: 212605.6 and C-index: 0.809 for BCSS). When assessed among patients with metastatic LN, LODDS outperformed other staging systems including AJCC N stage, LNR, NRLNs and NNLNs, whenever assessed using continuous (AIC: 428626.2 and C index: 0.728 for OS; AIC: 296886.8 and C index: 0.770 for BCSS) or categorical (AIC: 429527.5 and C index: 0.722 for OS; AIC: 297796.6 and C index: 0.762 for BCSS) cutoff values. Conclusions: Although LNR assessed as a continuous variable was the most potent method to stratify patients regardless of LN status, the prognostic superiority of LNR is confounded by a limited LN harvest. LODDS was a better and more powerful predictor of survival when patients were LN positive, especially among those patients with either very low or high LNR. Citation Format: Gong Y, Ji P, Jiang Y-Z, Hu X, Shao Z-M. Comparing prognostic performance of different lymph node staging systems among patients with breast cancer [abstract]. In: Proceedings of the 2018 San Antonio Breast Cancer Symposium; 2018 Dec 4-8; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2019;79(4 Suppl):Abstract nr P2-08-36.

Time series study is of paramount importance especially in developing countries such as India. The objective of this study is to forecast the number of dengue cases by applying time series models. Furthermore, this study also aims to uncover the correlation of ecological variables with dengue cases using a p value test. The ecological variables considered are the rainfall (with a 2-month lag period) and average relative humidity. An evaluation of host factors such as age and gender is done with the number of cases to determine the most affected strata. Monthly data on dengue cases from 2014 to 2017 are taken from the integrated disease surveillance programme, Government of India, and data on ecological variables, humidity and rainfall are taken from www.Indiastat.com . A 2-month lag period is considered for rainfall. These data were analysed by the autoregressive integrated moving average (ARIMA) and exponential smoothing model (ES). Predictive performance of these techniques was assessed by evaluation parameters such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), mean absolute error and mean absolute scaled error. Applying time series models was useful in the prediction. There were many more males than females according to the data for 2014–2017. The Akaike information criterion (AIC), Bayesian information criterion (BIC) and mean absolute scaled error for ARIMA are 48.6526, 448.333333, 449.30008, 453.08 and 0.507826, respectively. Based on model indices such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), mean absolute error and mean absolute scaled error, the ARIMA performed better than the ES model. Rainfall at the 2-month lag was correlated with the number of cases.

Prognostic indices combining several clinical and laboratory parameters have been proposed for prognostication in chronic lymphocytic leukemia (CLL). Recently, international consortium on CLL proposed an international prognostic index (CLL-IPI) integrating clinical, molecular, and genetic parameters. The present study was designed to evaluate the reproducibility of CLL-IPI in Indian CLL cohort. The prognostic ability of CLL-IPI in terms of overall survival (OS) and time to first treatment (TTFT) was investigated in treatment-naive CLL patients and also compared with other existing prognostic scores. For assigning scores, clinical and laboratory details were obtained from medical records, and IGHV gene mutation status, β2-microglobulin levels, and copy number variations were determined using c-DNA, ELISA, and multiplex ligation-dependent probe amplification (MLPA), respectively. The scores were generated as per the weighted grades assigned to each variable involved in score categorization. The predictive value of prognostic models was assessed and compared using Harrell's C-index and Akaike's information criterion (AIC). Stratification of patients according to CLL-IPI yielded significant differences in terms of OS and TTFT (p < 0.001). Comparative assessment of scores for OS suggested better performance of CLL-IPI (C = 0.64, AIC = 740) followed by Barcelona-Brno (C = 0.61, AIC = 754) and MDACC score (C = 0.59, AIC = 759). Comparison of predictive value of prognostic scores for TTFT illustrated better performance of CLL-IPI (C = 0.72, AIC = 726) followed by Barcelona-Brno (C = 0.68, AIC = 743), modified GCLLSG (C = 0.66, AIC = 744), and O-CLL1 index (C = 0.55, AIC = 773). The results suggest better performance of CLL-IPI in terms of both OS and TTFT as compared to other available scores in our cohort.

When inferring phylogenies, one important decision is whether and how nucleotide substitution parameters should be shared across different subsets or partitions of the data. One sort of partitioning error occurs when heterogeneous subsets are mistakenly lumped together and treated as if they share parameter values. The opposite kind of error is mistakenly treating homogeneous subsets as if they result from distinct sets of parameters. Lumping and splitting errors are not equally bad. Lumping errors can yield parameter estimates that do not accurately reflect any of the subsets that were combined whereas splitting errors yield estimates that did not benefit from sharing information across partitions. Phylogenetic partitioning decisions are often made by applying information criteria such as the Akaike information criterion (AIC). As with other information criteria, the AIC evaluates a model or partition scheme by combining the maximum log-likelihood value with a penalty that depends on the number of parameters being estimated. For the purpose of selecting an optimal partitioning scheme, we derive an adjustment to the AIC that we refer to as the AIC$^{(p)}$ and that is motivated by the idea that splitting errors are less serious than lumping errors. We also introduce a similar adjustment to the Bayesian information criterion (BIC) that we refer to as the BIC$^{(p)}$. Via simulation and empirical data analysis, we contrast AIC and BIC behavior to our suggested adjustments. We discuss these results and also emphasize why we expect the probability of lumping errors with the AIC$^{(p)}$ and the BIC$^{(p)}$ to be relatively robust to model parameterization.

The detection of multiple outliers can be interpreted as a model selection problem. Models that can be selected are the null model, which indicates an outlier free set of observations, or a class of alternative models, which contain a set of additional bias parameters. A common way to select the right model is by using a statistical hypothesis test. In geodesy data snooping is most popular. Another approach arises from information theory. Here, the Akaike information criterion (AIC) is used to select an appropriate model for a given set of observations. The AIC is based on the Kullback-Leibler divergence, which describes the discrepancy between the model candidates. Both approaches are discussed and applied to test problems: the fitting of a straight line and a geodetic network. Some relationships between data snooping and information criteria are discussed. When compared, it turns out that the information criteria approach is more simple and elegant. Along with AIC there are many alternative information criteria for selecting different outliers, and it is not clear which one is optimal.

The shape of the dose-response relation between particulate matter air pollution and mortality is crucial for public health assessment, and departures of this relation from linearity could have important regulatory consequences. A number of investigators have studied the shape of the particulate matter-mortality dose-response relation and concluded that the relation could be adequately described by a linear model. Some of these researchers examined the hypothesis of linearity by comparing Akaike's Information Criterion (AIC) values obtained under linear, piecewise linear, and spline alternative models. However, at the current time, the efficacy of the AIC in this context has not been assessed. The authors investigated AIC as a means of comparing competing dose-response models, using data from Cook County, Illinois, for the period 1987-2000. They found that if nonlinearities exist, the AIC is not always successful in detecting them. In a number of the scenarios considered, AIC was equivocal, picking the correct simulated dose-response model about half of the time. These findings suggest that further research into the shape of the dose-response relation using alternative model selection criteria may be warranted.

This study systematically examines the robustness of the Akaike Information Criterion (AIC) in determining the optimal order (p) of an autoregressive (AR) model applied to the RR interval time series of the PhysioNet healthy subject database. The AR approach is widely used to estimate the power spectral density (PSD) of heart rate variability (HRV), and accurate order selection is essential for model stability and reliable spectral estimation. Although the AIC is designed to balance model fit and complexity, it suffers from the problem of arbitrary model selection. This study provides a quantitative robustness analysis of information-criterion-based AR order selection under controlled expansion of the search space. Specifically, we investigated the behavior of the AIC using the PhysioNet database (N = 1257) under conditions where the maximum search order was set to an excessively high value (p = 50), far exceeding the commonly recommended range. Our analysis suggested that the AR model began to capture subtle noise and nonstationary components rather than the intrinsic HRV structure, leading to overfitting and excessive order selection, resulting in false peaks in the PSD and reduced robustness. In conclusion, order decisions based solely on information criteria such as the AIC become unstable when the search range is too large. To ensure robustness, it is recommended to complement the AIC with more stringent criteria such as the Bayesian Information Criterion (BIC) or Final Prediction Error (FPE), in addition to the traditional maximum order restriction.

Only unstructured single-path model selection techniques, i.e., Information Criteria, are used by Bounds test of cointegration for model selection. The aim of this paper was twofold; one was to evaluate the performance of these five routinely used information criteria {Akaike Information Criterion (AIC), Akaike Information Criterion Corrected (AICC), Schwarz/Bayesian Information Criterion (SIC/BIC), Schwarz/Bayesian Information Criterion Corrected (SICC/BICC), and Hannan and Quinn Information Criterion (HQC)} and three structured approaches (Forward Selection, Backward Elimination, and Stepwise) by assessing their size and power properties at different sample sizes based on Monte Carlo simulations, and second was the assessment of the same based on real economic data. The second aim was achieved by the evaluation of the long-run relationship between three pairs of macroeconomic variables, i.e., Energy Consumption and GDP, Oil Price and GDP, and Broad Money and GDP for BRICS (Brazil, Russia, India, China and South Africa) countries using Bounds cointegration test. It was found that information criteria and structured procedures have the same powers for a sample size of 50 or greater. However, BICC and Stepwise are better at small sample sizes. In the light of simulation and real data results, a modified Bounds test with Stepwise model selection procedure may be used as it is strongly theoretically supported and avoids noise in the model selection process.

The purpose of using the criteria for selecting models is to determine an appropriate model that leads to estimates that we can use in making future predictions. In this study, we presented a number of information criteria that help to choose the best model in the time series, such as Akaike information criterion (AIC), Bayesian information criterion ( BIC), Akaike corrected information criterion (AICC) and Pham information criterion (PIC). We aim to obtain a appropriate model for a time series used to predict the lowest temperatures in Erbil governorate for the next three years depending on the Box-Jenkins methodology for the purpose of building time series models. The best model was chosen from among the estimated models based on the value of the aforementioned statistical criteria. The model was used to predict the lowest temperatures for the next three years in Erbil governorate, as the predictive values ​​were consistent with the real values ​​of temperature ranges.

This report documents the Multi-Model Analysis (MMA) computer code. MMA can be used to evaluate results from alternative models of a single system using the same set of observations for all models. As long as the observations, the observation weighting, and system being represented are the same, the models can differ in nearly any way imaginable. For example, they may include different processes, different simulation software, different temporal definitions (for example, steady-state and transient models could be considered), and so on. The multiple models need to be calibrated by nonlinear regression. Calibration of the individual models needs to be completed before application of MMA. MMA can be used to rank models and calculate posterior model probabilities. These can be used to (1) determine the relative importance of the characteristics embodied in the alternative models, (2) calculate model-averaged parameter estimates and predictions, and (3) quantify the uncertainty of parameter estimates and predictions in a way that integrates the variations represented by the alternative models. There is a lack of consensus on what model analysis methods are best, so MMA provides four default methods. Two are based on Kullback-Leibler information, and use the AIC (Akaike Information Criterion) or AICc (second-order-bias-corrected AIC) model discrimination criteria. The other two default methods are the BIC (Bayesian Information Criterion) and the KIC (Kashyap Information Criterion) model discrimination criteria. Use of the KIC criterion is equivalent to using the maximum-likelihood Bayesian model averaging (MLBMA) method. AIC, AICc, and BIC can be derived from Frequentist or Bayesian arguments. The default methods based on Kullback-Leibler information have a number of theoretical advantages, including that they tend to favor more complicated models as more data become available than do the other methods, which makes sense in many situations. Many applications of MMA will be well served by the default methods provided. To use the default methods, the only required input for MMA is a list of directories where the files for the alternate models are located. Evaluation and development of model-analysis methods are active areas of research. To facilitate exploration and innovation, MMA allows the user broad discretion to define alternatives to the default procedures. For example, MMA allows the user to (a) rank models based on model criteria defined using a wide range of provided and user-defined statistics in addition to the default AIC, AICc, BIC, and KIC criteria, (b) create their own criteria using model measures available from the code, and (c) define how each model criterion is used to calculate related posterior model probabilities. The default model criteria rate models are based on model fit to observations, the number of observations and estimated parameters, and, for KIC, the Fisher information matrix. In addition, MMA allows the analysis to include an evaluation of estimated parameter values. This is accomplished by allowing the user to define unreasonable estimated parameter values or relative estimated parameter values. An example of the latter is that it may be expected that one parameter value will be less than another, as might be the case if two parameters represented the hydraulic conductivity of distinct materials such as fine and coarse sand. Models with parameter values that violate the user-defined conditions are excluded from further consideration by MMA. Ground-water models are used as examples in this report, but MMA can be used to evaluate any set of models for which the required files have been produced. MMA needs to read files from a separate directory for each alternative model considered. The needed files are produced when using the Sensitivity-Analysis or Parameter-Estimation mode of UCODE_2005, or, possibly, the equivalent capability of another program. MMA is constructed using

This article presents a simple and efficient method to detect multiple outliers using a modification of the Akaike's Information Criterion. The proposed method is exemplified using various examples and implemented in the statistical package R. The core features of the method are that it depends on simple computations, it does not depend on traditional p values, and it signals the absence of outliers. The proposed procedure is applicable to analyse data in all sciences.