Subset selection by Mallows’ [formula omitted]: A mixed integer programming approach

Mixed integer second-order cone programming formulations for variable selection in linear regression

The problem of variable selection is one of the most pervasive model selection problems in statistical applications. Often referred to as the problem of subset selection, it arises when one wants to model the relationship between a variable of interest and a subset of potential explanatory variables or predictors, but there is uncertainty about which subset to use. Several papers have dealt with various aspects of the problem but it appears that the typical regression user has not benefited appreciably. One reason for the lack of resolution of the problem is the fact that it is has not been well defined. Indeed, it is apparent that there is not a single problem, but rather several problems for which different answers might be appropriate. The intent of this chapter is not to give specific answers but merely to present a new simple multiplicative variable selection criterion based on the parametrically penalized residual sum of squares to address the subset selection problem in multiple linear regression analysis, where the objective is to select a minimal subset of predictor variables without sacrificing any explanatory power. The variables, which optimize this criterion, are chosen to be the best variables. The authors find that the proposed criterion performs consistently well across a wide variety of variable selection problems. Practical utility of this criterion is demonstrated by numerical examples.

Home care patients experience a high illness burden and a heightened risk of negative health outcomes. There is a need to understand how clinician behaviors, including notetaking and care coordination activities, can be better integrated into risk modelling in home care settings. This study predicted hospitalization/ED visits using data from 86,866 home care episodes from an agency in the Northeastern United States. We began our analysis with nearly 200 variables, including patient-related factors (e.g., comorbidities, types of medications taken, functional status), and factors relating to clinician behavior, such as problems documented in clinical notes, and calling patterns to the patient or medical care providers. We used least absolute shrinkage and selection operator (LASSO) to aid in the selection of the best subset of variables. In our preliminary analysis, we found that two clinician behavior-related factors were the strongest risk factors for experiencing hospitalization/ED visits, specifically the proportion of concerning notes generated during the care episode (OR 8.97, 95% CI 7.64, 10.52), and a measure reflecting frequent calls to the patient’s family (OR 4.51, 95% CI 3.80, 5.37). Clinician documentation of specific problems, and the clinician’s decision to conduct a medication review were also risk factors. Patient risk factors included prior hospitalization, health conditions such as cancer and renal disease), functional limitations, certain medication classes (e.g., anticoagulants, diuretics), the use of oxygen, and the use of a urinary catheter. This study demonstrates that clinician behaviors and documentation patterns can be incorporated in risk models to yield important insights.

This paper presents a novel Bayesian variable selection approach that accounts for the sign of the regression coefficients based on multivariate one-sided tests. We propose a truncated g prior to specify a prior distribution of coefficients with anticipated signs in a given model. Informative priors for the direction of the effects can be incorporated into prior model probabilities. The best subset of variables is selected by comparing the posterior probabilities of the possible models. The new Bayesian one-sided variable selection procedure has higher chance to include relevant variables and therefore select the best model, if the anticipated direction is accurate. For a large number of candidate variables, we present an adaptation of a Bayesian model search method for the one-sided variable selection problem to ensure fast computation. In addition, a fully Bayesian approach is used to adjust the prior inclusion probability of each one-sided model to correct for multiplicity. The performance of the proposed method is investigated using several simulation studies and two real data examples.

We consider the fundamental problem of estimating the mean of a vector y=Xβ+z, where X is an n×p design matrix in which one can have far more variables than observations, and z is a stochastic error term -— the so-called “p>n” setup. When β is sparse, or, more generally, when there is a sparse subset of covariates providing a close approximation to the unknown mean vector, we ask whether or not it is possible to accurately estimate Xβ using a computationally tractable algorithm. We show that, in a surprisingly wide range of situations, the lasso happens to nearly select the best subset of variables. Quantitatively speaking, we prove that solving a simple quadratic program achieves a squared error within a logarithmic factor of the ideal mean squared error that one would achieve with an oracle supplying perfect information about which variables should and should not be included in the model. Interestingly, our results describe the average performance of the lasso; that is, the performance one can expect in an vast majority of cases where Xβ is a sparse or nearly sparse superposition of variables, but not in all cases. Our results are nonasymptotic and widely applicable, since they simply require that pairs of predictor variables are not too collinear.

Machine learning techniques are becoming indispensable tools for extracting useful information. Among many machine learning techniques, variable selection is a solution used for converting high-dimensional data into simpler data while still preserving the characteristics of the original data. Variable selection aims to find the best subset of variables that produce the smallest generalization error; it can also reduce computational complexity, storage, and costs. The variable selection method developed in this paper was part of a latent class cluster (LCC) analysis—i.e., it was not a pre-processing step but, instead, formed part of LCC analysis. Many studies have shown that variable selection in LCC analysis suffers from computational problems and has difficulty meeting local dependency assumptions—therefore, in this study, we developed a method for selecting variables using mutual information (MI) in LCC analysis. Mutual information (MI) is a symmetrical measure of information that is carried by two random variables. The proposed method was applied to MI-based variable selection in LCC analysis, and, as a result, four variables were selected for use in LCC-based village clustering.

Model selection is an important and challenging problem in statistics. The model selection is inevitable in a large number of applications including life sciences, social sciences, business, or economics. In this article, we propose a resampling-based information criterion called paired bootstrap criterion (PBC) for model selection. The proposed criterion is based on minimizing the conditional expected prediction loss for selecting the best subset of variables. We estimate the conditional expected prediction loss by using the out-of-bag (OOB) bootstrap approach. Other classical criteria for model selection such as AIC, BIC are also presented for comparison purpose. We demonstrate that the proposed paired bootstrap model selection criterion is effective in selecting accurate models via real and simulated data examples. The results confirm the satisfactory behavior of the proposed model selection criterion to select parsimonious models that fit the data well. We apply the proposed methodology to a real data example.

This paper first reduces the problem of detecting structural breaks in a random walk to that of finding the best subset of explanatory variables in a regression model and then tailors various subset selection criteria to this specific problem. Of particular interest are those new criteria, which are obtained by means of simulation using the efficient algorithm of Bai and Perron (J Appl Econom 18:1---22, 2003). Unlike conventional variable selection methods, which penalize new variables entering a model either in the same way (e.g., AIC and BIC) or milder (e.g., MRIC and $$\mathrm {FPE}_\mathrm{{sub}}$$ ) than already included variables, they do not follow any monotonic penalizing scheme. In general, their non-monotonicity is more pronounced in the case of fat tails. The characteristics of the different criteria are illustrated using bootstrap samples from the Nile data set.

The development of new survey data collection methods such as online surveys has been particularly advantageous for social studies in terms of reduced costs, immediacy and enhanced questionnaire possibilities. However, many such methods are strongly affected by selection bias, leading to unreliable estimates. Calibration and Propensity Score Adjustment (PSA) have been proposed as methods to remove selection bias in online nonprobability surveys. Calibration requires population totals to be known for the auxiliary variables used in the procedure, while PSA estimates the volunteering propensity of an individual using predictive modelling. The variables included in these models must be carefully selected in order to maximise the accuracy of the final estimates. This study presents an application, using synthetic and real data, of variable selection techniques developed for knowledge discovery in data to choose the best subset of variables for propensity estimation. We also compare the performance of PSA using different classification algorithms, after which calibration is applied. We also present an application of this methodology in a real-world situation, using it to obtain estimates of population parameters. The results obtained show that variable selection using appropriate methods can provide less biased and more efficient estimates than using all available covariates.

BackgroundBreast cancer (BC) survival prediction can be a helpful tool for identifying important factors selecting the effective treatment reducing mortality rates. This study aims to predict the time-related survival probability of BC patients in different molecular subtypes over 30 years of follow-up.Materials and methodsThis study retrospectively analyzed 3580 patients diagnosed with invasive breast cancer (BC) from 1991 to 2021 in the Cancer Research Center of Shahid Beheshti University of Medical Science. The dataset contained 18 predictor variables and two dependent variables, which referred to the survival status of patients and the time patients survived from diagnosis. Feature importance was performed using the random forest algorithm to identify significant prognostic factors. Time-to-event deep-learning-based models, including Nnet-survival, DeepHit, DeepSurve, NMLTR and Cox-time, were developed using a grid search approach with all variables initially and then with only the most important variables selected from feature importance. The performance metrics used to determine the best-performing model were C-index and IBS. Additionally, the dataset was clustered based on molecular receptor status (i.e., luminal A, luminal B, HER2-enriched, and triple-negative), and the best-performing prediction model was used to estimate survival probability for each molecular subtype.ResultsThe random forest method identified tumor state, age at diagnosis, and lymph node status as the best subset of variables for predicting breast cancer (BC) survival probabilities. All models yielded very close performance, with Nnet-survival (C-index=0.77, IBS=0.13) slightly higher using all 18 variables or the three most important variables. The results showed that the Luminal A had the highest predicted BC survival probabilities, while triple-negative and HER2-enriched had the lowest predicted survival probabilities over time. Additionally, the luminal B subtype followed a similar trend as luminal A for the first five years, after which the predicted survival probability decreased steadily in 10- and 15-year intervals.ConclusionThis study provides valuable insight into the survival probability of patients based on their molecular receptor status, particularly for HER2-positive patients. This information can be used by healthcare providers to make informed decisions regarding the appropriateness of medical interventions for high-risk patients. Future clinical trials should further explore the response of different molecular subtypes to treatment in order to optimize the efficacy of breast cancer treatments.

This brief studies the linear quadratic regulation problem of discrete-time switched linear systems via dynamic programming. Discrete decision variables (referred as switch inputs) are introduced to represent the subsystem selection. This formulation leads to a mixed integer quadratic programming problem, which is known to be computationally hard in general. Instead of deriving suboptimal policies with analytical bounds on its optimality, the optimal switching sequence is computed in a computationally efficient manner when the input matrix is restricted as a vector. The unique contribution of this brief is an analytical expression of both the optimal switching condition (determining the subsystem selection) and the optimal control law. Proof of optimality is presented by fractional optimization and the practical merits of our approach is validated through simulations on a second-order system in comparison with recent pruning methods.

This paper presents a deterministic optimization technique for the optimal location of normally open switch on a real distribution network, located in north of Portugal. The goal is to find the optimal radial topology that minimizes the power losses. The method is developed in TOMLAB software, and is formulated as a mixed integer quadratic programming problem. Given the current characteristics of the network, the method is based on optimal power flow through DC model. The power losses are given by the Joule effect, calculated according to each distribution line. In order to prove the viability of the obtained results, the proposed methodology is compared with a software used by Portuguese operator for distribution networks planning.

In this paper we investigate the stability of discrete-time PWA systems in closed-loop with quadratic cost based model predictive controllers (MPC) and we derive a priori sufficient conditions for Lyapunov asymptotic stability. We prove that Lyapunov stability can be achieved for the closed-loop system even though the considered Lyapunov function and the system dynamics may he discontinuous. The stabilization conditions are derived using a terminal cost and constraint set method. An S-procedure technique is employed to reduce conservativeness of the stabilization conditions and a linear matrix inequalities set-up is developed in order to calculate the terminal cost. A new algorithm for computing piecewise polyhedral positively invariant sets for PWA systems is also presented. In this manner, the on-line optimization problem associated with MPC leads to a mixed integer quadratic programming problem, which can be solved by standard optimization tools.

Model predictive control based on mixed logical dynamical model can deal with the nolinear effectively,avoid oscillation of subsystem switching, get better control effect. But the mixed integer quadratic programming problem caused by the logic variable leads to computation complexity and long resolving time, it is hard to meet the requirement of real-time. In this paper, logic variable in the optimal variable is handled properly, A parabolic function is added in the performance index. Resolving time is greatly reduced and controller performance is improved. Finally the improved algorithm is applied into the continuous stirred tank reactor, Showing that the method has good control effect.

This paper proposes a hybrid stochastic-deterministic approach to realize the observer-based active fault diagnosis (AFD) for linear time varying systems. A novel conception called hybrid zonotope-Gaussian dispersity for random variables of systems is introduced to establish an online optimization strategy for AFD. The optimal gain of observer and the optimal input are simultaneously designed at each step by solving a non-convex quadratic fractional programming problem, which is proved to be equivalent to a mixed integer quadratic programming problem. The proposed approach avoids the strict set-separation constraint conditions of traditional offline AFD technology by making full use of the online measured outputs such that the diagnosisability of systems has potential to be further improved. At the end, a quadrotor model is used to verify the effectiveness of our proposed method.