Graphical group ridge

ABSTRACTThis article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.

This paper introduces a new class of efficient and debiased two-step shrinkage estimators for a linear regression model in the presence of multicollinearity. We derive the proposed estimators’ mean square error and define the necessary and sufficient conditions for superiority over the existing estimators. In addition, we develop an algorithm for selecting the shrinkage parameters for the proposed estimators. The comparison of the new estimators versus the traditional ordinary least squares, ridge regression, Liu, and the two-parameter estimators is done by a matrix mean square error criterion. The Monte Carlo simulation results show the superiority of the proposed estimators under certain conditions. In the presence of high but imperfect multicollinearity, the two-step shrinkage estimators’ performance is relatively better. Finally, two real-world chemical data are analyzed to demonstrate the advantages and the empirical relevance of our newly proposed estimators. It is shown that the standard errors and the estimated mean square error decrease substantially for the proposed estimator. Hence, the precision of the estimated parameters is increased, which of course is one of the main objectives of the practitioners.

The present investigation deals with the use of ridge regression in the presence of high degree multicollinearity, to determine the major factors influencing the agricultural production in Haryana state, India. Time series data for the period (1980–1981 to 2017–2018) on area, production, yield and different factors influencing the production of wheat, gram, paddy and cotton have been used in this study. Time series data was divided into four sub-periods of 10 year interval to measure the variability in area, production and yield of selected crops into different periods. Coefficient of variation around trend was also observed as the stability (or instability) index. Estimates for the factors affecting the agricultural production were observed by using the ordinary least squares technique and ridge regression both. A comparison is made and ridge estimates were found appropriate to measure the effect of different factors on agricultural production in the presence of multicollinearity. Total study period has been separated into training and testing parts and validity of the developed ridge regression model has been done on the basis of adjusted R2, percent deviation of forecast values and RMSE (%).

Multicollinearity arises when two or more regressors are correlated in multiple linear regression model (MLRM) and in most cases, one regressor variable can be predicted from another. Multicollinearity majorly results in inefficient regression model estimates and poor performance of the regression model. However, multicollinearity problem can easily be handled using various methods such as ridge regression, lasso regression, principal components regression, etc. This study compared the effectiveness of two estimators in handling multicollinearity problem in a given dataset. The estimators being compared are ridge estimator (RE) and principal components estimator (PCE). This research uses secondary data obtained from World Bank database, International Monetary Fund (IMF) database, and the Nigerian Debt Management Office to compare the two approaches of handling multicollinearity problem in MLRM. The presence of multicollinearity in the dataset was established using the correlation matrix of predictors and the Variance Inflation Factors (VIF's). Then ridge regression and principal components regression methods were used to fit models to the dataset respectively and their mean squared errors (MSE) were obtained. The MSE was used as performance evaluation measure for the regression models. Both methods addressed the problem multicollinearity in the datasets but the ridge estimator performed better than PCE by having the smallest mean squared error.

This paper introduces the Lasso and Ridge Regression methods, which are two popular regularization approaches. The method they give a penalty to the coefficients differs in both of them. L1 Regularization refers to Lasso linear regression, while L2 Regularization refers to Ridge regression. As we all know, regression models serve two main purposes: explanation and prediction of scientific phenomena. Where prediction accuracy will be optimized by balancing each of the bias and variance of predictions, while explanation will be gained by constructing interpretable regression models by variable selection. The penalized regression method, also known as Lasso regression, adds bias to the model's estimates and reduces variance to enhance prediction. Ridge regression, on the other hand, introduces a minor amount of bias in the data to get long-term predictions. In the presence of multicollinearity, both regression methods have been offered as an alternative to the least square approach (LS). Because they deal with multicollinearity, they have the appropriate properties to reduce numerical instability caused by overfitting. As a result, prediction accuracy can be improved. For this study, the Corona virus disease (Covid-19) dataset was used, which has had a significant impact on global life. Particularly in our region (Kurdistan), where life has altered dramatically and many people have succumbed to this deadly sickness. Our data is utilized to analyze the benefits of each of the two regression methods. The results show that the Lasso approach produces more accurate and dependable or reliable results in the presence of multicollinearity than Ridge and LS methods when compared in terms of accuracy of predictions by using NCSS10, EViews 12 and SPSS 25.

An Australian risk prediction model for 30-day mortality after isolated coronary artery bypass: The AusSCORE

Logistic regression is a widely used method to model categorical response data, and maximum likelihood (ML) estimation has widespread use in logistic regression. Although ML method is the most used method to estimate the regression coefficients in logistic regression model, multicollinearity seriously affects the ML estimator. To remedy the undesirable effects of multicollinearity, estimators alternative to ML are proposed. Drawing on the similarities between the multiple linear and logistic regressions, ridge, Liu and two parameter estimators are proposed which are based on the ML estimator. On the other hand, first-order approximated ridge estimator is proposed for use in logistic regression. This study will present further solutions to the problem in the form of alternative estimators which reduce the effect of collinearity. Owing to this, first-order approximated Liu, iterative Liu and iterative two parameter estimators are proposed. A simulation study as well as real life application are carried out to ascertain the effect of sample size and degree of multicollinearity, in which the ML based, first-order approximated and iterative biased estimators are compared. Graphical representations are presented which support the effect of the shrinkage parameter on the mean square error and prediction mean square error of the biased estimators.

In Multiple regression analysis, it is assumed that the independent variables are uncorrelated with one another, when such happen, the problem of multicollinearity occurs. Multicollinearity can create inaccurate estimates of the regression coefficients, inflate the standard errors of the regression coefficients, deflate the partial t-tests for the regression coefficients, give false p-values and degrade the predictability of the model. There are several methods to get rid of this problem and one of the most famous one is the ridge regression. The purpose of this research is to study the performance of some popular ridge regression estimators based on the effects of sample sizes and correlation levels on their Average Mean Square Error (AMSE) for Poisson Regression models in the presence of multicollinearity. As performance criteria, average MSE of k was used. A Monte Carlo simulation study was conducted to compare performance of Fifty (50) k estimators under four experimental conditions namely: correlation, Number of explanatory variables, sample size and intercept. From the results of the analysis as summarized in the Tables, the MSE of the estimators performed better in a lower explanatory variables <i>p</i> and an increased intercept value. It was also observed that some estimators performed better on the average at all correlation levels, sample sizes, intercept values and explanatory variables than others.

In this paper, detailed comparisons are given between those estimators that can be derived from the principal component two-parameter estimator such as the ordinary least squares estimator, the principal components regression estimator, the ridge regression estimator, the Liu estimator, the r-k estimator and the r-d estimator by the prediction mean square error criterion. In addition, conditions for the superiority of the principal component two-parameter estimator over the others are obtained. Furthermore, a numerical example study is conducted to compare these estimators under the prediction mean squared error criterion.

We consider the polynomial regression model. In this model a hierarchical structure, or natural ordering, in the parameter space can be assumed. Maximum likelihood estimators may be found for the parameters of each order model in the hierarchy. We introduce the class of estimators given by weighted combinations of these maximum likelihood estimators under certain restrictions. This class is obtained by considering a Bayes estimator class and contains the subset regression estimator as a special case. The optimal weights which minimize the predictive mean squared error are obtained exactly, using an alternative method to that of Kanda (1985). The estimated weights which minimize the estimated predictive mean square error in a similar way to the Mallow’s Cp-statistic are also exactly presented and some numerical examples are shown.

Model selection is an important part of any statistical analysis. Many tools are suggested for selecting the best model including frequentist and Bayesian perspectives. There is often a considerable uncertainty in the selection of a particular model to be the best approximating model. Model selection uncertainty arises when the data are used for both model selection and parameter estimation. Bias in estimators of model parameters often arise when data based selection has been done. Therefore, model averaging of the parameter estimators will be done to alleviate the bias in model selection in a set of candidate models, by combining the information from a set of candidate models. This paper is two-fold, new criteria of model selection are proposed based on different averages of AIC, BIC, AICc, and HQC. Also, model averaging is introduced to compare the parameter estimators in model averaging with the ones in model selection. Two Simulation studies are considered, the first is for model selection and showed that the new proposed criteria are lies between some of the known criteria such as AIC, BIC, AICc, and HQC, and so they can be used as new criteria of model selection. The second simulation study is for model averaging and showed that the parameter estimators have less bias and less predicted mean square error (PMSE) compared with the parameter estimators in model selection.

Bridge regression, a special family of penalized regressions of a penalty function Σ|βj|γ with γ ≤ 1, considered. A general approach to solve for the bridge estimator is developed. A new algorithm for the lasso (γ = 1) is obtained by studying the structure of the bridge estimators. The shrinkage parameter γ and the tuning parameter λ are selected via generalized cross-validation (GCV). Comparison between the bridge model (γ ≤ 1) and several other shrinkage models, namely the ordinary least squares regression (λ = 0), the lasso (γ = 1) and ridge regression (γ = 2), is made through a simulation study. It is shown that the bridge regression performs well compared to the lasso and ridge regression. These methods are demonstrated through an analysis of a prostate cancer data. Some computational advantages and limitations are discussed.

SummaryThe objective of this study was to compare accuracies of different Bayesian regression models in predicting molecular breeding values for health traits in Holstein cattle. The dataset was composed of 2505 records reporting the occurrence of retained fetal membranes (RFM), metritis (MET), mastitis (MAST), displaced abomasum (DA), lameness (LS), clinical endometritis (CE), respiratory disease (RD), dystocia (DYST) and subclinical ketosis (SCK) in Holstein cows, collected between 2012 and 2014 in 16 dairies located across the US. Cows were genotyped with the Illumina BovineHD (HD, 777K). The quality controls for SNP genotypes were HWE P‐value of at least 1 × 10−10; MAF greater than 0.01 and call rate greater than 0.95. The fimpute program was used for imputation of missing SNP markers. The effect of each SNP was estimated using the Bayesian Ridge Regression (BRR), Bayes A, Bayes B and Bayes Cπ methods. The prediction quality was assessed by the area under the curve, the prediction mean square error and the correlation between genomic breeding value and the observed phenotype, using a leave‐one‐out cross‐validation technique that avoids iterative cross‐validation. The highest accuracies of predictions achieved were: RFM [Bayes B (0.34)], MET [BRR (0.36)], MAST [Bayes B (0.55), DA [Bayes Cπ (0.26)], LS [Bayes A (0.12)], CE [Bayes A (0.32)], RD [Bayes Cπ (0.23)], DYST [Bayes A (0.35)] and SCK [Bayes Cπ (0.38)] models. Except for DA, LS and RD, the predictive abilities were similar between the methods. A strong relationship between the predictive ability and the heritability of the trait was observed, where traits with higher heritability achieved higher accuracy and lower bias when compared with those with low heritability. Overall, it has been shown that a high‐density SNP panel can be used successfully to predict genomic breeding values of health traits in Holstein cattle and that the model of choice will depend mostly on the genetic architecture of the trait.

This study relates to predicting hydrogen production using deep learning models. The co-gasification of biomass and plastics dataset used gasification temperature, particle size of biomass rubber seed shell (RSS) and High-Density Polyethylene (HDPE), and the amount of plastic in the mixture as the independent variables, and the amount of hydrogen produced as the dependent variable. It was found that during the co-gasification particle size is a controlling factor for hydrogen production due to the influence on surface reactions, while temperature had no significant effect. The neural network models were developed using Keras and two different architectures were compared with and without L1 and L2 regularizers. The values for L1 and L2 are determined using the gridserach: for the 1 archtecture, the ideal L1 value = 0.010; and the ideal L2 value = 0.000001 and for the 2nd architecture, The ideal L1 value is 0.100; and the ideal L2 value is 0.000010 using the lowest mean squared error values for the test sets. The mean cross-validation scores indicated that the second architecture performed better. The mean cross_val_score using the negative mean square error, for the 1st architecture, with l2 regularizers (0.000001) is determined as −20.05 (13.10) nMSE for Kfold, 10; and for the 2nd architecture l2 regularizers (0.000010) as −8.22 (7.77) nMSE for Kfold, 10, indicate the 2nd architecture performs better. The best model parameters for both architectures were determined using Grid Search CV. The best model hyperparameters using Grid Search is batch_size, 3; epochs,100; optimizer, rmsprop for the first architechure with negative mean square error, −20.95; and for the 2nd architecture, batch_size, 5; epochs,100; optimizer, adam with negative mean square error, −7.38, indicating the 2nd architecture to be a better model. The Keras Wrapper improved the performance of the model for the first architecture, but not for the second architecture. The permutation feature importance for architecture 1 (in descending order) is: size of RSS, size of HDPE, per cent plastics in mixture and temperature. For architecture 2, in descending order: size of HDPE, size of RSS, per cent plastics in mixture and temperature. Overall, the study demonstrates the potential of deep learning models for predicting hydrogen production.

We present a decomposition of prediction error for the multilevel model in the context of predicting a future observable y *j in the jth group of a hierarchical dataset. The multilevel prediction rule is used for prediction and the components of prediction error are estimated via a simulation study that spans the various combinations of level-1 (individual) and level-2 (group) sample sizes and different intraclass correlation values. Additionally, analytical results present the increase in predicted mean square error (PMSE) with respect to prediction error bias. The components of prediction error provide information with respect to the cost of parameter estimation versus data imputation for predicting future values in a hierarchical data set. Specifically, the cost of parameter estimation is very small compared to data imputation.