Alternative ridge robust regression estimator for dealing with collinear influential data points

Say that a regression method is "unstable" at a data set if a small change in the data can cause a relatively large change in the fitted plane. A well-known example of this is the instability of least squares regression (LS) near (multi)collinear data sets. It is known that least absolute deviation (LAD) and least median of squares (LMS) linear regression can exhibit instability at data sets that are far from collinear. Clear-cut instability occurs at a "singularity"--a data set, arbitrarily small changes to which can substantially change the fit. For example, the collinear data sets are the singularities of LS. One way to measure the extent of instability of a regression method is to measure the size of its "singular set" (set of singularities). The dimension of the singular set is a tractable measure of its size that can be estimated without distributional assumptions or asymptotics. By applying a general theorem on the dimension of singular sets, we find that the singular sets of LAD and LMS are at least as large as that of LS and often much larger. Thus, prima facie, LAD and LMS are frequently unstable. This casts doubt on the trustworthiness of LAD and LMS as exploratory regression tools.

Natural mortality rate (M) and fishing catchability coefficient (q) were estimated by using the Paloheimo pairwise linear regression model with simulated and real catch–effort data. The four estimation methods used were least squares (LS), least absolute value (LAV), least median of squares (LMS), and LMS-based reweighted least squares (RLS). When catch or effort data or both contained homogeneous error variances, mean squared errors (MSE) of LS and RLS were significantly smaller than those of LAV and LMS. However, there was no significant difference in MSE between LS and RLS. When catch–effort data contained variances inconsistent among years, MSEs of the robust methods were much smaller than those of LS. Reweighted least squares substantially reduced the variances in the LS-estimated q and M, and it had the smallest MSE among the four estimation methods. We suggest using the following procedures in estimating q and M based on the Paloheimo pairwise regression model: (1) applying LMS to the model to identify outliers in the data; and (2) weighting the defined outliers by a factor of 0 and normal data points by a factor of 1 and applying LS to the model to estimate q and M and their associated variances.

Multicollinearity arises when two or more regressors are correlated in multiple linear regression model (MLRM) and in most cases, one regressor variable can be predicted from another. Multicollinearity majorly results in inefficient regression model estimates and poor performance of the regression model. However, multicollinearity problem can easily be handled using various methods such as ridge regression, lasso regression, principal components regression, etc. This study compared the effectiveness of two estimators in handling multicollinearity problem in a given dataset. The estimators being compared are ridge estimator (RE) and principal components estimator (PCE). This research uses secondary data obtained from World Bank database, International Monetary Fund (IMF) database, and the Nigerian Debt Management Office to compare the two approaches of handling multicollinearity problem in MLRM. The presence of multicollinearity in the dataset was established using the correlation matrix of predictors and the Variance Inflation Factors (VIF's). Then ridge regression and principal components regression methods were used to fit models to the dataset respectively and their mean squared errors (MSE) were obtained. The MSE was used as performance evaluation measure for the regression models. Both methods addressed the problem multicollinearity in the datasets but the ridge estimator performed better than PCE by having the smallest mean squared error.

In the multiple linear regression analysis, the ridge regression estimator and the Liu estimator are often used to address multicollinearity. Besides multicollinearity, outliers are also a problem in the multiple linear regression analysis. We propose new biased estimators based on the least trimmed squares (LTS) ridge estimator and the LTS Liu estimator in the case of the presence of both outliers and multicollinearity. For this purpose, a simulation study is conducted in order to see the difference between the robust ridge estimator and the robust Liu estimator in terms of their effectiveness; the mean square error. In our simulations, the behavior of the new biased estimators is examined for types of outliers: X-space outlier, Y-space outlier, and X-and Y-space outlier. The results for a number of different illustrative cases are presented. This paper also provides the results for the robust ridge regression and robust Liu estimators based on a real-life data set combining the problem of multicollinearity and outliers.

The weighted total least-squares (WTLS) estimate for the partial EIV (PEIV) model is very susceptible to outliers. In view of outliers in observations and coefficient matrix, a robust estimate of WTLS which makes using of iteratively reweighted technique (IRTLS) for the PEIV model is developed by combining a new two-step iterated algorithm of the WTLS estimate with the sensitivity-analysis based on the robust M-estimation. The uniformly most powerful test statistics are constructed to determine the down-weighting factor and the variance of unit weight is estimated by least median squares (LMS) method possessing high break-down point. Depended on the PEIV model, two different down-weighting schemes are presented. In the first scheme down-weighting is only implemented for the coefficient matrix and not for observations when the elements of the coefficient matrix are estimated, and the second scheme is contrary. The most attractive aspect is that the suggested computational formulae are the same with the traditional robust least-squares (LS) methods. The two-dimensional affine transformation and liner fitting example are analyzed, and some comparisons are performed with different available weight functions. The proposed approach (Scheme 1) proves to be a powerful tool in detecting outliers for the PEIV model.

Local Walsh-average regression

The aim of this study is to assess the possibility of forecasting water level fluctuations in a relatively small (<100 km2), post-glacial lake located in a temperate climate zone by means of artificial neural networks and multiple linear regression. The area of study was Lake Serwy, located in northeastern Poland. Two artificial neural network (ANN) multilayer perceptron (MLP) and multiple linear regression (MLR) models were built. The following explanatory variables were considered: maximal and minimal temperature (Tmax, Tmin) wind speed (WS), vertical circulation (VC) and water level from previous periods (WL). Additionally, a binary variable describing the period of the year (winter, summer) has been considered in one of the two MLP and MLR models. The forecasting models have been assessed based on selected criteria: mean absolute percentage error (MAPE), root mean squared error (RMSE), coefficient of determination (R2) and mean biased error. Considering their values and absolute deviations from observed values it was concluded that the ANN model using an additional binary variable (MLP_B+) has the best forecasting performance. Absolute deviations from observed values were the determining factor which made this model the most efficient. In the case of the MLP_B+ model, those values were about 10% lower than in other models. The conducted analyses indicated good performance of ANN networks as a forecasting tool for relatively small lakes located in temperate climate zones. It is acknowledged that they enable water level forecasting with greater precision and lower absolute deviations than the use of multiple linear regression models.

Influential observations (IO) are those observations that are responsible for misleading conclusions about the fitting of a multiple linear regression model. The existing IO identification methods such as influential distance (ID) is not very successful in detecting IO. It is suspected that the ID employed inefficient method with long computational running time for the identification of the suspected IO at the initial step. Moreover, this method declares good leverage observations as IO, resulting in misleading conclusion. In this paper, we proposed fast improvised influential distance (FIID) that can successfully identify IO, good leverage observations, and regular observations with shorter computational running time. Monte Carlo simulation study and real data examples show that the FIID correctly identify genuine IO in multiple linear regression model with no masking and a negligible swamping rate.

Summary In this paper we derive an asymptotic normality result for an adaptive trimmed likelihood estimator of regression starting from initial high breakdownpoint robust regression estimates. The approach leads to quickly and easily computed robust and efficient estimates for regression. A highlight of the method is that it tends automatically in one algorithm to expose the outliers and give least squares estimates with the outliers removed. The idea is to begin with a rapidly computed consistent robust estimator such as the least median of squares (LMS) or least trimmed squares (LTS) or for example the more recent MM estimators of Yohai. Such estimators are now standard in statistics computing packages, for example as in SPLUS or R. In addition to the asymptotics we provide data analyses supporting the new adaptive approach. This approach appears to work well on a number of data sets and is quicker than the related brute force adaptive regression approach described in Clarke (2000). This current approach builds on the work of Bednarski and Clarke (2002) which considered the asymptotics for the location estimator only.

By constructing a correlation network between global sea surface temperature anomalies (SSTAs) and summer precipitation anomalies in the Yunnan–Guizhou Plateau, key SST regions influencing summer precipitation anomalies in the Yunnan–Guizhou Plateau were selected. It was found that spring SSTAs in the Bay of Bengal, southwestern Atlantic, and eastern Pacific are crucial for influencing summer precipitation anomalies in the Yunnan–Guizhou Plateau. Setting SSTAs from these three regions as predictor variables 3 months in advance, we constructed multiple linear regression (MLR), ridge regression (RR), and lasso regression (LR) models to predict summer precipitation anomalies over the Yunnan–Guizhou region. The training phase involved data spanning from 1961 to 2005, which aimed to predict precipitation anomalies in the Yunnan–Guizhou Plateau for the period extending from 2006 to 2022. Based on MLR, RR, and LR models, the correlations between predicted values and observed summer precipitation anomalies in Yunnan–Guizhou were 0.48, 0.46, and 0.46, respectively. These values were all higher than the correlation coefficients of the NCC_CSM model’s predicted and observed values. Additionally, its performance in predicting summer precipitation anomalies over the Yunnan–Guizhou region, based on key SST regions, was assessed using performance metrics such as anomaly correlation coefficient (ACC), anomaly sign consistency rate (PC), and trend anomaly comprehensive score (PS score). The average ACC of MLR, RR, and LR models was higher than that of the NCC_CSM model’s predictions. For MLR, RR, LR, and NCC_CSM models, the PCs exceeding 50% of the year were 14, 14, 11, and 10, respectively. Furthermore, the average PS score for predicting summer precipitation anomalies over the Yunnan–Guizhou region using MLR, RR, and LR was approximately 73 points; 8 higher than the average PS score of the NCC_CSM model. Therefore, predicting summer precipitation anomalies over the Yunnan–Guizhou region based on key SST regions is of great significance for improving the prediction skills of precipitation anomalies in this region.

A brief review of Ridge Regression (RR) and PLS Regression (PLS) is presented. Process and Spectral data are used in the analysis. Both are low-rank data, which is common in chemometric work. The Ridge constant k is determined by minimizing the size of the residuals in Leave-one-out RR. When RR and PLS are applied to the data, we find that RR gives almost the same fit and cross-validation as PLS. However, in applications to test sets the picture is unclear; the results depend both on the test sets and how the analysis is carried out. When comparing RR to PLS, we use from the average of 20 cross-validations. Efficient variable deletion/selection procedures are presented that are based on average cross-validation. When RR and PLS are applied, we find a very small and insignificant difference between RR and PLS. It is shown that RR amounts to adding small ‘noise’ values to X. The Ordinary Least Squares (OLS) solution for the modified X gives the same solution (regression coefficients) as RR. The theory of OLS shows that the RR estimates of the variance of the regression coefficients are too small. It means that we cannot apply the theory of RR in, for instance, analyzing the parameter estimates. RR can be carried out by the same algorithm as PLS. This can be used to show that the same graphic methods can be used at RR as those that are popular in chemometrics. The algorithm can be used to estimate the appropriate dimension to use at RR in a similar way as for PLS. In conclusion, we don’t find a significant difference between RR and PLS, when they are applied to the two types of data. However, the theory of RR does not apply to the analysis of parameter estimates.

This study compares the proposed hybrid method (wavelet robust M-estimation) to the traditional method (wavelet ordinary least square) when there are de-noising or outlier problems for estimating multiple linear regression models using the statistical criterion root mean square error (RMSE). According to simulated and real data, the proposed hybrid method (wavelet robust M-estimation) is better than the classical method (Wavelet Ordinary Least Square) and more accurate. The root mean square error of the proposed hybrid method (wavelet robust M-estimation) is less than the Wavelet Ordinary Least Square. Therefore, it is recommended to use the hybrid proposed method to reduce the problem of outliers and de-noise data.

Objective Knowledge-based radiotherapy (KBRT) can reduce the plan quality variability induced by different experiences between physicians and improve the quality of treatment plans. Methods The Varian Rapid Plan system was used to train a dose-volume histogram (DVH) prediction model. The obtained model was preliminarily applied to semi-automatic design of the preoperative treatment plans for rectal cancer. Eighty high-quality volumetric modulated arc therapy plans were imported into the model training set of the Rapid Plan system. The structures of the plans were matched to the corresponding labels and codes as listed in the library. The training started after the verification of prescription. The residual plots, regression curves, geometric plots for organ at risk (OAR), in-field DVH plots, and model training logs were examined. After removal of the mismatch, the original plans were assessed to rule out outliers and influential data points. More similar plans may be added for another round of training. Ten KBRT plans were designed using the final model and compared with the clinical plans. Results For the two major OARs, the femoral head and bladder, the average goodness of fit of the principal component were 0.999415/1.0 and 0.999963/1.0 for the DVH model, and 0.999651/1.0 and 0.999945/1.0 for geometry-based expected dose model, respectively. In all the plans, 11 had Cook’s distance values exceeding the tolerance and 4 had studentized residual values exceeding the tolerance. The outliers were all kept in the training set to generalize the scope of the model. The 10 KBRT plans had significantly improved homogeneity indices for PGTV and PTV than the original plans (P=0.00, 0.04). The 10 KBRT plans also had significantly reduced D50% to the femoral head and bladder as well as significantly reduced mean doses to the bladder than the original plans (P=0.042, 0.000, 0.005). Conclusions In this study, the Rapid Plan system is used to train a KBRT model for design of preoperative radiotherapy plans for rectal cancer. The results of preliminary application meet the clinical requirements. Key words: Radiotherapy planning; Model training; Model verification

회귀모형을 이용하여 자료를 분석하는 경우 이상점이나 영향점과 같은 특이값들의 유무를 검정하는 회귀진단기법은 모형의 적합성을 체크하기 위한 필수적인 도구로 잡은 지 오래이다. 이러한 점들이 존재 하는 경우 회귀분석의 결과가 왜곡되어 해석이 된다. Jang과 Anderson-Cook (2013)은 불꽃그림이란 이름을 붙인 그림도구를 발표하였는데 관측값에 부여된 가중치를 1에서 0으로 변화함에 따라 이상점이나 영향점이 회귀계수 및 잔차제곱합(SSE)에 어떠한 영향을 미치는지 3차원 그림에 추적곡선을 그려 보았을 뿐 아니라 쌍으로 대비시켜 봄으로써 분석의 시각적인 효과를 증대시켰다. 본 연구에서는 더 나아가 이러한 시도가 기존 방법과 어떤 차이점이 있는지 2013년에는 반영치 않은 통계량을 포함해서 더 많은 해석이 가능한지 혼합물 실험 계획을 통해 다양한 통계량의 민감도 분석을 실행하였다. 왜냐하면 작은 혼합물실험인 자료인 경우 더욱 세밀한 통계량에 대한 민감도 분석이 필요하기 때문이다. It is common to check the validity of an assumed model with the heavy use of diagnostics tools when conducting data analysis with regression techniques; however, outliers and influential data points often distort the regression output in undesired manner. Jang and Anderson-Cook (2013) proposed a graphical method called a firework plot for exploratory analysis that could visualize the trace of the impact of possible outlying and/or influential data points on individual regression coefficients and the overall residual sum of squares(SSE) measure. They developed 3-D plot as well as pair-wise plot for the appropriate measures of interest. In this paper, the approach was extended further to tell the strength of their approach; in addition, a more meaningful interpretation was possible by adding a measure not mentioned in their paper. This approach was applied to the mixture experiment because we felt that a detailed analysis of statistical measure sensitivity is required in a small experiment.

Leverage and influential observations on the Liu type estimator in the linear regression model with the severe collinearity