Ridge Regression and PLS Regression

ABSTRACTThis article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.

Abstract. Accurate quantification of long-term trends in stratospheric ozone can be challenging due to their sensitivity to natural variability, the quality of the observational datasets, and non-linear changes in forcing processes as well as the statistical methodologies. Multivariate linear regression (MLR) is the most commonly used tool for ozone trend analysis; however, the complex coupling in many atmospheric processes can make it prone to the issue of over-fitting when using the conventional ordinary-least-squares (OLS) approach. To overcome this issue, here we adopt a regularized (ridge) regression method to estimate ozone trends and quantify the influence of individual processes. We use the Stratospheric Water and OzOne Satellite Homogenized (SWOOSH) merged dataset (v2.7) to derive stratospheric ozone profile trends for the period 1984–2020. Besides SWOOSH, we also analyse a machine-learning-based satellite-corrected gap-free global stratospheric ozone profile dataset from a chemical transport model (ML-TOMCAT) and output from a chemical transport model (TOMCAT) simulation forced with European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 reanalysis. For 1984–1997, we observe smaller negative trends in the SWOOSH stratospheric ozone profile using ridge regression compared to OLS. Except for the tropical lower stratosphere, the largest differences arise in the mid-latitude lowermost stratosphere (>4 % per decade difference at 100 hPa). From 1998 and the onset of ozone recovery in the upper stratosphere, the positive trends estimated using the ridge regression model (∼1 % per decade near 2 hPa) are smaller than those using OLS (∼2 % per decade). In the lower stratosphere, post-1998 negative trends with large uncertainties are observed and ridge-based trend estimates are somewhat smaller and less variable in magnitude compared to the OLS regression. Aside from the tropical lower stratosphere, the largest difference is around 2 % per decade at 100 hPa (with ∼3 % per decade uncertainties for individual trends) in northern mid-latitudes. For both time periods the SWOOSH data produce large negative trends in the tropical lower stratosphere with a correspondingly large difference between the two trend methods. In both cases the ridge method produces a smaller trend. The regression coefficients from both OLS and ridge models, which represent ozone variations associated with natural processes (e.g. the quasi-biennial oscillation, solar variability, El Niño–Southern Oscillation, Arctic Oscillation, Antarctic Oscillation, and Eliassen–Palm flux), highlight the dominance of dynamical processes in controlling lower-stratospheric ozone concentrations. Ridge regression generally yields smaller regression coefficients due to correlated explanatory variables, and care must be exercised when comparing fit coefficients and their statistical significance across different regression methods. Comparing the ML-TOMCAT-based trend estimates with the ERA5-forced model simulation, we find ML-TOMCAT shows significant improvements with much better consistency with the SWOOSH dataset, despite the ML-TOMCAT training period overlapping with SWOOSH only for the Microwave Limb Sounder (MLS) measurement period. The largest inconsistencies with respect to SWOOSH-based trends post-1998 appear in the lower stratosphere where the ERA5-forced model simulation shows positive trends for both the tropics and the mid-latitudes. The large differences between satellite-based data and the ERA5-forced model simulation confirm significant uncertainties in ozone trend estimates, especially in the lower stratosphere, underscoring the need for caution when interpreting results obtained with different regression methods and datasets.

Common framework for linear regression

Unification of neural and statistical methods as applied to materials structure-property mapping

We present four unique prediction techniques, combined with multiple data pre-processing methods, utilizing a wide range of both oil types and oil peroxide values (PV) as well as incorporating natural aging for peroxide creation. Samples were PV assayed using a standard starch titration method, AOCS Method Cd 8-53, and used as a verified reference method for PV determination. Near-infrared (NIR) spectra were collected from each sample in two unique optical pathlengths (OPLs), 2 and 24 mm, then fused into a third distinct set. All three sets were used in partial least squares (PLS) regression, ridge regression, LASSO regression, and elastic net regression model calculation. While no individual regression model was established as the best, global models for each regression type and pre-processing method show good agreement between all regression types when performed in their optimal scenarios. Furthermore, small spectral window size boxcar averaging shows prediction accuracy improvements for edible oil PVs. Best-performing models for each regression type are: PLS regression, 25 point boxcar window fused OPL spectral information RMSEP = 2.50; ridge regression, 5 point boxcar window, 24 mm OPL, RMSEP = 2.20; LASSO raw spectral information, 24 mm OPL, RMSEP = 1.80; and elastic net, 10 point boxcar window, 24 mm OPL, RMSEP = 1.91. The results show promising advancements in the development of a full global model for PV determination of edible oils.

A MATLAB toolbox for multivariate regression coupled with variable selection

Partial Least Squares - Diffusion Tensor Imaging (PLS-DTI): A novel approach for biomarker imaging in breast cancer

In the contemporary real estate market, housing prices are fiercely competitive and influenced by various factors, such as the criminal activities existing in the community and the quality of the surrounding environment. However, inaccuracies in predicting house prices often lead to problems such as excessively high prices. The application of corresponding machine learning models plays a significant role in improving predictive accuracy in this domain. This study compares multiple models, including Ordinary Least Squares (OLS), Ridge regression, Lasso, Elastic Net, Random Forest, and Multilayer Perceptron (MLP), using a real-world housing price dataset. Model performance is evaluated through the accuracy, R², RMSE, and MSE. The final results indicate that the Random Forest outperforms the other methods, achieving the highest predictive accuracy, while Ordinary Least Squares produces the lowest accuracy. Additionally, the study examines the applicability of these techniques across various scenarios and their potential to enhance the accuracy of housing price predictions.

Application of the Internet of things (IoT) for data collection in solar drying can be very efficient in collecting big data of drying parameters. There are many variables involved so it is hard to find a model to predict the moisture content of the food product during drying. In model building, interaction terms should be incorporated because they also contribute to the model. Eight selection criteria (8SC) is a very useful method in model building. This study applied ordinary least squares (OLS) regression and ridge regression with 8SC in model building to predict the moisture content of drying fish. A total of eighty models were considered in this study. One best model was chosen each from OLS regression and ridge regression. M78.7.3 with a total of eleven independent variables was the best OLS model after conducting multicollinearity and coefficient test. Next, the best ridge model M56.0.0 was obtained after the coefficient test. The mean absolute percentage error (MAPE) was used to measure the accuracy of the prediction model. For OLS model M78.7.3, the MAPE value was 15.7342. The MAPE value for ridge model M56.0.0 was 17.4054. From the MAPE value, OLS model M78.7.3 provided a better estimation than the ridge model M56.0.0. However, OLS model M78.7.3 violated the normality assumptions of residuals. This is highly caused by the outlier problem. So, due to non- normality of the residuals and presence of outliers in the dataset, ridge regression is preferred for the best forecast model.

Regional regression models can be employed to estimate hydrologic statistics at ungauged river sites. The increased availability of watershed characteristics and the use of GIS-based methods to process this data have created situations where a large number of highly correlated watershed characteristics are available as potential model explanatory variables. The use of ordinary least squares (OLS) regression procedures with highly correlated variables can produce multicollinearity, creating highly sensitive parameter estimators with inflated variances, and improper model selection. A Monte Carlo simulation is developed to compare four techniques for handling multicollinearity: OLS, OLS with Variance Inflation Factor screening (VIF), principal component regression (PCR), and partial least squares regression (PLS). Results show the impact of multicollinearity is magnified at smaller samples sizes, higher correlations, and larger model error variances. Although PCR and PLS yield parameter estimators with reduced variances, there is no improvement in model prediction compared to OLS and VIF. Using VIF for screening variables produced small improvements in model predictions. The use of OLS appears warranted as the complexity of using biased regression techniques to address multicollinearity does little to improve model predictions.

ABSTRACTWe present algorithms that substantially accelerate partition‐based cross‐validation for machine learning models that require matrix products and . Our algorithms have applications in model selection for, for example, principal component analysis (PCA), principal component regression (PCR), ridge regression (RR), ordinary least squares (OLS), and partial least squares (PLS). Our algorithms support all combinations of column‐wise centering and scaling of and , and we demonstrate in our accompanying implementation that this adds only a manageable, practical constant over efficient variants without preprocessing. We prove the correctness of our algorithms under a fold‐based partitioning scheme and show that the running time is independent of the number of folds; that is, they have the same time complexity as that of computing and and space complexity equivalent to storing , and . Importantly, unlike alternatives found in the literature, we avoid data leakage due to preprocessing. We achieve these results by eliminating redundant computations in the overlap between training partitions. Concretely, we show how to manipulate and using only samples from the validation partition to obtain the preprocessed training partition‐wise and . To our knowledge, we are the first to derive correct and efficient cross‐validation algorithms for any of the 16 combinations of column‐wise centering and scaling, for which we also prove only 12 give distinct matrix products.

Maintaining multivariate calibrations is essential and involves keeping models developed on an instrument applicable to predicting new samples over time. Sometimes a primary instrument model is needed to predict samples measured on secondary instruments. This situation is referred to as calibration transfer. This paper reports on using a Tikhonov regularization (TR) based method in both cases. A distinction of the TR design for calibration maintenance and transfer is a defined weighting scheme for a small set of new (transfer or standardization) samples augmented to the full set of calibration samples. Because straight application of basic TR theory is not always possible with calibration maintenance and transfer, this paper develops a generic solution to always enable application of TR. Harmonious (bias/variance tradeoff) and parsimonious (effective rank) considerations for TR are compared with the same TR format applied to partial least squares (PLS), showing that both approaches are viable solutions to the calibration maintenance and transfer problems.

Improving plant biomass estimation in the field using partial least squares regression and ridge regression

The paper considers various adhoc assumptions to estimate the ridge parameter k, when the mean squared error (MSE) of ridge estimator is less than the ordinary least squares (OLS) estimator. We consider the optimal ridge parameter k in the jackknife

The objective of using regression is to explain the variation in one or more response variables by associating the variation with proportional variation in one or more explanatory variables. However, if the number of independent variables is multiple, they tend to be highly collinear and this contributes to multicollinearity problem. For instance, Ridge Regression (RR), Principal Component Regression (PCR), and Partial Least Squares Regression (PLSR) are some of the prediction methods used to handle dataset with multicollinearity. In addition, another problem that arises is the existence of outlying objects in a dataset. The effect of outlying data points in the presence of multicollinearity problem could be reduced with the implementation of robust regression method. A recently studied robust PLSR, which is called Partial Robust M-Regression (PRM) is found to be able in dealing with multicollinearity and outliers simultaneously. This method was employed in this study. Throughout this study, five methods of regression were chosen; OLS, RR, PCR, PLSR, and PRM, to compare which is the best method in their predictive ability. To compare these five regression methods, a simulation study had been conducted, and the mean square error of β estimate (MSE(β)) was calculated. The simulation results show that PRM outperforms other method in the presence of multicollinearity, outliers and leverage points.