Common framework for linear regression

A brief review of Ridge Regression (RR) and PLS Regression (PLS) is presented. Process and Spectral data are used in the analysis. Both are low-rank data, which is common in chemometric work. The Ridge constant k is determined by minimizing the size of the residuals in Leave-one-out RR. When RR and PLS are applied to the data, we find that RR gives almost the same fit and cross-validation as PLS. However, in applications to test sets the picture is unclear; the results depend both on the test sets and how the analysis is carried out. When comparing RR to PLS, we use from the average of 20 cross-validations. Efficient variable deletion/selection procedures are presented that are based on average cross-validation. When RR and PLS are applied, we find a very small and insignificant difference between RR and PLS. It is shown that RR amounts to adding small ‘noise’ values to X. The Ordinary Least Squares (OLS) solution for the modified X gives the same solution (regression coefficients) as RR. The theory of OLS shows that the RR estimates of the variance of the regression coefficients are too small. It means that we cannot apply the theory of RR in, for instance, analyzing the parameter estimates. RR can be carried out by the same algorithm as PLS. This can be used to show that the same graphic methods can be used at RR as those that are popular in chemometrics. The algorithm can be used to estimate the appropriate dimension to use at RR in a similar way as for PLS. In conclusion, we don’t find a significant difference between RR and PLS, when they are applied to the two types of data. However, the theory of RR does not apply to the analysis of parameter estimates.

ABSTRACTThis article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.

we propose a regression method using partial least square (PLS) technique and kernel method, which we refer to as kernel partial least square regression II to distinguish the conventional kernel PLS regression. The motivation of the conventional kernel PLS regression is to establish a coordinator system where the independent and dependent variables have a stronger correlation, and which models a regression model using the coordinator system. However, it projects independent variables into a reproducing kernel Hilbert space (RKHS) alone. The proposal extends the basic framework of conventional kernel PLS regression. The proposed method does not only project independent variables into the RKHS, but also dependent variables, and establishes a coordinator system where the independent and dependent variables have a stronger correlation. We use two function regression cases to evaluate the proposed method compared with the conventional kernel PLS regression. The regression performance of the proposed method has almost the same regression accuracy arising from the evaluation result, and this depends on the regression tasks. We explain the correlation calculations of our proposed method, conventional kernel PLS regression, and PLS regression. The meaning of correlation depends on the application in question. We also analyse and discuss the algorithm implementation, correlation meaning, and other issues for further development of the proposal.

The objective of using regression is to explain the variation in one or more response variables by associating the variation with proportional variation in one or more explanatory variables. However, if the number of independent variables is multiple, they tend to be highly collinear and this contributes to multicollinearity problem. For instance, Ridge Regression (RR), Principal Component Regression (PCR), and Partial Least Squares Regression (PLSR) are some of the prediction methods used to handle dataset with multicollinearity. In addition, another problem that arises is the existence of outlying objects in a dataset. The effect of outlying data points in the presence of multicollinearity problem could be reduced with the implementation of robust regression method. A recently studied robust PLSR, which is called Partial Robust M-Regression (PRM) is found to be able in dealing with multicollinearity and outliers simultaneously. This method was employed in this study. Throughout this study, five methods of regression were chosen; OLS, RR, PCR, PLSR, and PRM, to compare which is the best method in their predictive ability. To compare these five regression methods, a simulation study had been conducted, and the mean square error of β estimate (MSE(β)) was calculated. The simulation results show that PRM outperforms other method in the presence of multicollinearity, outliers and leverage points.

Minimum variance based-Bayes Combination for prediction of soil properties on Vis-NIR reflectance spectroscopy

In recent years, there has been increased interest in the development and use of quantitative structure-activity/property relationship (QSAR/QSPR) models. For the most part, this is due to the fact that experimental data is sparse and obtaining such data is costly, while theoretical structural descriptors can be obtained quickly and inexpensively. In this study, three linear regression methods, viz. principal component regression (PCR), partial least squares (PLS), and ridge regression (RR), were used to develop QSPR models for the estimation of human blood:air partition coefficient (logPblood:air) for a group of 31 diverse low-molecular weight volatile chemicals from their computed molecular descriptors. In general, RR was found to be superior to PCR or PLS. Comparisons were made between models developed using parameters based solely on molecular structure and linear regression (LR) models developed using experimental properties, including saline:air partition coefficient (logPsaline:air) and olive oil:air partition coefficient (logPolive oil:air), as independent variables, indicating that the structure-property correlations are comparable to the property-property correlations. The best models, however, were those that used rat logPblood:air as the independent variable. Haloalkane subgroups were modeled separately for comparative purposes and, although models based on the congeneric compounds were superior, the models developed on the complete set of diverse compounds were of acceptable quality. The structural descriptors were placed into one of three classes based on level of complexity: topostructural (TS), topochemical (TC), or three-dimensional/geometrical (3D). Modeling was performed using the structural descriptor classes both in a hierarchical fashion and separately. The results indicate that highest quality structure-based models, in terms of descriptor classes, were those derived using TC descriptors.

Improving plant biomass estimation in the field using partial least squares regression and ridge regression

We present four unique prediction techniques, combined with multiple data pre-processing methods, utilizing a wide range of both oil types and oil peroxide values (PV) as well as incorporating natural aging for peroxide creation. Samples were PV assayed using a standard starch titration method, AOCS Method Cd 8-53, and used as a verified reference method for PV determination. Near-infrared (NIR) spectra were collected from each sample in two unique optical pathlengths (OPLs), 2 and 24 mm, then fused into a third distinct set. All three sets were used in partial least squares (PLS) regression, ridge regression, LASSO regression, and elastic net regression model calculation. While no individual regression model was established as the best, global models for each regression type and pre-processing method show good agreement between all regression types when performed in their optimal scenarios. Furthermore, small spectral window size boxcar averaging shows prediction accuracy improvements for edible oil PVs. Best-performing models for each regression type are: PLS regression, 25 point boxcar window fused OPL spectral information RMSEP = 2.50; ridge regression, 5 point boxcar window, 24 mm OPL, RMSEP = 2.20; LASSO raw spectral information, 24 mm OPL, RMSEP = 1.80; and elastic net, 10 point boxcar window, 24 mm OPL, RMSEP = 1.91. The results show promising advancements in the development of a full global model for PV determination of edible oils.

In the domain of chemometrics and multivariate data analysis, partial least squares (PLS) modelling is a widely used technique. PLS gains its beauty by handling the high collinearity found in multivariate data by replacing highly covarying variables with common subspaces spanned by orthogonal latent variables. Furthermore, all can be achieved with simple steps of linear algebra requiring minimal computation power and time usage compared to current high-end computing and substantial hyperparameter tuning required by methods such as deep learning. PLS can be used for a wide variety of tasks, for example, single block modelling, multiblock modelling, multiway data modelling and for task such as regression and classification. Furthermore, new PLS based approaches can also incorporate meta information to improve the PLS subspace extraction. However, in the current scenario, there is a wide range of separate tools and codes available to perform different PLS tasks. Often when the user needs to perform a new PLS task, they need to start with a separate mathematical implementation of the PLS techniques. This study aims to provide a single solution, i.e., the Swiss knife PLS (SKPLS) modelling approach to enable a single mathematical implementation to perform analyses of single block, multiblock, multiway, multiblock multiway, multi-response, and incorporation of meta information in PLS modelling. It contains all that is needed for any PLS practitioner to perform both classification and regression tasks. The SKPLS backbone is the stepwise PLS strategy called response oriented sequential alternation (ROSA) which we generalize to enable all the mentioned analysis possibilities. The basic structure of the algorithm is highlighted, and some example cases of performing single block, multiblock, multiway, multiblock multiway, multi-response PLS modelling and the incorporation of meta information in PLS modelling are included.

Abstract. Accurate quantification of long-term trends in stratospheric ozone can be challenging due to their sensitivity to natural variability, the quality of the observational datasets, and non-linear changes in forcing processes as well as the statistical methodologies. Multivariate linear regression (MLR) is the most commonly used tool for ozone trend analysis; however, the complex coupling in many atmospheric processes can make it prone to the issue of over-fitting when using the conventional ordinary-least-squares (OLS) approach. To overcome this issue, here we adopt a regularized (ridge) regression method to estimate ozone trends and quantify the influence of individual processes. We use the Stratospheric Water and OzOne Satellite Homogenized (SWOOSH) merged dataset (v2.7) to derive stratospheric ozone profile trends for the period 1984–2020. Besides SWOOSH, we also analyse a machine-learning-based satellite-corrected gap-free global stratospheric ozone profile dataset from a chemical transport model (ML-TOMCAT) and output from a chemical transport model (TOMCAT) simulation forced with European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 reanalysis. For 1984–1997, we observe smaller negative trends in the SWOOSH stratospheric ozone profile using ridge regression compared to OLS. Except for the tropical lower stratosphere, the largest differences arise in the mid-latitude lowermost stratosphere (>4 % per decade difference at 100 hPa). From 1998 and the onset of ozone recovery in the upper stratosphere, the positive trends estimated using the ridge regression model (∼1 % per decade near 2 hPa) are smaller than those using OLS (∼2 % per decade). In the lower stratosphere, post-1998 negative trends with large uncertainties are observed and ridge-based trend estimates are somewhat smaller and less variable in magnitude compared to the OLS regression. Aside from the tropical lower stratosphere, the largest difference is around 2 % per decade at 100 hPa (with ∼3 % per decade uncertainties for individual trends) in northern mid-latitudes. For both time periods the SWOOSH data produce large negative trends in the tropical lower stratosphere with a correspondingly large difference between the two trend methods. In both cases the ridge method produces a smaller trend. The regression coefficients from both OLS and ridge models, which represent ozone variations associated with natural processes (e.g. the quasi-biennial oscillation, solar variability, El Niño–Southern Oscillation, Arctic Oscillation, Antarctic Oscillation, and Eliassen–Palm flux), highlight the dominance of dynamical processes in controlling lower-stratospheric ozone concentrations. Ridge regression generally yields smaller regression coefficients due to correlated explanatory variables, and care must be exercised when comparing fit coefficients and their statistical significance across different regression methods. Comparing the ML-TOMCAT-based trend estimates with the ERA5-forced model simulation, we find ML-TOMCAT shows significant improvements with much better consistency with the SWOOSH dataset, despite the ML-TOMCAT training period overlapping with SWOOSH only for the Microwave Limb Sounder (MLS) measurement period. The largest inconsistencies with respect to SWOOSH-based trends post-1998 appear in the lower stratosphere where the ERA5-forced model simulation shows positive trends for both the tropics and the mid-latitudes. The large differences between satellite-based data and the ERA5-forced model simulation confirm significant uncertainties in ozone trend estimates, especially in the lower stratosphere, underscoring the need for caution when interpreting results obtained with different regression methods and datasets.

The body condition of dairy cows is a crucial health and welfare indicator that is widely acknowledged. Dairy herds with a well-management body condition tend to have more fertile and functional cows. Therefore, routine recording of high-quality body condition phenotypes is required. Automated prediction of body condition from 3D images can be a cost-effective approach to current manual recording by technicians. Using 3D-images, we aimed to build a reliable prediction model of body condition for Jersey cows. The dataset consisted of 808 individual Jersey cows with 2,253 phenotypes from three herds in Denmark. Body condition was scored on a 1 to 9 scale and transformed into a 1 to 5 scale with 0.5-unit differences. The cows' back images were recorded using a 3D camera (Microsoft Xbox One Kinect v2). We used contour and back height features from 3D-images as predictors, together with class predictors (evaluator, herd, evaluation round, parity, lactation week). The performance of machine learning algorithms was assessed using H2O AutoML algorithm (h2o.ai). Based on outputs from AutoML, DeepLearning (DL; multi-layer feedforward artificial neural network) and Gradient Boosting Machine (GBM) algorithms were implemented for classification and regression tasks and compared on prediction accuracy. In addition, we compared the Partial Least Square (PLS) method for regression. The training and validation data were divided either through a random 7:3 split for 10 replicates or by allocating two herds for training and one herd for validation. The accuracy of classification models showed the DL algorithm performed better than the GBM algorithm. The DL model achieved a mean accuracy of 48.1% on the exact phenotype and 93.5% accuracy with a 0.5-unit deviation. The performances of PLS and DL regression methods were comparable, with mean coefficient of determination of 0.67 and 0.66, respectively. When we used data from two herds for training and the third herd as validation, we observed a slightly decreased prediction accuracy compared to the 7:3 split of the dataset. The accuracies for DL and PLS in the herd validation scenario were > 38% on the exact phenotype and > 87% accuracy with 0.5-unit deviation. This study demonstrates the feasibility of a reliable body condition prediction model in Jersey cows using 3D-images. The approach developed can be used for reliable and frequent prediction of cows' body condition to improve dairy farm management and genetic evaluations.

Unemployment is one of the most important issues in macro economics. Unemployment creates many economic and social problems in the economy. The condition and qualification of labor force in a country show economical developments. In the light of these facts, a developing country should overcome the problem of unemployment. In this study, the performance of robust biased Robust Ridge Regression (RRR), Robust Principal Component Regression (RPCR) and RSIMPLS methods are compared with each other and their classical versions known as Ridge Regression (RR), Principal Component Regression (PCR) and Partial Least Squares Regression (PLSR) in terms of predictive ability by using trimmed Root Mean Squared Error (TRMSE) statistic in case of both of multicollinearity and outliers existence in an unemployment data set of Turkey. Analysis results show that RRR model is chosen as the best model for determining unemployment rate in Turkey for the period of 1985-2012. Robust biased RRR method showed that the most important independent variable effecting the unemployment rate is Purchasing Power Parities (PPP). The least important variables effecting the unemployment rate are Import Growth Rate (IMP) and Export Growth Rate (EXP). Hence, any increment in PPP cause an important increment in unemployment rate, however, any increment in IMP causes an unimportant increase in unemployment rate. Any increment in EXP causes an unimportant decrease in unemployment rate.

From a wide range of techniques appropriate to relate spectra measurements with soil properties, partial least squares (PLS) regression and support vector machines (SVM) are most commonly used. This is due to their predictive power and the availability of software tools. Both represent exclusively statistically based approaches and, as such, benefit from multiple responses of soil material in the spectrum. However, physical-based approaches that focus only on a single spectral feature, such as simple linear regression using selected continuum-removed spectra values as a predictor variable, often provide accurate estimates. Furthermore, if this approach extends to multiple cases by taking into account three basic absorption feature parameters (area, width, and depth) of all occurring features as predictors and subjecting them to best subset selection, one can achieve even higher prediction accuracy compared with PLS regression. Here, we attempt to further extend this approach by adding two additional absorption feature parameters (left and right side area), as they can be important diagnostic markers, too. As a result, we achieved higher prediction accuracy compared with PLS regression and SVM for exchangeable soil pH, slightly higher or comparable for dithionite-citrate and ammonium oxalate extractable Fe and Mn forms, but slightly worse for oxidizable carbon content. Therefore, we suggest incorporating the multiple linear regression approach based on absorption feature parameters into existing working practices.

In this paper, a graph regularized nonlinear ridge regression (RR) model is proposed for remote sensing data analysis, including hyper-spectral image classification and atmospheric aerosol retrieval. The RR is an efficient linear regression method, especially in handling cases with a small number of training samples or with correlated features. However, large amounts of unlabeled samples exist in remote sensing data analysis. To sufficiently explore the information in unlabeled samples, we propose a graph regularized RR (GRR) method, where the vertices denote labeled or unlabeled samples and the edges represent the similarities among different samples. A natural assumption is that the predict values of neighboring samples are close to each other. To further address the nonlinearly separable problem in remote sensing data caused by the complex acquisition process as well as the impacts of atmospheric and geometric distortions, we extend the proposed GRR into a kernelized nonlinear regression method, namely KGRR. To evaluate the proposed method, we apply it to both classification and regression tasks and compare with representative methods. The experimental results show that KGRR can achieve favorable performance in terms of predictability and computation time.

QSAR study of the non-peptidic inhibitors of procollagen C-proteinase based on Multiple linear regression, principle component regression, and partial least squares