Model selection in kernel ridge regression

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a variety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression. Our theoretical results are supported by numerical experiments.

Analytic tools are useful for detecting patterns in education data and providing insights about student performance and learning. This study compared six supervised learning algorithms (linear regression, ridge regression, the lasso, regression trees, random forests regression, gradient boosted regression) and identified features important for predicting student performance. The dataset consisted of N=1044 observations from two secondary schools in Portugal (UCI-MLR, Cortez & Silva, 2008). Performance was assessed by final grades (range: 0-20) in two courses, mathematics and Portugese. The models were fit to training data with 27 independent variables and evaluated on a testing subset. Overall, performance was lower for students in mathematics than Portugese. The models selected a similar set of variables as important for predicting performance: mother's education level, student plans for higher education, and weekly study time were positively related to predicted performance, whereas course subject, school educational support, and romantic relationships were associated with decreased student performance. The models differed in the number, weighting, order and importance given to predictor variables. Linear regression provided a model with 13 predictors. Ridge regression shrank the coefficient estimates toward zero; the lasso performed variables selection for a model with 20 predictors. There was a tradeoff between model complexity and interpretability. The single pruned regression tree provided a simple, interpretable non-linear model with four features. Random forests regression and gradient boosting reduced overfitting, but were more difficult to interpret. Advantages and limitations of the different models are discussed. Applications for educational data mining (EDM) and learning analytics (LA) are considered.

Kernel Ridge Regression combines the principles of machine learning supervision and ridge regression by employing the kernel trick. This approach is particularly effective for regression problems with non-linear relationships between inputs and outputs. The kernel trick allows Kernel Ridge Regression (KRR) to perform ridge regression by learning non-linear functions in a high-dimensional space using ridge regression's regularization techniques. The success of KRR depends on the hyper-parameter settings, which determine the type of kernel used. Current methods for determining hyper-parameter values encounter three primary challenges: high computational costs, large memory demands, and low accuracy. This research introduces a significant improvement to the golden eagle optimization framework by incorporating elite opposite-based learning (EOBL) to enhance population diversity in the search space. We apply this strategy to efficiently select optimal hyper-parameters. Combining EOBL with KRR can lead to improved predictive accuracy. By selecting elite solutions and incorporating opposition-based methodologies, the model can circumvent local optima and broaden the range of potential solutions, leading to improved results, especially in complex datasets. The proposed enhancement to Kernel Ridge Regression was evaluated on ten publicly available multi-class datasets to demonstrate its effectiveness. The results from various evaluation criteria showed that the proposed enhancement achieved superior classification performance compared to all baseline techniques.

Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration

Kernel ridge regression (KRR) is a well-known and popular nonparametric regression approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, is computationally intensive for large data sets, due to the need to operate on a dense n×n kernel matrix, where n is the sample size. Recently, various approximation schemes for solving KRR have been considered, and some analyzed. Some approaches such as Nyström approximation and sketching have been shown to preserve the rate optimality of KRR. In this paper, we consider the simplest approximation, namely, spectrally truncating the kernel matrix to its largest r<n eigenvalues. We derive an exact expression for the maximum risk of this truncated KRR, over the unit ball of the RKHS. This result can be used to study the exact trade-off between the level of spectral truncation and the regularization parameter. We show that, as long as the RKHS is infinite-dimensional, there is a threshold on r, above which, the spectrally-truncated KRR surprisingly outperforms the full KRR in terms of the minimax risk, where the minimum is taken over the regularization parameter. This strengthens the existing results on approximation schemes, by showing that not only one does not lose in terms of the rates, truncation can in fact improve the performance, for all finite samples (above the threshold). Moreover, we show that the implicit regularization achieved by spectral truncation is not a substitute for Hilbert norm regularization. Both are needed to achieve the best performance.

Kernel ridge regression (KRR) is a powerful method for nonparametric regression. The time and space complexity of computing the KRR estimate directly are O(n3) and O(n2) , respectively, which are prohibitive for large-scale data sets, where n is the number of data. In this article, we propose a novel random sketch technique based on the circulant matrix that achieves savings in storage space and accelerates the solution of the KRR approximation. The circulant matrix has the following advantages: It can save time complexity by using the fast Fourier transform (FFT) to compute the product of matrix and vector, its space complexity is linear, and the circulant matrix, whose entries in the first column are independent of each other and obey the Gaussian distribution, is almost as effective as the i.i.d. Gaussian random matrix for approximating KRR. Combining the characteristics of the circulant matrix and our careful design, theoretical analysis and experimental results demonstrate that our proposed sketch method, making the estimate kernel methods scalable and practical for large-scale data problems, outperforms the state-of-the-art KRR estimates in time complexity while retaining similar accuracies. Meanwhile, our sketch method provides the theoretical bound that keeps the optimal convergence rate for approximating KRR.

Chapter 14 - Forecasting

A central problem in learning is selection of an appropriate model. This is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate. In this article, we discuss the problem of model selection and generalization error estimation in the context of kernel regression models, e.g., kernel ridge regression, kernel subset regression or Gaussian process regression. Previously, a non-asymptotic generalization error estimator called the subspace information criterion (SIC) was proposed, that could be successfully applied to finite dimensional subspace models. SIC is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel Hilbert space (RKHS) H and the reproducing kernels centered on training sample points span the whole space H. These conditions hold only if dim H < l, where l < ∞ is the number of training examples. Therefore, SIC could be applied only to finite dimensional RKHSs. In this paper, we extend the range of applicability of SIC, and show that even if the reproducing kernels centered on training sample points do not span the whole space H, SIC is an unbiased estimator of an essential part of the generalization error. Our extension allows the use of any RKHSs including infinite dimensional ones, i.e., richer function classes commonly used in Gaussian processes, support vector machines or boosting. We further show that when the kernel matrix is invertible, SIC can be expressed in a much simpler form, making its computation highly efficient. In computer simulations on ridge parameter selection with real and artificial data sets, SIC is compared favorably with other standard model selection techniques for instance leave-one-out cross-validation or an empirical Bayesian method.

Improving MRI-based analysis of brain structural changes in patients with hypertension via a privileged information learning algorithm.

Kernel ridge regression for out-of-sample mapping in supervised manifold learning

Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression

In general, the application conditions of linear regression models could be met after the natural logarithmic transformation of data. From the practical perspective, this paper introduced the linear regression models with natural logarithmic transformation of independent variable, dependent variable, and both independent and dependent variables in detail. The paper illustrated why the equation and coefficients could not be directly explained after the natural logarithmic transformation of data. The percentage changes of X and/or Y were applied to elaborate the principle and method for the explanation of the equation and coefficients. Three examples were used to fit simple linear models with natural logarithmic transformation of independent, dependent, and both independent and dependent variables and the results of theses models were explained in detail.

Recently, nonlinear vector autoregressive (NVAR) model based on Granger causality was proposed to infer nonlinear gene regulatory networks from time series gene expression data. Since NVAR requires a large number of parameters due to the basis expansion, the length of time series microarray data is insufficient for accurate parameter estimation and we need to limit the size of the gene set strongly. To address this limitation, we employ L1 regularization technique to estimate NVAR. Under L1 regularization, direct parents of each gene can be selected efficiently even when the number of parameters exceeds the number of data samples. We can thus estimate larger gene regulatory networks more accurately than those from existing methods. Through the simulation study, we verify the effectiveness of the proposed method by comparing its limitation in the number of genes to that of the existing NVAR. The proposed method is also applied to time series microarray data of Human hela cell cycle.

Advanced forecasting of COVID-19 epidemic: Leveraging ensemble models, advanced optimization, and decomposition techniques

We use machine learning to generate metamodels for sawing simulation. Simulation is widely used in the wood industry for decision making. These simulators are particular since their response for a given input is a structured object, i.e., a basket of lumbers. We demonstrate how we use simple machine learning algorithms (e.g., a tree) to obtain a good approximation of the simulator's response. The generated metamodels are guaranteed to output physically realistic baskets (i.e., there exists at least one log that can produce the basket). We also propose to use kernel ridge regression. While having the power to exploit the structure of a basket, it can predict previously unseen baskets. We finally evaluate the impact of possibly predicting unrealistic baskets using ridge regression jointly with a nearest neighbor approach in the output space. All metamodels are evaluated using standard machine learning metrics and novel metrics especially designed for the problem.