Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators

Kernel Ridge Regression combines the principles of machine learning supervision and ridge regression by employing the kernel trick. This approach is particularly effective for regression problems with non-linear relationships between inputs and outputs. The kernel trick allows Kernel Ridge Regression (KRR) to perform ridge regression by learning non-linear functions in a high-dimensional space using ridge regression's regularization techniques. The success of KRR depends on the hyper-parameter settings, which determine the type of kernel used. Current methods for determining hyper-parameter values encounter three primary challenges: high computational costs, large memory demands, and low accuracy. This research introduces a significant improvement to the golden eagle optimization framework by incorporating elite opposite-based learning (EOBL) to enhance population diversity in the search space. We apply this strategy to efficiently select optimal hyper-parameters. Combining EOBL with KRR can lead to improved predictive accuracy. By selecting elite solutions and incorporating opposition-based methodologies, the model can circumvent local optima and broaden the range of potential solutions, leading to improved results, especially in complex datasets. The proposed enhancement to Kernel Ridge Regression was evaluated on ten publicly available multi-class datasets to demonstrate its effectiveness. The results from various evaluation criteria showed that the proposed enhancement achieved superior classification performance compared to all baseline techniques.

Dynamic textures (DTs) that represent moving scenes such as flames, smoke, and waves, exhibit fixed dynamics within a period of time and have been successfully modeled using linear dynamic systems (LDS). In this paper, we show that the widely used LDS model can be approximated using a principal component regression (PCR) model with the main advantage of simplicity. Furthermore, to capture the nonlinearity of training frames, we extend traditional PCR to its kernelized version and introduce kernel principal component regression (KPCR) to model and synthesize DTs. To ensure algorithm stability, we remove the standard state model and directly apply the quantized kernel least mean squares algorithm from signal processing domain to approximate the performance achieved with KPCR. We term this improvement kernel adaptive dynamic texture synthesis (KADTS), which also has the benefits of computational and memory efficiency. These advantages make KADTS ideally suited for real-world applications, since the majority of electronic devices, including cell phones and laptops, suffer from limited memory and real-time constraints. We demonstrate, via both theoretical and experimental analyses, the connections between DT synthesis using KPCR and KADTS with a regularization network theory. We also show the superiority of our proposed algorithms for DT synthesis compared with other dynamic system-based benchmarks. MATLAB code is available from our project homepage http://bmal.hust.edu.cn/project/dts.html .

Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration

The Gaussian mixture model (GMM) method is popular and efficient for voice conversion (VC), but it is often subject to overfitting. In this paper, the principal component regression (PCR) method is adopted for the spectral mapping between source speech and target speech, and the numbers of principal components are adjusted properly to prevent the overfitting. Then, in order to better model the nonlinear relationships between the source speech and target speech, the kernel principal component regression (KPCR) method is also proposed. Moreover, a KPCR combined with GMM method is further proposed to improve the accuracy of conversion. In addition, the discontinuity and oversmoothing problems of the traditional GMM method are also addressed. On the one hand, in order to solve the discontinuity problem, the adaptive median filter is adopted to smooth the posterior probabilities. On the other hand, the two mixture components with higher posterior probabilities for each frame are chosen for VC to reduce the oversmoothing problem. Finally, the objective and subjective experiments are carried out, and the results demonstrate that the proposed approach shows greatly better performance than the GMM method. In the objective tests, the proposed method shows lower cepstral distances and higher identification rates than the GMM method. While in the subjective tests, the proposed method obtains higher scores of preference and perceptual quality.

Model selection in kernel ridge regression

Multicollinearity arises when two or more regressors are correlated in multiple linear regression model (MLRM) and in most cases, one regressor variable can be predicted from another. Multicollinearity majorly results in inefficient regression model estimates and poor performance of the regression model. However, multicollinearity problem can easily be handled using various methods such as ridge regression, lasso regression, principal components regression, etc. This study compared the effectiveness of two estimators in handling multicollinearity problem in a given dataset. The estimators being compared are ridge estimator (RE) and principal components estimator (PCE). This research uses secondary data obtained from World Bank database, International Monetary Fund (IMF) database, and the Nigerian Debt Management Office to compare the two approaches of handling multicollinearity problem in MLRM. The presence of multicollinearity in the dataset was established using the correlation matrix of predictors and the Variance Inflation Factors (VIF's). Then ridge regression and principal components regression methods were used to fit models to the dataset respectively and their mean squared errors (MSE) were obtained. The MSE was used as performance evaluation measure for the regression models. Both methods addressed the problem multicollinearity in the datasets but the ridge estimator performed better than PCE by having the smallest mean squared error.

Models built with deep neural network (DNN) can handle complicated real-world data extremely well, seemingly without suffering from the curse of dimensionality or the non-convex optimization. To contribute to the theoretical understanding of deep learning, this work studies the nonparametric perspective of DNNs by considering the following questions: (1) What is the underlying estimation problem and what are the most appropriate data assumptions? (2) What is the corresponding optimal convergence rate and does the curse of dimensionality occur? (3) Is the optimal rate achievable for DNN estimators and is there any optimization guarantee? These questions are investigated on two of the most fundamental problems --- regression and classification. Specifically, statistical optimality of DNN estimators is established under various settings with special focuses on the curse of dimensionality and optimization guarantee.In the classic binary classification problem, statistical optimal convergence rates that suffer less from the curse of dimensionality are established under two settings:(1) Under the smooth boundary assumption, I show that DNN classifiers with proper architectures can benefit from the compositional smoothness structure underlying the high dimensional data in the sense that the optimal convergence rates only depend on some effective dimension d*, potentially much smaller than the data dimension d. (2) Under a novel teacher-student framework that assumes the Bayes classifier to be expressed as ReLU neural networks, I obtain a dimension-free rate of convergence O(n^{-2/3}) for DNN classifiers, which is also proven optimal. The optimization of DNN is highly complicated with generally no algorithmic guarantee on finding the global minimizer. To this end, I turn to the recently proposed neural tangent kernel (NTK) literature where the similarity between overparametrized DNN trained by gradient descent (GD) and kernel methods are established. Specifically, through a comprehensive analysis of L2-regularized GD trajectories, I prove that for overparametrized one-hidden-layer ReLU neural networks with L2 regularization, the output from GD is close to that from the kernel ridge regression with the corresponding NTK and optimal rate of L2 estimation error can be achieved.

Kernel ridge regression (KRR) is a well-known and popular nonparametric regression approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, is computationally intensive for large data sets, due to the need to operate on a dense n×n kernel matrix, where n is the sample size. Recently, various approximation schemes for solving KRR have been considered, and some analyzed. Some approaches such as Nyström approximation and sketching have been shown to preserve the rate optimality of KRR. In this paper, we consider the simplest approximation, namely, spectrally truncating the kernel matrix to its largest r<n eigenvalues. We derive an exact expression for the maximum risk of this truncated KRR, over the unit ball of the RKHS. This result can be used to study the exact trade-off between the level of spectral truncation and the regularization parameter. We show that, as long as the RKHS is infinite-dimensional, there is a threshold on r, above which, the spectrally-truncated KRR surprisingly outperforms the full KRR in terms of the minimax risk, where the minimum is taken over the regularization parameter. This strengthens the existing results on approximation schemes, by showing that not only one does not lose in terms of the rates, truncation can in fact improve the performance, for all finite samples (above the threshold). Moreover, we show that the implicit regularization achieved by spectral truncation is not a substitute for Hilbert norm regularization. Both are needed to achieve the best performance.

Kernel ridge regression (KRR) is a powerful method for nonparametric regression. The time and space complexity of computing the KRR estimate directly are O(n3) and O(n2) , respectively, which are prohibitive for large-scale data sets, where n is the number of data. In this article, we propose a novel random sketch technique based on the circulant matrix that achieves savings in storage space and accelerates the solution of the KRR approximation. The circulant matrix has the following advantages: It can save time complexity by using the fast Fourier transform (FFT) to compute the product of matrix and vector, its space complexity is linear, and the circulant matrix, whose entries in the first column are independent of each other and obey the Gaussian distribution, is almost as effective as the i.i.d. Gaussian random matrix for approximating KRR. Combining the characteristics of the circulant matrix and our careful design, theoretical analysis and experimental results demonstrate that our proposed sketch method, making the estimate kernel methods scalable and practical for large-scale data problems, outperforms the state-of-the-art KRR estimates in time complexity while retaining similar accuracies. Meanwhile, our sketch method provides the theoretical bound that keeps the optimal convergence rate for approximating KRR.

Chapter 14 - Forecasting

In the medical research domain, limited data and high annotation costs have made efficient classification under few-shot conditions a popular research area. This paper proposes a meta-learning framework, termed MedOptNet, for few-shot medical image classification. The framework enables the use of various high-performance convex optimization models as classifiers, such as multi-class kernel support vector machines, ridge regression, and other models. End-to-end training is then implemented using dual problems and differentiation in the paper. Additionally, various regularization techniques are employed to enhance the model's generalization capabilities. Experiments on the BreakHis, ISIC2018, and Pap smear medical few-shot datasets demonstrate that the MedOptNet framework outperforms benchmark models. Moreover, the model training time is also compared to prove its effectiveness in the paper, and an ablation study is conducted to validate the effectiveness of each module.

Improving MRI-based analysis of brain structural changes in patients with hypertension via a privileged information learning algorithm.

Kernel ridge regression for out-of-sample mapping in supervised manifold learning

Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression

In order to determine the proper multivariate calibration model, it is necessary to select the number of respective basis vectors (latent vectors, factors, etc.) when using principal component regression (PCR) or partial least squares (PLS). These values are commonly referred to as the prediction rank of the model. Comparisons between PCR and PLS models for a given data set are often made with the prediction rank to determine the more parsimonious model, ignoring the fact that the values have been obtained using different basis sets. Additionally, it is not possible to use this approach for determining the prediction rank of models generated by other modeling methods such as ridge regression (RR). This paper presents measures of effective rank for a given model that can be applied to all modeling methods, thereby providing inter‐model comparisons. A definition based on the regression vector norm and is compared with two alternative forms from the literature. With a proper definition of effective rank, a better assessment of degrees of freedom for statistical computations is possible. Additionally, the true nature of variable selection for improved parsimony can be properly assessed. Spectroscopic data sets are used as examples with PCR, PLS and RR. Copyright © 2004 John Wiley &amp; Sons, Ltd.