Online pairwise learning algorithms with convex loss functions

We consider the classification problem by learning from samples drawn from a non-identical sequence of probability measures. The learning algorithm is from Tikhonov regularization schemes associated with convex loss functions and reproducing kernel Hilbert spaces. Our main goal is to provide satisfactory estimates for the excess misclassification error of the produced classifiers.

To deal with the problem of insufficient labeled data in video object classification, one solution is to utilize additional pairwise constraints that indicate the relationship between two examples, i.e., whether these examples belong to the same class or not. In this paper, we propose a discriminative learning approach which can incorporate pairwise constraints into a conventional margin-based learning framework. Different from previous work that usually attempts to learn better distance metrics or estimate the underlying data distribution, the proposed approach can directly model the decision boundary and, thus, require fewer model assumptions. Moreover, the proposed approach can handle both labeled data and pairwise constraints in a unified framework. In this work, we investigate two families of pairwise loss functions, namely, convex and nonconvex pairwise loss functions, and then derive three pairwise learning algorithms by plugging in the hinge loss and the logistic loss functions. The proposed learning algorithms were evaluated using a people identification task on two surveillance video data sets. The experiments demonstrated that the proposed pairwise learning algorithms considerably outperform the baseline classifiers using only labeled data and two other pairwise learning algorithms with the same amount of pairwise constraints.

This work theoretically studies the problem of estimating a structured high-dimensional signal $\boldsymbol{x}_0 \in{\mathbb{R}}^n$ from noisy $1$-bit Gaussian measurements. Our recovery approach is based on a simple convex program which uses the hinge loss function as data fidelity term. While such a risk minimization strategy is very natural to learn binary output models, such as in classification, its capacity to estimate a specific signal vector is largely unexplored. A major difficulty is that the hinge loss is just piecewise linear, so that its ‘curvature energy’ is concentrated in a single point. This is substantially different from other popular loss functions considered in signal estimation, e.g. the square or logistic loss, which are at least locally strongly convex. It is therefore somewhat unexpected that we can still prove very similar types of recovery guarantees for the hinge loss estimator, even in the presence of strong noise. More specifically, our non-asymptotic error bounds show that stable and robust reconstruction of $\boldsymbol{x}_0$ can be achieved with the optimal oversampling rate $O(m^{-1/2})$ in terms of the number of measurements $m$. Moreover, we permit a wide class of structural assumptions on the ground truth signal, in the sense that $\boldsymbol{x}_0$ can belong to an arbitrary bounded convex set $K \subset{\mathbb{R}}^n$. The proofs of our main results rely on some recent advances in statistical learning theory due to Mendelson. In particular, we invoke an adapted version of Mendelson’s small ball method that allows us to establish a quadratic lower bound on the error of the first-order Taylor approximation of the empirical hinge loss function.

Logistic classification with varying Gaussians

The paper studies convex stochastic optimization problems in a reproducing kernel Hilbert space (RKHS). The objective (risk) functional depends on functions from this RKHS and takes the form of a mathematical expectation (integral) of a nonnegative integrand (loss function) over a probability measure. The problem is generally ill-posed, a difficulty that in statistical learning is addressed through Tihonov regularization, with Monte Carlo approximation of integrals, which also makes it possible to solve the problem by finite dimensional (convex) quadratic optimization. The approximate solutions are referred to as kernel learning estimators and are expressed as a linear combination of kernels evaluated at the sample points. They are functional random variables that depend on the full sample. The paper studies a probabilistic convergence of these approximate solutions under a gradual elimination of the regularization parameter with rising number of observations. Its intended contribution is to derive novel nonasymptotic bounds on the minimization error and exponential bounds on the tail distribution of errors and to establish novel sufficient conditions for uniform convergence of kernel estimators to the true (normal) solution with probability one, jointly with a rule for downward adjustment of the regularization factor with increasing sample size. Applications to least squares, median, and quantile regression estimation, as well as to binary classification, are discussed.

A modified local quadratic approximation algorithm for penalized optimization problems

We obtain estimation and excess risk bounds for Empirical Risk Minimizers (ERM) and minmax Median-Of-Means (MOM) estimators based on loss functions that are both Lipschitz and convex. Results for the ERM are derived under weak assumptions on the outputs and subgaussian assumptions on the design as in Alquier et al. (Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions. arXiv:1702.01402, 2017). The difference with Alquier et al. (2017) is that the global Bernstein condition of this paper is relaxed here into a local assumption. We also obtain estimation and excess risk bounds for minmax MOM estimators under similar assumptions on the output and only moment assumptions on the design. Moreover, the dataset may also contains outliers in both inputs and outputs variables without deteriorating the performance of the minmax MOM estimators. Unlike alternatives based on MOM’s principle (Lecue and Lerasle in Ann Stat, 2017; Lugosi and Mendelson in JEMS, 2016), the analysis of minmax MOM estimators is not based on the small ball assumption (SBA) of Koltchinskii and Mendelson (Int Math Res Not IMRN 23:12991–13008, 2015). In particular, the basic example of non parametric statistics where the learning class is the linear span of localized bases, that does not satisfy SBA (Saumard in Bernoulli 24(3):2176–2203, 2018) can now be handled. Finally, minmax MOM estimators are analysed in a setting where the local Bernstein condition is also dropped out. It is shown to achieve excess risk bounds with exponentially large probability under minimal assumptions insuring only the existence of all objects.

Regularization is used to find a solution that both fits the data and is sufficiently smooth, and thereby is very effective for designing and refining learning algorithms. But the influence of its exponent remains poorly understood. In particular, it is unclear how the exponent of the reproducing kernel Hilbert space (RKHS) regularization term affects the accuracy and the efficiency of kernel-based learning algorithms. Here we consider regularized least squares regression (RLSR) with an RKHS regularization raised to the power of m, where m is a variable real exponent. We design an efficient algorithm for solving the associated minimization problem, we provide a theoretical analysis of its stability, and we compare it with respect to computational complexity and prediction accuracy to the classical kernel ridge regression algorithm where the regularization exponent m is fixed at 2. Our results show that the m-power RLSR problem can be solved efficiently, and support the suggestion that one can use a regularization term that grows significantly slower than the standard quadratic growth in the RKHS norm.

Multi-kernel regularized classifiers

Robust support vector regression in the primal

Bayesian decision problems require subjective elicitation of the inputs: beliefs and preferences. Sometimes, elicitation methods may not represent perfectly the judgements of the decision maker. Several foundations propose to overlay this problem using robust approaches. In these models, beliefs are modelled by a class of probability distributions and preferences by a class of loss functions. Then, we are in the conditions of a Pareto order. Hence the solution concept is the set of non dominated alternatives. In this article we focus on the computation of the efficient set when the preferences are modeled by a class of convex loss functions.

Kernel adaptive filters are a class of powerful nonlinear filter developed in reproducing kernel Hilbert space (RKHS).The Gaussian kernel is usually the default kernel in KAF algorithm, because the Gaussian kernel has the universal approximation. However, in previous research the kernel adaptive filtering algorithms were mostly based on mean square error criterion and assumed to be in a Gaussian noise environment. When environmental noise is changed, the performance of conventional kernel adaptive filtering algorithm based on mean square error criterion is seriously reduced to failure due to the interference of non-Gaussian noise and the influence of inappropriate non-Gaussian modeling. Therefore, it is important to develop a new method of suppressing the noise of non-Gaussian signals. In this paper, a new kernel fractional lower power adaptive filtering algorithm is proposed by combining the benefits of the kernel method and a new loss function which is robust against non-Gaussian impulsive interferences and has fast convergence under a similar stability condition. The proposed SP-KFLP algorithm generates a new framework of cost function which combines the Softplus function with the KFLP algorithm by updating its weight vector according to the gradient estimation while nonlinear saturation characteristics of output error are used. Compared with the features of sigmoid function the features of the Softplus function guarantee the SP-KFLP an excellent performance for combatting impulsive interference and speeding up the convergence rate. In the kernel fractional low power criterion the reciprocal of the system error is used as the coefficient of the weight vector update formula, and the method of error burst is used to make the weight vector not update to resist the impulse noise. The mean square convergence analysis for SP-KFLP is conducted, and a sufficient condition for guaranteeing convergence is therefore obtained by using the energy conservation relation. The proposed algorithm is very simple computationally. Simulations in a system identification show that the proposed SP-KFLP algorithm outperforms the kernel least-mean-square algorithm, kernel fractional lower power algorithm, and sigmoid kernel fractional lower algorithm in terms of convergence rate and the robustness of against impulsive interference. The proposed algorithm improves not only the capability of resisting impulsive interference, but also the convergence rate. In other words, the contradiction between convergence and tracking performance stability is well taken into account, and the performance under Gaussian noise is also better than the performance of the traditional kernel adaptive algorithm.

In classification, the use of 0-1 loss is preferable since the minimizer of 0-1 risk leads to the Bayes optimal classifier. However, due to the nonconvexity of 0-1 loss, this optimization problem is NP-hard. Therefore, many convex surrogate loss functions have been adopted. Previous works have shown that if a Bayes-risk consistent loss function is used as a surrogate, the minimizer of the empirical surrogate risk can achieve the Bayes optimal classifier as the sample size tends to infinity. Nevertheless, the comparison of convergence rates of minimizers of different empirical surrogate risks to the Bayes optimal classifier has rarely been studied. Which characterization of the surrogate loss determines its convergence rate to the Bayes optimal classifier? Can we modify the loss function to achieve a faster convergence rate? In this article, we study the convergence rates of empirical surrogate minimizers to the Bayes optimal classifier. Specifically, we introduce the notions of consistency intensity and conductivity to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer to the Bayes optimal classifier, enabling fair comparisons of the excess risks of different surrogate risk minimizers. The main result of this article has practical implications including: 1) showing that hinge loss (SVM) is superior to logistic loss (Logistic regression) and exponential loss (Adaboost) in the sense that its empirical minimizer converges faster to the Bayes optimal classifier and 2) guiding the design of new loss functions to speed up the convergence rate to the Bayes optimal classifier with a data-dependent loss correction method inspired by our theorems.

Classification and Regression are commonly used techniques in machine learning. The Loss functions are foundational to machine learning, serving as quantifiable measures of prediction error though which model optimization is lead. This study presents a comprehensive mathematical analysis of prominent loss functions utilized across classification and regression tasks. We present formal definitions and investigate inherent mathematical properties such as convexity and differentiability. Their implications for the learning process, specifically within gradient-based optimization frameworks. The analysis begins with fundamental regression losses such as Mean Squared Error and Mean Absolute Error. The analysis further extends to robust alternatives such as Huber and Quantile Loss. For classification, we explore binary and categorical Cross-Entropy, Hinge Loss, Exponential Loss by clarifying their characteristics. Also, we discuss practical considerations in selecting appropriate loss functions, importance of regularization, custom loss functions design, and the critical distinction between optimization objectives and evaluations metrics. The work aims to create understanding for readers with deeper theoretical understanding of how different loss functions shape model behaviour and performance in diverse machine learning applications.

We present two new algorithms for online learning in reproducing kernel Hilbert spaces. Our first algorithm, ILK (implicit online learning with kernels), employs a new, implicit update technique that can be applied to a wide variety of convex loss functions. We then introduce a bounded memory version, SILK (sparse ILK), that maintains a compact representation of the predictor without compromising solution quality, even in non-stationary environments. We prove loss bounds and analyze the convergence rate of both. Experimental evidence shows that our proposed algorithms outperform current methods on synthetic and real data.