The Kernel Least-Mean-Square Algorithm

We propose a data analysis method that combines the objectives of nonlinear principal component analysis and nonlinear discriminant analysis with the kernel method in a reproducing kernel Hilbert space. This method addresses nonlinear data analysis problems in high-dimensional spaces, specifically the reproducing kernel Hilbert space, through the use of the kernel trick. Our proposed method can be considered as a semi-supervised data analysis approach. We evaluate our proposed method using various kernel functions and datasets, both visually and quantitatively. The evaluation results demonstrate that our proposal outperforms kernel principal component analysis and generalized discriminant analysis in terms of classification performance. This indicates the advantages and originality of our proposed method. Furthermore, we analyze and discuss our findings based on the evaluation results, and highlight potential areas for further research and future work related to our proposal.

Kernel-based methods have had hefty success in a wide range of fields over the past decade, they are founded on the robust mathematical framework of reproducing kernel Hilbert spaces (RKHS), this space provides an interesting framework for the development of adaptive nonlinear filters. In this paper, we present a comparative study between the kernel method in Hilbert space with a reproducing kernel, and linear adaptive algorithms that is least mean square (LMS), normalized least mean square (NLMS) and recursive least square (RLS) algorithms. Simulation results show excellent performance of the kernel algorithm for identification of single-input single-output (SISO) systems, compared to the linear adaptive algorithms, this by adopting the very fast fading channels called Broadband Radio Access Network (BRAN A and BRAN C).

we propose a regression method using partial least square (PLS) technique and kernel method, which we refer to as kernel partial least square regression II to distinguish the conventional kernel PLS regression. The motivation of the conventional kernel PLS regression is to establish a coordinator system where the independent and dependent variables have a stronger correlation, and which models a regression model using the coordinator system. However, it projects independent variables into a reproducing kernel Hilbert space (RKHS) alone. The proposal extends the basic framework of conventional kernel PLS regression. The proposed method does not only project independent variables into the RKHS, but also dependent variables, and establishes a coordinator system where the independent and dependent variables have a stronger correlation. We use two function regression cases to evaluate the proposed method compared with the conventional kernel PLS regression. The regression performance of the proposed method has almost the same regression accuracy arising from the evaluation result, and this depends on the regression tasks. We explain the correlation calculations of our proposed method, conventional kernel PLS regression, and PLS regression. The meaning of correlation depends on the application in question. We also analyse and discuss the algorithm implementation, correlation meaning, and other issues for further development of the proposal.

The general perception is that kernel methods are not scalable, so neural nets become the choice for large-scale nonlinear learning problems. Have we tried hard enough for kernel methods? In this paper, we propose an approach that scales up kernel methods using a novel concept called doubly stochastic functional gradients''. Based on the fact that many kernel methods can be expressed as convex optimization problems, our approach solves the optimization problems by making two unbiased stochastic approximations to the functional gradient---one using random training points and another using random features associated with the kernel---and performing descent steps with this noisy functional gradient. Our algorithm is simple, need no commit to a preset number of random features, and allows the flexibility of the function class to grow as we see more incoming data in the streaming setting. We demonstrate that a function learned by this procedure after t iterations converges to the optimal function in the reproducing kernel Hilbert space in rate O(1/t), and achieves a generalization bound of O(1/\sqrt{t}). Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show competitive performances of our approach as compared to neural nets in datasets such as 2.3 million energy materials from MolecularSpace, 8 million handwritten digits from MNIST, and 1 million photos from ImageNet using convolution features.

This paper presents a kernel-based adaptive filter that is applied for the digital domain self-interference cancellation (SIC) in a transceiver operating in full-duplex (FD) mode. In FD, the benefit of simultaneous transmission and receiving of signals comes at the price of strong self-interference (SI). In this work, we are primarily interested in suppressing the SI using an adaptive filter namely adaptive projected subgradient method (APSM) in a reproducing kernel Hilbert space (RKHS) of functions. Using the projection concept as a powerful tool, APSM is used to model and consequently remove the SI. A low-complexity and fast-tracking algorithm is provided taking advantage of parallel projections as well as the kernel trick in RKHS. The performance of the proposed method is evaluated on real measurement data. The method illustrates the good performance of the proposed adaptive filter, compared to the known popular benchmarks. They demonstrate that the kernel-based algorithm achieves a favorable level of digital SIC while enabling parallel computation-based implementation within a rich and nonlinear function space, thanks to the employed adaptive filtering method.

In the machine learning based on kernel tricks, people often put one variable of a kernel function on the given samples to produce the basic functions of a solution space of learning problem. If the collection of the given samples deviates from the data distribution, the solution space spanned by these basic functions will also deviate from the real solution space of learning problem. In this paper a multikernel-like learning algorithm based on data probability distribution (MKDPD) is proposed, in which the parameters of a kernel function are locally adjusted according to the data probability distribution, and thus produces different kernel functions. These different kernel functions will generate different Reproducing Kernel Hilbert Spaces (RKHS). The direct sum of the subspaces of these RKHS constitutes the solution space of learning problem. Furthermore, based on the proposed MKDPD algorithm, a new algorithm for labeling new coming data is proposed, in which the basic functions are retrained according to the new coming data, while the coefficients of the retrained basic functions remained unchanged to label the new coming data. The experimental results presented in this paper show the effectiveness of the proposed algorithms.

Modeling data relation as a hierarchical structure has proven beneficial for many learning scenarios, and the hyperbolic space, with negative curvature, can encode such data hierarchy without distortion. Several recent studies also show that the representation power of the hyperbolic space can be further improved by endowing the kernel methods. Unfortunately, the known kernel methods, developed in hyperbolic space, are limited by the adaptation capacity or distortion issues. This paper addresses the issues through a novel embedding function. To this end, we propose a curvature-aware isometric embedding, which establishes an isometry from the Poincar\'e model to a special reproducing kernel Hilbert space (RKHS). Then we can further define a series of kernels on this RKHS, including several positive definite kernels and an indefinite kernel. Thorough experiments are conducted to demonstrate the superiority of our proposals over existing-known hyperbolic and Euclidean kernels in various learning tasks, e.g., graph learning and zero-shot learning.

In this paper, we investigate the enhancement of speech by applying kernel adaptive filter. Noise removal is very important in many applications like telephone conversation, speech recognition, etc. Kernel methods have shown good results for other applications like handwriting recognition, inverse distance weightings, etc. To improve the speech quality and intelligibility, we can process the signals in new domain like Reproducing Kernel Hilbert Space (RKHS) unlike time and frequency domains. We have used the noisy speech corpus (NOIZEUS) for the experiments. The experimental results shown the noise removal in RKHS has good improvement in the Signal to Noise Ratio (SNR) values as compared the traditional methods.

For a certain scaling of the initialization of stochastic gradient descent (SGD), wide neural networks (NN) have been shown to be well approximated by reproducing kernel Hilbert space (RKHS) methods. Recent empirical work showed that, for some classification tasks, RKHS methods can replace NNs without a large loss in performance. On the other hand, two-layers NNs are known to encode richer smoothness classes than RKHS and we know of special examples for which SGD-trained NN provably outperform RKHS. This is true even in the wide network limit, for a different scaling of the initialization. How can we reconcile the above claims? For which tasks do NNs outperform RKHS? If covariates are nearly isotropic, RKHS methods suffer from the curse of dimensionality, while NNs can overcome it by learning the best low-dimensional representation. Here we show that this curse of dimensionality becomes milder if the covariates display the same low-dimensional structure as the target function, and we precisely characterize this tradeoff. Building on these results, we present the spiked covariates model that can capture in a unified framework both behaviors observed in earlier work. We hypothesize that such a latent low-dimensional structure is present in image classification. We test numerically this hypothesis by showing that specific perturbations of the training distribution degrade the performances of RKHS methods much more significantly than NNs.

A local–global mixed kernel with reproducing property

The reproducing kernel Hilbert space (RKHS) methodology has shown to be a suitable tool for the resolution of a wide range of problems in statistical signal processing both in the real and complex domains. It relies on the idea of transforming the original functional data into an infinite series representation by projection onto an specific RKHS, which usually simplifies the statistical treatment without any loss of efficiency. Moreover, the advantages of quaternion algebra over real-valued three and four-dimensional vector algebra in the modelling of multidimensional data have been proven useful in much relatively recent research. This paper accordingly proposes a generic RKHS framework for the statistical analysis of augmented quaternion random vectors, which provide a complete description of their second order characteristics. It will allow us to exploit the full advantages of the RKHS theory in widely linear processing applications, such as signal detection. In particular, we address the detection of a quaternion signal disturbed by additive Gaussian noise and the discrimination between two quaternion Gaussian signals in continuous time.

Adaptive channel equalization is a signal processing technique to mitigate inter-symbol interference in a time dispersive channel. For adaptive equalization, minimum mean square error (MMSE) criterion-based reproducing kernel Hilbert spaces (RKHS) approaches such as the kernel least mean squares (KLMS) algorithm and its variants have been suggested in the literature for nonlinear channels. Another optimality criterion, based on minimum bit/symbol error rate (MBER/MSER), is a better choice for adapting an equalizer as compared to MMSE criterion. A kernel-based minimum symbol error rate (KMSER) equalization algorithm combines minimum symbol error rate (MSER)-based approaches with RKHS techniques. However, most algorithms in RKHS such as KMSER/KLMS require infinite storage requirement and hence cannot be practically implemented. To curtail the infinite memory requirement, and make adaptive algorithm suitable for implementation with finite memory and processing power, we propose quantized KMSER (QKMSER) and fixed-budget quantized KMSER (FBQKMSER)-based equalizers in this paper. In this paper, we derive the dynamical equation for MSE evolution of the QKMSER and FBQKMSER and find their performance to be asymptotically close to the MSE behavior of the KMSER. Also, it is found via simulations that the tracking performance of FBQKMSER is better than all the compared algorithms in this paper which is particularly useful for non-stationary channels.

The dynamic time warping (DTW) is state-of-the-art distance measure widely used in sequential pattern matching and it outperforms Euclidean distance in most cases because its matching is elastic and robust. It is tempting to substitute DTW distance for Euclidean distance in the Gaussian RBF kernel and plug it into the state-of-the art classifier support vector machines (SVMs) for sequence classification. However, it is not straightforward that DTW also outperforms Euclidean distance in kernel machines. While counter-examples can be found to numerically prove that DTW is not positive definite symmetric (PDS)acceptable by SVM, little is known why it can not be PDS theoretically. We analyze the DTW kernel and complete a theoretical proof via the connection between PDS kernel and reproducing kernel Hilbert space (RKHS). Our analysis leads to a better understanding that all Hilbertian metrics can be be converted to a PDS kernel in the Gaussian form, while the reverse is not true. The proof can be extended to conclude that elastic matching distance is not eligible to construct PDS kernels (e.g., Edit distance). Experiments were conducted to compare the RBF-kernel and DTW kernel in SVM classifications and the results show that simple Euclidean distance outperforms DTW in kernel machines.

The complex kernel adaptive filter (CKAF) has been widely applied to the complex-valued nonlinear problem in signal processing and machine learning. However, most of the CKAF applications involve the complex kernel least mean square (CKLMS) algorithms, which work in a pure complex or complexified reproducing kernel Hilbert space (RKHS). In this paper, we propose the generalized complex kernel affine projection (GCKAP) algorithms in the widely linear complex-valued RKHS (WL-RKHS). The proposed algorithms have two main notable features. One is that they provide a complete solution for both circular and non-circular complex nonlinear problems and show many performance improvements over the CKAP algorithms. The other is that the GCKAP algorithms inherit the simplicity of the CKLMS algorithm while reducing its gradient noise and boosting its convergence. The second-order statistical characteristics of WL-RKHS have also been developed. An augmented Gram matrix consists of a standard Gram matrix and a pseudo-Gram matrix. This decomposition provides more underlying information when the real and imaginary parts of the signal are correlated and learning is independent. In addition, some online sparsification criteria are compared comprehensively in the GCKAP algorithms, including the novelty criterion, the coherence criterion, and the angle criterion. Finally, two nonlinear channel equalization experiments with non-circular complex inputs are presented to illustrate the performance improvements of the proposed algorithms.

Continuous wavelet transform and linear time-frequency transforms are coefficients of continuous unitary group representations of the affine and Heisenberg groups. Many properties of these transforms that are important for wideband radar and sonar signal processing follow directly from group representation theory. These properties include volume invariance and variance of narrowband and wideband ambiguity functions and wavelet transform domain implementations of detects and signal estimators. For several radar, sonar, and array processing applications, the basic definition of wavelet and time-frequency representations must be generalized by using unitary representations of other groups and using reproducing kernel Hilbert space (RKHS) inner products in the definition of the linear transforms. The general definition then leads to weighted continuous wavelet transforms where the RKHS is determined by a nonstationary covariance function; generalized wideband ambiguity functions also follow from this general definition along with other important generalizations that arise in wideband array processing and model based signal processing in complex scattering and propagation media. This paper presents the generalized wavelet transform along with the weighted wavelet transforms. The classical narrowband and wideband ambiguity functions are then special cases. The application of generalized transform to multidimensional transforms for space-time processing are also presented, along with the application to conformal groups for detections and estimation of accelerating scatterers.© (1998) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.