Towards Unbiased Random Features with Lower Variance For Stationary Indefinite Kernels

In this paper, we investigate the nonlinear, finite dimensional and data independent random Fourier feature expansions that can approximate the popular Gaussian kernel. With recursive least squares algorithm, we develop the Random Fourier Feature Recursive Least Squares algorithm (RFF-RLS), which shows significant performance improvements in simulations when compared with several other online kernel learning algorithms such as Kernel Least Mean Square (KLMS) and Kerne Recursive Least Squares (KRLS). Our results confirm that the RFF-RLS can achieve desirable performance with low computational cost. As for the random Fourier features, the randomization generally results in redundancy. We use an algorithm, namely, Vector Quantization with Information Theoretic Learning (VQIT) to decrease the dictionary size. The resulting sparse dictionary can match the original data distribution well. The RFF-RLS with VQIT can outperform the RFF-RLS without VQIT.

Advanced Metering Infrastructure (AMI) of the power consumption side, serving as the neural hub of modern power systems, orchestrates real time energy data flows between distributed smart meters and centralized grid control centers, yet this pervasive connectivity introduces critical vulnerabilities to cyber attacks. Traditional intrusion detection systems struggle with the high-dimensional traffic patterns and severe class imbalance inherent in AMI networks, particularly in identifying handful attack variants amidst overwhelming normal traffic. To address these challenges, this paper proposes a detection model incorporating Random Fourier Features (RFF), Adaptive Weighted Focal Loss (AWFL), and a dual-scale convolutional neural network. The model first employs RFF to construct a low-dimensional explicit mapping, replacing the implicit kernel function in kernel PCA. This transforms the original nonlinear eigenvalue problem into linear PCA on random features, thereby reducing computational complexity significantly while preserving the capability of nonlinear dimensionality reduction. Subsequently, AWFL dynamically adjusts the model’s focus on different class samples during training, significantly improving class imbalance handling in network intrusion datasets. Finally, a dual-scale depthwise separable convolution module (DDSCM) is adopted to extract spatial features from input information, enabling more accurate identification of intrusion types. Evaluations on the NSL-KDD dataset and TON_IoT dataset demonstrate that the proposed model achieves superior intrusion detection performance, outperforming various comparative models in multiclass classification tasks regarding accuracy, precision, recall and F1-score.

Kernel based learning is very popular in machine learning but often quite costly with at least quadratic runtime complexity. Random Fourier features and related techniques have been proposed to provide an explicit kernel expansion such that standard techniques with low runtime and memory complexity can be used. This strategy leads to rather high dimensional datasets which is a drawback in many cases. Here, we combine a recently proposed unsupervised selection strategy for random Fourier features [1] with the very efficient supervised relevance learning given by Matrix LVQ. The suggested technique provides reasonable small but very discriminative features sets <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1.</sup>

Kernel based learning is very popular in machine learning, but many classical methods have at least quadratic runtime complexity. Random fourier features are very effective to approximate shift-invariant kernels by an explicit kernel expansion. This permits to use efficient linear models with much lower runtime complexity. As one key approach to kernelize algorithms with linear models they are successfully used in different methods. However, the number of features needed to approximate the kernel is in general still quite large with substantial memory and runtime costs. Here, we propose a simple test to identify a small set of random fourier features with linear costs, substantially reducing the number of generated features for low rank kernel matrices, while widely keeping the same representation accuracy. We also provide generalization bounds for the proposed approach.

The complex kernel adaptive filters (CKAFs) developed in complex reproducing kernel Hilbert space (RKHS) improve the performance of complex linear adaptive filters, but result in large burdens of computation and memory. To address these issues, a novel complex random Fourier features mapping (CRFFM) is proposed to approximate the kernel-induced mapping by applying the complexification of real RKHSs, and thus projects the complex-valued data from the original data space into the fixed-dimensional complex random Fourier features space (RFFS). To combat complex-valued non-Gaussian noises, a complex Cauchy loss function is presented, and the complex random Fourier features recursive complex Cauchy (CRFFRCC) algorithm is therefore proposed by the combination of CRFFM and the stochastic recursive method. The proposed CRFFRCC with a linear filter structure can reduce the computational and space complexities of CKAFs, significantly. Monte Carlo simulations conducted in the complex-valued nonlinear channel equalization validate the superiorities of CRFFRCC.

Fault diagnosis plays a vital role in ensuring the operation safety of blast furnaces and improving the quality of molten iron in the ironmaking and steelmaking industry. The blast furnace ironmaking process (BFIP) is intrinsically nonlinear. To address the nonlinearity issue of BFIP, a novel fault diagnosis approach that combines the randomized method, local structure information, and Fisher discriminant analysis is proposed in this paper. Using a randomized feature map, the process data is first mapped onto a randomized explicit low‐dimensional feature space. Compared to kernel methods, explicit low‐dimensional random Fourier features considerably reduce the computational cost, particularly for real‐time fault diagnosis for a large training dataset or a large‐scale process. Additionally, the local structure information contained in the randomized low‐dimensional feature space is extracted. The fault diagnosis performance is improved through the exploration of the local structure of random Fourier features. Finally, the blast furnace iron‐marking process state is determined using Bayesian inference. Case studies on a real‐world BFIP are carried out to demonstrate the superior performance of the proposed method in comparison with other related methods.

Random Fourier feature kernel recursive maximum mixture correntropy algorithm for online time series prediction

Slim-RFFNet: Slim deep convolution random Fourier feature network for image classification

We live in the era of big data with dataset sizes growing steadily over the past decades. In addition, obtaining expert labels for all the instances is time-consuming and in many cases may not even be possible. This necessitates the development of advanced semi-supervised models that can learn from both labeled and unlabeled data points and also scale at worst linearly with the number of examples. In the context of kernel based semi-supervised models, constructing the training kernel matrix for the large training dataset is expensive and memory inefficient. This paper investigates the scalability of the recently proposed multi-class semi-supervised kernel spectral clustering model (MSSKSC) by means of random Fourier features. The proposed model maps the input data into an explicit low-dimensional feature space. Thanks to the explicit feature maps, one can then solve the MSSKSC optimization formation in the primal, making the complexity of the method linear in number of training data points. The performance of the proposed model is compared with that of recently introduced reduced kernel techniques and Nystrom based MSSKSC approaches. Experimental results demonstrate the scalability, efficiency and faster training computation times of the proposed model over conventional large scale semi-supervised models on large scale real-life datasets.

Convolutional kernel networks based on a convex combination of cosine kernels

Fuzzy regression based on asymmetric support vector machines

Kernel density estimation (KDE) is one of the most widely used nonparametric density estimation methods. The fact that it is a memory-based method, i.e., it uses the entire training data set for prediction, makes it unsuitable for most current big data applications. Several strategies, such as tree-based or hashing-based estimators, have been proposed to improve the efficiency of the kernel density estimation method. The novel density kernel density estimation method (DMKDE) uses density matrices, a quantum mechanical formalism, and random Fourier features, an explicit kernel approximation, to produce density estimates. This method has its roots in the KDE and can be considered as an approximation method, without its memory-based restriction. In this paper, we systematically evaluate the novel DMKDE algorithm and compare it with other state-of-the-art fast procedures for approximating the kernel density estimation method on different synthetic data sets. Our experimental results show that DMKDE is on par with its competitors for computing density estimates and advantages are shown when performed on high-dimensional data. We have made all the code available as an open source software repository.

Although kernel machines allow a non-linear analysis through the transformation of their input data, their computational complexity makes them inefficient in terms of time and memory for the analysis of very large databases. Several attempts have been made to improve kernel methods performance, many of which are focused on approximate the kernel matrix or the feature mapping associated to it. Current trends in machine learning demands the capacity of dealing with large data sets while exploiting the capabilities of massively parallel architectures based on GPUs. This has been mainly accomplished by a combination of gradient descent optimization and online learning. This paper presents an online kernel-based model based on the dual formulation of Least Squared Support Vector Machine method, using the Learning on a Budget strategy to lighten the computational cost. This extends the algorithm capability to analyze very large or high-dimensional data without requiring high memory resources. The method was evaluated against other kernel approximation techniques: Nystrom approximation and Random Fourier Features. Experiments made with different datasets show the effectiveness of the Learning on a Budget strategy compared with the other approximation techniques.

Although fuzzy c-means algorithm has shown great capability to spherical clusters, it can not perform very well on non-spherical data sets yet. To deal with this problem, kernel-based fuzzy clustering has been presented by mapping data points into a high-dimensional Hilbert space with kernel functions. However, the computational complexity of kernel matrix is always quadratic, usually makes kernel fuzzy clustering non-scalable to large data sets. Recently, random features display a strong capability in approximating kernel functions while satisfying appropriate memory requirements. Therefore, we introduce two fuzzy clustering schemes in this paper based on random feature mapping and dimension reduction methods to approximate kernel fuzzy c-means clustering on general size data sets and one large data set. The clustering results illustrate the great potential of the proposed approaches.

A Gaussian process (GP)-based methodology is proposed to emulate complex dynamical computer models (or simulators). The method relies on emulating the numerical flow map of the system over an initial (short) time step, where the flow map is a function that describes the evolution of the system from an initial condition to a subsequent value at the next time step. This yields a probabilistic distribution over the entire flow map function, with each draw offering an approximation to the flow map. The model output times series is then predicted (under the Markov assumption) by drawing a sample from the emulated flow map (i.e., its posterior distribution) and using it to iterate from the initial condition ahead in time. Repeating this procedure with multiple such draws creates a distribution over the time series. The mean and variance of this distribution at a specific time point serve as the model output prediction and the associated uncertainty, respectively. However, drawing a GP posterior sample that represents the underlying function across its entire domain is computationally infeasible, given the infinite-dimensional nature of this object. To overcome this limitation, one can generate such a sample in an approximate manner using random Fourier features (RFF). RFF is an efficient technique for approximating the kernel and generating GP samples, offering both computational efficiency and theoretical guarantees. The proposed method is applied to emulate several dynamic nonlinear simulators including the well-known Lorenz and van der Pol models. The results suggest that our approach has a promising predictive performance and the associated uncertainty can capture the dynamics of the system appropriately.