Hashing Based Prediction for Large-Scale Kernel Machine

Kernel Ridge Regression combines the principles of machine learning supervision and ridge regression by employing the kernel trick. This approach is particularly effective for regression problems with non-linear relationships between inputs and outputs. The kernel trick allows Kernel Ridge Regression (KRR) to perform ridge regression by learning non-linear functions in a high-dimensional space using ridge regression's regularization techniques. The success of KRR depends on the hyper-parameter settings, which determine the type of kernel used. Current methods for determining hyper-parameter values encounter three primary challenges: high computational costs, large memory demands, and low accuracy. This research introduces a significant improvement to the golden eagle optimization framework by incorporating elite opposite-based learning (EOBL) to enhance population diversity in the search space. We apply this strategy to efficiently select optimal hyper-parameters. Combining EOBL with KRR can lead to improved predictive accuracy. By selecting elite solutions and incorporating opposition-based methodologies, the model can circumvent local optima and broaden the range of potential solutions, leading to improved results, especially in complex datasets. The proposed enhancement to Kernel Ridge Regression was evaluated on ten publicly available multi-class datasets to demonstrate its effectiveness. The results from various evaluation criteria showed that the proposed enhancement achieved superior classification performance compared to all baseline techniques.

Kernel methods have long played a central role in machine learning by enabling nonlinear modeling in high dimensional feature spaces. Among them, the Gaussian Radial Basis Function (RBF) kernel is widely used due to its smoothness, universality, and excellent performance in supervised learning. In contrast, the Jaccard/Tanimoto coefficient, originally defined over sets and binary vectors, has achieved notable success in chemoinformatics and information retrieval. Recent studies have shown that the Jaccard/Tanimoto coefficient can be extended to real-valued vectors and interpreted as a valid kernel. In this paper, we propose a natural generalization of the Tanimoto kernel by introducing a bandwidth-like parameter, leading to the Bandwidth-Adjustable Tanimoto (BAT) kernel. We focus on the shared structure of the BAT and the Gaussian RBF kernels in terms of their infinite polynomial expansions and investigate the universality of the BAT kernels, clarifying their relationship to the universal Taylor kernels. In particular, we show that by incorporating a bias term, the BAT kernel attains the universal approximation property comparable to that of the Gaussian kernel. Through experiments on classification, regression, and clustering tasks using benchmark datasets, we demonstrate that the BAT kernel performs comparably to the Gaussian kernel, while even the original Tanimoto kernel achieves competitive results. In particular, the BAT kernel attains performance on par with the Gaussian kernel in classification tasks, and it slightly surpasses the Gaussian kernel in regression and clustering tasks. A comparison of their functional forms, together with experiments on both regression and classification tasks involving nonlinear structures, further shows that the BAT kernel yields smoother predictive surfaces than the RBF kernel. Overall, our results indicate that the BAT kernel offers universality and practical performance, making it an alternative to the Gaussian kernel.

Kernel methods are an extremely popular set of techniques used for many important machine learning and data analysis applications. In addition to having good practical performance, these methods are supported by a well-developed theory. Kernel methods use an implicit mapping of the input data into a high dimensional feature space defined by a kernel function, i.e., a function returning the inner product between the images of two data points in the feature space. Central to any kernel method is the kernel matrix, which is built by evaluating the kernel function on a given sample dataset. In this paper, we initiate the study of non-asymptotic spectral properties of random kernel matrices. These are n x n random matrices whose (i,j)th entry is obtained by evaluating the kernel function on x_i and x_j, where x_1,..,x_n are a set of n independent random high-dimensional vectors. Our main contribution is to obtain tight upper bounds on the spectral norm (largest eigenvalue) of random kernel matrices constructed by using common kernel functions such as polynomials and Gaussian radial basis. As an application of these results, we provide lower bounds on the distortion needed for releasing the coefficients of kernel ridge regression under attribute privacy, a general privacy notion which captures a large class of privacy definitions. Kernel ridge regression is standard method for performing non-parametric regression that regularly outperforms traditional regression approaches in various domains. Our privacy distortion lower bounds are the first for any kernel technique, and our analysis assumes realistic scenarios for the input, unlike all previous lower bounds for other release problems which only hold under very restrictive input settings.

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a variety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression. Our theoretical results are supported by numerical experiments.

Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration

The future forecasting ability of machine learning (ML) makes ML a promising tool for predicting long-time quantum dissipative dynamics of open systems. In this article, we employ nonparametric ML algorithm (kernel ridge regression as a representative of the kernel methods) to study the quantum dissipative dynamics of the widely-used spin-boson (SB) model. Our ML model takes short-time dynamics as an input and is used for fast propagation of the long-time dynamics, greatly reducing the computational effort in comparison with the traditional approaches. Presented results show that the ML model performs well in both symmetric and asymmetric SB models. Our approach is not limited to SB model and can be extended to complex systems.

Kernel ridge regression (KRR) is a powerful method for nonparametric regression. The time and space complexity of computing the KRR estimate directly are O(n3) and O(n2) , respectively, which are prohibitive for large-scale data sets, where n is the number of data. In this article, we propose a novel random sketch technique based on the circulant matrix that achieves savings in storage space and accelerates the solution of the KRR approximation. The circulant matrix has the following advantages: It can save time complexity by using the fast Fourier transform (FFT) to compute the product of matrix and vector, its space complexity is linear, and the circulant matrix, whose entries in the first column are independent of each other and obey the Gaussian distribution, is almost as effective as the i.i.d. Gaussian random matrix for approximating KRR. Combining the characteristics of the circulant matrix and our careful design, theoretical analysis and experimental results demonstrate that our proposed sketch method, making the estimate kernel methods scalable and practical for large-scale data problems, outperforms the state-of-the-art KRR estimates in time complexity while retaining similar accuracies. Meanwhile, our sketch method provides the theoretical bound that keeps the optimal convergence rate for approximating KRR.

Laser-based metal additive manufacturing (MAM), particularly laser powder bed fusion (LPBF) and laser directed energy deposition (L-DED), is a transformative technology enabling the production of complex, high-performance metal components with precision and minimal waste. However, its wider adoption is hindered by challenges in ensuring repeatability and consistency due to the complex, dynamic nature of the process. Closed-loop systems with in-situ monitoring and control have gained attention for quality assurance, where machine learning (ML) models are crucial for correlating sensor data with process state diagnosis and facilitating control actions. This review adopts an ML perspective to elucidate the role of ML in closed-loop control systems. It first outlines sensor data types and preprocessing methods used as ML datasets, then discusses roles and applications of ML models for in-situ monitoring, highlighting limitations and potential solutions. An ML-based closed-loop control framework is also presented, with emphasis on the integration of ML in the control component. Finally, future development directions of advanced ML approaches are envisioned. This review provides new insights into enhancing the role of ML models in laser-based MAM closed-loop control systems, serving as a crucial aid in the development of intelligent control systems.

As synthetic biology expands and accelerates into real-world applications, methods for quantitatively and precisely engineering biological function become increasingly relevant. This is particularly true for applications that require programmed sensing to dynamically regulate gene expression in response to stimuli. However, few methods have been described that can engineer biological sensing with any level of quantitative precision. Here, we present two complementary methods for precision engineering of genetic sensors: in silico selection and machine-learning-enabled forward engineering. Both methods use a large-scale genotype-phenotype dataset to identify DNA sequences that encode sensors with quantitatively specified dose response. First, we show that in silico selection can be used to engineer sensors with a wide range of dose-response curves. To demonstrate in silico selection for precise, multi-objective engineering, we simultaneously tune a genetic sensor's sensitivity (EC50) and saturating output to meet quantitative specifications. In addition, we engineer sensors with inverted dose-response and specified EC50. Second, we demonstrate a machine-learning-enabled approach to predictively engineer genetic sensors with mutation combinations that are not present in the large-scale dataset. We show that the interpretable machine learning results can be combined with a biophysical model to engineer sensors with improved inverted dose-response curves.

In the contemporary landscape, information has emerged as a vital resource for entities, fuelling creativity, guiding choices, and enhancing operational effectiveness. Nonetheless, the rapid increase in the volume, variety, and velocity of data poses significant challenges for thorough analysis and effective utilization. Conventional analytical techniques frequently prove inadequate, highlighting the necessity for sophisticated tools to reveal actionable insights. Machine learning (ML) has surfaced as a groundbreaking approach, facilitating predictive analytics and streamlining decision-making processes. Although it holds significant promise, the incorporation of machine learning into predictive data strategies faces obstacles including data quality, model scalability, interpretability, and the ability to adapt to changing data streams. This study tackles these challenges by introducing a thorough framework for developing an efficient predictive data strategy utilizing machine learning. The framework outlines a series of steps encompassing data preprocessing, feature selection, model training, evaluation, and integration into organizational workflows. A variety of machine learning algorithms, including Random Forest, XGBoost, KNN, SVM, and Decision Trees, are examined and evaluated using metrics such as accuracy, precision, recall, and F1-score. Furthermore, ensemble methods are employed to improve the performance and stability of the model. The investigation highlights the importance of scalability and adaptability, guaranteeing that the framework remains pertinent in ever-changing data contexts. The findings indicate the success of the proposed framework, as XGBoost attained the highest predictive accuracy at 98.69%, surpassing other algorithms. This study connects theoretical progress in machine learning with real-world applications, enabling organizations to manage uncertainty, enhance operations, and promote innovation. The results highlight the promise of machine learning-based predictive data approaches in converting data into a valuable resource, facilitating more informed decision-making in competitive environments.

Navigating through the Maze of Homogeneous Catalyst Design with Machine Learning

There is a trend in recent machine learning community to construct a nonlinear version of a linear algorithm using the 'kernel method', e.g. Support Vector Machines (SVMs), kernel principal component analysis, kernel fisher discriminant analysis and the recent kernel clustering algorithms. In unsupervised clustering algorithms using kernel method, typically, a nonlinear mapping is used first to map the data into a potentially much higher feature space, where clustering is then performed. A drawback of these kernel clustering algorithms is thatthe clustering prototypes lie in high dimensional feature space and hence lack clear and intuitive descriptions unless using additional projection approximation from the feature to the data space as done in the existing literatures. In this paper, a novel clustering algorithm using the 'kernel method' based on the classical fuzzy clustering algorithm (FCM) is proposed and called as kernel fuzzy c-means algorithm (KFCM). KFCM adopts a new kernel-induced metric in the data space to replace the original Euclidean norm metric in FCM and the clustered prototypes still lie in the data space so that the clustering results can be reformulated and interpreted in the original space. Our analysis shows that KFCM is robust to noise and outliers and also tolerates unequal sized clusters. And finally this property is utilized to cluster incomplete data. Experiments on two artificial and one real datasets show that KFCM hasbetter clustering performance and more robust than several modifications of FCM for incomplete data clustering.

Machine Learning is a modern and actively developing field of computer science, devoted to extracting and estimating dependencies from empirical data. It combines such fields as statistics, optimization theory and artificial intelligence. In practical tasks, the general aim of Machine Learning is to construct algorithms able to generalize and predict in previously unseen situations based on some set of examples. Given some finite information, Machine Learning provides ways to exract knowledge, describe, explain and predict from data. Kernel Methods are one of the most successful branches of Machine Learning. They allow applying linear algorithms with well-founded properties such as generalization ability, to non-linear real-life problems. Support Vector Machine is a well-known example of a kernel method, which has found a wide range of applications in data analysis nowadays. In many practical applications, some additional prior knowledge is often available. This can be the knowledge about the data domain, invariant transformations, inner geometrical structures in data, some properties of the underlying process, etc. If used smartly, this information can provide significant improvement to any data processing algorithm. Thus, it is important to develop methods for incorporating prior knowledge into data-dependent models. The main objective of this thesis is to investigate approaches towards learning with kernel methods using prior knowledge. Invariant learning with kernel methods is considered in more details. In the first part of the thesis, kernels are developed which incorporate prior knowledge on invariant transformations. They apply when the desired transformation produce an object around every example, assuming that all points in the given object share the same class. Different types of objects, including hard geometrical objects and distributions are considered. These kernels were then applied for images classification with Support Vector Machines. Next, algorithms which specifically include prior knowledge are considered. An algorithm which linearly classifies distributions by their domain was developed. It is constructed such that it allows to apply kernels to solve non-linear tasks. Thus, it combines the discriminative power of support vector machines and the well-developed framework of generative models. It can be applied to a number of real-life tasks which include data represented as distributions. In the last part of the thesis, the use of unlabelled data as a source of prior knowledge is considered. The technique of modelling the unlabelled data with a graph is taken as a baseline from semi-supervised manifold learning. For classification problems, we use this apporach for building graph models of invariant manifolds. For regression problems, we use unlabelled data to take into account the inner geometry of the input space. To conclude, in this thesis we developed a number of approaches for incorporating some prior knowledge into kernel methods. We proposed invariant kernels for existing algorithms, developed new algorithms and adapted a technique taken from semi-supervised learning for invariant learning. In all these cases, links with related state-of-the-art approaches were investigated. Several illustrative experiments were carried out on real data on optical character recognition, face image classification, brain-computer interfaces, and a number of benchmark and synthetic datasets.

Fog computing is a fundamental facilitating technology in the future of networks. It broadens cloud computing services to its network edge, so that fog computing is able to bear the load of different emerging applications like IoT, blockchain, and big data without incurring high latency or cost of bandwidth consumption. In order to reach the optimal capacity of fog computing, designing an incentive mechanism for its service provider is essential. Over the recent years, we have noticed steady growth in the adoption of Machine Learning not only for the enhancement of fog computing applications but also for providing fog services, which include better resource management, improved security, reduced energy consumption, latency and traffic modelling. There hasn’t been a study yet, which investigates the role of Machine Learning in fog computing paradigm and this is what our current research aims to shed light on. Machine Learning application used for fog computing needs high layers services profound analytics, a strong end-user as well as smart responses for its designated tasks. Here, in this paper we work on presenting a comprehensive study that underlines the current advancements in Machine Learning techniques associated with the management of three important aspects of fog computing: accuracy, resource, security, and, as well as highlighting the role of Machine Learning in edge computing.

An efficient stripped cover-based accelerator for reduction of attributes in incomplete decision tables