Importance sampling policy gradient algorithms in reproducing kernel Hilbert space

Regularization is used to find a solution that both fits the data and is sufficiently smooth, and thereby is very effective for designing and refining learning algorithms. But the influence of its exponent remains poorly understood. In particular, it is unclear how the exponent of the reproducing kernel Hilbert space (RKHS) regularization term affects the accuracy and the efficiency of kernel-based learning algorithms. Here we consider regularized least squares regression (RLSR) with an RKHS regularization raised to the power of m, where m is a variable real exponent. We design an efficient algorithm for solving the associated minimization problem, we provide a theoretical analysis of its stability, and we compare it with respect to computational complexity and prediction accuracy to the classical kernel ridge regression algorithm where the regularization exponent m is fixed at 2. Our results show that the m-power RLSR problem can be solved efficiently, and support the suggestion that one can use a regularization term that grows significantly slower than the standard quadratic growth in the RKHS norm.

This article proposes a novel approach to linear dimension reduction for regression using nonparametric estimation with positive-definite kernels or reproducing kernel Hilbert spaces (RKHSs). The purpose of the dimension reduction is to find such directions in the explanatory variables that explain the response sufficiently: this is called sufficient dimension reduction. The proposed method is based on an estimator for the gradient of the regression function considered for the feature vectors mapped into RKHSs. It is proved that the method is able to estimate the directions that achieve sufficient dimension reduction. In comparison with other existing methods, the proposed one has wide applicability without strong assumptions on the distributions or the type of variables, and needs only eigendecomposition for estimating the projection matrix. The theoretical analysis shows that the estimator is consistent with certain rate under some conditions. The experimental results demonstrate that the proposed method successfully finds effective directions with efficient computation even for high-dimensional explanatory variables.

The general perception is that kernel methods are not scalable, so neural nets become the choice for large-scale nonlinear learning problems. Have we tried hard enough for kernel methods? In this paper, we propose an approach that scales up kernel methods using a novel concept called doubly stochastic functional gradients''. Based on the fact that many kernel methods can be expressed as convex optimization problems, our approach solves the optimization problems by making two unbiased stochastic approximations to the functional gradient---one using random training points and another using random features associated with the kernel---and performing descent steps with this noisy functional gradient. Our algorithm is simple, need no commit to a preset number of random features, and allows the flexibility of the function class to grow as we see more incoming data in the streaming setting. We demonstrate that a function learned by this procedure after t iterations converges to the optimal function in the reproducing kernel Hilbert space in rate O(1/t), and achieves a generalization bound of O(1/\sqrt{t}). Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show competitive performances of our approach as compared to neural nets in datasets such as 2.3 million energy materials from MolecularSpace, 8 million handwritten digits from MNIST, and 1 million photos from ImageNet using convolution features.

Laplacian least squares twin support vector machine for semi-supervised classification

The combination of the famed kernel trick and the least-mean-square (LMS) algorithm provides an interesting sample-by-sample update for an adaptive filter in reproducing kernel Hilbert spaces (RKHS), which is named in this paper the KLMS. Unlike the accepted view in kernel methods, this paper shows that in the finite training data case, the KLMS algorithm is well posed in RKHS without the addition of an extra regularization term to penalize solution norms as was suggested by Kivinen [Kivinen, Smola and Williamson, ldquoOnline Learning With Kernels,rdquo IEEE Transactions on Signal Processing, vol. 52, no. 8, pp. 2165-2176, Aug. 2004] and Smale [Smale and Yao, ldquoOnline Learning Algorithms,rdquo Foundations in Computational Mathematics, vol. 6, no. 2, pp. 145-176, 2006]. This result is the main contribution of the paper and enhances the present understanding of the LMS algorithm with a machine learning perspective. The effect of the KLMS step size is also studied from the viewpoint of regularization. Two experiments are presented to support our conclusion that with finite data the KLMS algorithm can be readily used in high dimensional spaces and particularly in RKHS to derive nonlinear, stable algorithms with comparable performance to batch, regularized solutions.

We investigate finding a map $g$ within a function class $G$ that minimises an Optimal Transport (OT) cost between a target measure $\nu$ and the image by $g$ of a source measure $\mu$. This is relevant when an OT map from $\mu$ to $\nu$ does not exist or does not satisfy the desired constraints of $G$. We address existence and uniqueness for generic subclasses of $L$-Lipschitz functions, including gradients of (strongly) convex functions and typical Neural Networks. We explore a variant that approaches a transport plan, showing equivalence to a map problem in some cases. For the squared Euclidean cost, we propose alternating minimisation over a transport plan $\pi$ and map $g$, with the optimisation over $g$ being the $L^2$ projection on $G$ of the barycentric mapping $\overline{\pi}$. In dimension one, this global problem equates the $L^2$ projection of $\overline{\pi^*}$ onto $G$ for an OT plan $\pi^*$ between $\mu$ and $\nu$, but this does not extend to higher dimensions. We introduce a simple kernel method to find $g$ within a Reproducing Kernel Hilbert Space in the discrete case. We present numerical methods for $L$-Lipschitz gradients of $\ell$-strongly convex potentials, and study the convergence of SGD methods for Neural Networks.

How to evaluate Pareto front approximations generated by multi/many-objective optimizers is a critical issue in the field of multiobjective optimization. Currently, there exist two types of comprehensive quality indicators (i.e., volume-based and distance-based indicators). Distance-based indicators, such as inverted generational distance (IGD), are usually computed by summing up the distance of each reference point to its nearest solution. Their high computational efficiency leads to their prevalence in many-objective optimization. However, in the existing distance-based indicators, the distributions of the solution sets are usually neglected, leading to their lack of ability to well distinguish between different solution sets. This phenomenon may become even more severe in high-dimensional space. To address such an issue, a kernel-based indicator (KBI) is proposed as a comprehensive indicator. Different from other distance-based indicators, a kernel-based maximum mean discrepancy is adopted in KBI for directly measuring the difference that can characterize the convergence, spread, and uniformity of two sets, i.e., the solution set and reference set, by embedding them in reproducing kernel Hilbert space (RKHS). As a result, KBI not only reflects the distance between the solution set and the reference set but also can reflect the distribution of the solution set itself. In addition, to maintain the desirable weak Pareto compliance property of KBI, a nondominated set reconstruction approach is also proposed to shift the original solution set. The detailed theoretical and experimental analysis of KBI is provided in this article. The properties of KBI have also been analyzed by the optimal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu $ </tex-math></inline-formula> -distribution.

Active learning based on minimization of the expected path-length of random walks on the learned manifold structure

Dimensionality reduction has emerged as one of the prominent fields of research since it provides a solution to a wide class of problems, such as compression, classification, regression, feature analysis, and visual recognition. Generally, subspace learning algorithms find a low-dimensional subspace from a given high-dimensional data, wherein samples from different classes can be well separated. In the past decades, several types of dimensionality reduction and subspace selection algorithms have been widely used for visual data analysis. Conventional algorithms such as principal component analysis (PCA) and linear discriminant analysis (LDA), perform well under the assumption that training and test data follow similar distributions. However, the main drawback still remains in the sense that the distribution of the training data and that of the test data are mismatched when samples are drawn from different but related sources. They also fail to incorporate temporal statistical variations in data distribution when applied to time-series data(videos). Consequently they result in unsatisfactory recognition performance when they deal with real world visual analysis problems. To overcome the above issues, in this research, our aim is to consider invariance and stationarity in subspace analysis pertaining to many computer vision applications such as video retrieval, human behaviour analysis, event analysis and activity recognition. More specifically, we will address the challenging and fundamental tasks of 1. Video Classification (Scene, dynamic texture and action recognition) 2. Visual Domain Adaptation. To address the first task, we propose a subspace learning approach that focuses on extracting the stationary parts of all videos in the same class. The notion of stationarity is intuitively well-adapted to model the temporal nature of the video signal and lets us make use of many image features. Instead of modeling temporal information in the features, our method explicitly accounts for it when learning the latent space. As a consequence, the resulting video representation is particularly well-suited for classification purpose. Our experimental evaluation shows that our approach outperforms baselines from the different groups of methods in several video classification tasks such as dynamic texture recognition, scene classification and action recognition.The second task addressed by this work is domain adaptation for visual recognition. In our proposed approach, we follow the intuitive idea of learning the domain invariant subspace by matching the distributions of the transformed training and test data using Maximum Mean Discrepancy (MMD). This, we believe, makes better use of the expressive power of the kernel in MMD compared to the other approaches like Transfer Component Analysis (TCA). Although motivated by MMD, in TCA, the distance between the sample means is measured in a lower-dimensional space rather than in Reproducing Kernel Hilbert Space (RKHS), which somewhat contradicts the intuition behind the use of kernels. Furthermore, we extend the framework for the semi-supervised scenario. Experiments on benchmark domain adaptation dataset for visual recognition, show that in comparison to well-known methods, the proposed approach obtains a significant improvement in classification accuracy. To address the second task, we propose another domain adaptation method that exploits the Riemannian structure of the statistical manifold when learning the invariant subspaces or samples. To this end, we introduce the use of the Hellinger distance which is related to the geodesic distance on the space of probability distributions. While the Hellinger distance has been employed for dimensionality reduction, to the best of our knowledge, our approach is the first attempt at exploiting the Riemannian geometry of the statistical manifold for domain adaptation. We showed that our sample-selection and subspace-based approaches in conjunction with Hellinger distance for distribution matching consistently outperform the similar approaches on the tasks of visual domain adaptation and WiFi localization.

In this paper we investigate and compare different gradient algorithms designed for the domain expression of the shape derivative. Our main focus is to examine the usefulness of kernel reproducing Hilbert spaces for PDE-constrained shape optimization problems. We show that radial kernels provide convenient formulas for the shape gradient that can be efficiently used in numerical simulations. The shape gradients associated with radial kernels depend on a so-called smoothing parameter that allows a smoothness adjustment of the shape during the optimization process. Besides, this smoothing parameter can be used to modify the movement of the shape. The theoretical findings are verified in a number of numerical experiments.

Functional data classification has gained increasing utility in the modern statistical learning era. In this article, we investigate the generalization ability and the Fisher consistency of the smoothed score (SS) classifier on the intrinsic infinite-dimensional functional data both theoretically and numerically. Formulating the empirical risk minimization based on a regularized smoothed non-convex loss function, we first establish a rigorous error bound on the misclassification rate. The theoretical results reveal a tradeoff between the choice of a tuning parameter and the size of a candidate function class. Additionally, a nonregular convergence rate of the SS estimation is derived in L 2 norm that aligns with h n − ( 1 − ν ) n − μ 2 ( μ + 1 ) , as the bandwidth h n of the smoothed loss is shrunk to 0. By projecting the functional data onto one specific direction over a reproducing kernel Hilbert space where the estimated function is expected to deliver desirable performance, we address the problem of slope function estimation. Computationally, we tackle the nonconvex optimization by developing an efficient proximal gradient algorithm. Finally, the finite sample results in simulation studies as well as a real data analysis from the ADNI study demonstrate the favorable performance of the proposed method compared with some popular classifiers in terms of prediction and estimation.

Piezoelectric inkjet printing has important applications in high-precision manufacturing, but precise control of droplet quality faces challenges such as the nonlinear coupling between drive waveform parameters and droplet characteristics, as well as environmental disturbances. To address these issues, this paper proposes a closed-loop droplet control method based on a curriculum deep deterministic policy gradient (DDPG) algorithm, with the standard deviation of droplet volume and velocity as the main optimization objectives. To tackle the complex mapping relationships and boost training efficiency, we first built a high-fidelity simulation environment using a four-layer deep residual network trained on real-world industrial ink droplet data. Following that, we designed a dynamic weight-adaptive composite reward mechanism and integrated a curriculum learning strategy with the Deep Deterministic Policy Gradient (DDPG) algorithm. Relying on our self-developed droplet observation platform, we achieve closed-loop control of the standard deviation of droplet volume and velocity. Experimental results show that this method can precisely control the droplet volume within ±0.1 pl of the target value, while keeping the velocity standard deviation below 40 mm/s. The adjustment process is stable without severe oscillations, and the method demonstrates strong robustness against environmental disturbances such as ink supply pressure. This provides an efficient and practical solution for the automated control of droplet quality in piezoelectric inkjet printing.

The design and implementation of a deep reinforcement learning and quantum finance theory-inspired portfolio investment management system

Multi-agent reinforcement learning excels at addressing group intelligent decision-making problems involving sequential decision-making. In particular, in complex, high-dimensional state and action spaces, it imposes higher demands on the reliability, stability, and adaptability of decision algorithms. The reinforcement learning algorithm based on the multi-agent deep strategy gradient incorporates a function approximation method using discriminant networks. However, this can lead to estimation errors when agents evaluate action values, thereby reducing model reliability and stability and resulting in challenging convergence. With the increasing complexity of the environment, there is a decline in the quality of experience collected by the experience playback pool, resulting in low efficiency of the sampling stage and difficulties in algorithm convergence. To address these challenges, we propose an innovative approach called the empirical clustering layer-based multi-agent dual dueling policy gradient (ECL-MAD3PG) algorithm. Experimental results demonstrate that our ECL-MAD3PG algorithm outperforms other methods in various complex environments, demonstrating a remarkable 9.1% improvement in mission completion compared to MADDPG within the context of complex UAV cooperative combat scenarios.

In today's logistics and delivery landscape, mobile robot delivery systems have attracted considerable attention due to their efficiency and adaptability. Nevertheless, current robotic delivery solutions encounter various obstacles in complex and dynamically changing environments. Traditional algorithms, for instance, struggle with processing high-dimensional and unstructured data, resulting in inefficient adaptation to real-time environmental changes, which compromises accuracy and efficiency in path planning and task execution. Moreover, the lack of robust perception and decision-making mechanisms limits the robots' ability to handle intricate scenarios and fluctuating delivery demands. To tackle these challenges, this paper proposes an optimization approach for mobile robot delivery systems that leverages deep learning. The study initially integrates a spatial attention mechanism within the model, enabling the robot to focus on critical environmental regions and dynamically adjust attention points, thus enhancing obstacle recognition and avoidance in complex settings, which improves navigation accuracy and path planning. Furthermore, the Deep Deterministic Policy Gradient (DDPG) algorithm is utilized to optimize policies, supporting efficient learning in high-dimensional continuous spaces and empowering robots to acquire effective delivery strategies in challenging environments. Finally, an end-to-end optimization approach allows the system to convert sensor inputs directly into control commands, reducing intermediate complexity and minimizing error accumulation, thereby streamlining the system’s structure. Experimental results confirm that the proposed method substantially boosts delivery system performance, excelling in key metrics like path planning accuracy, task efficiency, and system robustness. The successful integration of the spatial attention mechanism with the deep policy gradient algorithm demonstrates a valuable new approach for advancing future robot delivery system optimizations.