Structural and Statistical Analysis of Multidimensional Linear Approximations of Random Functions and Permutations

Residual gradient (RG) was proposed as an alternative to TD(0) for policy evaluation when function approximation is used, but there exists little formal analysis comparing them except in very limited cases. This paper employs techniques from online learning of linear functions and provides a worst-case (non-probabilistic) analysis to compare these two types of algorithms when linear function approximation is used. No statistical assumptions are made on the sequence of observations, so the analysis applies to non-Markovian and even adversarial domains as well. In particular, our results suggest that RG may result in smaller temporal differences, while TD(0) is more likely to yield smaller prediction errors. These phenomena can be observed even in two simple Markov chain examples that are non-adversarial.

This paper presents an approach to combinatorial optimization problems using linear function approximation (LFA) to solve the Travelling Salesman Problem (TSP). We create a reinforcement learning model in which we parameterize our policy using linear function approximation instead of the more commonly used neural networks. We then evaluated our models based on two factors: training time and optimality. When we compared our results with a state-of-the-art neural network solver, we found that our model was able to solve the TSP accurately while using drastically less computational resources and time to train than the neural network algorithm (Kool et al., 2019).

In reinforcement learning, when the state space is enormous or infinite, it is not feasible to find the exact value for each state in the memory. A common way to tackle this problem is to adopt linear value function approximation technique. In this paper, we review some commonly used linear value function approximation algorithms, which divided into two classes. The first class is the so-called on-policy algorithms that mainly include temporal difference and least-squares-based methods. The second is the off-policy algorithms that use two strategies to solve the exploration-exploitation problem. From the perspective of algorithm design, on-policy is simple, but it does not take both exploration and exploitation into consideration when collecting data, while the off-policy can well balance the relation between exploration and exploitation. In the end, we first present some experiments to demonstrate the usefulness of linear value function approximation technique in the continuous and high dimensional scenes, and then in-class and out-class experiments are respectively conducted to further illustrate the performance of different approximation algorithms.KeywordsReinforcement learningLinear value function approximationLeast-squares temporal difference

Finite-Time Performance of Distributed Temporal-Difference Learning with Linear Function Approximation

We introduce a diffusion-based algorithm in which multiple agents cooperate to predict a common and global state-value function by sharing local estimates and local gradient information among neighbors. Our algorithm is a fully distributed implementation of the gradient temporal difference with linear function approximation, to make it applicable to multiagent settings. Simulations illustrate the benefit of cooperation in learning, as made possible by the proposed algorithm.

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD( λ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an ϵ accuracy, the number of iterations needed is [Formula: see text] in the general case, where ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD( λ), which is referred to as AdaTD( λ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD( λ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches.

Energy harvesting (EH) two-hop communications are considered. The transmitter and the relay harvest energy from the environment and use it exclusively for transmitting data. A data arrival process is assumed at the transmitter. At the relay, a finite data buffer is used to store the received data. We consider a realistic scenario in which the EH nodes have only local causal information, i.e., at any time instant, each EH node only knows the current value of its EH process, channel state and data arrival process. Our goal is to find a power allocation policy to maximize the throughput at the receiver. We show that because the EH nodes have local causal information, the two-hop communication problem can be separated into two point-to-point problems. Consequently, independent power allocation problems are solved at each EH node. To find the power allocation policy, reinforcement learning with linear function approximation is applied. Moreover, to perform function approximation two feature functions which consider the data arrival process are introduced. Numerical results show that the proposed approach has only a small degradation as compared to the offline optimum case. Furthermore, we show that with the use of the proposed feature functions a better performance is achieved compared to standard approximation techniques.

Many authors work on constrained approximation such as monotonicity, convexity and k monotoni city, but little works introduced in positive collinear approximation. The aim of our paper is to investigate linear and positive approximation for real functions in , saturation problem between degree of best positive and linear approximation, and find collinear positive best approximation for measurable function in Lebesgue quasi normed spaces.

Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning

A higher-order version of the linear approximation of smooth functions of means proposed in Pallini (2002) is defined and studied. This version is shown to improve over the error of order Op(n -2) in probability, as the sample size n diverges, yielding a smaller error of order Op(n -3), as n diverges. Both linear approximations are shown to have a normal distribution, as diverges. Empirical results of a simulation study on the ratio of means example are presented.

In the paper the differential and the linear approximations of two classes of S-box functions are considered. The classes are the permutations and arbitrary functions with n binary inputs and m binary outputs, where 1 ≤ n=m ≤10. For randomly chosen functions from each of the classes, the distribution of the best nonzero approximations is investigated. The based on the definitions of differential and linear approximation algorithms to compute a single element of the approximation tables, are of exponential complexity. The presented in the paper fast algorithms compute the best nonzero approximations in at worst linear time for a single element, without memory needed for storage of the whole table.

On the accuracy of covariance matrix: Hessian versus Gauss–Newton methods in atmospheric remote sensing with infrared spectroscopy

In the paper two methods of linear approximation of n-bit arithmetic subtraction function are considered. In the first method, called the model of approximation of a single S-box, approximations are calculated for arbitrary m consecutive bits, where m ≤ n is limited by the size of so-called table of pairs TP, used during calculation. In the second method, called the model of exact composition of approximations, the subtraction approximations are calculated as a composition of k approximations of m-bit subtraction cells, where m ≤ n is limited by the size of the same table of pairs TP. In the first method, the set of nonzero approximations is limited to approximations in the range of m consecutive bits while in the second method is not limited. For n-bit arithmetic subtraction function however, the approximation probability can be calculated with use of the methods in time O(l) and O(k), respectively.

We describe the Fourier basis, a linear value function approximation scheme based on the Fourier series. We empirically demonstrate that it performs well compared to radial basis functions and the polynomial basis, the two most popular fixed bases for linear value function approximation, and is competitive with learned proto-value functions.

The task of simultaneous localization and mapping (SLAM) is to build environmental map and locate the position of mobile robot at the same time. FastSLAM 2.0 is one of powerful techniques to solve the SLAM problem. However, there are two obvious limitations in FastSLAM 2.0, one is the linear approximations of nonlinear functions which would cause the filter inconsistent and the other is the “particle depletion” phenomenon. A kind of PSO & H∞-based FastSLAM 2.0 algorithm is proposed. For maintaining the estimation accuracy, H∞ filter is used instead of EKF for overcoming the inaccuracy caused by the linear approximations of nonlinear functions. The unreasonable proposal distribution of particle greatly influences the pose state estimation of robot. A new sampling strategy based on PSO (particle swarm optimization) is presented to solve the “particle depletion” phenomenon and improve the accuracy of pose state estimation. The proposed approach overcomes the obvious drawbacks of standard FastSLAM 2.0 algorithm and enhances the robustness and efficiency in the parts of consistency of filter and accuracy of state estimation in SLAM. Simulation results demonstrate the superiority of the proposed approach.