ON THE ALMOST SURE RATE OF CONVERGENCE OF TEMPORAL-DIFFERENCE LEARNING ALGORITHMS

We give a new take on the error analysis of approximations of stochastic differential equations (SDEs), utilizing and developing the stochastic sewing lemma of Lê (Electron J Probab 25:55, 2020. https://doi.org/10.1214/20-EJP442). This approach allows one to exploit regularization by noise effects in obtaining convergence rates. In our first application we show convergence (to our knowledge for the first time) of the Euler–Maruyama scheme for SDEs driven by fractional Brownian motions with non-regular drift. When the Hurst parameter is Hin (0,1) and the drift is mathcal {C}^alpha , alpha in [0,1] and alpha >1-1/(2H), we show the strong L_p and almost sure rates of convergence to be ((1/2+alpha H)wedge 1) -varepsilon , for any varepsilon >0. Our conditions on the regularity of the drift are optimal in the sense that they coincide with the conditions needed for the strong uniqueness of solutions from Catellier and Gubinelli (Stoch Process Appl 126(8):2323–2366, 2016. https://doi.org/10.1016/j.spa.2016.02.002). In a second application we consider the approximation of SDEs driven by multiplicative standard Brownian noise where we derive the almost optimal rate of convergence 1/2-varepsilon of the Euler–Maruyama scheme for mathcal {C}^alpha drift, for any varepsilon ,alpha >0.

Eweda and Macchi [IEEE Trans. Automat. Control, 29 (1984), pp. 119–127] and Watanabe [IEEE Trans. Inform. Theory, 30 (1984), pp. 134–140] show that thee sequence of random vectors generated by a stochastic gradient adaptive filtering algorithm converges almost surely and in $L_p $ (for p an even integer) to the solution of the associated Wiener–Hopf equation when the driving data process is stationary and weakly dependent. Under strong (i.e., Rosenblatt or $\alpha $) and $\psi $ -mixing conditions, together with various moment bounds on the driving data process, an almost sure functional invariance principle is obtained that approximates the sample paths of the random process generated by the stochastic gradient algorithm with the sample paths of a particular Gauss–Markov process. Almost sure rates of convergence in the form of laws of the iterated logarithm follow from the functional invariance principle. As a byproduct a functional central limit theorem is also obtained for a sequence of processes derived by suitably scaling the sequence of iterations generated by the algorithm.

We consider a simple estimator of the density mode using the frequency polygon estimate. We investigate strong consistency of the estimator for strong mixing sequence of real variables under mild assumptions. We study the almost sure rate of convergence and we show that the estimator can achieve optimal almost sure rates of convergence for appropriate choices of the bin widths. The asymptotic normality of the simple estimator is given and a simulation study is performed. Our asymptotic results are obtained without any differentiability condition assumed on the density around the mode.

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD( λ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an ϵ accuracy, the number of iterations needed is [Formula: see text] in the general case, where ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD( λ), which is referred to as AdaTD( λ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD( λ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches.

In this paper we provide the asymptotic theory of the general of φ-divergences measures, which include the most common divergence measures : R´enyi and Tsallis families and the Kullback-Leibler measure. We are interested in divergence measures in the discrete case. One sided and two-sided statistical tests are derived as well as symmetrized estimators. Almost sure rates of convergence and asymptotic normality theorem are obtained in the general case, and next particularized for the R´enyi and Tsallis families and for the Kullback-Leibler measure as well. Our theoretical results are validated by simulations.

In this article we quantify almost sure martingale convergence theorems in terms of the tradeoff between asymptotic almost sure rates of convergence (error tolerance) and the respective modulus of convergence.For this purpose we generalize an elementary quantitative version of the first Borel-Cantelli lemma on the statistics of the deviation frequencies, which was recently established by the authors.First we study martingale convergence in L2, and in the setting of the Azuma-Hoeffding inequality.In a second step we study the strong law of large numbers for martingale differences.Applications are the tradeoff for the multicolor generalized Plya urn processes, the generalized Chinese restaurant process, statistical M-estimators, as well as excursion frequencies of the Galton-Watson branching process.

For pt.I see ibid., vol.9, p.11 (1976). When conventional variational trial functions are used to solve systems such as Schrodinger's equation with discontinuous potentials, it is found that the convergence rate is very slow when compared with that for continuous potentials. In a previous paper, the authors demonstrated how this convergence rate could be improved using local variational methods (i.e. finite elements). In this paper they demonstrate the construction of trial functions for global variational methods which results in a very substantial improvement in the obtained convergence rate. The technique, which is numerically stable, is based on orthogonal polynomials with suitable core functions incorporated.

In the present study, a temporal difference (TD) learning algorithm is proposed to solve the optimal control problem for semi‐Markov jump linear systems (S‐MJLSs). The proposed scheme is TP‐free so that it can be applied in cases without pre‐known transition probabilities of embedded Markov chain. Coupled algebraic Riccati equations (CAREs) implied with the analytical solution of control gains are derived by utilizing a S‐MJLS augmented with maximum sojourn time, which contributes to develop the TD learning algorithm. It is proved that for sufficiently rich enough jumping modes and jumping numbers observed online, the value function in TD algorithm converges to CAREs solutions. Finally, an example is carried out to evaluate the learning capability of TD algorithm and the effectiveness of the proposed control method.

We consider the random continued fraction where s n are independent random variables with the same gamma distribution. Every realization of the sequence defines a Stieltjes function that can be expressed as for some measure σ on the positive half-line. We study the convergence of the finite truncations of the continued fraction or, equivalently, of the diagonal Padé approximants of the function S . Using the Dyson–Schmidt method for an equivalent one-dimensional disordered system and the results of Marklof et al. , we obtain explicit formulae (in terms of modified Bessel functions) for the almost sure rate of convergence of these approximants, and for the almost sure distribution of their poles.

Soft sensing methods were widely studied due to their attractive properties. The soft sensing models based on supervised learning neural networks were well researched in the last decade. This paper proposes a soft sensing method based on the temporal difference (TD) learning. TD methods are more preferable to deal with multi-step prediction problems that involve temporal sequences of a dynamical process. The soft sensor is implemented with an Elman neural network, a multilayer network with local feedback, which is trained by the TD algorithm. Finally, the TD-based soft sensor is applied to the Kappa number prediction in the batch kraft pulping process

The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.

In this paper, an online selective kernel-based temporal difference (OSKTD) learning algorithm is proposed to deal with large scale and/or continuous reinforcement learning problems. OSKTD includes two online procedures: online sparsification and parameter updating for the selective kernel-based value function. A new sparsification method (i.e., a kernel distance-based online sparsification method) is proposed based on selective ensemble learning, which is computationally less complex compared with other sparsification methods. With the proposed sparsification method, the sparsified dictionary of samples is constructed online by checking if a sample needs to be added to the sparsified dictionary. In addition, based on local validity, a selective kernel-based value function is proposed to select the best samples from the sample dictionary for the selective kernel-based value function approximator. The parameters of the selective kernel-based value function are iteratively updated by using the temporal difference (TD) learning algorithm combined with the gradient descent technique. The complexity of the online sparsification procedure in the OSKTD algorithm is O(n). In addition, two typical experiments (Maze and Mountain Car) are used to compare with both traditional and up-to-date O(n) algorithms (GTD, GTD2, and TDC using the kernel-based value function), and the results demonstrate the effectiveness of our proposed algorithm. In the Maze problem, OSKTD converges to an optimal policy and converges faster than both traditional and up-to-date algorithms. In the Mountain Car problem, OSKTD converges, requires less computation time compared with other sparsification methods, gets a better local optima than the traditional algorithms, and converges much faster than the up-to-date algorithms. In addition, OSKTD can reach a competitive ultimate optima compared with the up-to-date algorithms.

Motivated by applications of distributed linear estimation, distributed control, and distributed optimization, we consider the question of designing linear iterative algorithms for computing the average of numbers in a network. Specifically, our interest is in designing such an algorithm with the fastest rate of convergence given the topological constraints of the network. As the main result of this paper, we design an algorithm with the fastest possible rate of convergence using a nonreversible Markov chain on the given network graph. We construct such a Markov chain by transforming the standard Markov chain, which is obtained using the Metropolis-Hastings method. We call this novel transformation <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pseudo-lifting</i> . We apply our method to graphs with geometry, or graphs with doubling dimension. Specifically, the convergence time of our algorithm (equivalently, the mixing time of our Markov chain) is proportional to the diameter of the network graph and hence optimal. As a byproduct, our result provides the fastest mixing Markov chain given the network topological constraints, and should naturally find their applications in the context of distributed optimization, estimation and control.

Preface. 1. Simulations and the Monte Carlo method. 1.1 Description of the method. 1.2 Convergence theorems. 1.3 Simulation of random variables. 1.4 Variance reduction techniques. 1.5 Exercises. 2. Markov chains. 2.1 Definitions and elementary properties. 2.2 Examples. 2.3 Strong Markov property. 2.4 Recurrent and transient states. 2.5 The irreducible and recurrent case. 2.6 The aperiodic case. 2.7 Reversible Markov chain. 2.8 Rate of convergence to equilibrium. 2.9 Statistics of Markov chains. 2.10 Exercises. 3. Stochastic algorithms. 3.1 Markov chain Monte Carlo. 3.2 Simulation of the invariant probability. 3.3 Rate of convergence towards the invariant probability. 3.4 Simulated annealing. 3.5 Exercises. 4. Markov chains and the genome. 4.1 Reading DNA. 4.2 The i.i.d. model. 4.3 The Markov model. 4.4 Hidden Markov models. 4.5 Hidden semi-Markov model. 4.6 Alignment of two sequences. 4.7 A multiple alignment algorithm. 4.8 Exercises. 5. Control and filtering of Markov chains. 5.1 Deterministic optimal control. 5.2 Control of Markov chains. 5.3 Linear quadratic optimal control. 5.4 Filtering of Markov chains. 5.5 The Kalman-Bucy filter. 5.6 Linear-quadratic control with partial observation. 5.7 Exercises. 6. The Poisson process. 6.1 Point processes and counting processes. 6.2 The Poisson process. 6.3 The Markov property. 6.4 Large time behaviour. 6.5 Exercises. 7. Jump Markov processes. 7.1 General facts. 7.2 Infinitesimal generator. 7.3 The strong Markov property. 7.4 Embedded Markov chain. 7.5 Recurrent and transient states. 7.6 The irreducible recurrent case. 7.7 Reversibility. 7.8 Markov models of evolution and phylogeny. 7.9 Application to discretized partial differential equations. 7.10 Simulated annealing. 7.11 Exercises. 8. Queues and networks. 8.1 M/M/1 queue. 8.2 M/M/1/K queue. 8.3 M/M/s queue. 8.4 M/M/s/s queue. 8.5 Repair shop. 8.6 Queues in series. 8.7 M/G/ queue. 8.8 M/G/1 queue. 8.9 Open Jackson network. 8.10 Closed Jackson network. 8.11 Telephone network. 8.12 Kelly networks. 8.13 Exercises. 9. Introduction to mathematical finance. 9.1 Fundamental concepts. 9.2 European options in the discrete model. 9.3 The Black-Scholes model and formula. 9.4 American options in the discrete model. 9.5 American options in the Black-Scholes model. 9.6 Interest rate and bonds. 9.7 Exercises. 10. Solutions to selected exercises. 10.1 Chapter 1. 10.2 Chapter 2. 10.3 Chapter 3. 10.4 Chapter 4. 10.5 Chapter 5. 10.6 Chapter 6. 10.7 Chapter 7. 10.8 Chapter 8. 10.9 Chapter 9. References Index.

In this paper, we investigate the convergence rates in geometric ergodicity for discrete-time Markov chains on general state spaces. When the state space contains an atom,we study the convergence rates by the spectral theory,which is similar to that on countable state spaces.However, atoms are less frequent. For the general case,we first use the Nummelin splitting technique to construct a newMarkov chain which contains an atom, and then establish theconnection of the geometrically ergodic rates between the newand the original Markov chains. We also obtain the convergence rate in geometric ergodicityfor the original Markov chain by the minorization conditionand the geometric moment of the first return time.Moreover, we apply the main result to study the Metropolis-Hastings algorithm.