Linear Least-Squares algorithms for temporal difference learning

We introduce two new temporal difference (TD) algorithms based on the theory of linear least-squares function approximation. We define an algorithm we call Least-Squares TD (LS TD) for which we prove probability-one convergence when it is used with a function approximator linear in the adjustable parameters. We then define a recursive version of this algorithm, Recursive Least-Square TD (RLS TD). Although these new TD algorithms require more computation per time-step than do Sutton‘s TD(λ) algorithms, they are more efficient in a statistical sense because they extract more information from training experiences. We describe a simulation experiment showing the substantial improvement in learning rate achieved by RLS TD in an example Markov prediction problem. To quantify this improvement, we introduce the TD error variance of a Markov chain, σTD, and experimentally conclude that the convergence rate of a TD algorithm depends linearly on σTD. In addition to converging more rapidly, LS TD and RLS TD do not have control parameters, such as a learning rate parameter, thus eliminating the possibility of achieving poor performance by an unlucky choice of parameters.

Sensitivity analysis of neural adaptive filters with respect to the slope parameter of a neuron activation function is performed. The analysis is provided both for a feedforward neural adaptive filter and a recurrent perceptron. The slope affects stability and convergence characteristics of a filter via inherent relationship between the slope and the learning rate parameter. In addition, it determines character of an activation function, i.e. whether it is contractive or expansive mapping. Presented analysis shows that gradient-descent based learning algorithms with an adaptive learning rate significantly reduce sensitivity of a neural adaptive filter with respect to the slope parameter, when compared with learning algorithms with a constant learning rate. Experimental results on the test speech and HRV signals support the analysis.

We study a model of on-line learning, the on-line Gibbs algorithm (OLGA), which is particularly suitable for supervised learning of realizable and unrealizable discrete valued functions. The learning algorithm is based on an on-line energy function E that balances the need to minimize the instantaneous error against the need to minimize the change in the weights. The relative importance of these terms in E is determined by a parameter \ensuremath{\lambda}, the inverse of which plays a similar role as the learning rate parameters in other on-line learning algorithms. In the stochastic version of the algorithm, following each presentation of a labeled example the new weights are chosen from a Gibbs distribution with the on-line energy E and a temperature parameter T. In the zero-temperature version of OLGA, at each step, the new weights are those that minimize the on-line energy E. The generalization performance of OLGA is studied analytically in the limit of small learning rate, i.e., $\stackrel{\ensuremath{\rightarrow}}{\mathrm{\ensuremath{\lambda}}}$\ensuremath{\infty}. It is shown that at finite temperature OLGA converges to an equilibrium Gibbs distribution of weight vectors with an energy function which is equal to the generalization error function. In its deterministic version OLGA converges to a local minimum of the generalization error. The rate of convergence is studied for the case in which the parameter \ensuremath{\lambda} increases as a power law in time. It is shown that in a generic unrealizable dichotomy, the asymptotic rate of decrease of the generalization error is proportional to the inverse square root of presented examples. This rate is similar to that of batch learning. The special cases of learning realizable dichotomies or dichotomies with output noise are also treated.

The Research aimst to compare backpropagation and Learning Vector Quantization (LVQ) methods in exploring the potential of new students at STMIK PalComTech. Comparisons in this study involve four input variables used which consist of four basic subjects of informatics engineering and information systems (math, basic programming, computer networks and management bases) which then make informatics techniques and information systems as outputs, to get the accuracy level high in this study, the researchers used several variations of parameters which eventually produced the best accuracy of the two methods. From 120 data tested using variations in test data and training data which are then processed using variations in the learning rate parameters and epochs. From the test results obtained the level of accuracy of pattern recognition in the backpropagation method is 99.17% with a learning rate variation of 0.1 and epoch 100, the learning vector quantization method has an accuracy rate of 96.67% with a variation of learning rate 1 and epoch 20 From the results of the comparison the Backpropagation method is superior in terms of accuracy so that it becomes the right method to use in exploring the potential of new students at STMIK PalComTech.

On the accuracy of covariance matrix: Hessian versus Gauss–Newton methods in atmospheric remote sensing with infrared spectroscopy

their Fourier series," Mat. Zametki, ~, 291-300 (1968). 13. S. B. Stechkin, "An estimation of the remainders of the Fourier series of differentiable functions," Tr. Mat. Inst. Akad. Nauk SSSR, 145, 126-151 (1980). 14. A. I. Stepanets, Classification and Approximation of Periodic Functions [in Russian], Naukova Dumka, Kiev (1987). 15. H. B. Dwight, Tables of Integrables and Other Mathematical Data, 4th edn., The Macmillan Company, New York (1961). 16. S. M. Nikol'skii, "Approximation of functions in the mean by trigonometric polynomials," Izv. Akad. Nauk SSSR, Ser. Mat., i0, 207-256 (1946). 17. N. I. Akhiezer, Lectures on Approximation Theory [in Russian], Nauka, Moscow (1965). 18. S. B. Stechkin and S. A. Telyakovskii, "On approximation of differentiable functions by trigonometric polynomials in the L metric," Tr. Mat. Inst. Akad. Nauk SSSR, 88, 20-29 (1967).

The approximation of functions and linear functionals: best vs. good approximation

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD( λ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an ϵ accuracy, the number of iterations needed is [Formula: see text] in the general case, where ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD( λ), which is referred to as AdaTD( λ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD( λ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches.


 
 
 
 One of the weakness in backpropagation Artificial neural network(ANN) is being stuck in local minima. Learning rate parameter is an important parameter in order to determine how fast the ANN Learning. This research is conducted to determine a method of finding the value of learning rate parameter using a genetic algorithm when neural network learning stops and the error value is not reached the stopping criteria or has not reached the convergence. Genetic algorithm is used to determine the value of learning rate used is based on the calculation of the fitness function with the input of the ANN weights, gradient error, and bias. The calculation of the fitness function will produce an error value of each learning rate which represents each candidate solutions or individual genetic algorithms. Each individual is determined by sum of squared error value. One with the smallest SSE is the best individual. The value of learning rate has chosen will be used to continue learning so that it can lower the value of the error or speed up the learning towards convergence.
 
 
 
 
 The final result of this study is to provide a new solution to resolve the problem in the backpropagation learning that often have problems in determining the learning parameters. These results indicate that the method of genetic algorithms can provide a solution for backpropagation learning in order to decrease the value of SSE when learning of ANN has been static in large error conditions, or stuck in local minima

Residual gradient (RG) was proposed as an alternative to TD(0) for policy evaluation when function approximation is used, but there exists little formal analysis comparing them except in very limited cases. This paper employs techniques from online learning of linear functions and provides a worst-case (non-probabilistic) analysis to compare these two types of algorithms when linear function approximation is used. No statistical assumptions are made on the sequence of observations, so the analysis applies to non-Markovian and even adversarial domains as well. In particular, our results suggest that RG may result in smaller temporal differences, while TD(0) is more likely to yield smaller prediction errors. These phenomena can be observed even in two simple Markov chain examples that are non-adversarial.

Jacobi–Poisson type operators as a method of summing Fourier series constructed using orthogonal polynomials play an important role in mathematical modeling problems and optimal solution theory. The work is devoted to the issue of approximation of functions, which are given on the segment [-1; 1] and satisfy the Lipschitz condition on it, by their Jacobi–Poisson type operators constructed using a system of orthogonal Jacobi polynomials. In particular, at each point x of the segment [-1; 1], we have established integral representations for the exact upper bounds of the deviations of Jacobi–Poisson type operators from functions of the class Lip[-1; 1] α for all 0<α ≤1. Solving many problems in both the theory of function approximation and systems analysis ultimately boils down to studying certain integral representations of the corresponding quantities. For this purpose, integral representations of the exact upper bounds of the deviations of Jacobi–Poisson type operators from functions of the Lipshitz class are established in the work. Keywords: optimal solution theory, Jacobi–Poisson type operators, Lipschitz class, integral representation.

Functional Analysis, Approximation Theory and Numerical Analysis

Backpropagation is a supervised learning algorithm for training multi-layer neural networks for function approximation and pattern classification by minimizing a suitably defined error metric (e.g., the mean square error between the desired and actual outputs of the network for a training set) using gradient descent. It does this by calculating the partial derivative of the overall error and changing each weight by a small amount (determined by the learning rate) in a direction that is expected to reduce the error. Despite its success on a number of real-world problems, backpropagation can be very slow (it requires hundreds of passes (epochs) through the training set). Also, its performance is extremely sensitive to the choice of parameters such as the learning rate. The mathematical considerations that go into the derivation of backpropagation require that the learning rate be as small as possible. On the other hand, in order reduce the number training epochs required to learn the desired input-output mapping, it is desirable to make the weight changes as large as possible without causing the error to increase. Also it is desirable to have an algorithm that can change the learning rate dynamically so that it is close to optimal (in terms of reducing the error as much as possible given the local gradient information) at each epoch (thereby eliminating the reliance on guesswork in the choice of the learning rate). Several authors have proposed methods for adaptively adjusting the learning rate based on certain assumptions about the shape of the error surface (e.g., quadratic) and/or estimation of higher order derivatives of the error with respect to the weights at a given point in the weight space. The primary disadvantage of such methods is that the estimation of second order derivatives is not only computationally expensive but also prone to inaccuracy due to the approximation of derivatives by discrete differences. In this paper we propose and evaluate a heuristically motivated method for adaptive modification of the learning rate in backpropagation that does not require the estimation of higher order derivatives. We present a modified version of the Backpropagation learning algorithm which uses a simple heuristic to come up with a learning parameter value at each epoch. We present numerous simulations on real-world data sets, using our modified algorithm. We compare these results with results got through standard backpropagation learning algorithm, and also various modifications of the standard backpropagation algorithm (e.g., flat-spot elimination methods) that have been discussed in the literature. Our simulation results suggest that the adaptive learning rate modification helps substantially speed up the convergence of backpropagation algorithm. Furthermore, it makes the initial choice of the learning rate fairly unimportant as our method allows the learning rate to change and settle at a reasonable value for the specific problem. As a standard against which to compare our results, we computed the quasi-optimal value of the learning parameter at each epoch. Simulation results indicate that our heuristic modification matches the performance of backpropagation with the quasi-optimal learning rate. The computational complexity of this algorithm is analysed and compared with that of standard backpropagation.

The learning rate is analyzed for linear learning algorithms in this paper. In the presence of outliers, the robustness of several linear learning algorithms is given and it is shown that an absolute criterion based learning algorithm is more robust than the corresponding quadratic criterion based learning algorithm.

In reinforcement learning, when the state space is enormous or infinite, it is not feasible to find the exact value for each state in the memory. A common way to tackle this problem is to adopt linear value function approximation technique. In this paper, we review some commonly used linear value function approximation algorithms, which divided into two classes. The first class is the so-called on-policy algorithms that mainly include temporal difference and least-squares-based methods. The second is the off-policy algorithms that use two strategies to solve the exploration-exploitation problem. From the perspective of algorithm design, on-policy is simple, but it does not take both exploration and exploitation into consideration when collecting data, while the off-policy can well balance the relation between exploration and exploitation. In the end, we first present some experiments to demonstrate the usefulness of linear value function approximation technique in the continuous and high dimensional scenes, and then in-class and out-class experiments are respectively conducted to further illustrate the performance of different approximation algorithms.KeywordsReinforcement learningLinear value function approximationLeast-squares temporal difference