Fast adaptive learning in repeated stochastic games by game abstraction

An agent must learn and adapt quickly when playing against other agents. This process is challenging in particular when playing in stochastic environments against other learning agents. In this paper, we introduce a fast and adaptive learning algorithm for repeated stochastic games (FAL-SG). FAL-SG utilizes lossy game abstraction to reduce the state space of the game and facilitate learning and adapting rapidly. We analyze FAL-SG's performance by proving bounds on the abstraction loss and prediction mistakes and show that FAL-SG satisfies three criteria prescribed for multiagent learning algorithms. We successfully establish the robustness and scalability of FAL-SG with extensive theoretical and experimental results.

Bayesian spiking neurons (BSNs) provide a probablisitic and intuitive interpretation of how spiking neurons could work and have been shown to be equivalent to leaky integrate-and-fire neurons under certain conditions [1]. The study of BSNs has been restricted mainly to small networks because online learning, which currently involves a maximum-likelihood-expectation-maximisation (ML-EM) approach [2,3], is quite slow. Here a new approach to estimating the parameters of Bayesian spiking neurons, referred to as fast learning (FL), is presented and compared to online ML-EM learning. Learning in a BSN is local to the neuron and involves estimation of the transition rate and observation rate parameters of an underlying implicit hidden Markov model (HMM), the hidden state of which the BSN output encodes [1]. Rather than estimating the parameters by maximizing the log-likelihood of the hidden states and the synaptic observations given the parameters as is done in ML-EM [2,3], the FL algorithm directly calculates statistics upon which the parameters depend. This is achieved by taking advantage of the relationship between the log-odds ratio of the hidden state computed by the BSN and the probability that the hidden state is ‘on’ given the past synaptic observations, . Online learning in a two BSN neuron hierarchy is explored, where the first neuron receives N=20 synapses driven by Poisson processes and the second neuron receives input from only the first neuron. Simulations were performed for a fixed set of transition rates and observation rates for 10 different perturbations of the initial transition and observation rate estimates: ±0-20%, ±20-40%,..., ±180-200%. Initial rates were not allowed to go below 10-6. Each perturbation condition was simulated 100 times by randomly selecting the initial parameter values. Although the FL algorithm is not as exact as the ML-EM at estimating the true parameter values for small perturbations of the initial rate estimates relative to the true rates, the FL algorithm is able to reliably estimate the parameters for initial perturbations of up to 200%, whereas the ML-EM algorithm estimates begin to deviate after perturbations of about 40-60%. Moreover, the simplicity of the FL algorithm means that it runs on the order of 25 times faster than the ML-EM implementation considered. These results hold true for both the first and second neurons in the two BSN neuron hierarchy. For the first neuron in the hierarchy, the RMS difference in the time series of the probability calculated for the estimated and the true parameter values, follows a similar pattern to the parameter estimates when the FL and ML-EM algorithms are compared, with average RMS errors of 0.2% obtained for the FL algorithm across the range of perturbations studied. Although we do not have a formal proof of convergence of the FL algorithm, we conclude that the FL algorithm can stably estimate the parameters over a large range of initial perturbations and it can do this very quickly. Thus the FL algorithm makes online learning in networks of BSNs much more tractable.

Two multi-agent policy iteration learning algorithms are proposed in this work. The two proposed algorithms use the exponential moving average approach along with the Q-learning algorithm as a basis to update the policy for the learning agent so that the agent's policy converges to a Nash equilibrium policy. The first proposed algorithm uses a constant learning rate when updating the policy of the learning agent, while the second proposed algorithm uses two different decaying learning rates. These two decaying learning rates are updated based on either the Win-or-Learn-Fast (WoLF) mechanism or the Win-or-Learn-Slow (WoLS) mechanism. The WoLS mechanism is introduced in this article to make the algorithm learn fast when it is winning and learn slowly when it is losing. The second proposed algorithm uses the rewards received by the learning agent to decide which mechanism (WoLF mechanism or WoLS mechanism) to use for the game being learned. The proposed algorithms have been theoretically analyzed and a mathematical proof of convergence to pure Nash equilibrium is provided for each algorithm. In the case of games with mixed Nash equilibrium, our mathematical analysis shows that the second proposed algorithm converges to an equilibrium. Although our mathematical analysis does not explicitly show that the second proposed algorithm converges to a Nash equilibrium, our simulation results indicate that the second proposed algorithm does converge to Nash equilibrium. The proposed algorithms are examined on a variety of matrix and stochastic games. Simulation results show that the second proposed algorithm converges in a wider variety of situations than state-of-the-art multi-agent reinforcement learning algorithms.

Bayesian spiking neurons (BSNs) provide a probabilistic interpretation of how neurons perform inference and learning. Online learning in BSNs typically involves parameter estimation based on maximum-likelihood expectation-maximization (ML-EM) which is computationally slow and limits the potential of studying networks of BSNs. An online learning algorithm, fast learning (FL), is presented that is more computationally efficient than the benchmark ML-EM for a fixed number of time steps as the number of inputs to a BSN increases (e.g., 16.5times faster run times for 20 inputs). Although ML-EM appears to converge 2.0 to 3.6times faster than FL, the computational cost of ML-EM means that ML-EM takes longer to simulate to convergence than FL. FL also provides reasonable convergence performance that is robust to initialization of parameter estimates that are far from the true parameter values. However, parameter estimation depends on the range of true parameter values. Nevertheless, for a physiologically meaningful range of parameter values, FL gives very good average estimation accuracy, despite its approximate nature. The FL algorithm therefore provides an efficient tool, complementary to ML-EM, for exploring BSN networks in more detail in order to better understand their biological relevance. Moreover, the simplicity of the FL algorithm means it can be easily implemented in neuromorphic VLSI such that one can take advantage of the energy-efficient spike coding of BSNs.

A fast learning algorithm is proposed for training of multilayer feedforward neural networks, based on a combination of optimal linear Kalman filtering theory and error propagation. In this algorithm, all the information available from the start of the training process to the current training sample is exploited in the update procedure for the weight vector of each neuron in the network in an efficient parallel recursive method. This innovation is a massively parallel implementation and has better convergence properties than the conventional backpropagation learning technique. Its performance is illustrated on some examples, such as a XOR logical operation and a nonlinear mapping of two continuous signals. >

Learning in multi-agent settings has recently garnered much interest, the result of which has been the development of somewhat effective multi-agent learning (MAL) algorithms for repeated normal-form games. However, general-purpose MAL algorithms for richer environments, such as general-sum repeated stochastic (Markov) games (RSGs), are less advanced. Indeed, previously created MAL algorithms for RSGs are typically successful only when the behavior of associates meets specific game theoretic assumptions and when the game is of a particular class (such as zero-sum games). In this paper, we present a new algorithm, called Pepper, that can be used to extend MAL algorithms designed for repeated normal-form games to RSGs. We demonstrate that Pepper creates a family of new algorithms, each of whose asymptotic performance in RSGs is reminiscent of its asymptotic performance in related repeated normal-form games. We also show that some algorithms formed with Pepper outperform existing algorithms in an interesting RSG.

The paper proposes a fast online learning method for neural network structures by using genetic algorithm (GA) and dynamic back propagation algorithm (BP) jointly. GA is used in the coarse tuning process which adjusts interconnection weights of the neural network. The dynamic back propagation algorithm is subsequently applied to achieve fine adjusting of the network weights. The fitness function, based on the squared error between the teaching signal and the network output value, is redefined at every time step and the proposed GA based algorithm solves a nonstationary function optimization task. At every time step the solution with the best fitness function is used for current representation of the neural network weights and biases. It is shown through the simulations and real time temperature control of drying oven that this learning algorithm has faster convergence ability and better performance on reducing mapping error in the online learning neural network structures. This leads to an improvement of the transient response of neuro adaptive systems. The proposed method has the potential to be applied to many practical areas such as system modeling and control, signal processing and pattern recognition.

Online adaptation is a key requirement for image processing applications when used in dynamic environments. In contrast to batch learning, where retraining is required each time a new observation occurs, sequential learning algorithms offer the ability to iteratively adapt the existing classifier. In this paper, we present a neural network architecture and a fast online learning algorithm that allow to use the class of resource allocation networks for such adaptive image processing applications. The network is based on receptive fields that are processed by RBF sub-nets. The learning algorithm builds such networks online by adding new units to the sub-nets each time novel input data is observed. For this, we define a global and a local novelty criterion. Experimental results show that the proposed network outperforms existing RAN algorithms when used for face detection and recognition and is competitive with existing classifiers.

A fast and robust algorithm for training feedforward neural networks (FNNs) by using a variable forgetting factor and U-D factorization-based recursive prediction error (RPE) method is proposed. In comparison with the backpropagation (BP) and RPE based learning algorithms, the proposed algorithm, called UD-RPE, can provide much more accurate learning results in fewer iterations with fewer hidden nodes and improve convergence rate and numerical stability (robustness). In addition, it is less sensitive to start-up parameters, such as initial weights and initial covariance matrix, and the randomness in the observed data. It also has good generalization ability and needs less learning time. Simulation results of nonlinear dynamic system modeling and identification show that the algorithm proposed here is an effective and efficient learning algorithm for FNNs.

This paper attempts to perform text-to-phoneme conversion by using recurrent neural networks trained with the real time recurrent learning (RTRL) algorithm. As recurrent neural networks deal well with spatial temporal problems, they are proposed to tackle the problem of converting English text streams into their corresponding phonetic transcriptions. We found that, due to the high computational complexity, the original RTRL algorithm takes a long time to finish the learning. We propose a fast RTRL algorithm (FRTRL), with a lower computational complexity, to shorten the time consumed in the learning process.

This paper investigates a relatively new direction in Multiagent Reinforcement Learning. Most multiagent learning techniques focus on Nash equilibria as elements of both the learning algorithm and its evaluation criteria. In contrast, we propose a multiagent learning algorithm that is optimal in the sense of finding a best-response policy, rather than in reaching an equilibrium. We present the first learning algorithm that is provably optimal against restricted classes of non-stationary opponents. The algorithm infers an accurate model of the opponentýs non-stationary strategy, and simultaneously creates a best-response policy against that strategy. Our learning algorithm works within the very general framework of n-player, general-sum stochastic games, and learns both the game structure and its associated optimal policy.

Unobtrusive indoor location systems must rely on methods that avoid the deployment of large hardware infrastructures or require information owned by network administrators. Fingerprinting methods can work under these circumstances by comparing the real-time received RSSI values of a smartphone coming from existing Wi-Fi access points with a previous database of stored values with known locations. Under the fingerprinting approach, conventional methods suffer from large indoor scenarios since the number of fingerprints grows with the localization area. To that aim, fingerprinting-based localization systems require fast machine learning algorithms that reduce the computational complexity when comparing real-time and stored values. In this paper, popular machine learning (ML) algorithms have been implemented for the classification of real time RSSI values to predict the user location and propose an intelligent indoor positioning system (I-IPS). The proposed I-IPS has been integrated with multi-agent framework for betterment of context-aware service (CAS). The obtained results have been analyzed and validated through established statistical measurements and superior performance achieved.

While the Resilient Backpropagation (RPROP) method can be extremely fast in converging to a solution, it suffers from the local minima problem. In this paper, a fast and reliable learning algorithm for multi-layer artificial neural networks is proposed. The learning model has two phases: the RPROP phase and the gradient ascent phase. The repetition of two phases can help the network get out of local minima. The proposed algorithm is tested on some benchmark problems. For all the above problems, the systems are shown to be capable of escaping from the local minima and converge faster than the Backpropagation with momentum algorithm and the simulated annealing techniques.KeywordsLocal MinimumGain ParameterSimulated Annealing MethodGradient AscentLocal Minimum ProblemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper presents as a contribution the formulation of the recursive least squares neuronal (RLSN) learning algorithm for teaching a neural network of the multilayer perceptron type. With this algorithm, the time of convergence achieved in the neural network is less than the time of convergence achieved when teaching the network with gradient methods. The procedure allows the application of neural network techniques to nonlinear and time dependant systems which are difficult to model and control. >

We consider fast convergence learning algorithms for multi-valued and universal binary neurons. These neurons are suggested to be used for design of neural networks based on CNN paradigm. On the basis of such networks we offer to solve some problems of image processing. For instance, high efficient method for contours detection obtained by learning algorithm described in the paper is presented. Also solution of the XOR-problem on the single neuron is described. >