Monotone solutions for mean field games master equations: continuous state space and common noise

<p style='text-indent:20px;'>In the paper, we study the dependence of solutions of the continuous-time finite state mean field game on initial distribution of players. Our approach relies on the concept of value multifunction of the mean field game that is a mapping assigning to an initial time and an initial distribution a set of expected outcomes of the representative player corresponding to solutions of the mean field game. Using the reformulation of the finite state mean field game as a control problem with mixed constraints, we give the sufficient condition on a given multifunction to be a value multifunction in the terms of the viability theory. The maximal multifunction (i.e., the mapping assigning to an initial time and an initial distribution the whole set of values corresponding to solutions of the mean field game) is characterized via the backward attainability set for the certain control system.</p>

This paper presents a method for behavior learning of an autonomous agent using modified Learning Vector Quantization (LVQ) with fuzzy sets in continuous state space. When the environment is a continuous state space, it has infinitely many state values. So, it is impossible to learn a good action to take in each of the state values. This paper uses a function approximation technique based on the LVQ algorithm to learn actions of agent in continuous state space. An advantage of this technique is that it can represent the mapping between the continuous-valued state space and appropriate actions with a finite number of parameters. An example illustrates its validity in continuous space problems.

This paper considers the optimization problem for partially observable Markov decision processes (POMDPs) with the continuous state, observation, and action spaces. POMDPs with the discrete spaces have emerged as a promising approach to the decision systems with imperfect state information. However, in recent applications of POMDPs, there are many problems that have continuous states, observations, and actions. For such problems, due to the infinite dimensionality of the belief space, the existing studies usually discretize the continuous spaces with the sufficient or nonsufficient statistics, which may cause the curse of dimensionality and performance degradation. In this paper, based on the sensitivity analysis of the performance criteria, we have developed a simulation-based policy iteration algorithm to find the local optimal observation-based policy for POMDPs with the continuous spaces. The proposed algorithm needs none of the specific assumptions and prior information, and has a low computational complexity. One numerical example of the complicated multiple-input multiple-output beamforming problem shows that the algorithm has a significant performance improvement.

Acceleration of cellular automata algorithms using genetic algorithms

In order to deal with the sequential decision problems with large or continuous state spaces, feature representation and function approximation have been a major research topic in reinforcement learning (RL). In this paper, a clustering-based graph Laplacian framework is presented for feature representation and value function approximation (VFA) in RL. By making use of clustering-based techniques, that is, K-means clustering or fuzzy C-means clustering, a graph Laplacian is constructed by subsampling in Markov decision processes (MDPs) with continuous state spaces. The basis functions for VFA can be automatically generated from spectral analysis of the graph Laplacian. The clustering-based graph Laplacian is integrated with a class of approximation policy iteration algorithms called representation policy iteration (RPI) for RL in MDPs with continuous state spaces. Simulation and experimental results show that, compared with previous RPI methods, the proposed approach needs fewer sample points to compute an efficient set of basis functions and the learning control performance can be improved for a variety of parameter settings.

One of the difficulties encountered in the application of reinforcement learning methods to real-world problem is the generalization of large-scale or continuous state space. In order to solve the curse of dimensionality problem caused by discretizing continuous state space, a kind of Q-learning method for continuous state space based on a dynamic Elman neural network was proposed in this paper. The inputs and the output of Elman network are the system state-action pair and the corresponding Q-value. That is, Elman network is used to estimate the Q-value of state-action pair on-line. Eligibility trace for connecting weights is introduced by borrowing ideas from the eligibility trace mechanism of state in temporal difference algorithm to improve the learning speed of neural network. Computer simulations on mountain car control illustrate the performance and applicability of the proposed Q-learning scheme.

We consider a simplified Ginzburg–Landau model of superconductivity in a two-dimensional infinite strip domain under the assumption of the periodicity in the infinite direction. This model equation has two physical parameters, λ, h, coming from the Ginzburg–Landau parameter and the strength of an applied magnetic field, respectively. We study the bifurcation of non-trivial solutions in the parameter space (h, λ), in particular through a bifurcation of the existence of a vortex solution, that is, a solution with isolated zeros. We first observe that in the parameter space there is a smooth (bifurcation) curve on which a solution with k-mode in the periodic direction takes place. This bifurcating solution, however, is vortexless. Then analysing the local bifurcation structure around the critical point at which two bifurcation curves for k and m(>k) intersect, we prove the existence of vortex solutions under a generic condition. Moreover, we show that the solutions have vortices lying on a boundary if the parameters belong to a certain curve emanating from the critical point. The stability of such solutions is also discussed.

The nonmonotonic differentiable solutions of equations involving iterated functional series are investigated. Conditions for the existence, uniqueness, and stability of such solutions are given. These extend earlier results due to Murugan and Subrahmanyam.

We develop a splitting method to prove the well-posedness, in short time, of solutions for two master equations in mean field game (MFG) theory: the second order master equation, describing MFGs with a common noise, and the system of master equations associated with MFGs with a major player. Both problems are infinite-dimensional equations stated in the space of probability measures. Our new approach simplifies and generalizes previous existence results for second order master equations and provides the first existence result for systems associated with MFG problems with a major player.

The sample complexity of a reinforcement-learning algorithm is highly coupled to how proficiently it explores, which in turn depends critically on the effective size of its state space. This paper proposes a new exploration mechanism for model-based algorithms in continuous state spaces that automatically discovers the relevant dimensions of the environment. We show that this information can be used to dramatically decrease the sample complexity of the algorithm over conventional exploration techniques. This improvement is achieved by maintaining a low-dimensional representation of the transition function. Empirical evaluations in several environments, including simulation benchmarks and a real robotics domain, suggest that the new method outperforms state-of-the-art algorithms and that the behavior is robust and stable.

The problem of optimal feedback planning under prediction uncertainties among static obstacles is considered. A discrete-time stochastic state transition model is defined over a continuous state space. A relation to a continuous “nearby” deterministic model is proven for small time steps; the cost-to-go function of the stochastic model is approximated with that of the deterministic model, and the approximation error is found to be proportional to the time step. This motivates using numerical methods, which are vastly available for solving deterministic problems, to approximate the original stochastic problem. We demonstrate this application using a Simplicial Label Correcting Algorithm. This algorithms uses a piecewise linear discretization to compute the shortest-path plan on a simplicial complex. Additionally, the theoretical error bound between the approximate solution and the exact solution is derived and confirmed during numerical experiments. This paper provides a rigorous analysis as well as algorithmic and implementation details of the proposed model for the stochastic shortest path problem in continuous spaces with obstacles.

Propp & Wilson (1996) described a protocol, called coupling from the past, for exact sampling from a target distribution using a coupled Markov chain Monte Carlo algorithm. In this paper we extend coupling from the past to various MCMC samplers on a continuous state space; rather than following the monotone sampling device of Propp & Wilson, our approach uses methods related to gamma‐coupling and rejection sampling to simulate the chain, and direct accounting of sample paths.

Swarm intelligence has brought a new paradise for function optimization, structural optimization, multi-agent systems and other study fields. In our previous work, we proposed a neuro-fuzzy system using reinforcement learning algorithm (actor-critic method with TD error learning algorithm) to acquire optimized swarm behaviors. This paper improves the conventional learning system, which only deals with discrete state space and action space, to solve how a swarm to learn and obtain its adaptive behaviors in the continuous state space. The improved system adopts a new policy function of action which is possible to yield continuous actions corresponding to continuous states. The effectiveness of proposed system is investigated by computer simulations with more kinds of environments for the goal-exploration problem.Keywordsneuro-fuzzy netswarm behaviorreinforcement learningmulti-agent systemactor-critic algorithmgoal-exploration problem

Feature representation is critical not only for pattern recognition tasks but also for reinforcement learning (RL) methods to solve learning control problems under uncertainties. In this paper, a manifold-based RL approach using the principle of locally linear reconstruction (LLR) is proposed for Markov decision processes with large or continuous state spaces. In the proposed approach, an LLR-based feature learning scheme is developed for value function approximation in RL, where a set of smooth feature vectors is generated by preserving the local approximation properties of neighboring points in the original state space. By using the proposed feature learning scheme, an LLR-based approximate policy iteration (API) algorithm is designed for learning control problems with large or continuous state spaces. The relationship between the value approximation error of a new data point and the estimated values of its nearest neighbors is analyzed. In order to compare different feature representation and learning approaches for RL, a comprehensive simulation and experimental study was conducted on three benchmark learning control problems. It is illustrated that under a wide range of parameter settings, the LLR-based API algorithm can obtain better learning control performance than the previous API methods with different feature representation schemes.

We have obtained the conditions of existence and a scheme for constructing solutions of a weakly nonlinear boundary value problem for a degenerate differential-algebraic system in the noncritical case. The boundary condition is determined by a weakly nonlinear vector functional. The linear part of the problem is a linear boundary value problem for a degenerate differential-algebraic system. Linear differential-algebraic boundary value problems have been studied in monographs by S. Campbell, J.R. Magnus, A.M. Samoilenko and V.P. Yakovets. In the works of A.M. Samoilenko and O.A. Boichuk, using the central canonical form, the necessary and sufficient conditions for the existence of solutions of nonlinear differential-algebraic boundary value problems were obtained. We have obtained necessary and sufficient conditions for the existence of solutions of nonlinear differential-algebraic systems without using the central canonical form, which allows us to study the solvability of differential-algebraic boundary value problems that depend on arbitrary continuous functions. This approach significantly varies the classification of nonlinear differential-algebraic boundary value problems in critical and noncritical cases. Our formulation of the weakly nonlinear differential-algebraic boundary value problem generalises the boundary value problems studied in the works of Yu.O. Mitropolsky, A.M. Samoilenko, and O.A. Boichuk. The case when a differential-algebraic system is not solvable with respect to the derivative is considered, and substitutions of the unknown are proposed. It leads the original system to a nonlinear differential-algebraic system solvable with respect to the derivative. Finally, we present an example of a nonlinear differential-algebraic antiperiodic boundary value problem for a Riccati-type equation, which demonstrates the constructiveness of the obtained necessary and sufficient conditions for the existence of solutions of nonlinear differential-algebraic systems. The obtained results can be transferred to the problems of finding conditions for the existence and schemes for constructing solutions of nonlinear degenerate differential-algebraic boundary value problems in critical cases, as well as to the problems of finding conditions for the stability of such solutions.