Two-layer piecewise deterministic games

This paper considers the optimization problem for partially observable Markov decision processes (POMDPs) with the continuous state, observation, and action spaces. POMDPs with the discrete spaces have emerged as a promising approach to the decision systems with imperfect state information. However, in recent applications of POMDPs, there are many problems that have continuous states, observations, and actions. For such problems, due to the infinite dimensionality of the belief space, the existing studies usually discretize the continuous spaces with the sufficient or nonsufficient statistics, which may cause the curse of dimensionality and performance degradation. In this paper, based on the sensitivity analysis of the performance criteria, we have developed a simulation-based policy iteration algorithm to find the local optimal observation-based policy for POMDPs with the continuous spaces. The proposed algorithm needs none of the specific assumptions and prior information, and has a low computational complexity. One numerical example of the complicated multiple-input multiple-output beamforming problem shows that the algorithm has a significant performance improvement.

The tasks with continuous state and action spaces are difficult to be solved with high sample efficiency. Model learning and planning, as a well-known method to improve the sample efficiency, is achieved by learning a system dynamics model first and then using it for planning. However, the convergence of the algorithm will be slowed if the system dynamics model is not captured accurately, with the consequence of low sample efficiency. Therefore, to solve the problems with continuous state and action spaces, a model-learning-based actor-critic algorithm with the Gaussian process approximator is proposed, named MLAC-GPA, where the Gaussian process is selected as the modeling method due to its valuable characteristics of capturing the noise and uncertainty of the underlying system. The model in MLAC-GPA is firstly represented by linear function approximation and then modeled by the Gaussian process. Afterward, the expectation value vector and the covariance matrix of the model parameter are estimated by Bayesian reasoning. The model is used for planning after being learned, to accelerate the convergence of the value function and the policy. Experimentally, the proposed method MLAC-GPA is implemented and compared with five representative methods in three classic benchmarks, Pole Balancing, Inverted Pendulum, and Mountain Car. The result shows MLAC-GPA overcomes the others both in learning rate and sample efficiency.

The electricity market has provided a complex economic environment, and consequently has increased the requirement for advancement of learning methods. In the agent-based modeling and simulation framework of this economic system, the generation company's decision-making is modeled using reinforcement learning. Existing learning methods that model the generation company's strategic bidding behavior are not adapted to the non-stationary and non-Markovian environment involving multidimensional and continuous state and action spaces. This paper proposes a reinforcement learning method to overcome these limitations. The proposed method discovers the input space structure through the self-organizing map, exploits learned experience through Roth-Erev reinforcement learning and explores through the actor critic map. Simulation results from experiments show that the proposed method outperforms Simulated Annealing Q-Learning and Variant Roth-Erev reinforcement learning. The proposed method is a step towards more realistic agent learning in Agent-based Computational Economics.

In this paper, we consider the problem of inverse reinforcement learning for a particular class of continuous-time stochastic systems with continuous state and action spaces, under the assumption that both the cost function and the optimal control policy are parametric with known basis functions. Our goal is to produce a cost function for which a given policy, observed in experiment, is optimal. We proceed by enforcing a constraint on the relationship between input noise and input cost that produces a maximum entropy distribution over the space of all sample paths. We apply maximum likelihood estimation to approximate the parameters of this distribution (hence, of the cost function) given a finite set of sample paths. We iteratively improve our approximation by adding to this set the sample path that would be optimal given our current estimate of the cost function. Preliminary results in simulation provide empirical evidence that our algorithm converges.

Representation learning for continuous action spaces is beneficial for efficient policy learning

Reinforcement distribution in fuzzy Q-learning

Fuzzy Q–learning extends the Q–learning algorithm to work in presence of continuous state and action spaces. A Takagi–Sugeno Fuzzy Inference System (FIS) is used to infer the continuous executed action and its action–value, by means of cooperation of several rules. Different kinds of evolution of the parameters of the FIS are possible, depending on different strategies of distribution of the reinforcement signal. In this paper, we compare two strategies: the classical one, focusing on rewarding the rules that have proposed the actions composed to produce the actual action, and a new one we are introducing, where reward goes to the rules proposing actions closest the ones actually executed.KeywordsReinforcement LearningFuzzy Q–learningFuzzy logiccontinuous state-action spacereinforcement distribution

Q-learning can be used to learn a control policy that maximises a scalar reward through interaction with the environment. Q- learning is commonly applied to problems with discrete states and actions. We describe a method suitable for control tasks which require continuous actions, in response to continuous states. The system consists of a neureil network coupled with a novel interpolator. Simulation results are presented for a non-holonomic control task. Advantage Learning, a variation of Q-learning, is shown enhance learning speed and reliability for this task.

With the rapid development of Internet of vehicles (IoV), the emerging in-vehicle applications meet people’s demand for intelligent experience, while also increasing the cost of network communication. Driven by the fact that a few popular content accounted for most of the traffic load, caching popular content at the wireless edge has become a trend of content delivery. However, the limitation of cache resources and the ever-changing mobile environment bring a severe challenge to content delivery cost optimization. In this paper, we first investigate the joint optimization of content placement and bandwidth allocation (JOCB) in IoV. In order to minimize the long-term system cost in the mobile scenario, we first establish an innovative vehicle mobility model based on the probability distribution. Then we use the Lagrangian multiplier method to solve the bandwidth allocation sub-problem to reduce the algorithm complexity. Considering the high-dimensional continuity of action and state spaces, the original problem is transformed into a Markov Decision Process (MDP), and we propose a Deep reinforcement learning based Dynamic Content Placement (DDCP) algorithm. DDCP algorithm has low algorithm complexity and can handle high-dimensional continuous state and action space. The simulation results show that compare with other three benchmark algorithms, DDCP algorithm can reduce the system cost effectively.

In this paper, a Reinforcement Learning (RL)-based approach to optimally dispatch PV inverters in unbalanced distribution systems is presented. The proposed approach exploits a decentralized architecture in which PV inverters are operated by agents that perform all computational processes locally; while communicating with a central agent to guarantee voltage magnitude regulation within the distribution system. The dispatch problem of PV inverters is modeled as a Markov Decision Process (MDP), enabling the use of RL algorithms. A rolling horizon strategy is used to avoid the computational burden usually associated with continuous state and action spaces, coupled with a computationally efficient learning algorithm to model the action-value function for each PV inverter. The effectiveness of the proposed decentralized RL approach is compared with the optimal solution provided by a centralized nonlinear programming (NLP) formulation. Results showed that within several executions, the proposed approach converges either to the optimal solution or to solutions with a PV curtailment excess of less than 2.5% while still enforcing voltage magnitude regulation.

The incorporation of macro-actions (temporally extended actions) into multi-agent decision problems has the potential to address the curse of dimensionality associated with such decision problems. Since macro-actions last for stochastic durations, multiple agents executing decentralized policies in cooperative environments must act asynchronously. We present an algorithm that modifies generalized advantage estimation for temporally extended actions, allowing a state-of-the-art policy optimization algorithm to optimize policies in Dec-POMDPs in which agents act asynchronously. We show that our algorithm is capable of learning optimal policies in two cooperative domains, one involving real-time bus holding control and one involving wildfire fighting with unmanned aircraft. Our algorithm works by framing problems as "event-driven decision processes," which are scenarios in which the sequence and timing of actions and events are random and governed by an underlying stochastic process. In addition to optimizing policies with continuous state and action spaces, our algorithm also facilitates the use of event-driven simulators, which do not require time to be discretized into time-steps. We demonstrate the benefit of using event-driven simulation in the context of multiple agents taking asynchronous actions. We show that fixed time-step simulation risks obfuscating the sequence in which closely separated events occur, adversely affecting the policies learned. In addition, we show that arbitrarily shrinking the time-step scales poorly with the number of agents.

In this paper, we present an approach that applies the reinforcement learning principle to the problem of learning height control policies for aerial blimps. In contrast to previous approaches, our method does not require sophisticated hand- tuned models, but rather learns the policy online, which makes the system easily adaptable to changing conditions. The blimp we apply our approach to is a small-scale vehicle equipped with an ultrasound sensor that measures its elevation relative to the ground. The major problem in the context of learning control policies lies in the high-dimensional state-action space that needs to be explored in order to identify the values of all state-action pairs. In this paper, we propose a solution to learning continuous control policies based on the Gaussian process model. In practical experiments carried out on a real robot we demonstrate that the system is able to learn a policy online within a few minutes only.

Reinforcement learning is a field of artificial intelligence that is continuously evolving and has a wide variety of applications. In recent years major progress has been made in the application of deep reinforcement learning to high-dimensional problems with continuous state and action spaces. This paper presents a complete analysis of the application of the soft actor-critic algorithm to teach a four legged robot with three joints on each leg how to move towards the center of a virtually simulated environment. The general formulation of the reinforcement learning problem is first presented, followed by the description of the environment under analysis and the applied algorithm. Afterwards, the obtained results are compared against those of a manually programmed policy, closing with a discussion of some key design choices and common challenges.

This paper proposes a new Q-learning which can construct the mapping function between the continuous state and action space. The proposed method can calculate the expected values of each action by using neural networks and CMAC which is one kind of tiling technics. The proposed method is evaluated through a navigation problem of an autonomous mobile robot.

Pang, Bo Jiang, Zhong-PingThis chapter studies the robustnessRobustness of reinforcement learningReinforcement Learning (RL) for discrete-time linear stochastic systemsStochastic system with multiplicative noiseMultiplicative noise evolving in continuous state and action spaces. As one of the popular methods in reinforcement learningReinforcement Learning (RL), the robustnessRobustness of policy iterationPolicy iteration is a longstanding open problem for the stochastic linear quadratic regulator (LQR)Linear Quadratic Regulator (LQR) problem with multiplicative noiseMultiplicative noise. A solution in the spirit of input-to-state stabilityInput-to-state stability is given, guaranteeing that the solutions of the policy iterationPolicy iteration algorithm are bounded and enter a small neighborhood of the optimal solution, whenever the error in each iteration is bounded and small. In addition, a novel off-policyOff-policy multiple-trajectory optimistic least-squaresLeast-squares algorithm policy iterationPolicy iteration algorithm is proposed, to learn a near-optimal solution of the stochastic LQRLinear Quadratic Regulator (LQR) problem directly from online input/state data, without explicitly identifying the system matrices. The efficacy of the proposed algorithm is supported by rigorous convergence analysis and numerical results on a second-order example.