Actor-critic reinforcement learning for the feedback control of a swinging chain

This paper investigates an echo state network (ESN) (Jaeger, 2001 and Maass and Markram, 2002) architecture as the approximation of the Q-function for temporally dependent rewards embedded in a linear dynamical system, the mass-spring-damper (MSD). This problem has been solved utilizing feed-forward neural networks (FNN) when all state information necessary to specify the dynamics is provided as input (Kretchmar, 2000). Time-delayed neural networks (TDNN) solve this problem with finite-size windows of incomplete state information. Our research demonstrates that the ESN architecture represents the Q-function of the MSD system given incomplete state information as well as current feed forward neural networks given either perfect state or a temporally-windowed, incomplete state vector. The remainder of this paper is organized as follows. We introduce basic concepts of reinforcement learning and the echo state network architecture. The MSD system simulation is defined in section IV. Experimental results for learning state quality given incomplete state information are presented in section V. Results for learning estimates of all future state qualities for incomplete state information is presented in section VI. Section VII discusses the potential of the ESN for use in reinforcement learning and provides current and future directions of research.

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming “beliefs”—optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN’s learned representation encodes belief information, but only when the RNN’s capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

In this work, an approach to stabilizing compensator design is presented that is suitable for a large class of discrete-time non-linear stochastic systems with incomplete state information. The non-linearity is contained either in the state or the measurement equation and it involves a white-noise vector sequence. The proposed compensator is a combination of a constant gain observer and a constant feedback control gain. The stabilizability conditions depend upon the system and chosen design parameter characteristics and are given for both mean-square and almost sure stochastic boundedness and stability of the controlled system. A specific subclass of these systems, subject to certain control laws, is shown to possess stabilizability conditions that can be tested a priori.

An unconstrained stochastic optimization problem involving a discrete-time linear process with a normally distributed initial condition and subject to additive gaussian state and measurement noise is formulated in terms of a quite general finite horizon, discrete-time quadratic cost criterion and solved when there is either complete or incomplete state information. It is shown that both the stochastic sampled-data optimal tracker and the stochastic sampled-data optimal regulator are special cases of this problem. A breakdown of the minimum cost for both sampled-data controllers is given.

Designing mobile robots that navigate indoor environments autonomously is known to be a difficult problem. A critical issue is in the formulation of robust motion control algorithms capable of reliably sensing partial or incomplete information from the environment and using this information to choose appropriate actions to achieve its designed goals. As an example, suppose we wish to deploy a mobile robot that autonomously patrols defined locations at a hazardous high-security facility. The robot must maintain accurate knowledge of its location, while using sensory data to recognize objects and obstacles in its immediate vicinity. Its task is to inspect the desired locations within a defined time period and provide real-time data in the event of an incident. The problem is thus to choose appropriate actions that result in accomplishing the patrol in a minimal amount of time in the partially structured environment. To solve this problem we adopt the Partially Observable Markov Decision Processes (POMDP) formalism to find near-optimal and efficient policies that provides a description the robot's motion in environments with incomplete state information. POMDP is a generalization of Markov Decision Processes (MDPs). It models a system as a coupling of an agent/decision maker (robots in our case) and an environment. We also present a methodology called Goal-Specific Representation (GSR) to reduce the size of the state-space for computational efficiency and propose an extension to the methodology.

Aiming at the problem of anti-swing and positioning of underactuated crane, this paper proposes a control strategy of underactuated bridge crane based on Linear Active Disturbance Rejection Control and controller parameter optimization method. Distinguished from other conventional control strategies of bridge crane, this method does not require any approximate decoupling or linearization of the crane model and allows the model to have some certain uncertainty, friction and air resistance of the system are also considered. At the same time, for the parameter tuning problem of the sensors, the bird swarm optimization algorithm is applied to optimize the parameters. And the excellent control performance of this method has been proved by simulation.

Numerous phenomena in the theory of automatic regulation, mechanics, radiophysics, biology, immunology, economics, robototechnics, etc. need for theiccorrect qualitative and quantitative description to use equations with time lag. One problem which here arises is connected with states estimate of stochastic hereditary systems [1]. This paper is a survey of some results and problems of filtering, interpolation and smoothing for arbitrary Gaussian stochastic processes for the case of linear observations with delays. In particular, the estimate problems of solutions of linear stochastic differential equations with delays and integral equations of Volterra are considered. Besides new method for solution of optimal control problem with incomplete state information is adduced. The survey is based on the papers [2-14]

In heterogeneous networks, interference management is very important since the interference due to spectrum-sharing can significantly decrease the overall performance. In the existing work, various resource allocation methods are proposed to either improve energy efficiency or alleviate interference. Very little research has addressed their joint interaction. Furthermore, most existing work assumes that all of the channel state information is known, which might not be realistic. In this paper, we propose a game-theoretical scheme using energy-efficient resource allocation and interference pricing for an interference-limited environment in two-tier heterogeneous networks without complete channel state information. We formulate the problems of resource allocation and interference management as a two-stage Stackelberg game with incomplete information. A backward induction method is used to analyze the proposed game, since it can capture the sequential dependencies of the decisions in the stages of the game.

Bridge crane is one of the most widely used cranes in our country, which is indispensable equipment for material conveying in the modern production. The most important indicator of crane performances is energysaving. So it is of importance to research on the bridge cranes energysaving assessment. Thus the establishment of assessment index system is a necessary task. Rough set theory is applied widely for researching on incomplete and uncertain information processing. Through reducing the properties by rough set theory, redundant properties are removed and an optimized index assessment system can be obtained.In this paper,establishing the index system of the bridge crane based on the full life cycle of the bridge crane firstly. Secondly,defining the decision attribute and the condition attributes. Thirdly,using rough set theory to reduce the index system of the bridge crane.Finally,an optimized index assessment system of the bridge can be obtained. The research lays the foundation for solving weight for the next step. DOI: http://dx.doi.org/10.11591/telkomnika.v11i12.2793 Full Text: PDF

Spare representations can help improve value prediction and control performances in Reinforcement Learning (RL), by capturing most essential features from states and ignoring unnecessary ones to avoid interference. However, existing sparse coding-based RL methods for control problems are optimized in the neural network methodology, which can not guarantee convergence. To this end, we propose a dictionary learning-based RL with the non-convex sparsity regularizer for RL control. To avoid the black-box optimization with the SGD, we employ the dictionary learning model in RL control, guaranteeing efficient convergence in control experiments. To obtain accurate representations in RL, we employ the non-convex \(\ell _p\) norm (0 < p < 1) beyond the convex \(\ell _1\) norm as the sparsity regularizer in dictionary learning-based RL, for capturing more essential features from states. To obtain solutions efficiently, we employ the proximal splitting method to update the multivariate optimization problem. Hence, the non-convex sparsity regularized dictionary learning-based RL is developed and validated in different benchmark RL environments. The proposed algorithm can obtain the best control performances among compared sparse coding-based RL methods with around 10% increases in reward. Moreover, the proposed method can obtain higher sparsity in representations in different environments.KeywordsReinforcement learningDictionary learningNon-convex sparsity regularizer

This paper presents the analyses of a deep reinforcement learning, DRL approach for maximum power point tracking (MPPT) of a photovoltaic PV boost converter system. The DRL controller used is the Deep Deterministic Policy Gradient (DDPG) algorithm, which is designed to address the limitations of conventional perturb and observe, P&amp;O methods with PID control, such as reduced efficiency under non – uniform irradiance or load variations. The DDPG framework offers an adaptive and computationally efficient alternative by learning the nonlinear mapping between PV states and duty cycle control. Thus, a comparative study of the DDPG with the conventional approach is performed under non – uniform irradiance and varying load resistance. Parameters of power and voltage were evaluated as well as the oscillatory performance for both controllers. The outcome of the results shows that the DDPG controller can achieve higher power extraction with significantly reduced under higher load impedances The conventional approach, however, achieves a better performance under uniform irradiation and low resistance conditions. This is due to the absence of multiple peaks that allows the conventional approach to rapidly converge to the MPP, while the exploratory nature DDPG introduces minor bias due to its exploratory learning process. These findings highlight the potential of DDPG to enhance the efficiency and adaptability of MPPT, particularly relevant in standalone DC microgrid applications.

Dynamic Decision Model in Evolutionary Games Based on Reinforcement Learning

In this work, the problem of stabilizing a discrete stochastic-parameter system with incomplete state information is addressed. In order to solve this problem, a time-invariant compensator system composed of a constant gain observer and a constant feedback gain, is proposed. The stabilizability conditions depend both on the first and second moments of the stochastic parameters.

Optimal control of Markov processes with incomplete state information

This paper presents a study on the use of reinforcement learning to control the torque vectoring of a small electric racecar aiming to improve vehicle handling and vehicle stability. The reinforcement-learning algorithm used is Neural Fitted Q Iteration, and the sampling of experiences is based on simulations using the software CarMaker. The cost function is based on the position of the states on the phase-plane of sideslip angle and angular velocity. To investigate the maximum ratio of torque distribution that should be set to guarantee stability as well as the effectiveness of the controller inputs in the learning process, two experiments were done (A and B) with different states and possibilities of torque distribution. The controller A is able to improve the vehicle handling and stability with a significant reduction in vehicle sideslip angle. And it also showed that 70% of torque distribution is enough to keep the vehicle stable.