Deep Deterministic Policy Gradient to Regulate Feedback Control Systems Using Reinforcement Learning

Two wheeled self balancing robots have dynamic behaviour and are unstable in nature. Due to this, stabilizing of these robots has become a field of interest to many researchers. Around the equilibrium point of two wheeled self balancing mobile robot, the pitch angles are very small. Hence the balancing is achieved commonly by using linear control methods such as LQR(Linear Quadratic Regulator) control, PID(Proportional Integral Derivative) control etc. But the robots with large pitch angles tends to be in a region of high nonlinear behavior. Hence the linear control techniques fails in maintaining the balance of the robot. To remedy this problem and for improving the performances of the robot, an SDRE(State Dependent Riccati Equation) controller is designed along with SDC (State Dependent Coefficient) Matrix. Simulations are done to check the stability using SDRE controllers. A comparison is done using SDRE, PID and LQR controllers for the system with nonlinear dynamics. Also a comparison is done using PID and LQR controllers for the system with linear dynamics. From the simulations it is clear that SDRE controller shows excellent balancing for the two wheeled self balancing robot when compared to PID and LQR controllers.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

P2P power trading based on reinforcement learning for nanogrid clusters

This paper presents the designing and simulation for vertical moving control of Unmanned Air Vehicles (UAV) system using Proportional Integral Derivative (PID) control method, Linear Quadratic Regulator (LQR) control method and fuzzy logic control technique.The UAV system pose a challenger control problem that in it precise model of system is non-linear. The conventional PID controller, the LQR controller, Fuzzy Logic Controller (FLC) and Self-Tuning Fuzzy PID controller are designned based on the dynamic modeling of the system is derived from a suitable mathematical model to describe the vertical motion of a UAV. Simulation results in Matlab and Simulink show that by using self-tuning Fuzzy PID, the performance of vertical moving control system is improved significantly compared to Other controllers. DOI: http://dx.doi.org/10.11591/ijece.v3i5.3440

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

This study investigates energy efficient control system for a small scale rocket recovery, utilizing thrust vector control (TVC) and reaction gas control (RGC) approaches. The dynamics of the rocket is managed through control input thruster forces and gimbaled angles, with the output parameters expressed in terms of velocity. Leveraging this model, several controllers: the linear quadratic regulator (LQR), linear quadratic Gaussian (LQG), sliding mode control (SMC), and proportional integral derivative (PID) controllers were designed and implemented. A linearized version of the nonlinear dynamic model of the rocket, obtained through Jacobian linearization, facilitates the application of LQR and LQG controllers. For precise pitch and yaw control, the architecture incorporates a single-nozzle TVC actuation and four-sided gas thrusters. Simulation studies, conducted in MATLAB/Simulink, evaluated the energy consumption profiles of each control strategy during rocket recovery, with the PID controller used as a baseline for comparison. Without disturbances, the LQG controller reduced energy consumption by 17.63% compared to PID, while SMC and LQR achieved reductions of 14.65% and 14.24%, respectively. Under disturbed conditions, LQG, SMC, and LQR reduced energy consumption by 30.39%, 27.38%, and 27.30%, respectively. These results highlight the potential of advanced control techniques to optimize energy efficiency. Thus, this study provides valuable insights for the development of more sustainable and cost-effective reusable rocket systems by emphasizing energy-efficient recovery control.

This paper presents a comparison of optimization for vehicle steering controls system simulation using several Artificial Intelligence (AI) for optimizing Proportional Integral Derivative (PID) control parameters to suppress errors on lateral motion and the yaw motion of vehicles. This paper compares five kinds of tuning methods of parameter for PID controller, among other are Firefly Algorithm (FA), Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO), Bat Algorithm (BA) and Imperialist Competitive Algorithm (ICA). The vehicles are represented in the model vehicle with 10 degrees of Freedom of vehicle dynamics system. The simulation results show that the PID control tuned by AI in the vehicle steering control system can adjust the plant output to the desired trajectory so that the stability of the vehicle is maintained. Vehicle yaw error and lateral error can be reduced by using ICA to determine PID parameter. The main advantage of proposed optimization is faster and more accurate compared with PID controller. So the error of the controller is reduced too. The results obtained are of vehicle motion can be maintained in accordance with the desired trajectory with smaller error and was able to achieve higher speeds than with the control system using optimized without parameters. This paper only deals with software simulation to proof the effect of AI optimization. The hardware implementation will be investigated in the next future.

Recent advancement on reinforcement learning (RL) algorithms shows that effective learning of parametric action- selection policies can often be achieved through direct opti- mization of a performance lower bound subject to pre-defined policy behavioral constraints. Driven by this understanding, this paper seeks to develop new policy search techniques where RL is achieved through maximizing a performance lower bound ob- tained originally based on an Expectation-Maximization method. For reliable RL, our new learning techniques must also simul- taneously guarantee constrained policy behavioral changes mea- sured through KL divergence. Two separate approaches will be pursued to tackle our constrained policy optimization problems, resulting in two new RL algorithms. The first algorithm utilizes a conjugate gradient technique and a Bayesian learning method for approximate optimization. The second algorithm focuses on minimizing a loss function derived from solving the Lagrangian for constrained policy search. Both algorithms have been experi- mentally examined on several benchmark problems provided by OpenAI GYM. The experiment results clearly demonstrate that our algorithms can be highly effective in comparison to several well-known RL algorithms.

Parrot Mambo mini-drone is a readily available commercial quadrotor platform to understand and analyze the behavior of a quadrotor both in indoor and outdoor applications. This study evaluates the performance of three alternative controllers on a Parrot Mambo mini-drone in an interior environment, including Proportional–Integral–Derivative (PID), Linear Quadratic Regulator (LQR), and Model Predictive Control (MPC). To investigate the controllers’ performance, initially, the MATLAB®/Simulink™ environment was considered as the simulation platform. The successful simulation results finally led to the implementation of the controllers in real-time in the Parrot Mambo mini-drone. Here, MPC surpasses PID and LQR in ensuring the system’s stability and robustness in simulation and real-time experiment results. Thus, this work makes a contribution by introducing the impact of MPC on this quadrotor platform, such as system stability and robustness, and showing its efficacy over PID and LQR. All three controllers demonstrate similar tracking performance in simulations and experiments. In steady state, the maximal pitch deviation for the PID controller is 0.075 rad, for the LQR, it is 0.025 rad, and for the MPC, it is 0.04 rad. The maximum pitch deviation for the PID-based controller is 0.3 rad after the take-off impulse, 0.06 rad for the LQR, and 0.17 rad for the MPC.

The area of Neurosymbolic Artificial Intelligence (Neurosymbolic AI) is rapidly developing and has become a popular research topic, encompassing sub-fields such as Neurosymbolic Deep Learning (Neurosymbolic DL) and Neurosymbolic Reinforcement Learning (Neurosymbolic RL). Compared to traditional learning methods, Neurosymbolic AI offers significant advantages by simplifying complexity and providing transparency and explainability. Reinforcement Learning(RL), a long-standing Artificial Intelligence(AI) concept that mimics human behavior using rewards and punishment, is a fundamental component of Neurosymbolic RL, a recent integration of the two fields that has yielded promising results. The aim of this paper is to contribute to the emerging field of Neurosymbolic RL by conducting a literature survey. Our evaluation focuses on the three components that constitute Neurosymbolic RL: neural, symbolic, and RL. We categorize works based on the role played by the neural and symbolic parts in RL, into three taxonomies:Learning for Reasoning, Reasoning for Learning and Learning-Reasoning. These categories are further divided into sub-categories based on their applications. Furthermore, we analyze the RL components of each research work, including the state space, action space, policy module, and RL algorithm. Additionally, we identify research opportunities and challenges in various applications within this dynamic field.

Reinforcement Learning (RL) has demonstrated significant potential in fields such as robotic control, autonomous driving, and financial decision-making. However, in complex environments, RL still faces challenges in stability and convergence. This study addresses this core issue by establishing a theoretical analysis framework that combines stochastic approximation theory and Lyapunov stability theory to rigorously analyze the convergence conditions and stability bounds of various RL algorithms in complex environments. Based on the theoretical analysis, we propose an Adaptive Stable RL (ASRL) algorithm, which employs dynamic regularized policy optimization and robust value function estimation to effectively mitigate policy oscillation and training divergence. Systematic experiments conducted in OpenAI Gym, MuJoCo, and customized non-stationary environments demonstrate that ASRL significantly outperforms baseline algorithms such as PPO and SAC in terms of convergence speed, final performance, and stability. Additionally, in industrial control and robotic navigation case studies, ASRL exhibits excellent adaptability and robustness. This research not only provides theoretical support for RL in complex environments but also offers optimization guidelines for algorithm design in practical applications.

This study introduces a novel controller based on a Reinforcement Learning (RL) algorithm for real-time adaptation of the stimulation pattern during FES-cycling. Core to our approach is the introduction of an RL agent that interacts with the cycling environment and learns through trial and error how to modulate the electrical charge applied to the stimulated muscle groups according to a predefined policy and while tracking a reference cadence. Instead of a static stimulation pattern to be modified by a control law, we hypothesized that a non-stationary baseline set of parameters would better adjust the amount of injected electrical charge to the time-varying characteristics of the musculature. Overground FES-assisted cycling sessions were performed by a subject with spinal cord injury (SCI AIS-A, T8). For tracking a predefined pedaling cadence, two closed-loop control laws were simultaneously used to modulate the pulse intensity of the stimulation channels responsible for evoking the muscle contractions. First, a Proportional-Integral (PI) controller was used to control the current amplitude of the stimulation channels over an initial parameter setting with predefined pulse amplitude, width and fixed frequency parameters. In parallel, an RL algorithm with a decayed-epsilon-greedy strategy was implemented to randomly explore nine different variations of pulse amplitude and width parameters over the same stimulation setting, aiming to adjust the injected electrical charge according to a predefined policy. The performance of this global control strategy was evaluated in two different RL settings and explored in two different cycling scenarios. The participant was able to pedal overground for distances over 3.5 km, and the results evidenced the RL agent learned to modify the stimulation pattern according to the predefined policy and was simultaneously able to track a predefined pedaling cadence. Despite the simplicity of our approach and the existence of more sophisticated RL algorithms, our method can be used to reduce the time needed to define stimulation patterns. Our results suggest interesting research possibilities to be explored in the future to improve cycling performance since more efficient stimulation cost dynamics can be explored and implemented for the agent to learn.

Recent research discovered that Reinforcement Learning (RL) algorithms with simple linear policies can achieve competitive performance as many state-of-the-art RL algorithms designed to train policies in the form of multi-layer neural networks. However, high learning performance is only achieved so far when policies are trained by jointly using multiple episodes of samples. An important research question remains as to whether linear policies can achieve cutting-edge performance when they are trained in a step-wise fashion (i.e., policies are iteratively updated based on every newly obtained sample). This paper presents a confirmatory answer to this question by developing a new RL algorithm called Primal-Dual Regular-gradient Actor-Critic (PD-RAC) as a generalization of RAC, which is a popular step-wise RL technique. Experiments on six benchmark control problems show that PD-RAC can achieve leading performance, in comparison to several recently developed baseline algorithms.

Reinforcement learning (RL) focuses on an essential aspect of intelligent behavior – how an agent can learn to make good decisions given experience and rewards in a stochastic<br>world. Yet popular RL algorithms that have enabled exciting successes in domains with good simulators (Go, Atari, etc) still often fail to learn in other domains because they rely on<br>simple heuristics for exploration. This provides additional empirical justification for essential questions around RL, specifically around algorithms that learn in a provably efficient manner through strategic exploration in any considered domain. This thesis provides new algorithms<br>and theory that enable good performance with respect to existing theoretical frameworks for evaluating RL algorithms (specifically, probably approximately correct) and introduces new stronger evaluation criteria, that may be particularly of interest as RL is applied to more real world problems.<br>For the first line of work on probably approximately correct (PAC) RL algorithms, we introduce a series of algorithms for episodic tabular domains with substantially better PAC<br>sample complexity bounds that culminate in a new algorithm with close to minimax optimal PAC and regret bounds. Look up tables are required by most sample efficient and computationally tractable algorithms, but cannot represent many practical domains. We therefore also present a new RL algorithm that can learn a good policy in environments with high dimensional observations and hidden deterministic states; unlike predecessors, this algorithm provably<br>explores not only in a statistically but also computationally efficient manner assuming access to function classes with efficient optimization oracles. To make progress it is critical to have the right measures of success. While empirical<br>demonstrations are quite clear, we find that for theoretical properties, two of the most commonly used learning frameworks, PAC guarantees and regret guarantees, each allow undesirable algorithm behavior (e.g. ignoring new observations that could improve the policy). We present<br>a new stronger learning framework called Uniform-PAC that unifies the existing frameworks and prevents undesirable algorithm properties. One caveat of all existing learning frameworks is that for any particular episode, we do not<br>know how well the algorithm will perform. To address this, we introduce the IPOC framework that requires algorithms to provide a certificate before each episode bounding how suboptimal the current policy can be. Such certifications may be of substantial interest in high stakes scenarios when an organization may wish to track or even pause an online RL system should the potential expected performance bound drop below a required expected outcome.

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.