Dynamic Event-Sampled Control of Interconnected Nonlinear Systems Using Reinforcement Learning.

This article considers the decentralized stabilization problem of continuous-time nonlinear systems subject to unmatched interconnections and asymmetric input constraints. Initially, with the nonquadratic value functions being introduced to constrained auxiliary subsystems, the decentralized stabilization problem is converted into an array of nonlinear optimal control problems. It is proved that all solutions of these nonlinear optimal control problems together assure asymptotic stability of the entire system. Then, in the framework of adaptive critics, the critic-only architecture is built to solve the Hamilton–Jacobi–Bellman equations associated with these solutions. The critic-only architecture is implemented via critic neural networks (NNs) with their weight vectors being tuned through an improved gradient descent method. A remarkable feature of the present gradient descent approach is that it simultaneously utilizes previously stored and instantaneous state data, which makes the persistence of excitation conditions relaxed. After that, with Lyapunov’s techniques being employed, asymptotic stability of the closed-loop auxiliary subsystems and uniform ultimate boundedness of the critic NNs’ weight estimation errors are demonstrated. Finally, simulations of an unmatched interconnected nonlinear plant are provided to validate the present decentralized control method.

Sliding-mode surface-based decentralized event-triggered control of partially unknown interconnected nonlinear systems via reinforcement learning

This paper presents a novel decentralized control strategy for a class of uncertain nonlinear large-scale systems with mismatched interconnections. First, it is shown that the decentralized controller for the overall system can be represented by an array of optimal control policies of auxiliary subsystems. Then, within the framework of adaptive dynamic programming, a simultaneous policy iteration (SPI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations associated with auxiliary subsystem optimal control policies. The convergence of the SPI algorithm is guaranteed by an equivalence relationship. To implement the present SPI algorithm, actor and critic neural networks are applied to approximate the optimal control policies and the optimal value functions, respectively. Meanwhile, both the least squares method and the Monte Carlo integration technique are employed to derive the unknown weight parameters. Furthermore, by using Lyapunov’s direct method, the overall system with the obtained decentralized controller is proved to be asymptotically stable. Finally, the effectiveness of the proposed decentralized control scheme is illustrated via simulations for nonlinear plants and unstable power systems.

Tangential hand velocity profiles of rapid human arm movements often appear as sequences of several bell-shaped acceleration-deceleration phases called submovements or movement units. This suggests how the nervous system might efficiently control a motor plant in the presence of noise and feedback delay. Another critical observation is that stochastic-ity in a motor control problem makes the optimal control policy essentially different from the optimal control policy for the deterministic case. We use a simplified dynamic model of an arm and address rapid aimed arm movements. We use reinforcement learning as a tool to approximate the optimal policy in the presence of noise and feedback delay. Using a simplified model we show that multiple submovements emerge as an optimal policy in the presence of noise and feedback delay. The optimal policy in this situation is to drive the arm's end point close to the target by one fast submovement and then apply a few slow submovementsto accurately drive the arm's end point into the target region. In our simulations, the controller sometimes generates corrective submovements before the initial fast submovement is completed, much like the predictive corrections observed in a number of psychophysical experiments.

In this paper, We consider an uncertain Markov decision process (MDP) with a control cost and a linear temporal logic (LTL) control specification. We propose a reinforcement learning (RL) based method for design of an optimal control policy by which the controlled MDP satisfies the control specification with probability 1 and minimizes an expected discounted sum of the control costs. First, we construct a deterministic Rabin automaton (DRA) that accepts all and only infinite words satisfying the LTL control specification. Second, we construct a product MDP of the MDP and the DRA to represent a dynamic control policy that satisfies the control specification. Third, we modify the product MDP in order to apply RL to the design of an optimal control policy. The control action of the modified product MDP is a pair of a pattern and an action, where the pattern is a set of actions. Moreover, we introduce a reward that represents both the satisfaction of the control specification and the minimally restrictiveness of the pattern. Finally, we proposed an algorithm for design of an optimal control policy that consists of a sequential decision making of two steps. At the first decision making, we select a pattern that maximizes a discounted sum of the reward. At the second one, we select an action from the pattern selected at the first one such that it minimizes the expected discounted sum of the costs. Moreover, we consider an illustrative example to show that the proposed algorithm can obtain an optimal control policy.

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

Manually designing an effective and efficient controller for complex mechanics, such as bio-inspired robots or underactuated mechanical system, typically are very difficult. It requires precise motion planning and dynamic control. Reinforcement learning or genetic algorithm based learning methods suffers from representing the high dimensional models. The combination of deep learning and reinforcement learning provide a feasible way to handle such difficulties. However, priori-less searching sometimes tends to be low efficient and usually finds the “mechanic” solution instead of the “natural” one. In this paper, the traditional nonlinear control concept is integrated into the deep reinforcement learning (DRL) framework. The whole process is implemented on the brachiation control problem of a two link bio-primate robot. Deep Deterministic Policy Gradient (DDPG) is used to search for the optimal control policy. The searching process is realized by interacting with the dynamic model instead of real robot. The energy based planning and control concept is adopted, which utilize the fact that when the shoulder joint angle is fixed, energy of the whole system keeps constant. By regulating the angle and energy, the robot can be restricted on a particular trajectory. The energy concept is encoded within the reward function and trained in the Gym environment. For varying targets point-to-point control, the network structure is also modified to accept the target coordinates. Effectiveness of the proposed methods are verified by simulation and experimental results.

The objective of this paper is to examine the use and applications of reinforcement learning (RL) techniques in the production planning and control (PPC) field addressing the following PPC areas: facility resource planning, capacity planning, purchase and supply management, production scheduling and inventory management. The main RL characteristics, such as method, context, states, actions, reward and highlights, were analysed. The considered number of agents, applications and RL software tools, specifically, programming language, platforms, application programming interfaces and RL frameworks, among others, were identified, and 181 articles were sreviewed. The results showed that RL was applied mainly to production scheduling problems, followed by purchase and supply management. The most revised RL algorithms were model-free and single-agent and were applied to simplified PPC environments. Nevertheless, their results seem to be promising compared to traditional mathematical programming and heuristics/metaheuristics solution methods, and even more so when they incorporate uncertainty or non-linear properties. Finally, RL value-based approaches are the most widely used, specifically Q-learning and its variants and for deep RL, deep Q-networks. In recent years however, the most widely used approach has been the actor-critic method, such as the advantage actor critic, proximal policy optimisation, deep deterministic policy gradient and trust region policy optimisation.

Flooding mitigation through safe & trustworthy reinforcement learning

This paper presents the design and implementation of a learning controller for the Automatic Generation Control (AGC) in power systems based on a reinforcement learning (RL) framework. In contrast to the recent RL scheme for AGC proposed by us, the present method permits handling of power system variables such as Area Control Error (ACE) and deviations from scheduled frequency and tie-line flows as continuous variables. (In the earlier scheme, these variables have to be quantized into finitely many levels). The optimal control law is arrived at in the RL framework by making use of Q-learning strategy. Since the state variables are continuous, we propose the use of Radial Basis Function (RBF) neural networks to compute the Q-values for a given input state. Since, in this application we cannot provide training data appropriate for the standard supervised learning framework, a reinforcement learning algorithm is employed to train the RBF network. We also employ a novel exploration strategy, based on a Learning Automata algorithm, for generating training samples during Q-learning. The proposed scheme, in addition to being simple to implement, inherits all the attractive features of an RL scheme such as model independent design, flexibility in control objective specification, robustness etc. Two implementations of the proposed approach are presented. Through simulation studies the attractiveness of this approach is demonstrated.

Nitrous oxide (N2O) emissions from agriculture are rising due to increased fertilizer use and intensive farming, posing a major challenge for climate mitigation. This study introduces a novel reinforcement learning (RL) framework to optimize farm management strategies that balance crop productivity with environmental impact, particularly N2O emissions. By modeling agricultural decision-making as a partially observable Markov decision process (POMDP), the framework accounts for uncertainties in environmental conditions and observational data. The approach integrates deep Q-learning with recurrent neural networks (RNNs) to train adaptive agents within a simulated farming environment. A Probabilistic Deep Learning (PDL) model was developed to estimate N2O emissions, achieving a high Prediction Interval Coverage Probability (PICP) of 0.937 within a 95% confidence interval on the available dataset. While the PDL model’s generalizability is currently constrained by the limited observational data, the RL framework itself is designed for broad applicability, capable of extending to diverse agricultural practices and environmental conditions. Results demonstrate that RL agents reduce N2O emissions without compromising yields, even under climatic variability. The framework’s flexibility allows for future integration of expanded datasets or alternative emission models, ensuring scalability as more field data becomes available. This work highlights the potential of artificial intelligence to advance climate-smart agriculture by simultaneously addressing productivity and sustainability goals in dynamic real-world settings.

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem

In this paper, a novel online Synchronous Policy Iteration algorithm based on Concurrent Learning (CLSPI) is presented to solve the continuous-time optimal control problem. We design this algorithm based on actor-critic architecture. Original Synchronous Policy Iteration (SPI) algorithm just updates parameters of actor-critic simultaneously, which is rather different to the way standard policy iteration updates. In the scheme of SPI, only current estimation error is utilized to update weights while previous information can also contribute to weights update. Concurrent learning is a new parameters estimation method which combines previous information and current estimation error. CLSPI utilizes concurrent learning to train SPI, in order to improve the learning performance. Finally, two comparison experiments including linear and nonlinear systems are presented to demonstrate that CLSPI can obtain the optimal control policy with faster convergence rate, which shows that CLSPI is more efficient to solve continuous-time optimal control problems.

The rapid advancement of autonomous robots in domains such as healthcare, transportation, and disaster response necessitates robust ethical decision-making frameworks to mitigate risks associated with uncertain environments. This research delves deeply into the application of reinforcement learning (RL) for enabling "safe" decisions in robots facing multiple risky outcomes. By synthesizing over 80 peer- reviewed sources, including recent 2024-2025 surveys on ethical RL and safe RL benchmarks, it systematically addresses three pivotal research questions: (1) How should a robot choose between two risky paths? (2) Can RL augmented with reward shaping effectively reduce harm in unpredictable environments? (3) How do prominent RL algorithms compare in terms of safety efficacy in robotic applications? Key findings indicate that multi-objective RL (MORL) and constrained policy optimization variants, such as Proximal Policy Optimization (PPO) with Lagrangian constraints, outperform traditional RL in balancing ethical imperatives with task performance, achieving up to 40% reductions in violation rates in simulated benchmarks. Reward shaping, particularly potential-based and ethics-informed variants, accelerates ethical convergence by 3-6x while curbing reward hacking—a common pitfall where agents exploit reward loopholes. A novel student-led simulation in a custom gridworld environment demonstrates that shaped rewards enhance average returns by approximately 12% with near-zero harm incidents over 1,000 episodes. This expanded study proposes an integrated Ethical Safe RL (ESRL) framework, incorporating human-in-the-loop feedback and verifiable constraints, tailored for accessible implementation in educational settings. It underscores persistent challenges like distributional shifts in real-world deployment and calls for interdisciplinary ethics integration. By bridging theoretical surveys with empirical simulations, this work empowers high-school researchers to contribute to responsible AI robotics.