Optimization-Based Design and Control of Dynamic Systems

: Chemical model systems were studied to develop insights into the behavior and control of spatiotemporal dynamical systems. Noise supported wave propagation has been characterized as well as power-law relations that are characteristic of avalanche behavior. Studies were also carried out of periodically driven excitable media, spatial symmetry breaking, and the evolution of three-dimensional scroll waves. Theoretical studies were carried out on exclusive product selectivity and self-segregation in systems with competitive autocatalysis as well as coherent structure analysis of spatiotemporal chaos. Topographic organization of Hebbian neural connections, uncertain destination dynamics in coupled Lorenz systems, and normal modes for chemical reactions from time series analysis were also studied.

In recent years, multilayer neural networks and recurrent networks have emerged as important components for representing nonlinear transformations and have proved particularly successful in pattern recognition and optimization problems. The authors explore methods for incorporating such networks in adaptive systems for the identification and control of complex nonlinear dynamical systems. >

On model reference adaptive control for uncertain dynamical systems, it is well know that there exists a fundamental stability limit, where the closed-loop dynamical system subject to this class of control laws remains stable either if there does not exist significant unmodeled dynamics or the effect of system uncertainties is negligible. Specifically, this implies that model reference adaptive controllers cannot tolerate large system uncertainties even when unmodeled dynamics satisfy a set of conditions. Motivated from this standpoint, this paper proposes a model reference adaptive control approach to relax this fundamental stability limit, where an adaptive control signal is augmented with an adaptive robustifying term. The key feature of our framework allows the closed-loop dynamical system to remain stable in the presence of large system uncertainties when the unmodeled dynamics satisfy a set of conditions. An illustrative numerical example is provided to demonstrate the efficacy of the proposed approach.

A novel neural network approximation scheme that is especially appropriate for adaptive control of nonlinear dynamical systems is proposed. In light of the new function approximation characterization, a neuro adaptive control framework for continuous- and discrete-time nonlinear uncertain dynamical systems is also presented. Specifically, the proposed neural network control framework is Lyapunov-based and, unlike standard neural network controllers guaranteeing ultimate boundedness, the framework guarantees partial asymptotic stability of the closed-loop system, that is, asymptotic stability with respect to part of the closed-loop system states associated with the system plant dynamics. The neuro adaptive controllers are constructed without requiring explicit knowledge of the system dynamics other than the assumption that the plant dynamics are continuous and piecewise continuously differentiable, and that the approximation error of uncertain system nonlinearities lie in a small gain-type norm bounded conic sector. This allows us to show that the standard neural network controllers are in fact capable of achieving partial asymptotic stability around the system equilibrium point for continuous-and discrete-time uncertain systems.

An Overview of Filtering and Stochastic Control in Dynamic Systems

Simultaneous design and control under uncertainty: A back-off approach using power series expansions

Approaches to keeping a dynamical system within state constraints typically rely on a model-based safety condition to limit the control signals. In the face of significant modeling uncertainty, the system can suffer from important performance penalties due to the safety condition becoming overly conservative. Machine learning can be employed to reduce the uncertainty around the system dynamics, and allow for higher performance. In this article, we propose the safe uncertainty-learning principle, and argue that the learning must be properly structured to preserve safety guarantees. For instance, robust safety conditions are necessary, and they must be initialized with conservative uncertainty bounds prior to learning. Also, the uncertainty bounds should only be tightened if the collected data sufficiently captures the future system behavior. To support the principle, two example problems are solved with control barrier functions: a lane-change controller for an autonomous vehicle, and an adaptive cruise controller. This work offers a way to evaluate whether machine learning preserves safety guarantees during the control of uncertain dynamical systems. It also highlights challenging aspects of learning for control.

Linear quadratic regulator (LQR) and proportional-integral-derivative (PID) control methods, which are generally used for control of linear dynamical systems, are used in this paper to control the nonlinear dynamical system. LQR is one of the optimal control techniques, which takes into account the states of the dynamical system and control input to make the optimal control decisions. The nonlinear system states are fed to LQR which is designed using a linear state-space model. This is simple as well as robust. The inverted pendulum, a highly nonlinear unstable system, is used as a benchmark for implementing the control methods. Here the control objective is to control the system such that the cart reaches a desired position and the inverted pendulum stabilizes in the upright position. In this paper, the modeling and simulation for optimal control design of nonlinear inverted pendulum-cart dynamic system using PID controller and LQR have been presented for both cases of without and with disturbance input. The Matlab-Simulink models have been developed for simulation and performance analysis of the control schemes. The simulation results justify the comparative advantage of LQR control method.

Safety is a critical feature of controller design for physical systems. When designing control policies, several approaches to guarantee this aspect of autonomy have been proposed, such as robust controllers or control barrier functions. However, these solutions strongly rely on the model of the system being available to the designer. As a parallel development, reinforcement learning provides model-agnostic control solutions but in general, it lacks the theoretical guarantees required for safety. Recent advances show that under mild conditions, control policies can be learned via reinforcement learning, which can be guaranteed to be safe by imposing these requirements as constraints of an optimization problem. However, to transfer from learning safety to learning safely, there are two hurdles that need to be overcome: (i) it has to be possible to learn the policy without having to re-initialize the system; and (ii) the rollouts of the system need to be in themselves safe. In this paper, we tackle the first issue, proposing an algorithm capable of operating in the continuing task setting without the need of restarts. We evaluate our approach in a numerical example, which shows the capabilities of the proposed approach in learning safe policies via safe exploration.

A new recurrent neural-network predictive feedback control structure for a class of uncertain nonlinear dynamic time-delay systems in canonical form is developed and analyzed. The dynamic system has constant input and feedback time delays due to a communications channel. The proposed control structure consists of a linearized subsystem local to the controlled plant and a remote predictive controller located at the master command station. In the local linearized subsystem, a recurrent neural network with on-line weight tuning algorithm is employed to approximate the dynamics of the time-delay-free nonlinear plant. No linearity in the unknown parameters is required. No preliminary off-line weight learning is needed. The remote controller is a modified Smith predictor that provides prediction and maintains the desired tracking performance; an extra robustifying term is needed to guarantee stability. Rigorous stability proofs are given using Lyapunov analysis. The result is an adaptive neural net compensation scheme for unknown nonlinear systems with time delays. A simulation example is provided to demonstrate the effectiveness of the proposed control strategy.

Modeling, stabilization, and identification processes are significant stages in the process of developing knowledge about chaotic dynamical systems which entail the effective prediction depending on the degree of uncertainty toleration in the forecast, accuracy of the current state to be measured as well as a time scale resting on the dynamics of the system. Control of under-activated dynamical systems has been considered substantially, and it is for periods and is currently developing in various domains such as biology, data analysis, computing systems, and so forth. Dynamic systems of growing population signifies a model describing the way a population evolves over time during which population goes through major life events, split into discrete time periods. The size of the population at a given time period is determined by the rate of growth as well as other related factors. Most progress has been made in model-based control theory, which has drawbacks when the system under consideration is exceedingly complicated, and no model can be constructed. Accordingly, a 3D-discrete and dynamic human population growth system with many controllers is proposed by examining the stability and symmetry of controller system clarifications. The symmetric stability control results are presented by considering a special parametric dynamic system in its coefficients besides suggesting periodic functional coefficients in terms of sin and cos functions. The controllers have the ability to reduce population growth rate unpredictability or enhance system stability under various external conditions. The unique and very effective strategies in relevant domains could provide a deeper understanding of their impact as well as the theoretical or technological innovations thereof. These controllers are capable of reducing population growth rate unpredictability or improving system stability under various external conditions, and applicable strategies in the relevant domains can provide profound comprehension over the impact along with the theoretical as well as technological advancements.

In this chapter, the use of principles of reinforcement learning to design a new class of feedback controllers for continuous-time dynamical systems is presented. This chapter also reviews current technology, showing that for discrete-time dynamical systems, reinforcement learning methods allow the solution of HJB design equations online, forward in time and without knowing the full system dynamics.

In this article, a policy-iteration-based active disturbance rejection control (ADRC) is proposed for uncertain nonlinear systems to achieve real-time output tracking performance, regardless of the specific relative degree of the system. The approach integrates a partial control input generator with a policy-iteration-based reinforcement learning (RL) agent for degree weight adjustment. The partial control input generator includes each ith order partial control input, which is constructed following the ADRC design framework for an ith order system. The RL agent adjusts the degree weights (its actions) to enhance the dominance of the partial control input corresponding to the unknown relative degree through iterative policy refinement. The RL agent is designed to minimize the quadratic reward as the performance index function while enhancing the influence of the partial control input associated with the correct relative degree via the policy iteration procedure. All signals in the closed-loop system (including the time-varying degree weights) ensure semi-global uniformly ultimately boundness using the Lyapunov stability theorem and the affinely quadratically stable property. Consequently, the degree weight adjustments by the RL agent do not affect the closed-loop stability. The proposed method does not require system dynamics, specific relative degree, external disturbances, and other state variable sensing beyond output sensing. The performance of the proposed method was validated via simulations for two different-order uncertain nonlinear systems and experiments using a permanent magnet synchronous motor testbed.

In this chapter, a novel control method using a reinforcement learning (RL) (Sutton and Barto (1998)) with concept of sliding mode control (SMC) (Slotine and Li (1991)) for unknown dynamical system is considered. In designing the control system for unknown dynamical system, there are three approaches. The first one is the conventional model-based controller design, such as optimal control and robust control, each of which is mathematically elegant, however both controller design procedures present a major disadvantage posed by the requirement of the knowledge of the system dynamics to identify and model it. In such cases, it is usually difficult to model the unknown system, especially, the nonlinear dynamical complex system, to make matters worse, almost all real systems are such cases. The second one is the way to use only the soft-computing, such as neural networks, fuzzy systems, evolutionary systems with learning and so on. However, in these cases it is well known that modeling and identification procedures for the dynamics of the given uncertain nonlinear system and controller design procedures often become time consuming iterative approaches during parameter identification and model validation at each step of the iteration, and in addition, the control system designed through such troubles does not guarantee the stability of the system. The last one is the way to use the method combining the above the soft-computing method with the model-based control theory, such as optimal control, sliding mode control (SMC), H∞ control and so on. The control systems designed through such above control theories have some advantages, that is, the good nature which its adopted theory has originally, robustness, less required iterative learning number which is useful for fragile system controller design not allowed a lot of iterative procedure. This chapter concerns with the last one, that is, RL system, a kind of soft-computing method, supported with robust control theory, especially SMC for uncertain nonlinear systems. RL has been extensively developed in the computational intelligence and machine learning societies, generally to find optimal control policies for Markovian systems with discrete state and action space. RL-based solutions to the continuous-time optimal control problem have been given in Doya (Doya (2000). The main advantage of using RL for solving optimal

Despite the significant advancements in single-agent evolutionary reinforcement learning, research exploring evolutionary reinforcement learning within multi-agent systems is still in its nascent stage. The integration of evolutionary algorithms (EA) and reinforcement learning (RL) has partially mitigated RL's reliance on the environment and provided it with an ample supply of data. Nonetheless, existing studies primarily focus on the indirect collaboration between RL and EA, which lacks sufficient exploration on the effective balance of individual and team rewards. To address this problem, this study introduces game theory to establish a dynamic cooperation framework between EA and RL, and proposes a multi-agent evolutionary reinforcement learning based on cooperative games. This framework facilitates more efficient direct collaboration between RL and EA, enhancing individual rewards while ensuring the attainment of team objectives. Initially, a cooperative policy is formed through a joint network to simplify the parameters of each agent to speed up the overall training process. Subsequently, RL and EA engage in cooperative games to determine whether RL jointly optimizes the same policy based on Pareto optimal results. Lastly, through double objectives optimization, a balance between the two types of rewards is achieved, with EA focusing on team rewards and RL focusing on individual rewards. Experimental results demonstrate that the proposed algorithm outperforms its single-algorithm counterparts in terms of competitiveness.