Performance Evaluation of Direct Heuristic Dynamic Programming using Control-Theoretic Measures

In this paper a neural network-based approximate dynamic programming method, namely direct heuristic dynamic programming (direct HDP), is applied to power system stability control. Direct HDP is a learning and approximation based approach to address nonlinear system control under uncertainty. In the present paper, real-time system responses provided by wide area measurement system (WAMS) are used to construct such controllers which are uniquely tailored for the problems under consideration. In addition, the controller learning objective is formulated as a reward function that reflects global characteristics of the power system low frequency oscillation under the consideration of coupling effect among system components. The contribution of the paper includes a convergence proof of the direct HDP algorithm using an LQR framework, as well as case study to illustrate the proposed learning control algorithm. The case study aims at providing a new solution to a difficult large scale system coordination problem where the China Southern Power Grid is used for.

In this work, time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcement learning (RL) algorithms, the direct heuristic dynamic programming (dHDP) has been shown an effective tool as demonstrated in solving several complex learning control problems. It continuously updates the control policy and the critic as system states continuously evolve. It is therefore desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. Toward this goal, we propose a new event-driven dHDP. By constructing a Lyapunov function candidate, we prove the uniformly ultimately boundedness (UUB) of the system states and the weights in the critic and the control policy networks. Consequently, we show the approximate control and cost-to-go function approaching Bellman optimality within a finite bound. We also illustrate how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.

This chapter contains sections titled: Introduction Direct Heuristic Dynamic Programming A Control Theoretic View on the Direct HDP Direct HDP Design with Improved Performance Case 1��-��Design Guided by a Priori LQR Information Direct HDP Design with Improved Performance Case 2��-��Direct HDP for Coorindated Damping Control of Low-Frequency Oscillation Summary References ]]>

Approximate dynamic programming based approach to process control and scheduling

This paper presents a method of pole placement in the linear quadratic regulator and its application to the flight control system design. There are three features in the proposed method. First, a weighting matrix that gives desired closed-loop pole locations is obtained by solving a set of differential equations. They are derived from the characteristic equation of the Hamilton matrix. Second, poles can be placed exactly at the desired positions and arbitrarily except the symmetry of complex conjugate poles with respect to the real axis. Third, it is a diagonal weighting matrix that is obtained by the proposed method. The third feature makes output regulation with desired pole locations possible. After demonstrating the effectiveness of the method through a simple literature example, it is then applied to the F-4 aircraft's lateral dynamic model. HE optimal regulator, or linear quadratic regulator (LQR), is a typical state-space design method. It has several features. For example, by imposing penalty on state variables and control inputs in a quadratic performance index, one can investigate tradeoff between system performance and control efforts. Besides, it is well known that the LQR has infinite gain margin and 60-deg phase margin for single-input systems. However, by this technique one cannot assign closed-loop pole locations. Since pole locations have a large effect on time-response characteristics, such as overshoot, rise time, etc., it is desirable to assign pole positions in the LQR design. Pole placement in the LQR is a problem associated with the selection of appropriate weighting matrices, when a set of desired closed-loop eigenvalues (CLEs) are known. This problem has been studied by many researchers since early 1970s, and many papers113 have been published in this research area. Among them, the study of Hayase 1 provides a relation between the weighting matrix and the feedback gains for a single-input system in controllable canonical form. Recently, Ohta et al. 2 showed a simple derivation of the relation. Meanwhile, for multi-input systems, Saif3 established a simple method using aggregation to compute a weighting matrix by which arbitrary and exact pole placement can be attained in a general form. He integrated the previously proposed three approaches.46 All of the aforementioned methods for multi-input systems follow Solheim's approach for placing imaginary parts as well as for placing real parts using the mirror image property.7 Another approach, which is completely different from the preceding approaches, is that of Graupe. In his method, first, differential equations of a weighting matrix with respect to the CLEs are derived, and then a weighting matrix is found as a solution of an optimization problem, wherein the difference between CLEs and assigned eigenvalues is made as small as possible.8 However, it is complicated and does not seem practical. Besides the preceding approaches, there are many approaches such as assigning eigenvalues with prescribed degree of stability, 9 placing eigenvalues into a specified region,10'11 and achieving pole placement asymptotically.12'13 In this paper, we propose a method of pole placement that is very different from the previously proposed methods. Whereas Graupe's differential equations are derived from the Riccati equation, we

Eigenstructure assignment difficulties through linear quadratic regulator (LQR) control designs are associated to the weighting matrices Q and R determination for the algebraic Riccati equation (ARE), whose solutions give the controller K gains, that are able to assign specified eigenvalues and eigenvectors. This work proposes a methodology to solve these difficulties, and based on it and on parallel hybrid genetic algorithms and parallel solutions of the LQR problems, an efficient approach is also developed. Finally, the controller performance is verified on an aircraft model.

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

The Active Magnetic Bearing system (AMB) is a mechatronic device which is used to suspend rotating parts of a machine so that they rotate without contact to the stationary part of the machine. AMBs are highly nonlinear, non-minimum phase and inherently unstable. This paper has aimed to obtain a robust optimal state-feedback control system for the stabilization of the AMB System, with the help of Genetic algorithm (GA) as an optimization tool which will solve the tedious manual tuning of the weighting matrices in the design of linear quadratic regulator (LQR) and linear quadratic Gaussian (LQG).The system’s mathematical model has been developed and also the properties of the uncontrolled system have been analyzed. Since AMB is a MIMO system, the interaction of the inputs with the outputs has been analyzed using relative gain array analysis and frequency domain analysis of the system transfer functions. Then, the optimal state feedback controllers have been developed. Here, LQR and LQG controllers are developed.Finally, Comparative analysis between the controllers and between the design methods was carried out. The proposed GA based design methodology has resulted good Performance. In addition, the GA based design has also resulted improvements in robustness of the control systems. As far as gain margin (GM) and phase margin (PM) are concerned GA has resulted increase of 8.0*10–4 db and 6.02*10–3° in GM and PM respectively for LQR. Whereas, in LQG GA has resulted an increase of 3.8*10–5 db and 2.54*10–4°. KeywordsActive magnetic bearingGenetic algorithmLinear quadratic GaussianKalman filter

Parrot Mambo mini-drone is a readily available commercial quadrotor platform to understand and analyze the behavior of a quadrotor both in indoor and outdoor applications. This study evaluates the performance of three alternative controllers on a Parrot Mambo mini-drone in an interior environment, including Proportional–Integral–Derivative (PID), Linear Quadratic Regulator (LQR), and Model Predictive Control (MPC). To investigate the controllers’ performance, initially, the MATLAB®/Simulink™ environment was considered as the simulation platform. The successful simulation results finally led to the implementation of the controllers in real-time in the Parrot Mambo mini-drone. Here, MPC surpasses PID and LQR in ensuring the system’s stability and robustness in simulation and real-time experiment results. Thus, this work makes a contribution by introducing the impact of MPC on this quadrotor platform, such as system stability and robustness, and showing its efficacy over PID and LQR. All three controllers demonstrate similar tracking performance in simulations and experiments. In steady state, the maximal pitch deviation for the PID controller is 0.075 rad, for the LQR, it is 0.025 rad, and for the MPC, it is 0.04 rad. The maximum pitch deviation for the PID-based controller is 0.3 rad after the take-off impulse, 0.06 rad for the LQR, and 0.17 rad for the MPC.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

The sensitivity to initial conditions and the unpredictable nature of chaos makes chaos synchronization an important topic of both theory and practice. However they can be synchronized with appropriate control schemes. The paper suggests a novel approach of synchronizing chaotic systems using a Linear Quadratic Regulator (LQR) whose parameters are optimized using Particle Swarm Optimization (PSO). In the traditional LQR design, the weighting matrices <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Q$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$R$</tex> are tuned manually and this process can be difficult and suboptimal, particularly in highly nonlinear and sensitive systems like chaotic models. To address this, PSO is employed to automate the selection of these matrices by minimizing a cost function that depends on synchronization error. A master-slave synchronization scheme is designed and the LQR controller is designed to ensure that the slave system follows the master system trajectory. The results of the simulation show the effectiveness of the proposed method, as the error converges rapidly and the method performs better than the classical LQR designs.

The authors present a genetic approach to the optimal linear quadratic regulator (LQR) problem. Design of the optimal LQR depends on choosing weighting matrices. One must carry out a trial-and-error process to choose the proper weighting matrices in general. It is not a trivial problem. They show a new possibility using a genetic algorithm (GA) to find the proper weighting matrices and hence to obtain the optimal control gain. They apply the approach to the attitude control of an inverted pendulum system, which is known as the LQR problem. They perform computer simulations on the system with a 50 cm long inverted pendulum and obtain good results.

This paper discusses a control design for compensating reactive power requirement in power distribution systems using Distribution Static Compensator (DSTATCOM). The DSTATCOM is realized using voltage source inverter and is controlled using PWM pulses generated through indirect current control. The current control utilizes optimal proportional controller gains designed using Linear Quadratic Regulator (LQR) approach. The performance of the controller depends on the weighting matrix, which is chosen to ensure satisfactorily response. The DSTATCOM is controlled and designed using LQR based controller and Power Balance Theory (PBT) to generate the reference currents. The effectiveness of the control is verified through simulation results for the Matlab based model.

In the past decade, there has been considerable research interest in learning control methods based on reinforcement learning (RL) and approximate dynamic programming (ADP). As an important class of function approximation techniques, kernel methods have been recently applied to improve the generalization ability of RL and ADP methods but most previous works were only based on simulation. This paper focuses on experimental studies of real-time online learning control for nonlinear systems using kernel-based ADP methods. Specifically, the kernel-based dual heuristic programming (KDHP) method is applied and tested on real-time control systems. Two kernel-based online learning control schemes are presented for uncertain nonlinear systems by using simulation data and online sampling data, respectively. Learning control experiments were performed on a single-link inverted pendulum system as well as a double-link inverted pendulum system. From the experimental results, it is shown that both online learning control schemes, either using simulation data or using real sampling data, are effective for approximating near-optimal control policies of nonlinear dynamical systems with model uncertainties. In addition, it is demonstrated that KDHP can achieve better performance than conventional DHP, which uses multilayer perceptron neural networks.

A recursive design procedure is presented which assigns the closed-loop eigenstructure in linear quadratic regulators. At each stage, the required solution for the steady-state Riccati matrix which shifts a pole or pole pair to specified values is obtained. For pole pair placement, a free parameter in the solution permits selection of closed-loop eigenvectors. The interactive design procedure is summarized and illustrated by an example