Adaptive Dynamic Programming for Optimal Residential Energy Management

In this paper, an iterative adaptive dynamic programming (ADP) algorithm is developed to solve the optimal cooperative control problems for residential multi-battery systems. To avoid solving high-dimensional optimal control problems, we first constrain all the batteries at their worst performance, which transforms the multi-input optimal control problem into a single-input one. Based on the worst-performance optimal control law, the optimal cooperative control law for the residential multi-battery systems is obtained, where in each iteration, only a single-input optimization problem is implemented. Finally, numerical results are given to illustrate the performance of the developed algorithm.

In this paper, a new iterative optimal learning control scheme for discrete-time nonlinear systems using iterative adaptive dynamic programming (ADP) approach is developed to obtain the optimal control law for coal gasification control systems. For the unknown coal gasification process, neural networks (NNs) are introduced to reconstruct the dynamics of the gasification process, where the approximation errors of the reconstruction dynamics are considered. Via system transformation, the optimal tracking control problem with approximation errors is transformed into a two-person zero-sum optimal control problem. A new iterative ADP algorithm is then developed to obtain the optimal control law for the transformed system with convergence analysis. Finally, numerical results are given to illustrate the performance of the present method.

In this paper, an infinite horizon optimal robust guaranteed cost control scheme of a class of continuous-time uncertain nonlinear systems is established based on adaptive dynamic programming. The main idea lies in that the optimal robust guaranteed cost control problem can be transformed into an optimal control problem. Actually, the optimal cost function of the nominal system is nothing but the optimal guaranteed cost of the original uncertain system. A critic neural network is constructed to help solving the modified Hamilton-Jacobi-Bellman equation corresponding to the nominal system. Then, an additional stabilizing term is introduced to reinforce the updating process of the weight vector and reduce the requirement of an initial stabilizing control. An example is provided to illustrate the effectiveness of the present control approach.

Optimal control of aluminum electrolysis production process (AEPP) has long been a challenging industrial issue due to its inherent difficulty in establishing an accurate dynamic model. In this paper, a novel robust optimal control algorithm based on adaptive dynamic programming (ADP) is proposed for the AEPP, where the system subjects to input constraints. First, to establish an accurate dynamic model for the AEPP system, recursive neural network (RNN) is employed to reconstruct the system dynamic using the input-output production data. To ensure input constraints are not to exceed the bound of the actuator, the optimal control problem of the AEPP is formulated under a new nonquadratic form performance index function. Then, considering the perturbation of the AEPP, the robust control problem is effectively converted to the constrained optimal control problem via system transformation. Furthermore, a single critic network framework is developed to obtain the approximate solution of the Hamilton-Jacobi-Bellman (HJB) equation. Finally, the proposed ADP controller is applied to the AEPP system to validate the effectiveness and performance.

In this paper, a learning-based predictive control (LPC) scheme is proposed for adaptive optimal control of discrete-time nonlinear systems under stochastic disturbances. The proposed LPC scheme is different from conventional model predictive control (MPC), which uses open-loop optimization or simplified closed-loop optimal control techniques in each horizon. In LPC, the control task in each horizon is formulated as a closed-loop nonlinear optimal control problem and a finite-horizon iterative reinforcement learning (RL) algorithm is developed to obtain the closed-loop optimal/suboptimal solutions. Therefore, in LPC, RL and adaptive dynamic programming (ADP) are used as a new class of closed-loop learning-based optimization techniques for nonlinear predictive control with stochastic disturbances. Moreover, LPC also decomposes the infinite-horizon optimal control problem in previous RL and ADP methods into a series of finite horizon problems, so that the computational costs are reduced and the learning efficiency can be improved. Convergence of the finite-horizon iterative RL algorithm in each prediction horizon and the Lyapunov stability of the closed-loop control system are proved. Moreover, by using successive policy updates between adjoint time horizons, LPC also has lower computational costs than conventional MPC which has independent optimization procedures between two different prediction horizons. Simulation results illustrate that compared with conventional nonlinear MPC as well as ADP, the proposed LPC scheme can obtain a better performance both in terms of policy optimality and computational efficiency.

This paper builds up two equivalence theorems for different kinds of optimal control problems of internally controlled Schrodinger equations. The first one is an equivalence of the minimal norm and the minimal time control problems where the target sets are the origin of the state space. (The minimal time control problems are also called the first type of optimal time control problems.) The second one is an equivalence of three optimal control problems: optimal target control problems, optimal norm control problems, and the second type of optimal time control problems.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

Nearly finite-horizon optimal control for a class of nonaffine time-delay nonlinear systems based on adaptive dynamic programming

Previous article Next article The Reduction of Certain Control Problems to an “Ordinary Differential” TypeJ. WargaJ. Wargahttps://doi.org/10.1137/1010036PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout Previous article Next article FiguresRelatedReferencesCited byDetails Stationarity conditions in optimal control problems class related to dynamical objects group control Cross Ref On the First- and Second-Order Conditions in Optimal Control Problems Related to Autonomous Objects Group lifecycle Cross Ref Optimal Control Problems with Time Delays: Constancy of the HamiltonianRichard B. Vinter25 July 2019 | SIAM Journal on Control and Optimization, Vol. 57, No. 4AbstractPDF (525 KB)On obtaining the stationarity conditions in an optimal control problem for a trajectory with a smooth contact with the phase boundary Cross Ref On one Method to Obtain Stationarity Conditions for an Optimal Control Problem Trajectory With a Smooth Boundary ContactIFAC-PapersOnLine, Vol. 51, No. 32 Cross Ref The Maximum Principle for Optimal Control Problems with Time DelaysA. Boccia and R. B. Vinter19 September 2017 | SIAM Journal on Control and Optimization, Vol. 55, No. 5AbstractPDF (407 KB)Problems with relaxed shifted controls and the pointwise maximum principleNonlinear Analysis: Theory, Methods & Applications, Vol. 75, No. 3 Cross Ref The Instability of Optimal Control Problems to Time DelayAlexey S. Matveev26 July 2006 | SIAM Journal on Control and Optimization, Vol. 43, No. 5AbstractPDF (340 KB)On the Characterization of Properly Relaxed Delayed ControlsJournal of Mathematical Analysis and Applications, Vol. 209, No. 1 Cross Ref Approximation of strongly relaxed minimizers with ordinary delayed controlsApplied Mathematics & Optimization, Vol. 32, No. 1 Cross Ref Jack Warga: In Appreciation14 July 2006 | SIAM Journal on Control and Optimization, Vol. 33, No. 2AbstractPDF (809 KB)Minimizing approximate original solutions for commensurate delayed controlsApplied Mathematics Letters, Vol. 7, No. 5 Cross Ref How to Properly Relax Delayed Controls Cross Ref A proper relaxation of shifted and delayed controlsJournal of Mathematical Analysis and Applications, Vol. 169, No. 2 Cross Ref Optimal controls and their discontinuities in quadratic problems of delay systemsIEEE Transactions on Automatic Control, Vol. 30, No. 7 Cross Ref Optimal Controls with PseudodelaysJ. Warga18 July 2006 | SIAM Journal on Control, Vol. 12, No. 2AbstractPDF (1227 KB)A conservative bound on the estimation error covariance matrix in the presence of correlated driving noise and correlated discrete measurement noiseJournal of Mathematical Analysis and Applications, Vol. 43, No. 2 Cross Ref References Cross Ref Volume 10, Issue 2| 1968SIAM Review History Submitted:18 October 1967Published online:18 July 2006 InformationCopyright © 1968 Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/1010036Article page range:pp. 219-222ISSN (print):0036-1445ISSN (online):1095-7200Publisher:Society for Industrial and Applied Mathematics

In this paper, we propose a new approach to address robust control problem of nonlinear systems with uncertainties based on an adaptive dynamic programming (ADP) algorithm. After reformulating the robust control problem as an optimal control problem, we propose a modified ADP method to solve the derived Hamilton-Jacobi-Bellman (HJB) equation, where the optimal cost function is approximated by online training a critic neural network (NN). Then the approximated optimal control action can be derived to guarantee the stability of the controlled system with uncertainties. The closed-loop system stability and convergence have been proved. A simulation example is provided to illustrate the effectiveness of the method.

This thesis deals with the study of optimal control problems for the incompressible Magnetohydrodynamics (MHD) equations. Particular attention to these problems arises from several applications in science and engineering, such as fission nuclear reactors with liquid metal coolant and aluminum casting in metallurgy. In such applications it is of great interest to achieve the control on the fluid state variables through the action of the magnetic Lorentz force. In this thesis we investigate a class of boundary optimal control problems, in which the flow is controlled through the boundary conditions of the magnetic field. Due to their complexity, these problems present various challenges in the definition of an adequate solution approach, both from a theoretical and from a computational point of view. In this thesis we propose a new boundary control approach, based on lifting functions of the boundary conditions, which yields both theoretical and numerical advantages. With the introduction of lifting functions, boundary control problems can be formulated as extended distributed problems. We consider a systematic mathematical formulation of these problems in terms of the minimization of a cost functional constrained by the MHD equations. The existence of a solution to the flow equations and to the optimal control problem are shown. The Lagrange multiplier technique is used to derive an optimality system from which candidate solutions for the control problem can be obtained. In order to achieve the numerical solution of this system, a finite element approximation is considered for the discretization together with an appropriate gradient-type algorithm. A finite element object-oriented library has been developed to obtain a parallel and multigrid computational implementation of the optimality system based on a multiphysics approach. Numerical results of two- and three-dimensional computations show that a possible minimum for the control problem can be computed in a robust and accurate manner.

The authors present a state-space solution to the discrete H/sub 2/ control problem and give direct formulas for an H/sub 2/-optimal sampled-data control problem with state feedback and disturbance feedforward using the powerful lifting technique. The discrete-time H/sub 2/-optimal control is first treated via state feedback and disturbance feedforward and then via dynamic output feedback. New direct formulas for a sampled-data H/sub 2/ problem using state measurement are presented. The optimal sampled-data control is applied to a two-motor system and a comparison is made with the optimal analog control. >

This paper presents a functional model predictive control (MPC) approach based on an adaptive dynamic programming (ADP) algorithm with the abilities of handling control constraints and disturbances for the optimal control of nonlinear discrete-time systems. In the proposed ADP-based nonlinear MPC (NMPC) structure, a neural-network-based identification is established first to reconstruct the unknown system dynamics. Then, the actor-critic scheme is adopted with a critic network to estimate the index performance function and an action network to approximate the optimal control input. Meanwhile, as the MPC strategy can effectively determine the current control by solving a finite horizon open-loop optimal control problem, in the proposed algorithm, the infinite horizon is decomposed into a series of finite horizons to obtain the optimal control. In each finite horizon, the finite ADP algorithm solves the optimal control problem subject to the terminal constraint, the control constraint, and the disturbance. The uniform ultimate boundedness of the closed-loop system is verified by the Lyapunov approach. Finally, the ADP-based NMPC is conducted on two different cases and the simulation results demonstrate the quick response and strong robustness of the proposed method.

Discrete-time hybrid control in Borel spaces: Average cost optimality criterion

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.