On the Minimum Principle and Dynamic Programming for Hybrid Systems

Hybrid optimal control problems are studied for a general class of hybrid systems where autonomous and controlled state jumps are allowed at the switching instants and in addition to terminal and running costs switching between discrete states incurs costs. The statements of the Hybrid Minimum Principle and Hybrid Dynamic Programming are presented in this framework and it is shown that under certain assumptions the adjoint process in the Hybrid Minimum Principle and the gradient of the value function in Hybrid Dynamic Programming are governed by the same set of differential equations and have the same boundary conditions and hence are almost everywhere identical to each other along optimal trajectories. Analytic examples are provided to illustrate the results and, in particular, a Riccati formalism for linear quadratic hybrid tracking problems is presented.

This chapter is devoted upon optimality, a topic in which the central result is the Pontryagin’s minimum principle. This important result is briefly approached from the classical calculus of variation. We show that the classical calculus of variation has some major limitations to modern control problems and motivate Pontryagin’s minimum principle. The Hamilton–Jacobi–Bellman method is discussed as an alternative approach to gain first-order necessary conditions for optimality. It is shown that both approaches correspond to each other under restrictive assumptions. The original Pontryagin’s minimum principle for continuous optimal control problems is not suitable for hybrid optimal control problems. However, a quite natural reformulation of the hybrid optimal control problem admits the classical theory for deduction of first-order necessary conditions in the sense of Pontryagin. The charm of this methodology is its comprehensible derivation.

Many interesting problems in numerical optimization are hybrid optimal control problems. Hybrid optimal control (HOC) problems include both continuous-valued variables and categorical variables in the problem formulation. For the types of problems envisioned here the categorical variables will specify the structure or sequence of events that qualitatively describes the trajectory or mission. Continuous variables are used in the modeling of the continuous dynamics. In this work, the mission planning problem is of interest. The problem is qualitatively different from the typical interception of multiple targets as the discrete variables now represent events like impulses, coast and thrust arcs that change the structure of the problem. In addition, the number of events in the categorical sequence is not fixed. For the dynamical assembly of events required for the solution of the mission planning problem, a scheme that defines events as modules consisting of parameters and constraints is presented. The method assembles the respective events one next to the other in time according to the given mission structure. For the generation of the initial guess, two new methods were developed that approximate optimal low-thrust trajectories. The first method, based on genetic algorithms (GA), handles the rendezvous constraints explicitly using a conditional penalty function. The second method, feasible region analysis (FRA), is based on GA and nonlinear programming (NLP), which allows taking advantage of the GA capabilities in finding a global optimum and NLP ability in handling constraints. A rendezvous problem with known solution is solved.

A Class of Linear Quadratic Gaussian Hybrid Optimal Control Problems with Realization–Independent Riccati Equations

In this paper, an approach to the finite-horizon optimal state-feedback control problem of nonlinear, stochastic, discrete-time systems is presented. Starting from the dynamic programming equation, the value function will be approximated by means of Taylor series expansion up to second-order derivatives. Moreover, the problem will be reformulated, such that a minimum principle can be applied to the stochastic problem. Employing this minimum principle, the optimal control problem can be rewritten as a two-point boundary-value problem to be solved at each time step of a shrinking horizon. To avoid numerical problems, the two-point boundary-value problem will be solved by means of a continuation method. Thus, the curse of dimensionality of dynamic programming is avoided, and good candidates for the optimal state-feedback controls are obtained. The proposed approach will be evaluated by means of a scalar example system

In the paper we present a new method for solving optimal control problems of a class of hybrid systems. We describe the new effective algorithm based on memetic algorithm (MA) for optimal control of switched systems. We concentrate on a class of problems in which a pre-specified sequence of active subsystems is known. Our aim is to find both the optimal switching instants and the optimal continuous inputs. The new approach, which we propose, decomposes the cost functional of the basic optimal control problem in Bolza form in two terms. The first term depends explicitly on a value of state variables at the final time. The second term depends on state and control trajectories. In oreder to solve those two tasks we used MA as the multi-objective optimization algorithm. In this paper we considered a fundamental bi-criteria case with two mentioned before functions: the value of state variables at the final time and the state and control trajectories. In order to find an approximation of Pareto frontier, we proposed new effective method based on genetic algorithm (GA) and local search (LS). Problems properties were taken into consideration in the design of our new approach of solving it. They were used to construct new algorithm inspired by the LS NSGA-II, which performed rather well in multi-criteria scheduling problem. Since simple genetic algorithms are efficient heuristics in searching for optimal solutions, but lack the accuracy of some more computational complex algorithms, a hybrid algorithm was constructed. It uses fast non-dominated sorting, in order to evaluate child population and allocate solutions to corresponding Pareto frontiers. I addition a local search method was used, in order to find more differentiated and better solutions. Clustering solutions from Pareto frontiers also improved diversity of solutions in child population. This approach can be used as a start point for searching for algorithm for solving optimal control problems of switched systems without pre-specified sequence of active subsystems. The performance of the algorithm is illustrated by the examples: (a) a hybrid optimal control problem with nonlinear dynamics and (b) a switched linear quadratic optimal control problem with long switching time intervals.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

This paper is concerned with the relationship between maximum principle and dynamic programming in zero-sum stochastic differential games. Under the assumption that the value function is enough smooth, relations among the adjoint processes, the generalized Hamiltonian function and the value function are given. A portfolio optimization problem under model uncertainty in the financial market is discussed to show the applications of our result.

This paper deals with optimization techniques for linear impulsive hybrid systems (LIHSs). We study the LQ (linear quadratic) impulsive hybrid optimal control problem (OCP) and apply the corresponding Pontryagin-type maximum principle (MP) (see). Our aim is to investigate the natural relationship between the above MP and the Bellman dynamic programming (DP) approach to the above hybrid OCP under consideration. We derive the associated Riccati-type formalism and discuss some related numerical schemes.

Chapter 1 - Introduction and Motivation

A novel numerical method based on the differential transformation is proposed for solving nonlinear optimal control problems in this paper. The differential transformation is a linear operator that transforms a function from the original time and/or space domain into another domain in order to simplify the differential calculations. The optimality conditions for the optimal control problems can be represented by algebraic and differential equations. Using the differential transformation, these algebraic and differential equations with their boundary conditions are first converted into a system of nonlinear algebraic equations. Then the numerical optimal solutions are obtained in the form of finite-term Taylor series by solving the system of nonlinear algebraic equations. The differential transformation algorithm is similar to the spectral element methods in that the computational region splits into several subregions but it uses polynomials of high degrees by keeping a small number of subregions. The differential transformation algorithm could solve the finite- (or infinite-) time horizon optimal control problems formulated as either the algebraic and ordinary differential equations using Pontryagin’s minimum principle or the Hamilton–Jacobi–Bellman partial differential equation using dynamic programming in one unified framework. In addition, the differential transformation algorithm can efficiently solve optimal control problems with the piecewise continuous dynamics and/or nonsmooth control. The performance is demonstrated through illustrative examples.

Loss control regions in optimal control problems

Optimal minimal variation control with quality constraint for fed-batch fermentation processes involving multiple feeds

We consider a fairly general class of state-constrained nonlinear hybrid optimal control problems that are based on coordinatizing Sussmann's model. An event set generalizes the notion of a guard set, reset map, endpoint set as well as the switching set. We present a pseudospectral (PS) knotting method that discretizes the continuous-time variables of the problem. The discrete event conditions are imposed over the PS knots leading to a large, sparse, mixed-variable programming (MVP) problem. The Karush-Kuhn-Tucker conditions for the MVP are transformed in a manner that makes them closely resemble the discretized necessary conditions obtained from the hybrid minimum principle. A set of closure conditions are introduced to facilitate commuting the operations of dualization and discretization. An immediate consequence of this is a hybrid covector mapping theorem that provides an order-preserving transformation of the Lagrange multipliers associated with the discretized problem to the discretized covectors associated with the hybrid optimal control problem.

This paper addresses an optimal control problem for a robot that has to find and collect a finite number of objects and move them to a depot in minimum time. The objects are modeled by point masses with a priori unknown locations in a bounded two-dimensional space. The robot has forth-order dynamics that change instantaneously at any pick-up or drop-off of an object. The corresponding hybrid Optimal Control Problem (OCP) is solved by a receding horizon scheme, where the derived lower bound for the cost-to-go is evaluated for the worst- and a probabilistic case, assuming a uniform distribution of the objects. We first present a time-driven approximate solution based on time and position space discretization. Due to the high computational cost of this solution, we alternatively propose an event-driven approximate approach based on a suitable motion parameterization. The solutions are compared in a numerical example, suggesting that the latter approach offers a significant computational advantage while yielding similar qualitative results compared to the former.