Reach-Avoid Differential Graphical Games for Single Evader and Multiple Pursuers With Nonlinear Dynamics

We study a stochastic perishable inventory control problem with endogenous (decision-dependent) uncertainty in shelf life of units. Our primary motivation is determining ordering policies for blood platelets. Hospitals typically order their required platelets from a central supplier, and as such, the shelf life of units at the time of delivery can be subject to significant variability. Determining optimal ordering quantities is a challenging task because of the short maximum shelf life of platelets (three to five days after testing) and high uncertainty in daily demand. We formulate the problem as an infinite-horizon discounted Markov decision process (MDP). The model captures salient features observed in our data from a network of Canadian hospitals and allows for fixed ordering costs. We show that with uncertainty in shelf life, the value function of the MDP is nonconvex and key structural properties valid under deterministic shelf life no longer hold. Hence, we propose an approximate dynamic programming (ADP) algorithm to find approximate policies. We approximate the value function using a linear combination of basis functions and tune the parameters using a simulation-based policy iteration algorithm. We evaluate the performance of the proposed policy using extensive numerical experiments in parameter regimes relevant to the platelet inventory management problem. We further leverage the ADP algorithm to evaluate the impact of ignoring shelf-life uncertainty. Finally, we evaluate the out-of-sample performance of the ADP algorithm in a case study using real data and compare it with the historical hospital performance and other benchmarks. After tuning the parameters, the ADP policy can be computed online in a few seconds and results in more than 50% lower expiration and shortage rates compared with the historical rates. In addition, it performs better or as well as other benchmarks, including an exact policy that ignores uncertainty in shelf life and becomes hard to compute for larger instances of the problem. History: Accepted by Paul Brooks, Area Editor for Applications in Biology, Medicine, & Healthcare. Funding: V. Sarhangian was supported by the Natural Sciences and Engineering Research Council of Canada [Grant RGPIN-2018-04518]. H. Abouee-Mehrizi was supported by the Natural Sciences and Engineering Council of Canada [Grant RGPIN-2019-05625]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0245 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0245 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

This paper introduces novel Integral Reinforcement Learning solution to a class of differential games known as differential graphical games. The agents' error dynamics are coupled dynamical systems driven by the control input of each agent and the control inputs of its neighbors. A new class of control policies is developed to solve the differential graphical games with innovative performance index which is used to measure the system performance. The graphical game Integral Reinforcement Learning Bellman equations are shown to be equivalent to certain graphical game coupled Hamilton-Jacobi-Bellman equations developed herein. Online Policy Iteration algorithm is proposed to solve the differential graphical game in real-time. Convergence of the policy iteration algorithm is shown under mild assumptions about the inter-connectivity properties of the graph. Novel coupled Riccati formulation is developed to solve the differential graphical games.

We are interested in high-order numerical approximations for the solution of multi-dimensional Hamilton-Jacobi (HJ) equations of the form $$ {{\phi }_{t}} + H(\nabla \phi ) = 0,\quad x = ({{x}_{1}}, \ldots {{x}_{d}}) \in {{\mathbb{R}}^{d}}. $$ Here, H is the Hamiltonian, which we assume depends on ∇ϕ and possibly on x and t. The main difficulty in approximating the solutions of HJ equations is the discontinuous derivatives that develop in the solution in finite time even when the initial data is smooth. From an analytical point of view, solutions of HJ equations past the discontinuity are treated via the machinery of viscosity solutions (see [3, 4, 11] and the references therein). As far as the numerics is concerned, the first significant result were the converging first-order approximations by Souganidis [18]. High-order upwind methods, that are based on the ENO reconstruction of Harten et al [6], were introduced by Osher, Sethian and Shu [16, 17]. Jiang and Peng combined the Weighted ENO (WENO) interpolant [8, 14] with a monotone fiux obtaining the upwind-WENO schemes for HJ equations in [7].

ABSTRACTThis article investigates the optimal control problem (OCP) for a class of discrete‐time nonlinear systems with state constraints. First, to overcome the challenge caused by the constraints, the original constrained OCP is transformed into an unconstrained OCP by utilizing the system transformation technique. Second, a new cost function is designed to alleviate the effect of system transformation on the optimality of the original system. Further, a novel off‐policy deterministic approximate dynamic programming (ADP) scheme is developed to obtain a near‐optimal solution for the transformed OCP. Compared to existing off‐policy deterministic ADP schemes, the developed scheme relaxes the requirement on the learning data and saves computing resources from the perspective of training neural networks. Third, considering approximation errors, we analyze the convergence and stability of the developed ADP scheme. Finally, the developed ADP with the designed cost function is tested in two numerical cases, and simulation results confirm its effectiveness.

This paper seeks to combine differential game theory with the actor-critic-identifier architecture to determine forward-in-time, approximate optimal controllers for formation tracking in multi-agent systems, where the agents have uncertain heterogeneous nonlinear dynamics. A continuous control strategy is proposed, using communication feedback from extended neighbors on a communication topology that has a spanning tree. A model-based reinforcement learning technique is developed to cooperatively control a group of agents to track a trajectory in a desired formation. Simulation results are presented to demonstrate the performance of the developed technique.

Coordinated transportation of tethered multi-rotor UAVs based on differential graphical games

In this article, a class of the relay pursuit–evasion problem is dealt with. Multiple pursuers and a single evader take part in the considered game. We propose a region-based relay pursuit scheme for the pursuers to capture a single evader that adopts the pure evasion strategy. First, a round pursuit strategy, which is a quickest decent control law, is proposed as an alternative strategy to the pure evasion strategy. Next, a partition of the planar space is given by using the Voronoi diagram and some circular barriers to indicate the dominance region of each pursuer and each pursuit strategy. Then, the region-based relay pursuit scheme is put forward. Finally, two simulation examples are provided to verify the effectiveness and superiority of the proposed methodology.

Reinforcement learning (RL) and approximate dynamic programming (ADP) have been recently studied to solve nonlinear optimal control problems (OCPs) of continuous‐time (CT) systems. However, online learning efficiency and reliability are two major concerns to be further improved. Motivated by the above issues, in this paper we propose a receding‐horizon reinforcement learning (RHRL) algorithm for near‐optimal control of CT systems under control constraints. Different from classic RL and ADP, in the proposed approach, the infinite‐horizon OCP is decomposed as a series of finite‐horizon ones solved with an actor‐critic structure according to the receding horizon strategy, which can improve the online learning efficiency and reliability. The unknown dynamics of the system are identified offline using a sparse kernel‐based neural network structure whose weights are also updated online in the RHRL framework to improve the control performance. Moreover, the convergence of the modeling error is proven. To verify the effectiveness of our approach, we apply the RHRL algorithm to the autonomous ground vehicle for realizing near‐optimal path‐tracking control. Compared with CT model predictive control using a nominal model and other model‐free tracking controllers such as pure pursuit, heuristic dual programming, and the soft actor‐critic algorithm, RHRL performs better in terms of control performance.

Stability and robustness analysis of minmax solutions for differential graphical games

This paper focuses on economical operation of a grid-connected microgrid in real-time. An online optimization model is developed to achieve the microgrid operation in the process of dynamic optimal control, and through information feedback and online optimization to reduce the impact of uncertainty in a MG. The online optimization model has the characteristics of nonlinear and discontinuous, so an approximate dynamic programming (ADP) algorithm is used to solve this online model. Finally, the proposed online optimization model is tested in the European benchmark microgrid system. The simulation results show that the online optimization model overcomes the impacts of random fluctuations of renewable energy and loads and reduces the operating cost of the microgrid.

An adaptive sparse grid local discontinuous Galerkin method for Hamilton-Jacobi equations in high dimensions

This article is devoted to the study of fully nonlinear stochastic Hamilton-Jacobi (HJ) equations for the optimal stochastic control problem of ordinary differential equations with random coefficients. Under the standard Lipschitz continuity assumptions on the coefficients, the value function is proved to be the unique viscosity solution of the associated stochastic HJ equation.

This paper presents the hierarchical optimal multi-control algorithm for the partially unknown finite-horizon linear systems. The key to the multi-control problem is to find the solution to the coupled leader-follower Hamilton-Jacobi (HJ) equations. The coupled leader-follower HJ equations are derived and we define the Stackelberg equilibrium of the games. To solve the HJ equations for the follower, an integral reinforcement learning (IRL) approach is given on the basis of the PI algorithm, which uses partial knowledge of the dynamic systems. The Finite-Neural-Network (FNN) structure is used to approximate the value functions in the proposed algorithm. Simulation results are shown to verify the effectiveness of the scheme.

This paper presents a data-driven method for designing optimal controllers and robust controllers for unknown nonlinear systems. Mathematical models for the realization of the control are difficult to develop owing to a lack of knowledge regarding such systems. The proposed multidisciplinary method, based on optimal control theory and machine learning with kernel functions, facilitates designing appropriate controllers using a data set. Kernel-based system models are useful for representing nonlinear systems. An optimal and an H-infinity controller can be designed by solving Hamilton-Jacobi (HJ) equations, which unfortunately, are difficult to solve owing to the nonlinearity and complexity of the kernel-based models. The objective of this study consists of overcoming two challenges. The first challenge is to derive exact solutions to the HJ equations for a class of kernel-based system models. A key technique in overcoming this challenge is to reduce the HJ equations to easily solvable algebraic matrix equations, from which optimal and H-infinity controllers are designed. The second challenge is to control an unknown system using the obtained controllers, wherein the system is identified as a kernel-based model. Additionally, this study analyzes probabilistic stability of the feedback system with the proposed controllers. Numerical simulations demonstrate control performances of both the derived optimal and H-infinity controllers and stability of the feedback system.