Input-derivative-constrained approximate dynamic programming for unknown continuous-time linear systems

While Approximate Dynamic Programming has successfully been used in many applications involving discrete states and inputs such as playing the games of Tetris or chess, it has not been used in many continuous state and input space applications. In this paper, we combine Approximate Dynamic Programming techniques and apply them to the continuous, nonlinear and high dimensional dynamics of a quadcopter vehicle. We use a polynomial approximation of the dynamics and sum-of-squares programming techniques to compute a family of polynomial value function approximations for different tuning parameters. The resulting approximations to the optimal value function are combined in a point-wise maximum approach, which is used to compute the online policy. The success of the method is demonstrated in both simulations and experiments on a quadcopter. The control performance is compared to a linear time-varying Model Predictive Controller (MPC). The two methods are then combined to keep the computational benefits of a short horizon MPC and the long term performance benefits of the Approximate Dynamic Programming value function as the terminal cost.

While Approximate Dynamic Programming has successfully been used in many applications involving discrete states and inputs such as playing the games of Tetris or chess, it has not been used in many continuous state and input space applications. In this paper, we combine Approximate Dynamic Programming techniques and apply them to the continuous, nonlinear and high dimensional dynamics of a quadcopter vehicle. We use a polynomial approximation of the dynamics and sum-of-squares programming techniques to compute a family of polynomial value function approximations for different tuning parameters. The resulting approximations to the optimal value function are combined in a point-wise maximum approach, which is used to compute the online policy. The success of the method is demonstrated in both simulations and experiments on a quadcopter. The control performance is compared to a linear time-varying Model Predictive Controller (MPC). The two methods are then combined to keep the computational benefits of a short horizon MPC and the long term performance benefits of the Approximate Dynamic Programming value function as the terminal cost.

It has been shown that for a class of nonlinear systems x=f(x)+g(x)u, the solution of the infinite horizon optimal regulation problem leads to a state dependent Ricatti equation. Under appropriate assumptions the optimal control may be obtained from the point-wise solution of an algebraic Ricatti equation during state evolution. This method cannot be used in finite horizon optimal regulation and finite or infinite horizon of optimal tracking problem. To solve the nonlinear quadratic regulation or tracking problems, two forward and backward equations corresponding to the state and co-state systems respectively must be solved and then the optimal control can be derived. Since the co-state equation is state-dependent and it develops backward and the state is not accessible in whole time, then the control law cannot be calculated. To overcome this problem, an iterative procedure is proposed. This method can be applied to both finite and infinite horizon optimal regulation and tracking problems. Simulation results are given for the nonlinear benchmark problem introduced in and Lorenz attractor as a chaotic system.

In this paper, near optimal tracking of a class of nonlinear systems is addressed. Adaptive (approximate) dynamic programming approach is used to calculate the optimal control in closed form. ADP (Adaptive (approximate) dynamic programming) has been widely used to resolve optimal regulation and tracking problems of nonlinear control systems. Despite advances in the so called supervised and unsupervised ADP techniques for optimal tracking, they have a main draw back. That is, the optimal controller needs to be recalculated for every particular reference trajectory. The main goal of this work is to address this issue for a class of nonlinear systems. Finally, this approach is applied on a Delta robot and the performance of the method is analyzed experimentally.

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

This work presents an adaptive optimal control algorithm based integral sliding mode control law for a class of continuous-time systems with input disturbance or uncertain and unknown parameters. The main objective is to find a general form of integral sliding mode control law can assure that the system states are forced to reach a sliding surface in the initial finite time. Then, an adaptive optimal control based on the approximate dynamic programming method is responsible for the robust stability of the closed-loop system. Finally, the theoretical analysis and simulation results demonstrate the performance of the proposed algorithm for a wheel inverted pendulum system.

In this paper, the Bellman equation is used to solve the optimal adaptive control of quantized linear discrete-time system with unknown dynamics. To mitigate the effect of the quantization errors, the dynamics of the quantization error bound and an update law for tuning the range of the dynamic quantizer are derived. Subsequently, by using adaptive dynamic programming technique, the infinite horizon optimal regulation problem of the uncertain quantized linear discrete-time system is solved in a forward-in-time manner without using value and/or policy iterations. The asymptotic stability of the closed-loop system is verified by standard Lyapunov stability approach in the presence of state and input quantizers. The effectiveness of the proposed method is verified via simulation.

The general approach to solve digital control problems is to approximate them by discrete-time control problems, which consider the system behaviour at the sampling instants only, completely disregarding the inter-sample behaviour. In this paper we consider digital control problems without making any approximations, i.e. we solve problems involving continuous-time criteria taking explicitly into account the inter-sample behaviour, which relaxes the demand for a 'small' sampling time. We solve what we call the digital optimal regulator and tracking problem where the continuous-time system is linear time-varying, and disturbed by additive white noise, and the state information at the sampling instants incomplete, and corrupted by additive white noise. The control is piecewise constant, and the continuous-time criteria are quadratic. Both the regulator and tracking problem turn out to be certainty equivalent. The solutions to both the regulator and tracking problem therefore consist of the well-known discret...

The conventional coal miner cut-off section uses a single-motor system, which will stall when encountering a hard header surface due to power limitations. In this paper, we improve the multi-motor servo system of coal miner cut-off section and propose an adaptive optimal controller for multi-driven coal miner servo system. The gear power can be changed to be lager with the multi-driver servo motor, which can cut the hard header coal surface effectively. Firstly, the coal miner multi-driven servo system is approximated a neural network. Then, define the optimal performance index with the coal miner servo system states and torques. The adaptive torques are found using an approximation dynamic programming (ADP) technique based on a given performance function that can find the saddle point and minimize the coal miner performance index. The NN weights' convergence is investigated. Finally, a simulation is included to demonstrate the efficacy of the strategies for optimizing coal miner servo system performance.

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control

Taylor-series representations for the weighting matrices in the optimal sampled-data regulator problem are presented along with error bounds and doubling formulas for computer implementation.

Weather-impacted asset routing is a complex problem, involving nonlinear, non-convex costs due to spatiotemporally varying uncertainty of weather parameters over a finite time horizon. To this end, we have augmented a previously published tool for multiobjective planning and asset routing, termed TMPLAR, with a new approximate dynamic programming-based Pareto optimization (NAPO) algorithm. TMPLAR is a mixed-initiative tool to solve the asset routing problem in dynamic and uncertain environments. It is built upon multi-objective dynamic programming algorithms to route assets in a timely fashion, while considering objectives, such as fuel efficiency, voyage time, distance, and adherence to real world constraints (asset vehicle limits, navigator-specified deadlines, etc.). The asset routing problem is exacerbated by the need to address multiple conflicting objectives, spatial and temporal uncertainty associated with the weather and multiple constraints on asset operation. The NAPO algorithm optimizes weather-based objectives in a reasonable amount of time, optimizing arrival and departure times at waypoints, asset speed and bearing. The key algorithmic contribution is a fast approximate method for substantially containing the computational complexity by generating the Pareto-front of the multi-objective shortest path problem for networks with stochastic non-convex edge costs, utilizing approximate dynamic programming and clustering techniques. The proposed algorithm is validated and we compare its performance with the new approach to multi-objective A* (NAMOA*).

Reinforcement learning (RL) and approximate dynamic programming (ADP) have been recently studied to solve nonlinear optimal control problems (OCPs) of continuous‐time (CT) systems. However, online learning efficiency and reliability are two major concerns to be further improved. Motivated by the above issues, in this paper we propose a receding‐horizon reinforcement learning (RHRL) algorithm for near‐optimal control of CT systems under control constraints. Different from classic RL and ADP, in the proposed approach, the infinite‐horizon OCP is decomposed as a series of finite‐horizon ones solved with an actor‐critic structure according to the receding horizon strategy, which can improve the online learning efficiency and reliability. The unknown dynamics of the system are identified offline using a sparse kernel‐based neural network structure whose weights are also updated online in the RHRL framework to improve the control performance. Moreover, the convergence of the modeling error is proven. To verify the effectiveness of our approach, we apply the RHRL algorithm to the autonomous ground vehicle for realizing near‐optimal path‐tracking control. Compared with CT model predictive control using a nominal model and other model‐free tracking controllers such as pure pursuit, heuristic dual programming, and the soft actor‐critic algorithm, RHRL performs better in terms of control performance.

We use the dynamic programming and approximation techniques to prove the existence of an optimal synthesis for a class of linear discrete time systems with quadratic costs and a fixed target point, in which the optimal control law is given in the feedback form of time and state. Our result establishes an analogy in synthesizing fixed target point linear quadratic regulator problems between linear continuous time systems and this class of linear discrete time systems.