A data-driven energy management method for parallel PHEVs based on action dependent heuristic dynamic programming (ADHDP) model

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach.

In this paper an optimal controller based on Action Dependent Heuristic Dynamic Programming (ADHDP) approach has been proposed for multi area Automatic Generation Control (AGC) scheme. A competitive environment has been considered in the interconnected power system. Conventional AGC model has been modified to include bilateral transactions taking place in the energy market. To implement the ADHDP approach discrete model of modified AGC scheme has been utilized. ADHDP is a powerful technique of Approximate Dynamic Programming, used for providing optimal solution by minimizing the given objective function. The proposed approach has been tested on two area AGC scheme for different cases. The results have been compared with the discrete full state feedback controller.

In presented paper we propose a discrete tracking control algorithm for a two-wheeled mobile robot. The control algorithm consists of discrete Adaptive Critic Design (ACD) in Action Dependant Heuristic Dynamic Programming (ADHDP) configuration, PD controller and a supervisory term, derived from the Lyapunov stability theorem and based on the variable structure systems theory. Adaptive Critic Designs are a group of algorithms that use two independent structures for estimation of optimal value function from Bellman equation and estimation of optimal control law. ADHDP algorithm consists of Actor (ASE - Associate Search Element) that estimates the optimal control law and Critic (ACE - Adaptive Critic Element) that evaluates quality of control by estimation of the optimal value function from Bellman equation. Both structures are realized in a form of Neural Networks (NN). ADHDP algorithm does not require a plant model (the wheeled mobile robot (WMR) model) for ACE or ASE neural network weights update procedure (in contrast with other ACD configurations e.g. Heuristic Dynamic Programming or Dual Heuristic Programming that use the plant model). In presented control algorithm Actor-Critic structure is supported by PD controller and the supervisory term, that guarantee stable implementation of tracking in an initial adaptive critic neural networks learning phase, and robustness in a face of disturbances. Verification of proposed control algorithm was realized on the two-wheeled mobile robot Pioneer-2DX.

In the article a new approach to a reactive navigation of a wheeled mobile robot (WMR), using a neural dynamic programming algorithm (NPD), is presented. A proposed discrete hierarchical control system consists of a trajectory generator and a tracking control system. In the trajectory generator we used a sensor-based approach to path design for the WMR in an unknown 2-D environment with static obstacles. The main part of the navigator is an action dependant heuristic dynamic programming algorithm (ADHDP), that generates control signals used to design a collision-free trajectory, that makes reaching a goal possible. ADHDP is the discrete algorithm of actor-critic architecture, that works on-line and does not require a preliminary learning or a controlled system knowledge. The tracking control system realises the generated trajectory, it consists of dual-heuristic dynamic programming (DHP) structure, PD controller and the supervisory term derived from the Lyapunov stability theorem. Computer simulations have been conducted to illustrate the performance of the algorithm.

This paper is concerned with a new data-driven zero-sum neuro-optimal control problem for continuous-time unknown nonlinear systems with disturbance. According to the input-output data of the nonlinear system, an effective recurrent neural network is introduced to reconstruct the dynamics of the nonlinear system. Considering the system disturbance as a control input, a two-player zero-sum optimal control problem is established. Adaptive dynamic programming (ADP) is developed to obtain the optimal control under the worst case of the disturbance. Three single-layer neural networks, including one critic and two action networks, are employed to approximate the performance index function, the optimal control law, and the disturbance, respectively, for facilitating the implementation of the ADP method. Convergence properties of the ADP method are developed to show that the system state will converge to a finite neighborhood of the equilibrium. The weight matrices of the critic and the two action networks are also convergent to finite neighborhoods of their optimal ones. Finally, the simulation results will show the effectiveness of the developed data-driven ADP methods.

In this paper, a data-driven optimal control method is provided for the half-vehicle suspension system via adaptive dynamic programming. The main contribution of this paper is that the data-driven adaptive dynamic programming method is applied to the optimal control problem of half-vehicle suspension system, which only requires the input-state data of the system. First, the structure of the half-vehicle suspension system is analyzed, and the optimal control problem is introduced. Next, the model-based adaptive dynamic programming method is provided. Based on the model-based method, a data-driven adaptive dynamic programming method is given. The properties of the provided methods are analyzed. Finally, simulation example is given to show the effectiveness of the data-driven adaptive dynamic programming method.

A novel nonlinear optimal controller for a static compensator (STATCOM) connected to a power system, using artificial neural networks, is presented in this paper. The action dependent heuristic dynamic programming, a member of the adaptive critic designs family is used for the design of the STATCOM neurocontroller. This neurocontroller provides optimal control based on reinforcement learning and approximate dynamic programming. Using a proportional-integrator approach, the proposed neurocontroller is capable of dealing with actual rather than deviation signals. Simulation results are provided to show that the proposed controller outperforms a conventional PI controller for a STATCOM in a small and large multimachine power system during large-scale faults, as well as small disturbances

In order to distinguish faces of various angles during face recognition, an algorithm of the combination of approximate dynamic programming (ADP) which is called action dependent heuristic dynamic programming (ADHDP) and particle swarm optimization (PSO) is presented and used, that is to say, ADP is applied for dynamically changing the values of the PSO parameters. During the process of face recognition, the discrete cosine transformation (DCT) is first introduced to reduce negative effects. Then K-L transformation can be used to compress images and decrease data dimensions. According to principal component analysis (PCA), main parts of vectors are extracted for data representation. Finally, radial basis function (RBF) neural network is enrolled to recognize various faces. The training of RBF neural network is exploited by ADP-PSO. In terms of ORL face database, the experimental result gives a clear view of its highly accurate efficiency.

In this article, the optimal control scheme for ice-storage air conditioning (IAC) system is solved via a data-based adaptive dynamic programming (ADP) method. It is the first time that ADP is employed to design a self-learning scheme, which obtains the optimal control policy of IAC system. First, based on the data of the temperature, irradiance, and cooling load in an actual project, a prediction model of cooling load is built by a three-layer neural network with the performance verification. Second, the operation of the IAC system is analyzed. Third, a data-based ADP method is designed to realize a self-learning optimal control for the IAC system. Then, numerical results show that using the data-based optimal control method can reduce the operation costs. Finally, the comparison results show that the developed ADP method improves the system efficiency, minimizing the overall cost. Thus, the superiority of the developed algorithm is verified.

A new approach for automotive engine torque and air-fuel ratio control is presented in this paper. A class of adaptive critic designs that can be classified as (model-free) action-dependent heuristic dynamic programming is used in the present project. Adaptive critic designs are defined as designs that approximate dynamic programming in the general case, i.e., approximate optimal control over time in noisy, nonlinear environment. The present work uses a system, called "critic", to approximate the cost function in dynamic programming and thus to achieve optimal control. The goals of the present learning control design for automotive engines are emissions reduction and maintenance of optimal performance under various operating conditions. Using the data obtained from a test vehicle, we first develop a neural network model of the engine. A neural network controller is then designed based on the idea of approximate dynamic programming to achieve optimal control. In the simulation studies, the initial controller is trained using the neural network engine model developed rather than the actual engine. We have developed and tested self-learning neural network controllers for both engine torque and exhaust airfuel ratio control. For both control problems, good transient performance of the neural network controller has been observed. A distinct feature of the present technique is the controller's real-time adaptation capability based on real vehicle data which allows the neural network controller to be further refined and improved in real-time vehicle operation through continuous learning.

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

This paper presents the design and investigation of performance of a 3-DOF Quanser helicopter system using a learning optimal control approach that is grounded on approximate dynamic programming paradigms, specifically action-dependent heuristic dynamic programming (ADHDP). This approach results in an algorithm that is embedded in the actor-critic reinforcement learning architecture, that characterizes this design as a model-free structure. The developed methodology aims at implementing an optimal controller that acts in real-time in the plant control, using only the input and output signals and states measured along the system trajectories. The feedback control design technique is capable of an online tuning of the controller parameters according to the plant dynamics, which is subject to the model uncertainties and external disturbances. The experimental results demonstrate the desired performance of the proposed controller implemented on the 3-DOF Quanser helicopter.

This paper presents an optimal trajectory tracking control approach for wheeled mobile robots (WMRs) using the adaptive dynamic programming (ADP) method integrated with radial basis function neural network (RBFNN). Firstly, the kine-matic model and error dynamics of the wheeled mobile robots are derived, with the tracking error defined in a transformed coordinate system. Then, the control input is decomposed into feedforward and feedback components to track the reference trajectory. To circumvent the issue of solving the Hamilton-Jacobi-Bellman (HJB) equation, the critic learning approach based on RBFNN is employed to approximate the optimal value function. The modified critic network weight update rule is introduced, which combines stability indicators and concurrent learning terms. The feedback control strategy is formulated by combining the output of critic network with the error dynamics, ensuring convergence of tracking errors to the neighborhood of zero. Finally, the simulation experiments validate the effectiveness of the method.