Learning to Guide: Guidance Law Based on Deep Meta-Learning and Model Predictive Path Integral Control

This paper considers path following problem of an under-actuated unmanned surface vehicle (USV) being lack of model information. A fully data-driven path following algorithm is presented for the USV based on model-based reinforcement learning (RL) and model predictive path integral (MPPI) control. Specifically, a fully-connected neural network is trained to approximate the state transition model of the USV based on model-based RL. Next, based on the learned model, a sampling-based MPPI control is used to obtain the optimal actions. Simulation results demonstrate the performance of the proposed data-driven path following algorithm for the USV without using any model information.

This article proposes a new safety concept: dynamically formulated backup plan safety. The backup plan safety is defined as the ability to complete one of the alternative missions formulated in real-time in the case of primary mission abortion. To incorporate this new safety concept in control problems, we formulate a feasibility maximization problem that adopts additional (virtual) input horizons toward the alternative missions on top of the input horizon toward the primary mission. Cost functions for the primary and alternative missions construct multiple objectives, and multi-horizon inputs evaluate them. To address the feasibility maximization problem, we develop a multi-horizon multi-objective model predictive path integral control (3M) algorithm. Model predictive path integral control (MPPI) is a sampling-based scheme that can help the proposed algorithm deal with nonlinear dynamic systems and achieve computational efficiency by parallel computation. Simulations of the aerial vehicle control problems demonstrate the new concept of backup plan safety and the performance of the proposed algorithm.

This article addresses flocking control of a swarm of under-actuated unmanned surface vehicles (USVs) subject to stationary and moving obstacles guided by a dynamic leader. The leader and follower USVs suffer from fully unknown models. A fully data-driven learning and control method is proposed to achieve a stable flocking behavior via deep reinforcement learning (DRL) and model predictive path integral (MPPI) control. Specifically, by leveraging the recorded input and output data, a deep neural network is first trained for approximating the dynamics model of each USV. Next, based on the learned vehicle dynamics, MPPI control laws are designed such that flocking and collision avoidance tasks can be simultaneously achieved. It is indicated that after learning by utilizing random and new data, stable flocking can be achieved without using any model information. Simulation results validate the effectiveness of the proposed fully data-driven learning and control method for flocking of under-actuated USVs.

In this paper we present a model predictive control algorithm designed for optimizing non-linear systems subject to complex cost criteria. The algorithm is based on a stochastic optimal control framework using a fundamental relationship between the information theoretic notions of free energy and relative entropy. The optimal controls in this setting take the form of a path integral, which we approximate using an efficient importance sampling scheme. We experimentally verify the algorithm by implementing it on a Graphics Processing Unit (GPU) and apply it to the problem of controlling a fifth-scale Auto-Rally vehicle in an aggressive driving task.

The model predictive control (MPC) of voltage source pulse width modulation (PWM) rectififier is extremely dependent on system model in control process. System model mismatch will make the prediction results deviate from the actual value, seriously affect the system control results, even caused the system to be unstable. In addition, MPC is a control method based on state space model and predictive state of controlled system. In each control cycle, it performs an ergodic optimization to solve the online optimization, and thus determines optimal driving signal of the switching device. However, to obtain the optimal control variables online usually leads to a high amount of computation, which is one of the difficulties in the implementation of MPC on PWM rectifier. To solve these problems, neural network is used to approximate the unknown functions in system model. A finite set model predictive path integral control (FCS-MPPIC) based on neural network is proposed to effectively decrease computational complexity of control algorithm under premise of ensuring the distortion rate of the grid side current. At the same time, dynamic response speed of the system is improved. Finally, the availability of this method is verified through the simulation results.

High-speed autonomous driving in off-road environments has immense potential for various applications, but it also presents challenges due to the complexity of vehicle-terrain interactions. In such environments, it is crucial for the vehicle to predict its motion and adjust its controls proactively in response to environmental changes, such as variations in terrain elevation. To this end, we propose a method for learning terrain-aware kinodynamic model which is conditioned on both proprioceptive and exteroceptive information. The proposed model generates reliable predictions of 6-degree-of-freedom motion and can even estimate contact interactions without requiring ground truth force data during training. This enables the design of a safe and robust model predictive controller through appropriate cost function design which penalizes sampled trajectories with unstable motion, unsafe interactions, and high levels of uncertainty derived from the model. We demonstrate the effectiveness of our approach through experiments on a simulated off-road track, showing that our proposed model-controller pair outperforms the baseline and ensures robust high-speed driving performance without control failure.

This paper presents a reactive navigation method that leverages a Model Predictive Path Integral (MPPI) control enhanced with spline interpolation for the control input sequence and Stein Variational Gradient Descent (SVGD). The MPPI framework addresses a nonlinear optimization problem by determining an optimal sequence of control inputs through a sampling-based approach. The efficacy of MPPI is significantly influenced by the sampling noise. To rapidly identify routes that circumvent large and/or newly detected obstacles, it is essential to employ high levels of sampling noise. However, such high noise levels result in jerky control input sequences, leading to non-smooth trajectories. To mitigate this issue, we propose the integration of spline interpolation within the MPPI process, enabling the generation of smooth control input sequences despite the utilization of substantial sampling noises. Nonetheless, the standard MPPI algorithm struggles in scenarios featuring multiple optimal or near-optimal solutions, such as environments with several viable obstacle avoidance paths, due to its assumption that the distribution over an optimal control input sequence can be closely approximated by a Gaussian distribution. To address this limitation, we extend our method by incorporating SVGD into the MPPI framework with spline interpolation. SVGD, rooted in the optimal transportation algorithm, possesses the unique ability to cluster samples around an optimal solution. Consequently, our approach facilitates robust reactive navigation by swiftly identifying obstacle avoidance paths while maintaining the smoothness of the control input sequences. The efficacy of our proposed method is validated on simulations with a quadrotor, demonstrating superior performance over existing baseline techniques.

Recently, Model Predictive Path Integral (MPPI) control algorithm has been applied to autonomous navigation tasks. However, using the conventional MPPI algorithm for the autonomous navigation of a quadrotor, there are cases in which a quadrotor cannot stay around a target position or avoid a concave-shaped obstacle. In this paper, we propose two algorithms to solve those problems. The first one is the noise variance control algorithm. This algorithm controls the variance of noise added for sampling depending on the distance between the position of a quadrotor and the target position. The second one is the unapproachable point algorithm. This algorithm creates an unapproachable point if a quadrotor stays in the same position for a while because of unable to avoid an obstacle. An unapproachable point gives a cost depending on the distance from the current position of a quadrotor to keep the quadrotor apart from the unapproachable point. We implemented those two algorithms on simulations and found that the quadrotor can avoid concave-shaped obstacles and stay around the given target position.

Deep reinforcement learning control for non-stationary building energy management

This paper presents a system for enabling real-time synthesis of whole-body locomotion and manipulation policies for real-world legged robots. Motivated by recent advancements in robot simulation, we leverage the efficient parallelization capabilities of the MuJoCo simulator on a multi-core CPU to achieve fast sampling over the robot state and action trajectories. Our results show surprisingly effective real-world locomotion and manipulation capabilities with a very simple control strategy. We demonstrate our approach on several hardware and simulation experiments: robust locomotion over flat and uneven terrains, climbing over a box whose height is comparable to the robot, and pushing a box to a goal position. To our knowledge, this is the first successful deployment of whole-body sampling-based MPC on real-world legged robot hardware. Experiment videos and code can be found at: whole-body-mppi.github.io.

Selection of guidance law is an important factor that affects the tactical performance of surface to air missiles. In order to select a suitable guidance law, a method of comparing the normal available acceleration with demanded or required acceleration is formulated. Two guidance laws i.e., proportional navigation guidance law (PNC) and augmented proportional navigation guidance law (APNC) are tested by this method. These guidance laws are studied by the comparison of normal available acceleration and demanded acceleration. In order to simulate the missile-target engagements for maneuvering and non maneuvering targets, a mathematical model is formed and then it is simulated in SIMULINK/MATLAB to carry out extensive simulations to analyze the PNG and APNG against maneuvering and non maneuvering targets. By observing the simulation results, it can be seen that PNG is suitable for non-maneuvering targets and it is incapable for maneuvering targets. However, the APNG is an optimal guidance law for maneuvering and non-maneuvering targets both.

A guidance law with finite time convergence is designed using the sliding mode control method and finite time convergence control theory, accounting for the missile autopilot as second-order dynamics. The high-order derivatives of the line of sight (LOS) angle are avoided in the expression of guidance law such that it can be implemented in practical applications. The designed guidance law is effective in compensating the bad influence of the autopilot dynamics on guidance accuracy. In simulations of intercepting a non maneuvering target or a maneuvering target, respectively, the designed guidance law is compared with the adaptive sliding mode guidance (ASMG) law in the presence of missile autopilot lag. Simulation results show that the designed guidance law is able to guide a missile to accurately intercept a nonmaneuvering target or a maneuvering target with finite time, even if it escapes in a great and fast maneuver and the autopilot has a relatively large lag.

A Backstepping guidance (BSG) law with saturating actuator is proposed for nonlinear kinematics of missile-target motion in the three-dimensional (3D) spherical coordinates system. The most important characteristic of Backstepping control (BSC) is high robustness. The proposed guidance law can reduce interception time and increase robustness in comparison with other guidance laws. In the proposed guidance outline, a Backstepping guidance law is designed to eliminate the effect of external perturbation. Then, it is illustrated that derived guidance law stabilize system. Some scenarios are considered for the engagement and simulated. Simulation results confirm above pretensions.

In this work, design of three-dimensional (3D) proportional navigation guidance law based on input-output linearization (IOL) for tactical missiles is proposed. For the three-dimensional nonlinear missile-target engagement dynamics, the line-of-sight (LOS) rates in pitch and yaw planes are taken as outputs and the IOL theory is used to derive the guidance laws without ignoring the cross-coupling terms. Simulations are carried out by considering maneuvering target, and the results are presented to showcase the efficacy of the proposed design. Further, comparison of the performance of the proposed guidance law with augmented proportional navigation guidance(APNG) laws is carried out and the related results are presented. Lastly, implementation issues of the proposed guidance law are discussed.

P ROPORTIONAL navigation (PN) is a popular guidance law because of its robustness, simplicity, and ease of implementation. PN guidance law is capable of intercepting both nonmaneuvering and maneuvering targets [1]. PN is also an optimal control effort guidance law in the linearized setting [2]. Modern tactical guidance laws are designed not only to intercept a target but to satisfy other constraints [3] like impact time, impact angle, and other optimality constraints. These guidance laws can often be represented as being comprising of a PN component and other components designed to satisfy other constraints. As these guidance laws require an estimate of the time-to-go, it is important to develop methods to obtain an accurate estimate of the time-to-go for the PNguidance law. An interesting application requiring the estimation of time-to-go is in salvo attack problems [4], in which several missiles, launched at different times, need to achieve interception of the target simultaneously. Using PN guidance law to achieve this task requires an accurate estimation of time-to-go. Impact time, impact angle, and other optimal guidance laws also need information of the time-to-go to compute the guidance command. One of the simplest ways to estimate time-to-go is by computing range over closing velocity (or the negative of the range rate). The range over range rate estimate is accurate only when the closing velocity is constant and the initial heading error is small. Improved methods for time-to-go estimation by solving a linear quadratic control problem is discussed in York and Pastrick [5]. In Tahk et al. [6], a recursive time-to-go method is discussed in which the timeto-go estimates are updated recursively. The time-to-go estimate is updated in every guidance cycle time and a compensationmethod for the time-to-go error is obtained. The time-to-go estimate improves as the engagement proceeds because the guidance law used reduces the heading error. In Sang et al. [7] the commanded guidance history is considered to obtain the time-to-go estimate. The time-to-go estimate in Jeon et al. [4] is obtained as a function of the initial range and heading error. Our time-to-go estimation method also depends on the initial conditions and hence, our results are compared with Jeon et al. [4]. In this Note, an accurate method to obtain a time-to-go estimate of PN guidance law for a stationary target is proposed. The time-to-go for a stationary target is obtained accurately by a simple interpolation. It is shown that from the time-to-go estimate of a particular initial condition, the time-to-go estimate of any other initial condition can be obtained by a simple but elegant time scaling property of the PN guidance law. The time-to-go estimate as a function of the heading error for the particular initial condition is used as the base solution for estimating the time-to-go for other initial conditions. This method also gives estimates for initial conditions with large initial heading errors; even for cases when the missile is launched with a high heading error. Themethod can be used to obtain time-to-go estimates for any initial condition and navigation gain. The time-to-go estimation is extended to constant velocity targets by a simple iterative process. The time-to-go obtained after the initial iteration is exact and need not be computed in every guidance cycle time.