Model-Based Robot Learning Control with Uncertainty Directed Exploration

Deep reinforcement learning is the technology of artificial neural networks in the field of decision-making and control. The traditional model-free reinforcement learning algorithm requires a large amount of environment interactive data to iterate the algorithm. This model’s performance also suffers due to low utilization of training data, while the model-based reinforcement learning (MBRL) algorithm improves the efficiency of the data, MBRL locks into low prediction accuracy. Although MBRL can utilize the additional data generated by the dynamic model, a system dynamics model with low prediction accuracy will provide low-quality data and affect the algorithm’s final result. In this paper, based on the A3C (Asynchronous Advantage Actor-Critic) algorithm, an improved model-based deep reinforcement learning algorithm using a learning degree network (MBRL-LDN) is presented. By comparing the differences between the predicted states outputted by the proposed multidynamic model and the original predicted states, the learning degree of the system dynamics model is calculated. The learning degree represents the quality of the data generated by the dynamic model and is used to decide whether to continue to interact with the dynamic model during a particular episode. Thus, low-quality data will be discarded. The superiority of the proposed method is verified by conducting extensive contrast experiments.

Reinforcement learning (RL) for robotics is challenging due to the difficulty in hand-engineering a dense cost function, which can lead to unintended behavior, and dynamical uncertainty, which makes exploration and constraint satisfaction challenging. We address these issues with a new model-based reinforcement learning algorithm, Safety Augmented Value Estimation from Demonstrations (SAVED), which uses supervision that only identifies task completion and a modest set of suboptimal demonstrations to constrain exploration and learn efficiently while handling complex constraints. We then compare SAVED with 3 state-of-the-art model-based and model-free RL algorithms on 6 standard simulation benchmarks involving navigation and manipulation and a physical knot-tying task on the da Vinci surgical robot. Results suggest that SAVED outperforms prior methods in terms of success rate, constraint satisfaction, and sample efficiency, making it feasible to safely learn a control policy directly on a real robot in less than an hour. For tasks on the robot, baselines succeed less than 5% of the time while SAVED has a success rate of over 75% in the first 50 training iterations. Code and supplementary material is available at https://tinyurl.com/saved-rl.

The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as ‘perseveration’ errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides 2 general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior but they also provide a useful framework for understanding neural reward valuation and action selection.

Arbitration between model-based (MB) and model-free (MF) reinforcement learning (RL) is key feature of human reinforcement learning. The computational model of arbitration control has been demonstrated to outperform conventional reinforcement learning algorithm, in terms of not only behavioral data but also neural signals. However, this arbitration process does not take full account of contextual changes in environment during learning. By incorporating a Dirichlet process Gaussian mixture model into the arbitration process, we propose a meta-controller for RL that quickly adapts to contextual changes of environment. The proposed model performs better than a conventional model-free RL, model-based RL, and arbitration model.

Advanced control techniques for CHP-DH systems: A critical comparison of Model Predictive Control and Reinforcement Learning

Model-based reinforcement learning methods provide a promising direction for a range of automated applications, such as autonomous vehicles and legged robots, due to their sample-efficiency. However, their asymptotic performance is usually inferior compared to the state-of-the-art model-free reinforcement learning methods in locomotion control domains. One main challenge of model-based reinforcement learning is learning a dynamics model that is accurate enough for planning. This paper mitigates this issue by meta-reinforcement learning from an ensemble of dynamics models. A policy learns from dynamics models that hold different beliefs of a real environment. This procedure improves its adaptability and inaccuracy-tolerance ability. A proximal meta-reinforcement learning algorithm is introduced to improve computational efficiency and reduces variance of higher-order gradient estimation. A heteroscedastic noise is added to the training dataset, thus leading to a robust and efficient model learning. Subsequently, proximal meta-reinforcement learning maximizes the expected returns by sampling “imaginary” trajectories from the learned dynamics, which does not require real environment data and can be deployed on many servers in parallel to speed up the whole learning process. The aim of this work is to reduce the sample-complexity and computational cost of reinforcement learning in robot locomotion tasks. Simulation experiments show that the proposed algorithm achieves an asymptotic performance compared with the state-of-the-art model-free reinforcement learning methods with significantly fewer samples, which confirm our theoretical results.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

Recent evidence in neuroscience and psychology suggests that a single reinforcement learning (RL) algorithm only accounts for less than 60% of the variance of human choice behavior in an uncertain and dynamic environment, where the amount of uncertainty in state-action-state transitions drift over time. The prediction performance further decreases when the size of the state space increases. We proposed a hierarchical context-dependent RL control framework that dynamically exerted control weights on model-based (MB) and multiple model-free (MF) RL strategies associated with different task goals. To properly assess the validity of the proposed method, we considered a two-stage Markov decision task (MDT) in which the three different types of context changed over time. We trained 57 different RL control models on a Caltech MDT data set; then, we assessed their prediction performance using a Bayesian model comparison. This large-scale computer simulation analysis revealed that the model providing the most accurate prediction was the version that implemented the competition between the MB and multiple goal-dependent MF RL strategies. The present study demonstrates the applicability of the goal-driven RL control to a variety of real-world human-robot interaction scenarios.

Light Detection and Ranging (LIDAR)-based wind measurement system, positioned forward-facing, can gather information about the approaching wind. It proactively enables the wind turbine to adjust its operation via the feedforward (FF) loop. LIDAR technology can enhance wind turbine performance throughout its entire operational range. It can assist in torque control when wind speeds are below the rated level and in pitch control when wind speeds exceed the rated level. In this study, Model Predictive Control (MPC) is utilized. Within the field of wind turbine research, MPC has garnered significant interest in recent years due to its capability to handle both input and output constraints and leverage advanced on disturbances caused by the incoming wind, measured by the LIDAR. In this study, an FF-MPC is designed and compared with a more standard feedback (FB) MPC. A comparison is conducted in realistic gust wind conditions, considering below and above-rated wind ranges. These comparisons are performed in a realistic, high-fidelity aeroelastic simulation environment, i.e., DNV BLADED. Both controllers are designed for the DNV BLADED Supergen 5 MW wind turbine model. The control algorithm is implemented in C++, compiled into a dynamic link library (DLL), and integrated as an external controller within the DNV BLADED to enable accurate, high-fidelity simulations. Simulation results are presented to demonstrate the superiority of FF-MPC over the standard FB-MPC.

Hybrid actuators can provide better performances than hydraulic or electromechanical actuators alone; however, control of these hybrid systems is more challenging. This paper presents the application of model predictive control (MPC) on a hybrid actuator system to coordinate multiple control objectives. MPC has advantages over traditional controllers such as a PID in controlling non-linear multi-input, multi-output systems. MPC proactively updates the control inputs in a specific time interval by observing the current state of the system, predicting the future behavior, and solving an optimization problem repeatedly with consideration of pre-defined control objectives. In this paper, a high-fidelity dynamic model of the hybrid actuator system will be developed and a simplified, reduced-order model will be derived for use in the MPC formulation process. The detailed process of designing an MPC controller, which may consist of mathematical formulation of an optimization problem and numerical methods to find the optimal solution, will be discussed in detail. Simulation results will be presented to compare the performances of the MPC with PID controllers, and the effects of different controller parameters on the system performances will be discussed.

Reinforcement Learning (RL) is gaining much research attention because it allows the system to learn from interacting with the environment. Yet, with all these successful applications, the application of RL in direct joint torque control without the help of an underlining dynamic model is not reported in the literature. This study presents a split network structure that enables successful training of RL to learn the direct torque control for trajectory following a six-axis articulated robot without prior knowledge of the dynamic robot model. The training took a very long time to converge. However, we were able to show the successful control of four different trajectories without needing an accurate dynamics model and complex inverse kinematics computation. To show the RL-based control’s effectiveness, we also compare the RL control with the Model Predictive Control (MPC), another popular trajectory control method. Our results show that while the MPC achieves smoother and more accurate control, it does not automatically treat the singularity. In addition, it requires complex inverse dynamics calculations. On the other hand, the RL controller instinctively avoided the violent action around the singularities.

Reinforcement Learning (RL) has become a trending research topic with great success in outperforming humans on many tasks including video games, board games, and robotics control. By leveraging Deep Learning (DL), RL algorithms can consume a large volume of data without any prior knowledge of the system dynamics. However, requiring a large amount of data also limits the applicability in many fields where data is costly to obtain. Model-based Reinforcement Learning (MBRL) is regarded as a promising way to achieve high data efficiency while maintaining comparable performance. MBRL equips a dynamic transition model to facilitate and speed up the policy searching by learning the system dynamics. But there are no satisfying open-sourced libraries for the RL community to conduct MBRL research. Therefore, to fill the gap, we propose an open-sourced, flexible, and user-friendly MBRL library, Baconian, to facilitate the research on MBRL.

In this paper, we present a robotic model-based reinforcement learning method that combines ideas from model identification and model predictive control. We use a feature-based representation of the dynamics that allows the dynamics model to be fitted with a simple least squares procedure, and the features are identified from a high-level specification of the robot's morphology, consisting of the number and connectivity structure of its links. Model predictive control is then used to choose the actions under an optimistic model of the dynamics, which produces an efficient and goal-directed exploration strategy. We present real time experimental results on standard benchmark problems involving the pendulum, cartpole, and double pendulum systems. Experiments indicate that our method is able to learn a range of benchmark tasks substantially faster than the previous best methods. To evaluate our approach on a realistic robotic control task, we also demonstrate real time control of a simulated 7 degree of freedom arm.

Although model-free reinforcement learning methods are illustrated to own abilities to learn a wide range of control and decision-making tasks, it is time-consuming and needs a huge number of samples to learn optimal policy. Therefore, model-based reinforcement learning algorithms are always treated as the most efficient ways to settle robotic control problems. In this work, a model-based reinforcement learning tracking control method with free energy is proposed. The control objective is to make sure that the proposed controller can let a robotic manipulator with parameters uncertainties and disturbance follow the desired trajectory and obtain good tracking performance. The main contribution of this work is to employ the free energy principle to optimize the problem of robotic control. The probability distribution of robotic dynamic model is learnt by a evolving neural network, a free energy based reinforcement learning control method uses the learnt model to find optimal control policy. The experiment results demonstrate that the proposed tracking control method is effective in solving robotic tracking control problems.