Learning Control for Robotic Manipulator with Free Energy

Deep reinforcement learning is the technology of artificial neural networks in the field of decision-making and control. The traditional model-free reinforcement learning algorithm requires a large amount of environment interactive data to iterate the algorithm. This model’s performance also suffers due to low utilization of training data, while the model-based reinforcement learning (MBRL) algorithm improves the efficiency of the data, MBRL locks into low prediction accuracy. Although MBRL can utilize the additional data generated by the dynamic model, a system dynamics model with low prediction accuracy will provide low-quality data and affect the algorithm’s final result. In this paper, based on the A3C (Asynchronous Advantage Actor-Critic) algorithm, an improved model-based deep reinforcement learning algorithm using a learning degree network (MBRL-LDN) is presented. By comparing the differences between the predicted states outputted by the proposed multidynamic model and the original predicted states, the learning degree of the system dynamics model is calculated. The learning degree represents the quality of the data generated by the dynamic model and is used to decide whether to continue to interact with the dynamic model during a particular episode. Thus, low-quality data will be discarded. The superiority of the proposed method is verified by conducting extensive contrast experiments.

Learning control for robotic manipulators has been developed over the past decade and to the best of the authors’ knowledge, it is still in its infant development stage; the authors believe that it will become one of the most promising directions in the control area in robotic manipulators. Learning control in robotic manipulators is mainly used to address the issue that the friction at the joints of robotic mechanisms and other uncertainties may exist in the dynamic models, which are very complex and may even be impossible to model mathematically. In this paper, the authors review and discuss the learning control in robotic manipulators and some issues in learning control for robotic manipulators are also illustrated. This review is able to give a general guideline for future research in learning control for robotic manipulators.

Model-based reinforcement learning methods provide a promising direction for a range of automated applications, such as autonomous vehicles and legged robots, due to their sample-efficiency. However, their asymptotic performance is usually inferior compared to the state-of-the-art model-free reinforcement learning methods in locomotion control domains. One main challenge of model-based reinforcement learning is learning a dynamics model that is accurate enough for planning. This paper mitigates this issue by meta-reinforcement learning from an ensemble of dynamics models. A policy learns from dynamics models that hold different beliefs of a real environment. This procedure improves its adaptability and inaccuracy-tolerance ability. A proximal meta-reinforcement learning algorithm is introduced to improve computational efficiency and reduces variance of higher-order gradient estimation. A heteroscedastic noise is added to the training dataset, thus leading to a robust and efficient model learning. Subsequently, proximal meta-reinforcement learning maximizes the expected returns by sampling “imaginary” trajectories from the learned dynamics, which does not require real environment data and can be deployed on many servers in parallel to speed up the whole learning process. The aim of this work is to reduce the sample-complexity and computational cost of reinforcement learning in robot locomotion tasks. Simulation experiments show that the proposed algorithm achieves an asymptotic performance compared with the state-of-the-art model-free reinforcement learning methods with significantly fewer samples, which confirm our theoretical results.

Reinforcement Learning (RL) has become a trending research topic with great success in outperforming humans on many tasks including video games, board games, and robotics control. By leveraging Deep Learning (DL), RL algorithms can consume a large volume of data without any prior knowledge of the system dynamics. However, requiring a large amount of data also limits the applicability in many fields where data is costly to obtain. Model-based Reinforcement Learning (MBRL) is regarded as a promising way to achieve high data efficiency while maintaining comparable performance. MBRL equips a dynamic transition model to facilitate and speed up the policy searching by learning the system dynamics. But there are no satisfying open-sourced libraries for the RL community to conduct MBRL research. Therefore, to fill the gap, we propose an open-sourced, flexible, and user-friendly MBRL library, Baconian, to facilitate the research on MBRL.

The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as ‘perseveration’ errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides 2 general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior but they also provide a useful framework for understanding neural reward valuation and action selection.

Prioritized sweeping (PS) and its variants are model-based reinforcement-learning algorithms that have demonstrated superior performance in terms of computational and experience e‐ciency in practice. This note establishes the flrst|to the best of our knowledge|formal proof of convergence to the optimal value function when they are used as planning algorithms. We also describe applications of this result to provably e‐cient model-based reinforcement learning in the PAC-MDP framework. We do not address the issue of convergence rate in the present paper.

Arbitration between model-based (MB) and model-free (MF) reinforcement learning (RL) is key feature of human reinforcement learning. The computational model of arbitration control has been demonstrated to outperform conventional reinforcement learning algorithm, in terms of not only behavioral data but also neural signals. However, this arbitration process does not take full account of contextual changes in environment during learning. By incorporating a Dirichlet process Gaussian mixture model into the arbitration process, we propose a meta-controller for RL that quickly adapts to contextual changes of environment. The proposed model performs better than a conventional model-free RL, model-based RL, and arbitration model.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

Reinforcement learning (RL) is a simulation-based technique to solve Markov decision problems or processes (MDPs). It is especially useful if the transition probabilities in the MDP are hard to find or if the number of states in the problem is too large. In this paper, we present a new model-based RL algorithm that builds the transition probability model without the generation of the transition probabilities; the literature on model-based RL attempts to compute the transition probabilities. We also present a variance-penalized Bellman equation and an RL algorithm that uses it to solve a variance-penalized MDP. We conclude with some numerical experiments with these algorithms.

Recent evidence in neuroscience and psychology suggests that a single reinforcement learning (RL) algorithm only accounts for less than 60% of the variance of human choice behavior in an uncertain and dynamic environment, where the amount of uncertainty in state-action-state transitions drift over time. The prediction performance further decreases when the size of the state space increases. We proposed a hierarchical context-dependent RL control framework that dynamically exerted control weights on model-based (MB) and multiple model-free (MF) RL strategies associated with different task goals. To properly assess the validity of the proposed method, we considered a two-stage Markov decision task (MDT) in which the three different types of context changed over time. We trained 57 different RL control models on a Caltech MDT data set; then, we assessed their prediction performance using a Bayesian model comparison. This large-scale computer simulation analysis revealed that the model providing the most accurate prediction was the version that implemented the competition between the MB and multiple goal-dependent MF RL strategies. The present study demonstrates the applicability of the goal-driven RL control to a variety of real-world human-robot interaction scenarios.

A discrete-time learning control for robotic manipulators is studied using its pulse transfer function. Firstly, the discrete-time learning stability condition which is applicable to single-input two-outputs systems is derived. Secondly, the stability of the learning algorithm with a position signal is studied. In this case, when the sampling period is small, the algorithm is not stable because of an unstable zero of the system. Thirdly, the stability of the algorithm with position and velocity signals is studied. In this case, we can stabilize the learning control system which is unstable in the case of learning with only the position signal. Finally, simulation results on the trajectory control of robotic manipulators using the discrete-time learning control are shown. The simulation results agree well with the analytical results.

Triboelectric nanogenerators and sensors can be applied as human-machine interfaces to the next generation of intelligent and interactive products, where flexible tactile sensors exhibit great advantages for diversified applications such as robotic control. In this paper, we present a self-powered, flexible, triboelectric sensor (SFTS) patch for finger trajectory sensing and further apply the collected information for robotic control. This innovative sensor consists of flexible and environmentally friendly materials, i. e., starch-based hydrogel, polydimethylsiloxane (PDMS), and silicone rubber. The sensor patch can be divided into a two-dimensional (2D) SFTS for in-plane robotic movement control and a one-dimensional (1D) SFTS for out-of-plane robotic movement control. The 2D-SFTS is designed with a grid structure on top of the sensing surface to track the continuous sliding information on the fingertip, e. g., trajectory, velocity, and acceleration, with four circumjacent starch-based hydrogel PDMS elastomer electrodes. Combining the 2D-SFTS with the 1D-SFTS, three-dimensional (3D) spatial information can be generated and applied to control the 3D motion of a robotic manipulator, and the real-time demonstration is successfully realized. With the facile design and very low-cost materials, the proposed SFTS shows great potential for applications in robotics control, touch screens, and electronic skins.

Reinforcement learning (RL) for robotics is challenging due to the difficulty in hand-engineering a dense cost function, which can lead to unintended behavior, and dynamical uncertainty, which makes exploration and constraint satisfaction challenging. We address these issues with a new model-based reinforcement learning algorithm, Safety Augmented Value Estimation from Demonstrations (SAVED), which uses supervision that only identifies task completion and a modest set of suboptimal demonstrations to constrain exploration and learn efficiently while handling complex constraints. We then compare SAVED with 3 state-of-the-art model-based and model-free RL algorithms on 6 standard simulation benchmarks involving navigation and manipulation and a physical knot-tying task on the da Vinci surgical robot. Results suggest that SAVED outperforms prior methods in terms of success rate, constraint satisfaction, and sample efficiency, making it feasible to safely learn a control policy directly on a real robot in less than an hour. For tasks on the robot, baselines succeed less than 5% of the time while SAVED has a success rate of over 75% in the first 50 training iterations. Code and supplementary material is available at https://tinyurl.com/saved-rl.

Although providing exceptional asymptotic performance, trial-and-error based reinforcement learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. In this paper, we propose a data-efficient model-based reinforcement learning algorithm based on the Koopman operator theory. By representing the environment dynamics as a linear dynamical system in a high-dimensional space, the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning. However, when applying the theory for reinforcement learning, with the sparse and unevenly distributed trial data, it is difficult to learn globally linear representations thus leading to serious model bias. To overcome this problem, we devise a local Koopman operator approach that is tailored for the setup of reinforcement learning. By coupling with deep neural networks and the linear quadratic regulator control, we propose the first Koopman operator model-based reinforcement learning algorithm called deep Koopman reinforcement learning (DKRL). The simulation experiment results show the proposed method can outperform existing approaches and can learn robotics control tasks with a small amount of data.