Integration of Adaptive Control and Reinforcement Learning for Real-Time Control and Learning

In the past decade, there has been considerable research interest in learning control methods based on reinforcement learning (RL) and approximate dynamic programming (ADP). As an important class of function approximation techniques, kernel methods have been recently applied to improve the generalization ability of RL and ADP methods but most previous works were only based on simulation. This paper focuses on experimental studies of real-time online learning control for nonlinear systems using kernel-based ADP methods. Specifically, the kernel-based dual heuristic programming (KDHP) method is applied and tested on real-time control systems. Two kernel-based online learning control schemes are presented for uncertain nonlinear systems by using simulation data and online sampling data, respectively. Learning control experiments were performed on a single-link inverted pendulum system as well as a double-link inverted pendulum system. From the experimental results, it is shown that both online learning control schemes, either using simulation data or using real sampling data, are effective for approximating near-optimal control policies of nonlinear dynamical systems with model uncertainties. In addition, it is demonstrated that KDHP can achieve better performance than conventional DHP, which uses multilayer perceptron neural networks.

A neural network adaptive inverse controller structure is proposed to solve the problems that hypersonic aircraft with highly nonlinear and flight conditions of large span. OS-ELM algorithm is improved with RBF network for online learning and adaptive inverse controller for the on line training is used. Hypersonic aircraft with six degrees of freedom non-linear model is used to verify the performance of adaptive inverse controller. The controller based on flight conditions doing online learning in real time is shown with flight simulation in a typical condition. Aircraft orders tracking is achieved rapidly and steadyly and desired control effect is achieved.

Cognitive theorists believe that students should be active in organizing their learning. Thus learner control may assist learning directly and promote good strategies.A teaching programme is described that offers to the student control over content, style and level of difficulty. The material was delivered by PET microcomputer and the subject was binary arithmetic. Four groups of young secondary schoolchildren were exposed to different treatments of the same material.The treatments were: learner control; learner control with advice; random program control; adaptive program control. The effect of advice on pupil's choices was clearly evident. The random group performed less well than the other groups but no differences were observed between learner and adaptive control.

This paper proposes an adaptive learning control scheme to solve high-precision velocity tracking problem for tank gun control servo systems. Lyapunov approach is used to design the learning controller, with alignment condition used to cope with initial problem of iterative learning control. Robust control technique and adaptive learning control technique are synthesized to handle nonlinear uncertainties and external disturbances. The unknown parameters are estimated according to the full saturation difference learning strategy. As the iteration number increases, the system state can accurately track the reference signal over the whole time interval, and all signal are guaranteed to be bounded.

ABSTRACTA new type of saturated position (Proportional) and velocity (Differential) feedback (SPD servo-loop) induced by a quasi-natural potential is proposed for control of robot manipulators with all revolute-type joints. If the servo loop is composed of such SP-D feedback, then not only the original robot dynamics but also the residual error dynamics satisfy the passivity with appropriate choice of feedback gains. By virtue of the passivity, it is possible to show the existence of Lyapunov functions in a global sense for a variety of set-point, adaptive trajectory tracking, and learning controls. Proofs of convergence of tracking errors in model-based adaptive control and learning control are given on the basis of construction of such Lyapunov functions.

Adaptive iterative learning control is a popular approach to control parametric time-varying systems over finite time interval. However, initial resetting condition limits the application of a large amount of learning control algorithms. In this paper, we propose a suboptimal iterative learning control for a class of dynamic parametric systems to solve this problem. An error-tracking control strategy is used to deal with initial position problem, with suboptimal strategy used to accelerate convergence speed. As the iteration increases, the tracking error of the closed-loop system accurately follows the desired error trajectory over the entire time interval, and the system state tracks the reference signal perfectly on the pre-specified interval. The performance of the proposed suboptimal error-tracking learning control scheme is illustrated by an example involving both suboptimal errortracking algorithm and normal error-tracking algorithm.

In this work, path tracking control of autonomous vehicles (AVs) are studied under the framework of adaptive iterative learning control (AILC). In order to facilitate the controller design, the nonlinear vehicle dynamics are firstly transformed into a parametric form by utilizing proper variable transformations. Then a robust adaptive learning control approach is developed to achieve the path tracking tasks. Furthermore, to deal with the actuator constraints, an input-dependent auxiliary system is employed to work together with the proposed learning controller, which thus helps to reduce their influence to the control performance. The convergence analysis for the proposed method is also provided by virtue of the Lyapunov-like theory. It has been shown that with the proposed AILC approach, a good path tracking performance can be ensured in the presence of system uncertainties, exteranl disturbances and input constraints. A numerical example is illustrated to verify its effectiveness.

Learning the peg-into-hole assembly operation with a connectionist reinforcement technique

Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

In this paper, an adaptive iterative learning control (AILC) method combined with sliding mode technique is proposed to improve the force control performance for repeating tasks of fluidic muscle (FM) driven parallel manipulators. Different from the traditional iterative learning control method, the proposed AILC is to learn the controller time-varying parameters rather than to learn the control signals. Since the AILC is sensitive to non-repetitive disturbances, the sliding mode technique is introduced to enhance the robustness. Since no model information involved in the controller design, the proposed method is a complete data-driven method. Hence, the difficulty of obtaining accurate model is avoided. Simulation studies are performed on a two degrees of freedom FM driven parallel manipulator. Simulation results demonstrate that the proposed method can achieve high force tracking performance and robustness.

Advanced robotic prostheses are expensive considering the cost of human resources and the time spent on manually tuning the high-dimensional control parameters for individual users. To alleviate clinicians’ effort and promote the advanced robotic prosthesis, we implemented an optimal adaptive control algorithm, which fundamentally is a type of reinforcement learning method, to automatically tune the high-dimensional control parameters of a robotic knee prosthesis through interaction with a human-prosthesis system. The ‘human-in-the-loop’ term means that the learning controller tunes the control parameters based on the performance of the robotic knee prosthesis while an amputee subject walking with it. We validated the human-in-the-loop auto-tuner with one transfemoral amputee subject for 4 hour-long lab testing sessions. Our results demonstrated that this novel reinforcement learning controller was able to learn through interaction with the human-prosthesis system and discover a set of suitable control parameter for the amputee user to generate near-normative knee kinematics.

A practical reinforcement learning control design for nonlinear systems with input and output constraints

In this paper, we address the autonomous control of a 6-legged robot using reinforcement learning, and apply it to rubbles. In conventional framework of reinforcement learning, many degrees of freedom of the 6-legged robot cause the state explosion problem, and it prevents the robot from real-time learning. Furthermore, even if learning process is conducted in simulation, the acquired policy is effective only for the simple simulated environment, and therefore, it is impossible to apply the acquired policy to unknown complex environment such as rubbles. To solve these problems, we focus on the flexibility of the body and its body image. The body image solves the state explosion problem and the flexibility of the body compensates the difference between the simple simulated environment and the complex real environment. We developed a prototype of the robot and conducted experiments to confirm the effectiveness and the validity of the proposed framework. As a result, effective locomotion was realised.

A new paradigm for learning control of a generic nonlinear system is presented. The method is based on a special radial basis function network architecture and is adaptive as the input/output properties of the system are estimated and the learning control gain is computed online. The controlled system is described with a mapping between the feedforward control and an array of sampled output values over a given time interval. The network approximates the system input/output mapping as a function of the task parameter vector comprising the initial and a desired final system state. The approximation is affine and linear in the control input. For given network weights, this allows inverting the network approximation and an optimal control program. The network weights are updated depending on the results of the learning trials. The application of the algorithm to the control of an arbitrary planar motion of a two link arm demonstrates fast learning and the high accuracy of the method. >

As robots become more capable in terms of hardware, and more complex tasks are considered, optimality starts playing a more important role in the design of algorithms implemented in these systems. Optimality is a guiding principle that directs the computation of feasible and efficient solutions to different robotics tasks. In control theory, this principle is implemented online as a set of efficient numerical optimization algorithms, that in addition to solving the task, purports to save a suitably defined effort or energy term. This thesis investigates trajectory generation, learning and control for dynamic tasks from the unifying point of view of optimization. As an application, we focus on Table Tennis, a chal- lenging task where robots are yet to outperform humans. We believe that the required dexterity and accuracy for this dynamical task hinges on the developments in online optimization and efficient learning algorithms. We consider trajectory generation for table tennis in the first part of the thesis. In highly dynamic tasks like table tennis that involve moving targets, planning is necessary to figure out when, where and how to intercept the target. Motion planning can be very challenging in robotic table tennis in particular, due to time constraints, dimension of the search space and joint limits. Conventional planning algorithms often rely on a fixed virtual hitting plane to construct robot striking trajectories. These algorithms, however, generate restrictive strokes and can result in unnatural strategies when compared with human playing. In this thesis, we introduce a new trajectory generation framework for robotic table tennis that does not involve a fixed hitting plane. A free-time optimal control approach is used to derive two different trajectory optimizers. The resulting two algorithms, Focused Player and Defensive Player, encode two different play- styles. We evaluate their performance in simulation and in our robot table tennis platform with a high speed cable-driven seven DOF robot arm. The algorithms return the balls with a higher probability to the opponent’s court when compared with a virtual hitting plane based method. Moreover, both can be run online and the trajectories can be corrected with new ball observations. In the second part of the thesis, we look at how such trajectories, computed on the kine- matics level, can be tracked accurately with learning control based approaches. Highly dynamic tasks like table tennis require large accelerations and precise tracking for successful perfor- mance. To track desired trajectories well, such tasks usually rely on accurate models and/or high gain feedback. While kinematic optimization allows for efficient representation and online generation of hitting trajectories, learning to track such dynamic movements with inaccurate models remains an open problem. In particular, stability issues surrounding the learning per- formance, in the iteration domain, can prevent the successful implementation of model based learning approaches. To achieve accurate tracking for these tasks in a stable and efficient way, we propose a new adaptive Iterative Learning Control algorithm that is implemented efficiently using a recursive approach. Moreover, covariance estimates of model matrices are used to ex- ercise caution during learning. We evaluate the performance of the proposed approach in our robotic table tennis platform, where we show how the performance of two Barrett WAMs can be optimized. Our implementation on the table tennis platform compares favorably with two state-of-the-art approaches. Finally, we discuss an alternative learning from demonstrations approach, where we learn sparse representations from demonstrated movements. Learning from demonstrations is an easy and intuitive way to show examples of successful behavior to a robot. However, the fact that humans optimize or take advantage of their body and not of the robot, usually called the embodiment problem in robotics, often prevents industrial robots from executing the task in a straightforward way. The shown movements often do not or cannot utilize the degrees of freedom of the robot efficiently, and typically suffer from excessive execution errors. In the last chapter, we show a new approach that can alleviate some of these difficulties by learning sparse representations of movement. Moreover, the number of learned parameters are independent of the degrees of freedom of the robot. Sparsity is a desirable feature for policy search Rein- forcement Learning algorithms that adapt the parameters of these movement primitives. By ranking the learned parameters on the Elastic Net path in terms of importance, we note that our approach could be potentially useful to combat the curse of dimensionality in robot learning applications. We show preliminary results on the real robot setup, including a successful table tennis serve using our new movement primitive representation. Throughout the thesis, we present and analyze in detail new control and learning algo- rithms. Efficient online optimization approaches are presented that can be used to solve not just table tennis problems, but they can be adapted to solve different dynamic tasks.