Robotic Knee Prosthesis Real-Time Control Using Reinforcement Learning with Human in the Loop

Robotic prostheses provide new opportunities to better restore lost functions than passive prostheses for trans-femoral amputees. But controlling a prosthesis device automatically for individual users in different task environments is an unsolved problem. Reinforcement learning (RL) is a naturally promising tool. For prosthesis control with a user in the loop, it is desirable that the controlled prosthesis can adapt to different task environments as quickly and smoothly as possible. However, most RL agents learn or relearn from scratch when the environment changes. To address this issue, we propose the knowledge-guided Q-learning (KG-QL) control method as a principled way for the problem. In this report, we collected and used data from two able-bodied (AB) subjects wearing a RL controlled robotic prosthetic limb walking on level ground. Our ultimate goal is to build an efficient RL controller with reduced time and data requirements and transfer knowledge from AB subjects to amputee subjects. Toward this goal, we demonstrate its feasibility by employing OpenSim, a well-established human locomotion simulator. Our results show the OpenSim simulated amputee subject improved control tuning performance over learning from scratch by utilizing knowledge transfer from AB subjects. Also in this paper, we will explore the possibility of information transfer from AB subjects to help tuning for the amputee subjects.

This study aims to demonstrate reinforcement learning tracking control for automatically configuring the impedance parameters of a robotic knee prosthesis. While our previous studies involving human subjects have focused on tuning the impedance control parameters to meet a fixed, subjectively prescribed target motion profile to enable continuous walking with human-in-the-loop, in this paper we develop a new tracking control solution for a robotic knee to mimic the motion of the intact knee. As such, we replaced the prescribed target knee motion by an automatically generated profile based on the intact knee. As the profile of the intact knee varies over time due to human adaptation, we are presented with a challenging tracking control problem in the context of classical control theory. By formulating the “echo control” of the robotic knee as a reinforcement learning problem, we provide a promising new tool for real-time tracking control design without explicitly representing the underlying dynamics using a mathematical model, which can be difficult to obtain for a human-robot system. Additionally, our results may inspire future studies and new robotic prosthesis impedance control designs that can potentially coordinate between the intact and the robotic limbs toward daily use of the robotic device.

Various linear and nonlinear controllers have been developed to improve the dynamic performance of DC-DC converters. Most controllers can only be designed on the basis of understanding the mathematical model of DC-DC converter, but the inherent nonlinear and time-varying characteristics of DC-DC switching converter make it difficult to complete the precise modeling, so the model-based control design is complex and the control performance is limited. In order to overcome the problem, this paper proposes a reinforcement learning (RL) controller based on the twin-delayed deep deterministic policy gradient (TD3) algorithm. This controller does not need the model of the switching converter. The converter will be regarded as a black box model, the policy approximation function (policy neural network) can be trained and learned by constructing a Markov decision process interacting with the black box model in the control system, and the optimal control action can be output. The RL controller is developed based on actor critic architecture, and a TD3 algorithm with higher learning efficiency is proposed to improve the control performance of the RL controller. The proposed RL controller based on TD3 algorithm is compared with the traditional PI controller. The simulation results show that the RL controller has better dynamic performance when the converter starts and the load step changes.

Reinforcement learning (RL) has attracted great interest from researchers in recent years. RL performs as human or better in many fields such as games and robot control. Although this technology is booming in computer science, it has not been practically applied in industrial process control. Up to now, proportional–integral–derivative (PID) control is still the most dominating and popular control method in industrial control. In this paper, we propose a combination of deep reinforcement learning (DRL) and PID control for better process control performance. The idea is generated by the following observations: for PID controller, its transient performance is not usually well enough to meet a strict requirement or in complex signal tracking tasks; For RL technology, a perfectly designed reward function is required for training. However, in practice, the reward function needs to be tested through trial and error, which will lead to a waste of computational power and time. By combining these two strategies, PID controller can help to improve the steady-state performance of RL control by its integral term, while the trained RL agent is able to improve the transient performance of PID controller. Several case studies with the water tank system are presented to demonstrate the effectiveness of the combined PID + RL control strategy.

This paper considers the effectiveness of neural-network-based reinforcement learning (RLNN) controller in discrete time for the underactuated ships. The path following problem of underactuated ships is solved by utilizing RLNN controller together with the Line-of-Sight (LOS) guidance law. The proper form of variables in RLNN controller compatible with the variables in ship maneuvering model is derived and the calculation of desired yaw angle in LOS is implemented in a concise way. The proposed controller is model independent with the consideration of output constraints. Our approach is evaluated by simulations on different ship models with different way-point generated paths.

Micro aerial vehicles are widely being researched and employed due to their relative low operation costs and high flexibility in various applications. We study the under-actuated quadrotor perching problem, designing a trajectory planner and controller which generates feasible trajectories and drives quadrotors to desired state in state space. This paper proposes a trajectory generating and tracking method for quadrotor perching that takes the advantages of reinforcement learning controller and traditional controller. We demonstrate the performance of the trained reinforcement learning controller generated trajectory information and manipulated quadrotor toward the perching point (manually throwing it up in the air with an initial velocity of 1 m/s). We show that this approach permits the control structure of trajectories and controllers enabling such aggressive maneuvers perching on vertical surfaces with relatively accurate. Computation time of evaluating the policy is only 0.03 sec per trajectory, which is two orders of magnitude less than common trajectory optimization algorithms with an approximated model.

Research on the costs of delivering cancer chemotherapy has typically focused on drug administration, but assessing the overall costs must also consider the human resources necessary. Furthermore, supportive care, such as using colonystimulating factors to manage chemotherapy-induced neutropenia, can greatly affect the costs of care. This study evaluated the human resource costs associated with delivering chemotherapy and managing chemotherapy-induced neutropenia in the community oncology setting. A model of medical visits associated with 21-day chemotherapy regimens and common approaches to managing neutropenia was defined. Four hundred medical professionals at 20 community oncology practices in the United States completed a semistructured interview to provide data on practice patterns and the time and costs of performing medical tasks. The results indicate that chemotherapy- and neutropenia-related medical visits require substantial human resource time and costs. The human resource cost of a single treatment with pegfilgrastim or filgrastim was equivalent, but, because of the greater number of visits required with filgrastim, the total time and costs with pegfilgrastim (2.4 hours and $57.30) were less than those with filgrastim (14.8 hours and $364.66). This study provides important data on medical visits that can be used in future analyses to model the implications of chemotherapy and supportive care for anemia and neutropenia. Implementing strategies to lessen the number of visits can reduce human resource time and costs and help focus these resources on other treatment-related activities.

Autonomous digital twin framework for gas turbine combined cycle control loops: Comparative study of proportional-integral control, reinforcement learning, and reinforcement learning with agents

A practical reinforcement learning control design for nonlinear systems with input and output constraints

Reinforcement learning (RL) can provide a basic framework for autonomous robots to learn to control and maximize future cumulative rewards in complex environments. To achieve high performance, RL controllers must consider the complex external dynamics for movements and task (reward function) and optimize control commands. For example, a robot playing tennis and squash needs to cope with the different dynamics of a tennis or squash racket and such dynamic environmental factors as the wind. In addition, this robot has to tailor its tactics simultaneously under the rules of either game. This double complexity of the external dynamics and reward function sometimes becomes more complex when both the multiple dynamics and multiple reward functions switch implicitly, as in the situation of a real (multi-agent) game of tennis where one player cannot observe the intention of her opponents or her partner. The robot must consider its opponent's and its partner's unobservable behavioral goals (reward function). In this article, we address how an RL agent should be designed to handle such double complexity of dynamics and reward. We have previously proposed modular selection and identification for control (MOSAIC) to cope with nonstationary dynamics where appropriate controllers are selected and learned among many candidates based on the error of its paired dynamics predictor: the forward model. Here we extend this framework for RL and propose MOSAIC-MR architecture. It resembles MOSAIC in spirit and selects and learns an appropriate RL controller based on the RL controller's TD error using the errors of the dynamics (the forward model) and the reward predictors. Furthermore, unlike other MOSAIC variants for RL, RL controllers are not a priori paired with the fixed predictors of dynamics and rewards. The simulation results demonstrate that MOSAIC-MR outperforms other counterparts because of this flexible association ability among RL controllers, forward models, and reward predictors.

Although model-free reinforcement learning methods are illustrated to own abilities to learn a wide range of control and decision-making tasks, it is time-consuming and needs a huge number of samples to learn optimal policy. Therefore, model-based reinforcement learning algorithms are always treated as the most efficient ways to settle robotic control problems. In this work, a model-based reinforcement learning tracking control method with free energy is proposed. The control objective is to make sure that the proposed controller can let a robotic manipulator with parameters uncertainties and disturbance follow the desired trajectory and obtain good tracking performance. The main contribution of this work is to employ the free energy principle to optimize the problem of robotic control. The probability distribution of robotic dynamic model is learnt by a evolving neural network, a free energy based reinforcement learning control method uses the learnt model to find optimal control policy. The experiment results demonstrate that the proposed tracking control method is effective in solving robotic tracking control problems.

This paper presents the analyses of a deep reinforcement learning, DRL approach for maximum power point tracking (MPPT) of a photovoltaic PV boost converter system. The DRL controller used is the Deep Deterministic Policy Gradient (DDPG) algorithm, which is designed to address the limitations of conventional perturb and observe, P&O methods with PID control, such as reduced efficiency under non – uniform irradiance or load variations. The DDPG framework offers an adaptive and computationally efficient alternative by learning the nonlinear mapping between PV states and duty cycle control. Thus, a comparative study of the DDPG with the conventional approach is performed under non – uniform irradiance and varying load resistance. Parameters of power and voltage were evaluated as well as the oscillatory performance for both controllers. The outcome of the results shows that the DDPG controller can achieve higher power extraction with significantly reduced under higher load impedances The conventional approach, however, achieves a better performance under uniform irradiation and low resistance conditions. This is due to the absence of multiple peaks that allows the conventional approach to rapidly converge to the MPP, while the exploratory nature DDPG introduces minor bias due to its exploratory learning process. These findings highlight the potential of DDPG to enhance the efficiency and adaptability of MPPT, particularly relevant in standalone DC microgrid applications.

The complexity in planning and control of robot compliance tasks mainly results from simultaneous control of both position and force and inevitable contact with environments. It is quite difficult to achieve accurate modeling of the interaction between the robot and the environment during contact. In addition, the interaction with the environment varies even for compliance tasks of the same kind. To deal with these phenomena, in this paper, we propose a reinforcement learning and robust control scheme for robot compliance tasks. A reinforcement learning mechanism is used to tackle variations among compliance tasks of the same kind. A robust compliance controller that guarantees system stability in the presence of modeling uncertainties and external disturbances is used to execute control commands sent from the reinforcement learning mechanism. Simulations based on deburring compliance tasks demonstrate the effectiveness of the proposed scheme.

Recent evidence in neuroscience and psychology suggests that a single reinforcement learning (RL) algorithm only accounts for less than 60% of the variance of human choice behavior in an uncertain and dynamic environment, where the amount of uncertainty in state-action-state transitions drift over time. The prediction performance further decreases when the size of the state space increases. We proposed a hierarchical context-dependent RL control framework that dynamically exerted control weights on model-based (MB) and multiple model-free (MF) RL strategies associated with different task goals. To properly assess the validity of the proposed method, we considered a two-stage Markov decision task (MDT) in which the three different types of context changed over time. We trained 57 different RL control models on a Caltech MDT data set; then, we assessed their prediction performance using a Bayesian model comparison. This large-scale computer simulation analysis revealed that the model providing the most accurate prediction was the version that implemented the competition between the MB and multiple goal-dependent MF RL strategies. The present study demonstrates the applicability of the goal-driven RL control to a variety of real-world human-robot interaction scenarios.

Autonomous navigation at unsignalized intersections: A coupled reinforcement learning and model predictive control approach