Autonomous Learning of Internal Dynamic Models for Reaching Tasks

Dynamic movement primitives (DMPs) are a learning from demonstration (LfD) method that imitates movement primitives. Recently, reinforcement learning (RL) has been used to modify DMPs. In this regard, most methods have concentrated on improving DMPs with RL after DMPs are learned with supervised learning. In this way, there is no guarantee that the DMPs stay close to demonstrations while modified. To address this problem, in this study, a framework is proposed for learning DMPs with deep RL, which can be further used to modify DMPs. To do so, first, DMPs are modeled in the framework of deep RL. Then, after the reward function for learning is defined, they are learned with deep RL. The deep RL algorithm used in the present study is the twin-delayed deep deterministic policy gradient (TD3) algorithm. Our proposed algorithm is evaluated with various demonstrations, and a case study of a pick-and-place task is done. The results show that our method can learn all demonstrations with high accuracy.

In this article, we present a general internal model (GIM) approach for motion skill learning at the elementary level and coordination level. In the past, internal models with two different configurations are used to describe the two classes of dynamic movement primitives (DMPs): discrete and rhythmic movement. In this work, we developed a unified internal model which can describe both classes of DMPs. In particular, a discrete movement can be modeled as a fraction of a rhythmic movement. The general internal model retains the temporal and spatial scalabilities which are defined as the ability to generate similar movement patterns directly by means of tuning some parameters of the internal model. The advantage of scalability lies in that the learning or training process can be avoided while dealing with similar tasks. Complex motions require movement coordinations, hence coordination of multiple internal models. In the general internal model approach, the coordination is implemented with appropriate phase shifts among multiple internal models. Further in the GIM, the phase shift can be achieved by means of adjusting the initial state values of internal models. Through two illustrative examples, we show that the human behavior patterns with single or multiple limbs can be easily learned and established by the GIM at elementary and coordination levels.

Movement primitives are a well-established approach to robot motion planning, as they offer a modular basis for robot motion planning. Dynamic movement primitives (DMPs) are a popular control framework based on nonlinear differential equations which, when scaled in time, produce a smooth kinematic movement plan. In this letter, we introduce an adjusted movement primitive framework which replaces the stable equilibria of DMPs with saddle points. These saddle points are linked together in stable heteroclinic channels (SHCs) which are in turn part of dynamical systems called stable heteroclinic networks. SHC-based movement primitives (SMPs) form a framework where the weights of the kernel functions have spatial significance in the task space. In this letter, we show that SMPs and DMPs perform comparably according to a kinematic-based cost function. We also show that SMP kernel weights follow a given trajectory when plotted in the task space. The plotted kernel weights provide an intuitive tool for managing the controller.

In this paper, composite dynamic movement primitives (DMPs) based on radial basis function neural networks (RBFNNs) are investigated for robots’ skill learning from human demonstrations. The composite DMPs could encode the position and orientation manipulation skills simultaneously for human-to-robot skills transfer. As the robot manipulator is expected to perform tasks in unstructured and uncertain environments, it requires the manipulator to own the adaptive ability to adjust its behaviours to new situations and environments. Since the DMPs can adapt to uncertainties and perturbation, and spatial and temporal scaling, it has been successfully employed for various tasks, such as trajectory planning and obstacle avoidance. However, the existing skill model mainly focuses on position or orientation modelling separately; it is a common constraint in terms of position and orientation simultaneously in practice. Besides, the generalisation of the skill learning model based on DMPs is still hard to deal with dynamic tasks, e.g., reaching a moving target and obstacle avoidance. In this paper, we proposed a composite DMPs-based framework representing position and orientation simultaneously for robot skill acquisition and the neural networks technique is used to train the skill model. The effectiveness of the proposed approach is validated by simulation and experiments.

Recent advances in bioinspired robotics highlight the growing demand for dexterous, adaptive control strategies that allow robots to interact naturally, safely, and efficiently with dynamic, contact-rich environments. Yet, achieving robust adaptability and reflex-like responsiveness to unpredictable disturbances remains a fundamental challenge. This paper presents a bioinspired imitation learning framework that models human adaptive dynamics to jointly acquire and generalize motion and force skills, enabling compliant and resilient robot behavior. The proposed framework integrates hybrid force–motion learning with dynamic response mechanisms, achieving broad skill generalization without reliance on external sensing modalities. A momentum-based force observer is combined with dynamic movement primitives (DMPs) to enable accurate force estimation and smooth motion coordination, while a broad learning system (BLS) refines the DMP forcing function through style modulation, feature augmentation, and adaptive weight tuning. In addition, an adaptive radial basis function neural network (RBFNN) controller dynamically adjusts control parameters to ensure precise, low-latency skill reproduction, and safe physical interaction. Simulations and real-world experiments confirm that the proposed framework achieves human-like adaptability, robustness, and scalability, attaining a competitive learning time of 5.56 s and a rapid generation time of 0.036 s, thereby demonstrating its efficiency and practicality for real-time applications and offering a lightweight yet powerful solution for bioinspired intelligent control in complex and unstructured environments.

Kinesthetic teaching and Dynamic Movement Primitives (DMPs) enable fast and adaptable learning of robot tasks based on a human demonstration. A task encoded as a dynamic movement primitive can be reused with a different goal position, albeit with a resulting distortion in the approach trajectory with regards to the original task. While this is sufficient for some robotic applications, the accuracy requirements for assembly tasks in an industrial context, where tolerances are tight and workpieces are small, is much higher. In such a context it is also preferable to keep the number of demonstrations and of external sensors low. Our approach relies on a single demonstration and a single force-torque sensor at the robot tool. We make use of a Convolutional Neural Network (CNN) trained on the force-torque sensor data to segment the task into several movement primitives for the different phases: pickup - approach - insertion - retraction, allowing us to achieve better positional accuracy when generalizing the task primitives to new targets. To the best of our awareness, we are the first to utilize a CNN as a segmentation tool to improve the generalization performance of DMPs.

Movement generation, and especially generalization to unseen situations, plays an important role in robotics. Different types of movement generation methods exist such as spline based methods, dynamical system based methods, and methods based on Gaussian mixture models (GMMs). Using a large, new dataset on human manipulations, in this paper we provide a highly detailed comparison of five fundamentally different and widely used movement encoding and generation frameworks: dynamic movement primitives (DMPs), time based Gaussian mixture regression (tbGMR), stable estimator of dynamical systems (SEDS), Probabilistic Movement Primitives (ProMP) and Optimal Control Primitives (OCP). We compare these frameworks with respect to their movement encoding efficiency, reconstruction accuracy, and movement generalization capabilities. The new dataset consists of nine object manipulation actions performed by 12 humans: pick and place, put on top/take down, put inside/take out, hide/uncover, and push/pull with a total of 7,652 movement examples. Our analysis shows that for movement encoding and reconstruction DMPs and OCPs are the most efficient with respect to the number of parameters and reconstruction accuracy, if a sufficient number of kernels is used. In case of movement generalization to new start- and end-point situations, DMPs, OCPs and task parameterized GMM (TP-GMM, movement generalization framework based on tbGMR) lead to similar performance, which ProMPs only achieve when using many demonstrations for learning. All models outperform SEDS, which additionally proves to be difficult to fit. Furthermore we observe that TP-GMM and SEDS suffer from problems reaching the end-points of generalizations. These different quantitative results will help selecting the most appropriate models and designing trajectory representations in an improved task-dependent way in future robotic applications. • Comparison of five motion encoding frameworks on human manipulation action dataset. • Large dataset containing 7652 movement trajectories from 9 different action classes. • Comparison regarding encoding and generalization reveals model specific differences.

Addressing the limitations of traditional Dynamic Movement Primitives (DMP) obstacle avoidance methods-which rely on dynamic coupling terms leading to complex algorithmic structures, sluggish responses, and difficulties in autonomous obstacle avoidance-this study innovatively proposes a robotic arm obstacle avoidance trajectory planning framework that integrates DMP with the Informed RRT* algorithm. This framework uses the Informed RRT algorithm to do local trajectory replanning in areas near obstacles. It keeps the main dynamic features of DMP and creates obstacle-avoidance trajectories that are then smoothly added to the original DMP trajectory. This approach eliminates the need for complex dynamic coupling terms, significantly reducing algorithmic complexity. Its local replanning mechanism not only enables autonomous obstacle avoidance and effectively suppresses trajectory oscillations but also substantially enhances real-time response speed. The main contributions of this study include 1) proposing a decoupled DMP obstacle avoidance method based on local trajectory replanning combined with Informed RRT*, simplifying the algorithm structure; 2) the adopted local replanning strategy significantly shortens response time and improves obstacle avoidance efficiency. The effectiveness of the proposed algorithm was validated through two sets of comparative experiments (against the RRT method) conducted on a collaborative robot platform. Experimental results demonstrate that the algorithm reduces response time for obstacle avoidance by 33.2% and increases success rate by 19.5%, fully proving its potential and applicability in specific industrial scenarios.

Dynamic Movement Primitives (DMPs) are a well-known tool for encoding robotic motions. Their popularity stems from invariance properties in time and space, the ability to describe complex coordinated motions in multiple degrees of freedom with a relatively small number of parameters, and the linearity in the parameters that describe the motion. The latter allows easily fitting a DMP to motions e.g. demonstrated by a human. DMPs are at their core second order autonomous differential equations. However, feedforward controls of robots with flexible joints are known to require reference trajectories up to the fourth derivative of position. Consequently, classical DMPs are mechanically not compatible with flexible joint robots. In this paper, we propose an extension of DMPs by introducing FlexDMPs. This concept retains the structural properties and benefits of classical DMPs but generates trajectories up to the fourth derivative that can theoretically be tracked ideally (i.e. with zero tracking error) by flexible joint robots. The concept is demonstrated on a high fidelity simulation model of an industrial robot and in experimental results on a collaborative manipulator.

Dynamic movement primitives are one of key concepts for understanding dexterous and flexible movements of biological bodies. In the field of robotics engineering, simple types of nonlinear differential equations are used to generate movement primitives from demonstrations, but it remains unclear how nonlinear dynamics in the real brain can also generate movement primitives in biologically natural ways. The aim of this study is to investigate a possible role of nonlinear dynamics in random recurrent neural networks (RNNs) for skillful motor learning. We show that one-shot temporal patterns such arm reaching movements can be trained by a type of RNN-learning so-called FORCE-learning recently proposed by Sussillo and Abbott and a number of patterns are summarized as a manifold embedded in a space of synaptic weights of readout neurons. We also discuss how generalization of learning against untrained motor patterns can be achieved by identifying nonlinear coordinates (meta-parameters) on this manifold in a higher level of the central nervous system.

In order to properly function in real-world environments, the gait of a humanoid robot must be able to adapt to new situations as well as to deal with unexpected perturbations. A promising research direction is the modular generation of movements that results from the combination of a set of basic primitives. Humans seem to be able to adapt their motion in an effortless way and efficient way. In this work we present a methodology to extract a single demonstration signal from a human teacher that can be used in a biped robot to walk in an efficient way and quickly adapt to new situations. Instead of using a normal human gait, the demonstrator walks on a gait we called "robot-like". It is our belief that the retargeting task will be more efficient and easier by using this methodology. The acquired signal is then used to train a set of Dynamic Movement Primitives (DMP) that directly retrieves the trajectories of the end effector in the Cartesian space. In previous works we have used this methodology with efficiency in several tasks, e.g., walking over irregularities, overcoming an obstacle. Now, by combining rhythmic DMP with discrete DMP, we show that the adaptability performance to new situations that require precise foot placement is increased.

Forestry cranes are an important tool for safe and efficient timber harvesting with forestry machines. However, their complex manual control often led to inefficiencies and excessive energy usage, due to the many joysticks and buttons that must be used in a precise sequence to perform efficient movements. To address this, the industry is increasingly turning to partial automation, making manual control more intuitive for the operator and, consequently, achieving improvements in energy efficiency. This article introduces a novel approach to energy-optimal motion planning that can be used along with a feedback control system to automate crane motions, taking over portions of the operator’s work. Our method combines dynamic movement primitives (DMPs) and an energy-optimization algorithm. DMPs is a machine learning technique for motion planning based on human demonstrations, while the optimization algorithm exploits the crane’s redundancy to find energy-optimal trajectories. Simulation results show that DMPs can replicate human-like controlled motions with a 25% reduction in energy consumption. However, our energy optimization algorithm shows improvements of over 40%, providing substantial energy savings and a promising pathway towards environmentally friendly partially automated machines.

Maintaining the balance is always a challenge issue for bipedal humanoid robots in dealing with various locomotive tasks to serve human society, especially when the real environment the robot worked within exhibits to be very complex. Unlike plenty of previous successful approaches on humanoid balancing or push recovery, in this research, the Dynamical Movement Primitives (DMP) is employed to model several typical bio-inspired strategies. As humanoid balancing or push recovery could be regarded as a problem of how a robot to get back to its ongoing behavior when a break happens on account of external force or uneven terrain etc., the DMP model becomes an alternative ideal choice due to its promising nature of attractor. Meanwhile, the DMP composed of a set of differential equations provides a compact parameterized representation in modelling a motion strategy, and thus leads to a strategy model that is suitable to be fulfilled with machine learning techniques. In this research, the learning process for those bio-inspired strategies modeled with DMP are completed by adopting the stochastic policy gradient reinforcement learning and imitation learning separately. Furthermore, with Gaussian Process(GP) regression, push recovery strategies are generalized taking the advantages of the invariance properties of the DMP model. As a consequence, an online adaptive push recovery control strategy is finally achieved. Experimental results on both simulated robot and a real bipedal humanoid robot PKU-HR5 demonstrate the presented approach is effective and promising.

Purpose This paper aims to solve the problem of the inability to apply learning methods for robot motion skills based on dynamic movement primitives (DMPs) in tasks with explicit environmental constraints, while ensuring the reliability of the robot system. Design/methodology/approach The authors propose a novel DMP that takes into account environmental constraints to enhance the generality of the robot motion skill learning method. First, based on the real-time state of the robot and environmental constraints, the task space is divided into different regions and different control strategies are used in each region. Second, to ensure the effectiveness of the generalized skills (trajectories), the control barrier function is extended to DMP to enforce constraint conditions. Finally, a skill modeling and learning algorithm flow is proposed that takes into account environmental constraints within DMPs. Findings By designing numerical simulation and prototype demonstration experiments to study skill learning and generalization under constrained environments. The experimental results demonstrate that the proposed method is capable of generating motion skills that satisfy environmental constraints. It ensures that robots remain in a safe position throughout the execution of generation skills, thereby avoiding any adverse impact on the surrounding environment. Originality/value This paper explores further applications of generalized motion skill learning methods on robots, enhancing the efficiency of robot operations in constrained environments, particularly in non-point-constrained environments. The improved methods are applicable to different types of robots.

Imitation learning techniques have been used as a way to transfer skills to robots. Among them, dynamic movement primitives (DMPs) have been widely exploited as an effective and an efficient technique to learn and reproduce complex discrete and periodic skills. While DMPs have been properly formulated for learning point-to-point movements for both translation and orientation, periodic ones are missing a formulation to learn the orientation. To address this gap, we propose a novel DMP formulation that enables encoding of periodic orientation trajectories. Within this formulation we develop two approaches: Riemannian metric-based projection approach and unit quaternion based periodic DMP. Both formulations exploit unit quaternions to represent the orientation. However, the first exploits the properties of Riemannian manifolds to work in the tangent space of the unit sphere. The second encodes directly the unit quaternion trajectory while guaranteeing the unitary norm of the generated quaternions. We validated the technical aspects of the proposed methods in simulation. Then we performed experiments on a real robot to execute daily tasks that involve periodic orientation changes (i.e., surface polishing/wiping and liquid mixing by shaking).