An Integrated Framework of Grasp Detection and Imitation Learning for Space Robotics Applications

Soft sensor model development in multiphase/multimode processes based on Gaussian mixture regression

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

Learning from demonstration (LfD), which provides a natural way to transfer skills to robots, has been extensively researched for decades, and an army of methods and applications have been developed and investigated for learning an individual or low-level task. Nevertheless, learning long time sequential tasks is still very difficult as it involves task segmentation and sub-task clustering under an extremely large demonstration variance. Besides, the representation problem should be considered when doing segmentation. This paper presents a new unified framework to solve the problems of segmentation, clustering, and representation in a sequential task. The segmentation algorithm segments unstructured demonstrations into movement primitives (MPs). Then, the MPs are automatically clustered and labeled so that they can be reused in other tasks. Finally, the representation model is leveraged to encode and generalize the learned MPs in new contexts. To achieve the first goal, a change-point detection algorithm based on Bayesian inference is leveraged. It can segment unstructured demonstrations online with minimum prior knowledge requirements. By following the Gaussian distributed assumption in the segmentation model, MPs are encoded by Gaussians or Gaussian mixture models. Thus, the clustering of MPs is formulated as a clustering over cluster (CoC) problem. The Kullback–Leibler divergence is used to measure similarities between MPs, through which the MPs with smaller distance are clustered into the same group. To replay and generalize the task in novel contexts, we use task-parameterized regression models such as the Gaussian mixture regression. We implemented our framework on a sequential open-and-place task. The experiments demonstrate that the segmentation accuracy of our framework can reach 94.3% and the recognition accuracy can reach 97.1%. Comparisons with the state-of-the-art algorithm also indicate that our framework is superior or comparable to their results.

Wearable motion sensing in daily life has attracted attention in various disciplines. Especially, stretchable strain sensors have been instrumented into garments (e.g. brace). To estimate joint motions from such sensors, previous studies have modelled relationships between the sensor strains and motion parameters via supervised/semi-supervised learning. However, typically these only model a single relationship assuming the sensor to be located at a specific point on the body. Consequently, they exhibit reduced performance when the strain-parameter relationship varies due to sensor shifts caused by long-term wearing or donning/doffing of braces. This letter presents a shift-adaptive estimation of knee joint angle. First, a brace is instrumented with two stretch sensors placed at different heights. Next, the different strain-angle relationships at varying brace shift positions are learned using Gaussian mixture models (GMMs). The system then estimates the joint angle from the sensor strains through Gaussian mixture regression using a maximum likelihood shift GMM, which is identified by referring to the two strains in a previous 1 s period. Experimental results indicated that the proposed method estimates the joint angle at multiple shift positions (0-20 mm) with higher accuracy than methods using a single model, single sensor, or referring to the present sensor strains.

Learning from demonstrations (LfD) is gaining more popularity in robotics due to its promise of providing a human-friendly technique for teaching robots new skills by robotics-naive users. The two main approaches to LfD are dynamic motor primitives (DMP) which models demonstrated motions as dynamical systems with the advantage flexibility in changing the motion’s starting position, goal or speed and Gaussian Mixture Modelling/ Gaussian Mixture Regression (GMM/GMR) which represents demonstrated motions as mixtures of Gaussians with the advantage of keeping track of the correlations between different dimensions of learned motions and automatic extraction of motion variability along these dimensions. This paper introduces a third approach that relies on symbolization of demonstrated motions by extending the Symbolic Aggregate approXimation (SAX) to handle multiple dimensions of data. The proposed approach is shown through several real-world evaluations to be more resistant to confusing demonstrations that usually arise when action segmentation is automated. The paper also discusses a possible way to combine SAX based LfD withGMM/GMR in order to preserve the advantages of these two approaches while providing superior confusion resistance.

This paper proposes an enhanced robot skill learning system considering both motion generation and trajectory tracking. During robot learning demonstrations, dynamic movement primitives (DMPs) are used to model robotic motion. Each DMP consists of a set of dynamic systems that enhances the stability of the generated motion toward the goal. A Gaussian mixture model and Gaussian mixture regression are integrated to improve the learning performance of the DMP, such that more features of the skill can be extracted from multiple demonstrations. The motion generated from the learned model can be scaled in space and time. Besides, a neural-network-based controller is designed for the robot to track the trajectories generated from the motion model. In this controller, a radial basis function neural network is used to compensate for the effect caused by the dynamic environments. The experiments have been performed using a Baxter robot and the results have confirmed the validity of the proposed methods.

With the development of robotic technologies over last decades, the robotic mapping or the simultaneous localization and mapping (SLAM) has got matured correspondingly. The Large-Scale Direct Monocular SLAM (LSD-SLAM) is among the most promising methods in SLAM, which not only locally tracks the motion of the camera, but allows to build consistent and large-scale maps of the environment. In this paper, LSD-SLAM is implemented to model the surrounding environment of a three degrees-of-freedom robot manipulator. All the objects in the robot workspace have been recognized and located. And the robot will take the extracted information as visual input to perform a point-to-point approaching task A novel motion planning technique, programming by demonstrations (PbD), is used to transfer human approaching skills to the robot. Gaussian Mixture Models (GMM) and Gaussian Mixture Regression (GMR) are used to encode human demonstrations with a first-order differential dynamical systems model. Experimental results have verified the effectiveness of the proposed methods.

Cable-driven rehabilitation robots can be used to recover the upper-limb motor function of hemiplegia patients after stroke. In this study, a statistical approach was proposed to enable a 3-DOF upper-limb cable-driven rehabilitation robot (UCDRR) to predict the human intention by learning human skills from demonstrations. Specifically, in the learning phase, Gaussian mixture model (GMM) was exploited to statistically analyze the human-robot interaction features (i.e., Electromyography (EMG) and interaction force) based on the recorded demonstrator-robot interaction data. Then, during prediction, the robot estimated the interaction force from the measured EMG by using Gaussian mixture regression (GMR). The experimental results of the interaction force prediction in simulation demonstrated that the proposed approach could statistically extract the variable information from the multiple demonstrations successfully and estimate subject-robot interaction forces with high accuracy. This study will help the UCDRR to provide assistance as needed to hemiplegia patients during rehabilitation training, thereby promoting the active participation.

Detection of bimanual gestures everywhere: Why it matters, what we need and what is missing

This paper presents a novel force learning framework to learn fingertip force for a grasping and manipulation process from a human teacher with a force imaging approach. A demonstration station is designed to measure fingertip force without attaching force sensor on fingertips or objects so that this approach can be used with daily living objects. A Gaussian Mixture Model (GMM) based machine learning approach is applied on the fingertip force and position to obtain the motion and force model. Then a force and motion trajectory is generated with Gaussian Mixture Regression (GMR) from the learning result. The force and motion trajectory is applied to a robotic arm and hand to carry out a grasping and manipulation task. An experiment was designed and carried out to verify the learning framework by teaching a Fanuc robotic arm and a BarrettHand a pick-and-place task with demonstration. Experimental results show that the robot applied proper motions and forces in the pick-and-place task from the learned model.

This paper presents the design and application of the SAFER glove in the field of hand rehabilitation. The authors present preliminary results on a new hand grasping rehabilitation learning system that is designed to gather kinematic and force information of the human hand and to playback the motion to assist a user in common hand grasping movements, such as grasping a bottle of water. The fingertip contact forces during grasping have been measured by the SAFER Glove from 12 subjects. The measured fingertip contact forces were modeled with Gaussian Mixture Model (GMM) based on machine learning approach. The learned force distributions were then used to generate fingertip force trajectories with a Gaussian Mixture Regression (GMR) method. To demonstrate the glove’s potential to manipulate the hand, experiments with the glove fitted on a wooden hand to grasp various objects were performed. Instead of defining a grasping force, contact force trajectories were used to control the SAFER Glove to actuate/assist this hand while carrying out a learned grasping task. To prove that the hand can be driven safely by the haptic mechanism, force sensor readings placed between each finger and the mechanism have been plotted. The experimental results show the potential of the proposed system in future hand rehabilitation therapy.

Este trabajo propone el aprendizaje de una señal de control para un electrolizador tipo PEM, considerando variables de entrada y salida del sistema como la corriente eléctrica suministrada por una fuente fotovoltaica, la temperatura ambiente y la energía disipada por el caudal del sistema de refrigeración acoplado al electrolizador. Los datos de corriente eléctrica y temperatura ambiente son medidos por el sistema. Por otro lado, los datos de energía disipada por el caudal del sistema de refrigeración fueron obtenidos a partir de simulaciones de un controlador de temperatura que utiliza la técnica de Control Predictivo Basado en Modelo (MPC). Se proponen dos técnicas de aprendizaje: Gaussian Mixture Model (GMM) / Gaussian Mixture Regression (GMR) y un modelo NeuroFuzzy (NF). Los resultados de las simulaciones demuestran que las técnicas de aprendizaje modelaron con precisión el comportamiento de la señal de control en el electrolizador.

In the context of structural and industrial asset monitoring and twinning, the estimation of unknown inputs – typically reflecting the loads acting onto a system – stands as a critical factor in ensuring both the safety and performance of engineered systems. This research introduces a novel approach for inferring such unknown inputs from observed outputs, focusing particularly on dynamical systems, whether characterized by known or learned dynamics. Our proposed approach is situated within the framework of Imitation Learning from Observation (ILfO), which is here recast in the context of dynamical systems’ estimation. Our primary objective is to infer estimates of input signals on the basis of real-world measured outputs. The problem is formulated as a Partially Observable Markov Decision Process (POMDP), and we address it by utilizing established planning algorithms specifically tailored for ILfO applications. To address the POMDP, we harness the effectiveness and robustness of the cross-entropy method. We first verify the efficacy of the proposed approach on simulated case studies involving dynamical systems possessing well-defined dynamics. The proposed method is further validated on a practical scenario involving experimental data from a scaled wind turbine blade, leveraging dynamics learned through the Neural Extended Kalman Filter (Neural EKF) technique; an approach which leverages deep learning for inferring the dynamics of a system when this is imprecisely known. These examples demonstrate the flexibility of the proposed approach for use with systems featuring different degrees of prescribed dynamics. • A new perspective of input estimation, embedded within Imitation Learning from Observation methodology. • Recast the input estimation problem as a Partially Observable Markov Decision Process. • Integration of learning-based dynamics models.

Robust grasp pose detection is crucial for robotic manipulation. Antipodal grasps are often inadequate for spatial objects. However, insufficient demonstrations and the higher complexity compared to parallel grippers make it challenging to implement learning-based robotic grasping with multi-fingered hands. To address these deficiencies, we propose a multi-modal grasp detection method tailored for three-fingered grippers. We utilize a simplified unified grasp representation, which can represent both precision and power grasping modes. Additionally, we introduce a pipeline to generate a multi-modal grasp dataset with a three-point contact grasp metric. We also present the concept of confness to quantify the probabilities of different grasp configurations within a given graspable area, guiding the predicted grasp poses to be concentrated around on centroidal region of objects. Furthermore, we develop QcMNet, an end-to-end network that is used to predict confness and infer grasp parameters. Experiments conducted with a Robotiq 3-Finger Adaptive Gripper, both in simulation and real-world scenarios, demonstrate that multi-modal grasping can increase the percent-cleared rate and enhance anti-disturbance capabilities in cluttered scenes. Note to Practitioners—This paper was motivated by the problem of finding the proper gripper configurations to ensure the stable grasping of objects across diverse tasks in industrial automation. Most existing approaches to robotic grasping detection utilize parallel grippers. However, it is insufficient to achieve stable grasping for spatial objects only with two frictional point contacts. This paper presents a learning-based grasp pose detection method tailored for three-fingered grippers, enabling robots to achieve a stable grasp on objects. We first propose a unified grasp representation that can represent different grasp modes, including precision and power grasp modes. Subsequently, a densely annotated dataset for three-fingered grippers is obtained by employing a robust grasp metric and an auto-annotation pipeline in simulation. This approach is designed to reduce labor consumption. The proposed pipeline not only enhances the efficiency of the data collection process but also ensures its diversity. Furthermore, we introduced a neural network to quantify the probabilities of different grasp modes within a graspable area and to infer grasp poses. Extensive physical robotic experiments validate the feasibility of this approach for grasping various objects. Significantly, most grasp poses concentrate around the centroid of objects, enhancing grasp stability. In future research, we will involve further optimizing the generated grasping candidates using local geometry information of objects. Additionally, we intend to generate online collision-free trajectories for robotic grasping to adapt to more complex automation environments.

In rarefied gas flows, discontinuity phenomena such as velocity slip and temperature jump commonly appear in the gas layer adjacent to a solid boundary. Due to the physical complexity of the interactions at the gas-solid interface, particularly in the case of systems with local nonequilibrium state, boundary models with limited number of parameters cannot completely describe the reflection of gas molecules at the boundary. In this work, the Gaussian mixture (GM) model, which is an unsupervised machine learning technique, is employed to construct a statistical gas-solid surface scattering model based on the collisional data obtained from molecular dynamics (MD) simulations. The GM model is applied to study Couette flow for different inert gases (Ar and He) confined between two parallel infinite gold walls at different temperatures. A direct comparison between the results obtained from the GM model and the Cercignani-Lampis-Lord (CLL) scattering kernel against the MD collisional data in terms of the distribution of the predicted postcollisional velocities, and accommodation coefficients has shown that the results from the GM model are an excellent match with the MD results outperforming the CLL scattering kernel. As an example, for He gas, while the predicted energy accommodation coefficient by the CLL model is more than two times higher than the MD predictions, the value computed by the GM model is in excellent agreement with the MD results. This superior performance of the GM model confirms its high potential to derive a generalized boundary condition in systems encountered with highly nonequilibrium and complex gas flow conditions.