DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations

Grasping and manipulation are common robotic tasks which require to properly control the interaction forces and position of multiple robots end‐effectors. In this study, the authors focus on the problem of dexterous manipulation of rigid objects by means of a cooperative robotic system. Unlike several works, they consider the more complex case when the manipulated object is not mechanically attached to the robots' end‐effectors. In addition, they address the control problem of designing a control law with continuous and bounded input torques for cooperative robots in constrained motion. To guarantee a stable grasp and a fine manipulation a centralised control algorithm based on the Orthogonalisation principle and a dynamic sliding mode control is proposed. The control algorithm does not require the knowledge of the dynamic model of the robot manipulators and manipulated object for implementation. Experimental results are presented to show the good performance of the proposed controller. In addition, the stability analysis of the closed‐loop dynamics is developed.

Based at CERN, the European Organization for Nuclear Research, the Large Hadron Collider (LHC) is the world's largest and most powerful particle accelerator. From 2025, the LHC will be upgraded to allow it to achieve a factor of 10 higher luminosity, which increases the rate of collisions, essential for probing new physics phenomena in the future. The route to high luminosity LHC (HL-LHC) involves various detector upgrades and requires significant infrastructure changes. Recent measurements by CERN Radiation Protection, verifying previous calculations by The University of Sheffield (UoS), has raised awareness about the need to restrict human activity in the HL-LHC experimental, construction and maintenance areas due to exposure from high levels of radiation. Examining the case of the ATLAS detector upgrade, the collaborative partnership between UoS and UK industry is developing state-of-the-art robotic instrumentation, capable of tolerating high radiation levels. The main object of this research is a feasibility study with a TRL (technology readiness level) of three, to determine how materials and sub components of dexterous robotic systems behave after exposure to high levels of radiation. This is evaluation uses novel robotic irradiation equipment, techniques and test methods housed in the Birmingham University (UK) Irradiation Facility. One finger of an unmodified Shadow Robot Company “Hand”, a highly dexterous robotic manipulator, was exposed to specific doses of high radiation in a temperature controlled thermal chamber. Cooled by a liquid nitrogen evaporative system, the irradiation system moves samples continuously through a homogeneous proton beam. Movement is provided by a radiation hard pre-configured XY-Axis Cartesian Robot. The methods and techniques developed as a result of this TRL3 research will further aid the application and deployment of robotic and autonomous systems into highly radioactive environments. Based on preliminary findings it has been concluded that finger materials and basic electrical components can tolerate hazardous radiation environments, with careful selection and substitution of a minimal amount of materials, radiation hardness is also possible. Further work is scheduled for the irradiation of a fully instrumented and powered robotic hand to determine working hour tolerance.

Tactile sensation across the whole hand, including the fingers and palm, is essential for manipulation and, therefore, is expected to be similarly useful for enabling dexterous robot manipulation. Tactile sensation would ideally be distributed (over large surface areas), have a high precision, and provide measurements in multiple axes, allowing for effective manipulation and interaction with objects of varying shapes, textures, friction, and compliance. Given the complex geometries and articulation of state-of-the-art robotic grippers and hands, they would benefit greatly from their surface being instrumented with a thin, curved, and/or flexible tactile sensor technology. However, the majority of current sensor technologies measure tactile information across a planar sensing surface or instrument-curved skin using relatively bulky camera-based approaches; proportionally in the literature, thin and flexible tactile sensor arrays are an under-explored topic. This paper, presents a thin, flexible, non-camera-based optical tactile sensor design as an investigation into the feasibility of adapting our novel LiVec sensing principle to curved and flexible surfaces. To implement the flexible sensor, flexible PCB technology is utilized in combination with other soft components. This proof-of-concept design eliminates rigid circuit boards, creating a sensor capable of providing localized 3D force and 3D displacement measurements across an array of sensing units in a small-thickness, non-camera-based optical tactile sensor skin covering a curved surface. The sensor consists of 16 sensing units arranged in a uniform 4 × 4 grid with an overall size of 30 mm × 30 mm × 7.2 mm in length, width, and depth, respectively. The sensor successfully estimated local XYZ forces and displacements in a curved configuration across all sixteen sensing units, the average force bias values (μ¯) were -1.04 mN, -0.32 mN, and -1.31 mN, and the average precision (SD¯) was 54.49 mN, 55.16 mN and 97.15 mN, for the X, Y, Z axes, respectively, the average displacement bias values (μ¯) were 1.58 μm, 0.29 μm, and -1.99 μm, and the average precision values (SD¯) were 221.61 μm, 247.74 μm, and 44.93 μm for the X, Y, and Z axes, respectively. This work provides crucial insights into the design and calibration of future curved LiVec sensors for robotic fingers and palms, making it highly suitable for enhancing dexterous robotic manipulation in complex, real-world environments.

Dexterous robotic manipulation of unknown objects can open the way to novel tasks and applications of robots in semi-structured and unstructured settings, from advanced industrial manufacturing to exploration of harsh environments. However, it is challenging for at least three reasons: the desired motion of the object might be too complex to be described analytically, precise models of the manipulated objects are not available, the controller should simultaneously ensure both a robust grasp and an effective in-hand motion. To solve these issues we propose to learn in-hand robotic manipulation tasks from human demonstrations, using Dynamical Movement Primitives (DMPs), and to reproduce them with a robust compliant controller based on the Virtual Springs Framework (VSF), that employs real-time feedback of the contact forces measured on the robot fingertips. With this solution, the generalization capabilities of DMPs can be transferred successfully to the dexterous in-hand manipulation problem: we demonstrate this by presenting real-world experiments of in-hand translation and rotation of unknown objects.

Finding target molecules with specific chemical properties plays a decisive role in drug development. We proposed GEOM-CVAE, a constrained variational autoencoder based on geometric representation for molecular generation with specific properties, which is protein-context-dependent. In terms of machine learning, it includes continuous feature embedding encoder and molecular generation decoder. Our key contribution is to propose an efficient geometric embedding method, including the spatial structure representations of drug molecule (converting the 3-D coordinates into image) and the geometric graph representations of protein target (modeling the protein surface as a mesh). The 3-D geometric information is vital to successful molecular generation, which is different from previous molecular generative methods based on 1-D or 2-D. Our model framework generates specific molecules in two phases, by first generating special image with molecular 3-D information to learn latent representations and generating molecules with constrained condition based on geometric graph convolution for specific protein and then inputting the generated structural molecules into a parser network for obtaining Simplified Molecular Input Line Entry System (SMILES) strings. Our model achieves competitive performance that implies its potential effectiveness to enable the exploration of the vast chemical space for drug discovery.

SUMMARY In this study, we examined questions of spatial memory and the travel routestaken by Nicaraguan mantled howler monkeys ( Alouatta palliata ) when mov-ing between major feeding and resting sites. Studies of primate foraging andrangingbehaviorindicateevidenceofgoal-directedtravelandrelativelystraight-line movement between sequential feeding sites. In the case of mantled howlerson BCI, Panama, Milton (1980) has argued that over the course of severalweeks, group members center their feeding, resting, and ranging activities ona small set of pivotal trees that are visited several times daily. The degree towhich howlers use topological (route-based) spatial representations or geomet-ric (coordinate-based) spatial representations to locate and revisit these feedingand resting sites remains unclear.In order to address questions concerning goal-directed travel and spatialmemory, we mapped the travel routes taken by a troop of 26–29 mantledhowlersinhabitingEstaci´onBiol´ogicadeOmetepelocatedonIsladeOmetepe,Nicaragua. Behavioral data were collected during July and August 2002. Overthe course of 15 days and 103 h of observation, all trees the howlers wereobserved to travel, feed, and rest in were marked (

Robotic manipulation and assembly of micro and nanocomponents in confined spaces is still a challenge. Indeed, the current proposed solutions that are highly inspired by classical industrial robotics are not currently able to combine precision, compactness, dexterity, and high blocking forces. In a previous work, we proposed 2-D in-hand robotic dexterous manipulation methods of arbitrary shaped objects that considered adhesion forces that exist at the micro and nanoscales. Direct extension of the proposed method to 3-D would involve an exponential increase in complexity. In this paper, we propose an approach that allows to plan for 3-D dexterous in-hand manipulation with a moderate increase in complexity. The main idea is to decompose any 3-D motion into a 3-D translation and three rotations about specific axes related to the object. The obtained simulation results show that 3-D in-hand dexterous micro-manipulation of arbitrary objects in presence of adhesion forces can be planned in just few seconds.

Chapter 12 - A hand-arm teleoperation system for robotic dexterous manipulation

This paper describes the operations of the 5 degree-of-freedom instrument deployment device (IDD), a dexterous robotic manipulator on the Mars Exploration Rovers, spirit and opportunity. The unprecedented flawless operations of the IDD enabled precise and reliable placement of at least 3 in situ instruments in sequential order on a designated target position on Martian rock/soil any time during the Martian diurnal cycle (day or night). These placements demonstrated a repeatability of /spl sim/1 mm in position and /spl sim/1 degree in orientation. This operations breakthrough is underappreciated, but it alone enabled the scientist to characterize a wide range of rocks and soils in a timely manner in the hunt for geological clues that revealed that the planet was once rich in water. In this paper we describe the IDD planning and command sequence generation process used to place and hold in situ instruments directly against rock and soil targets of interest within the IDD work volume.

This paper presents a robotic capture concept that was developed as part of the e.deorbit study by ESA. The defective and tumbling satellite ENVISAT was chosen as a potential target to be captured, stabilized, and subsequently de-orbited in a controlled manner. A robotic capture concept was developed that is based on a chaser satellite equipped with a seven degrees-of-freedom dexterous robotic manipulator, holding a dedicated linear two-bracket gripper. The satellite is also equipped with a clamping mechanism for achieving a stiff fixation with the grasped target, following their combined satellite-stack de-tumbling and prior to the execution of the de-orbit maneuver. Driving elements of the robotic design, operations and control are described and analyzed. These include pre and post-capture operations, the task-specific kinematics of the manipulator, the intrinsic mechanical arm flexibility and its effect on the arm's positioning accuracy, visual tracking, as well as the interaction between the manipulator controller and that of the chaser satellite. The kinematics analysis yielded robust reachability of the grasp point. The effects of intrinsic arm flexibility turned out to be noticeable but also effectively scalable through robot joint speed adaption throughout the maneuvers. During most of the critical robot arm operations, the internal robot joint torques are shown to be within the design limits. These limits are only reached for a limiting scenario of tumbling motion of ENVISAT, consisting of an initial pure spin of 5 deg/s about its unstable intermediate axis of inertia. The computer vision performance was found to be satisfactory with respect to positioning accuracy requirements. Further developments are necessary and are being pursued to meet the stringent mission-related robustness requirements. Overall, the analyses conducted in this study showed that the capture and de-orbiting of ENVISAT using the proposed robotic concept is feasible with respect to relevant mission requirements and for most of the operational scenarios considered. Future work aims at developing a combined chaser-robot system controller. This will include a visual servo to minimize the positioning errors during the contact phases of the mission (grasping and clamping). Further validation of the visual tracking in orbital lighting conditions will be pursued.

The Japanese art of origami provides a framework to design and fabricate mechanical structures with unique properties that can be harnessed to constrain motion, reduce actuators, and mimic human locomotion. These properties are incredibly applicable to robot design, increasing the capabilities of robotic movement and manipulation relative to the cost. Utilizing origami structures, an omnidirectional locomotive climbing robot was developed to navigate complex three-dimensional (3D) spaces. The robot consists of a central body and two end effectors attached at either end. It’s intended to mimic how humans use their hands and feet for grip while simultaneously adjusting their body position to facilitate movement. The central body of the robot employs a bellows fold pattern that can be extended, contracted, and bent in order to position the end effectors at a desired location. The end effectors are constructed from a Miura-ori pattern that can grasp objects of varying diameters. The patterns are actuated with a tether and spool mechanism that is utilized to autonomously drive movement throughout the entire robot. Experiments were conducted with the climbing robot on various structures in order to validate the applicability of the design to real-world scenarios. An additional origami-inspired robotic manipulator design was also created in order to address the challenges of dexterous manipulation. The manipulator is designed to mimic the motion of a human finger, with the ability to conform to oddly-shaped objects. The finger is based off the waterbomb tesselation, which allows for human finger-like motion with only a single actuator. The tesselation is linkage-driven by a single motor, reducing the number of actuators relative to comparable designs. Both the climbing robot and the dexterous manipulator are manufactured using 3D printing, including multi-material hybrid prints with rigid panels and flexible folds.

International audience

Ophthalmic microsurgery is known to be a challenging operation, which requires very precise and dexterous manipulation. Image guided robot-assisted surgery (RAS) is a promising solution that brings significant improvements in outcomes and reduces the physical limitations of human surgeons. However, this technology must be further developed before it can be routinely used in clinics. One of the problems is the lack of proper calibration between the robotic manipulator and appropriate imaging device. In this work, we developed a flexible framework for hand-eye calibration of an ophthalmic robot with a microscope-integrated Optical Coherence Tomography (MIOCT) without any markers. The proposed method consists of three main steps: a) we estimate the OCT calibration parameters; b) with micro-scale displacements controlled by the robot, we detect and segment the needle tip in 3D-OCT volume; c) we find the transformation between the coordinate system of the OCT camera and the coordinate system of the robot. We verified the capability of our framework in ex-vivo pig eye experiments and compared the results with a reference method (marker-based). In all experiments, our method showed a small difference from the marker based method, with a mean calibration error of 9.2 $\mu$m and 7.0 $\mu$m, respectively. Additionally, the noise test shows the robustness of the proposed method.

Redundant robot manipulator is a kind of robot arm having more degrees-of-freedom (DOF) than required for a given task. Due to the extra DOF, it can be used to accomplish many complicated tasks, such as dexterous manipulation, obstacle avoidance, singularity avoidance, collision free, etc. However, modeling the inverse kinematics of such kind of robot manipulator remains challenging due to its property of null space motion. In this paper, support vector regression (SVR) is implemented to solve the inverse kinematics problem of redundant robotic manipulators. To further improve the prediction accuracy of SVR, a special machine learning technique called blending is used in this work. The proposed approach is verified in MATLAB with a seven DOF Mitsubishi PA-10 robot and the simulation results have proved its high accuracy and effectiveness.

We propose a model-based approach to design feedback policies for dexterous robotic manipulation. The manipulation problem is formulated as reaching the target region from an initial state for some non-smooth nonlinear system. First, we use trajectory optimization to find a feasible trajectory. Next, we characterize the local multi-contact dynamics around the trajectory as a piecewise affine system, and build a funnel around the linearization of the nominal trajectory using polytopes. We prove that the feedback controller at the vicinity of the linearization is guaranteed to drive the nonlinear system to the target region. During online execution, we solve linear programs to track the system trajectory. We validate the algorithm on hardware, showing that even under large external disturbances, the controller is able to accomplish the task.