Grasping model identification for ZJUT hand

To achieve high-accuracy grasping of unknown objects, we present novel multilevel convolutional neural networks (CNNs) for robotic grasping with a parallel gripper or multifingered dexterous hand. The multilevel CNNs include four levels with different structures and functions. The first level is constructed to get the approximate position of the grasped object. The second level aims to obtain the preselected grasping rectangles. The third level is constructed to re-evaluate the preselected grasping rectangles and obtain substantially detailed features with quite a large network, so as to assess each preselected grasping rectangle exactly. By using a selection algorithm, the optimal grasping rectangle can be determined and unknown object grasping can be achieved with a parallel gripper. The purpose of the fourth level is to obtain the finger position distribution to complete the accurate grasping of unknown objects with a multifingered dexterous hand. The test results indicate that, compared to state-of-the-art methods, the proposed multilevel CNNs can greatly increase the precision of the grasping rectangle. Grasping experiments were implemented on a Youbot arm with five degrees of freedom and a Shadow four-fingered dexterous hand. The results show that the multilevel CNNs can determine the optimal grasping rectangle and finger position distribution, thereby achieving high-accuracy grasping of various unknown objects, even under several complex environmental conditions. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners—Robot grasping of objects lags far behind human experiences and poses a significant challenge in the robotics area. To solve it, we present new multilevel convolutional neural networks (CNNs) to process red green blue-depth (RGB-D) images and realize optimal grasping detection of unknown objects. Moreover, we provide details of the network structure, network training, and network testing. The testing results obtained from the open grasping data set show that the multilevel CNNs can significantly increase the accuracy of the grasping rectangle compared to state-of-the-art methods. Experiments were implemented on different robotic platforms, including a five-degrees-of-freedom Youbot arm with a parallel gripper and a UR5 robot arm with a Shadow multifingered dexterous hand. The results validate that the multilevel CNNs offer excellent generalization and robustness for handling different sizes and shapes of unknown objects, as well as background disturbances, which are key problems in robotic manipulation.

Grasp with push policy for multi-finger dexterity hand based on deep reinforcement learning

This paper deals with two fundamental problems concerning a multi-fingered robotic hand with the capability of compliance control. One of them relates to developing a torque sensor useful for the tendon-pulley driving system and the other was a stable grasping and manipulating problem. In order to construct the finger joint actuation in multifingered systems, a tendon-pulley driving system has normally been used. In this kind of driving system a compact joint torque sensor plays an important role in achieving compliant motions at the finger tip and in turn constructing dextrous multifingered hands. It seems, however, that a satisfactory torque sensor has not yet been developed for such a driving system. To cope with this, a Tension Differential type Torque sensor (TDT sensor) is first proposed in this paper and applied to a newly designed robotic hand with two articulated fingers in experiments. Secondly, the stable grasping and manipulation problems in the multifingered hand are addressed assuming the existence of friction at the contact area of each finger tip and the object grasped. To formulate a stable grasping condition, the stiffness matrix of an object grasped by fingers with compliance adjustable joints was introduced. By using the stiffness components, the condition was described in a simple form. The stiffness matrix was resolved to the simpler form at the tip of each finger when the hand was grasping an object. To satisfy the desired stiffness matrix at the tip, the joint stiffness matrix at each finger was adjusted. To manipulate an object, the desired trajectories of the object were converted to joint trajectories using the inverse kinematics equations, and the position reference at each joint servomechanism was adjusted according to them, keeping the stable grasping condition. A robotic hand with two articulated fingers equipped with specially designed small TDT sensors was constructed. Using the hand, various experiments were carried out and the proposed methods were confirmed.

Current research on robotic dexterous hands mainly focuses on designing new finger and palm structures, as well as developing smarter control algorithms. Although the dimensional synthesis of dexterous hands with traditional rigid palms has been carried out, research on the dimensional synthesis of dexterous hands with metamorphic palms remains insufficient. This study investigated the dimensional synthesis of a palm of a novel metamorphic multi-fingered hand, and explored the geometric design for maximizing the precision manipulation workspace. Different indexes were used to value the workspace of the metamorphic hand, and the best proportions between the five links of the palm to obtain the optimal workspace of the metamorphic hand were explored. Based on the fixed total length of the palm member, four nondimensional design parameters that determine the size of the palm were introduced; through the discretization method, the influence of the four design parameters on the workspace of the metamorphic hand with full-actuated fingers and under-actuated fingers was analyzed. Based on the analysis of the metamorphic multi-fingered hand, the symmetrical structure of the palm was designed, resulting in the largest workspace of the multi-fingered hand, and proved that the metamorphic palm has a massive upgrade for the workspace of underactuated fingers. This research contributed to the dimensional synthesis of metamorphic dexterous hands, with practical significance for the design and optimization of novel metamorphic hands.

It is important to enable a robot to manipulate a target object that has no 3-D model information and is situated in an environment with other unknown objects nearby. This poses an open problem of how to combine perception and manipulation to enable the robot to build an appearance-based model of the target object on the spot to facilitate further manipulation of the object while avoiding the other unknown obstacles in the way. In this paper, we introduce an approach to enable a continuum manipulator, which is apt to maneuver through a crowded environment, to gradually build a 3-D surface model of the target object by moving an RGB-D sensor around the object while also detecting and avoiding surrounding unknown obstacles. Our approach interleaves perception and manipulation such that perception guides the manipulator movement, which in turn allows more perception of the target object for object model building and further manipulator motion. Our approach is characterized by a progressive strategy to register RGB-D images of the target object to build and extend a partial model of the object and the corresponding motion planning strategy for the continuum robot to carry out model building and avoid obstacles at the same time. To demonstrate the effectiveness of our approach, experiments on progressive model building of real objects from real RGB-D images are conducted, where a simulated continuum robot plans and executes its motion to carry the RGB-D camera around a target object for taking those images in an augmented reality setting.

Using a dextrous hand involves understanding the relation-ships between the three kinds of sensor outputs available: joint space positions, tendon forces, and tactile contacts. Dextrous manipulation then is a fusion of these three sensing modalities. This paper is an exploration of using a dextrous, multi-fingered hand (Utah/MIT hand) for high-level object recognition tasks. The paradigm is model-based recognition in which the objects are modeled and recovered as superquadrics, which are shown to have a number of important attributes that make them well suited for such a task. Experiments have been performed to recover the shape of objects using sparse contact point data from the hand kinematics and forces with promising results. We also present our approach to using tactile data in conjunction with the dextrous hand to build a library of grasping and exploration primitives that can be used in recognizing and grasping more complex multi-part objects.

Manipulation using dextrous robot hands has been an exciting yet frustrating research topic for the last several years. While significant progress has occurred in the design, construction, and low level control of robotic hands, researchers are up against fundamental problems in developing algorithms for real-time computations in multi-sensory processing and motor control. The aim of this book is to explore parallels in sensorimotor integration in dextrous robot and human hands, addressing the basic question of how the next generation of dextrous hands should evolve. By bringing together experimental psychologists, kinesiologists, computer scientists, electrical engineers, and mechanical engineers, the book covers topics that range from human hand usage in prehension and exploration, to the design and use of robotic sensors and multi-fingered hands, and to control and computational architectures for dextrous hand usage. While the ultimate goal of capturing human hand versatility remains elusive, this book makes an important contribution to the design and control of future dextrous robot hands through a simple underlying message: a topic as complex as dextrous manipulation would best be addressed by collaborative, interdisciplinary research, combining high level and low level views, drawing parallels between human studies and analytic approaches, and integrating sensory data with motor commands. As seen in this text, success has been made through the establishment of such collaborative efforts. The future will hold up to expectations only as researchers become aware of advances in parallel fields and as a common vocabulary emerges from integrated perceptions about manipulation.

A multi fingered dexterous anthropomorphic hand is being developed by the authors. The focus of the hand is the replacement of human operators in hazardous environments and also in environments where zero tolerance is observed for the human errors. The robotic hand will comprise of five fingers (four fingers and one thumb) each having four degrees of freedom (DOF) which can perform flexion, extension, abduction, adduction and also circumduction. For the actuation purpose pneumatic muscles and springs will be used. The paper exemplifies the mechanical design for the robotic hand. It also describes different mechanical designs that have been developed before date.

Model-based object-tracking can provide mobile robotic systems with real-time 6-dof pose information of a dynamic target object. However, model-based trackers typically require the model of the target to be known a-priori. This paper presents a novel method capable of building an approximate 3D geometric model of a target object in an on-line mode, fast enough for real-time use by a model-based object tracker. The algorithm constructs a 3D tessellated model and uses projective texture mapping to model the target object's surface features.

The interactive grasp of robots can grasp the corresponding objects according to the user’s choice. Most interactive grasp methods based on deep learning comprise visual language and grasp detection models. However, in existing methods, the trainability and generalization ability of the visual language model is weak, and the robot cannot cope well with grasping small target objects. Therefore, this paper proposes a 6-DoF grasp detection method guided by visual language, which converts text instructions and RGBD images of the scene to be grasped into inputs and outputs for the 6-DoF grasp posture of the object corresponding to the text instructions. In order to improve the trainability and feature extraction ability of the visual language model, a multi-head attention mechanism combined with hybrid normalization is designed. At the same time, a local attention mechanism is introduced into the grasp detection model to enhance the global and local information interaction ability of point cloud data, thereby improving the grasping ability of the grasp detection model for small target objects. The method proposed in this paper first uses the improved visual language model to predict the plane position information of the target object, then uses the improved grasp detection model to predict all the graspable postures in the scene, and finally uses the plane position information to filter out the graspable postures of the target object. The visual language model and grasp detection model proposed in this paper have achieved excellent performance in various scenarios of public datasets while ensuring a specific generalization ability. In addition, we also conducted real grasp experiments, and the 6-DoF grasp detection method based on visual language guidance proposed in this paper achieved a grasp success rate of 95%.

We have been studying an in-situ 3D modeling and authoring system, AR Diorama. In the AR Diorama system, a user is able to reconstruct a 3D model of a real object of concern and describe behaviors of the model by stroke input. In this article, we will introduce two ongoing studies on interactive 3D reconstruction techniques. First technique is feature-based. Natural feature points are first extracted and tracked. A convex hull is then obtained from the feature points based on Delaunay tetrahedralisation. The polygon mesh is carved to approximate the target object based on a feature-point visibility test. Second technique is region-based. Foreground and background color distribution models are first estimated to extract an object region. Then a 3D model of the target object is reconstructed by silhouette carving. Experimental results show that the two techniques can reconstruct a better 3D model interactively compared with our previous system.

Moving object tracking in video sequences is an important task in the field of computer vision. In this paper, we propose a new population-based algorithm namely simplified swarm optimization (SSO) for tracking arbitrary objects. In SSO, the object model is first projected into a high-dimensional feature space, then the particles will fly over image pixels to find an optimal match of the target. While searching for the optimum, SSO progressively analyzes the occlusion situation. If any occlusion or disappearance of the target object is detected, the movement rules for the searching particles will be adaptively adjusted to recapture the target object. Experimental results showed that the SSO can robustly track an arbitrary target in various challenging conditions. Furthermore, SSO is capable to have 40% faster in speed and 36% higher in accuracy rate than the traditional PSO for varied environment.

A method is presented for stably grasping two dimensional polygonal objects with a dextrous hand when object models are not available. Basic constraints on object vertex angles are found for feasible grasping with two fingers. Local tactile information can be used to determine the finger motion that will reach feasible grasping locations. With an appropriate choice of finger stiffnesses, a hand can automatically grasp these objects with two fingers. The bounded slip of a part in a hand is shown to be valuable for adapting the fingers and object to a stable situation. Examples are given to show the ability of this grasping method to accommodate disturbance forces and to perform simple part reorientations and regrasping operations.

The autonomous and precise grasping operation of robots is considered challenging in situations where there are different objects with different shapes and postures. In this study, we proposed a method of 6-D target pose estimation for robot autonomous manipulation. The proposed method is based on: 1) a fully convolutional neural network for scene semantic segmentation and 2) fast global registration to achieve target pose estimation. To verify the validity of the proposed algorithm, we built a robot grasping operation system and used the point cloud model of the target object and its pose estimation results to generate the robot grasping posture control strategy. Experimental results showed that the proposed method can achieve a six-degree-of-freedom pose estimation for arbitrarily placed target objects and complete the autonomous grasping of the target. Comparative experiments demonstrated that the proposed target pose estimation method achieved a significant improvement in average accuracy and real-time performance compared with traditional methods.

Image segmentation is a well-developing topic in the image processing, and a number of previous works have been proposed and achieved high performance. However, most previous works needed user-assistance to provide the prior information of the target object in the segmentation. In this paper we propose an unsupervised scheme, combining the salient object detection and segmentation method, to segment the target object without any prior information from users. The experimental results show that the proposed salient color model derived with salient features can provide a prior information with high confidence to generate precise segmentation automatically. The proposed color model of salient objects can not only be applied with Min-Cut algorithm, but also extended to more segmentation algorithms, like matting or non-parametric model.