In-hand recognition and manipulation of elastic objects using a servo-tactile control strategy

Tactile sensors are key components for a robot hand system, which are usually used to obtain the object’s features. The use of tactile sensors to obtain information from the objects is an open topic of research. In this paper, a new strategy for in-hand extraction of object’s properties and for control of the interaction forces with robot fingers, mainly based on tactile data, is presented. The scope of this strategy is to grasp and manipulate solid objects, including rigid and soft bodies. Assuming that the hand is in an initial configuration in which the object is grasped, the properties’ extraction approach is executed. After the extraction of properties is finished, the object can be classified in regard to a general body listing: rigid body, soft elastic body, or soft plastic object. Once the object is classified, for in-hand manipulation tasks, the contact points between the object grasped and the fingers are maintained using the information given by the tactile sensors in order to perform manipulation tasks. Each task is defined by a sequence of basic actions, in which the contact points and applied forces are adapted depending on the action to be performed and the estimated features for the object. The presented approach tries to imitate the behavior of human beings, in which the applied forces by the fingers are changed when the human estimates the rigidity of a body and when the fingers react to unexpected movements of the object to keep the contact points.

Anthropomorphic robotic hand-arm systems have the potential to manipulate various objects in the same manner as a human does. Generally, robot hand manipulation has been studied independently of the manipulation by using robot arms. As the hand and arm cooperate to achieve the same objective, controlling them as a single system is desirable. In this paper, an object manipulation strategy using whole hand-arm systems is proposed. The proposed manipulation strategy provides a method to translate and rotate an object while a robot holds the object based on the planning of the position of each fingertip. A control scheme of the whole hand-arm system to generate the required manipulation motion and the object grasping force is presented. The proposed strategy is experimentally verified using the KITECH upper-body robot with 8-DOF arms and 16-DOF multifingered hands.

Predicting instead of only reacting to the properties of objects we grasp is crucial to dexterous object manipulation. Although we normally plan our grasps according to well-learned associations, we rely on implicit sensorimotor memories when we learn to interact with novel or ambiguous objects. However, little is known about the influence of sensorimotor predictions on subsequent perception and action. Here, young and elderly subjects repeatedly lifted an object in which the center of mass (CoM) was randomly varied between trials straight upward with the aim of preventing object tilts. After each lift, subjects indicated the location of the perceived CoM and reported how heavy the object felt. Surprisingly, we found that sensorimotor torque memories eventually causing initial lifting errors had substantial effects on the perception of torques, weight, and the torque planning for the next lift. Whereas subjects tended to partly retain their previous erroneous sensorimotor memories (instead of solely relying on the previously encountered torque for the upcoming motor plan), they perceived encountered torques to be stronger when they erroneously predicted them. Additionally, we found that torque prediction errors, as well as the actual torques, made the object feel heavier. By contrast, perception did not influence upcoming motor control. There were no major differences observed between the age groups. The sensorimotor impact on torque perception can be explained by internal feedforward prediction highlighting task-relevant errors, while the partial retention and adaptation of sensorimotor torque memories is reconciled with the trial-to-trial learning rule for motor adaptation. NEW & NOTEWORTHY The current study is the first to demonstrate in an object manipulation task in uncertainty that errors in the sensorimotor prediction of torques influence the perception of both torques and weight, whereas sensorimotor torque memories are partly retained and partly adapted to planning errors. Our results provide novel insights into the predictive mechanisms underpinning the common everyday task of object manipulation and further support theories about the predictive modulation of perception established in other neuroscientific disciplines.

Introduction We can easily manipulate a variety of objects with our hands. When exploring an object, we gather rich sensory information through both haptics and vision. The haptic and visual information obtained through such exploration is, in turn, key for realizing dexterous manipulation. Reproducing such codevelopment of sensing and adaptive/dexterous manipulation by a robotic hand is one of the ultimate goals of robotics, and further, it would be essential for understanding human object recognition and manipulation. Although many robotic hands have been developed, their performance is by far inferior to that of human hands. One reason for this performance difference may be due to differences in grasping strategies. Historically, research on robotic hands has mainly focused on pinching manipulation (e.g. Nagai and Yoshikawa, 1993) because the analysis was easy with point-contact conditions. Based on the analysis, roboticists applied control schemes using force/touch sensors at the fingertips (Kaneko et al ., 2007; Liu et al ., 2008). Since the contact points are restricted to the fingertips, it is easy for the robot to calculate how it grasps an object (e.g. a holding polygon) and how large it should exert force based on friction analysis. However, the resultant grasping is very brittle since a slip of just one of the contacting fingertips may lead to dropping the object.

This paper details a study performed on a new proposed twelve degree-of-freedom dual robot arm, which is very light but capable of handling heavy loads. The proposed robot arm has a higher value for the ratio of the load capacity/robot weight than conventional robot arms, which are actuated by motors with speed reducers, such as a harmonic drives, since it adopts a new type of robot actuator based on a closed chain mechanism. Because of the high value of the ratio of the payload capacity/robot weight, it can be used as a robot arm for mobile robots and for walking robots. Analyses of the design scheme and of the mechanism of the joint actuator used for the robot arm are presented. Also, the control system developed for the robot arm is introduced. The superior characteristics of the new proposed robot arm, handling heavy payloads with light weight links compared to industrial robots, are presented through carrying out various payload capacity tests. Since the robot arm is designed with light links, it has some deflections and these deflections of the links are analyzed using the Finite Element Method (FEM). The results of performance tests are presented to check the correctness of the FEM analysis and to demonstrate the actual capability of handling heavy payloads applied to the robot arm.

Grasp control refers to the art of controlling the motion of an object by constraining its dynamics through contacts with a hand. The process of controlling the grasp is not limited to robotic hands only but also applies to human hands (Johansson and Edin 1991) and to all other mechanisms using contact constraints to control the motion of the manipulated object (Brost and Goldberg 1996). A crucial role in the control of grasping is played by contact constraints. All the interactions between the robotic hand and the grasped object occur at the contacts whose understanding is paramount (Salisbury and Roth 1983). The unilateral nature of contact interaction in grasping makes the control problems much more challenging than cooperative manipulation where multiple arms hold the object rigidly allowing bilateral force transmission at each contact point (Chiacchio et al. 1991). The importance of unilateral contact constraints in grasping led a large part of the literature to focus on the closure properties of the grasp (Bicchi 1995). Those properties refer to the ability of a grasp to prevent motions of the grasped object relying only on unilateral frictionless constraints in case of form closure (Reuleaux 1876) and on contact constraints with friction in case of force closure (Nguyen 1988). While form closure is a purely geometric property of the grasp and depends on where the unilateral contact points are on the object, force closure depends on the ability that the robotic hand has to resist and apply forces to the object through the contacts while satisfying the friction constraints. In other terms force closure directly involves the control of the robotic hand kinematics and not only the geometry of the contacts (Bicchi 1995). This entry focuses on force-closed grasps. The optimal choice of the contact points on the object surface is a critical issue known as grasp planning. Among the many optimal criteria that have been proposed in the literature to choose the contact points, I want to recall the one proposed in Ferrari and Canny (1992) where the grasping configuration is evaluated according to the magnitude of the largest worst-case disturbance wrench that can be resisted by the grasp. Many approaches have been studied in the literature on grasp planning in the presence of uncertainties. The uncertainty can be either due to the shape of the object which is partially known or partially sensed as in Goldfeder et al. (2009) or due to the errors in positioning the fingers on the object during the grasping (Roa and Suarez 2009). In what follows all the parameters of the grasp including those related to the hand, the object, and the contact points are assumed to be known with no uncertainties. The main objective of grasp control is that of tracking a desired trajectory with the grasped object by applying a set of contact forces satisfying the friction constraints (Bicchi and Kumar 2000). Complex in-hand object motions can be obtained by rolling and sliding the contact points on the object surface as proposed in Montana (1988) or by using finger gaiting to get large-scale motions (Han and Trinkle 1998). This entry deals with non-rolling and non-sliding contact points

In recent years, the computer vision applications in the robotics have been improved to approach human-like visual perception and scene/context understanding. Following this aspiration, in this study, we explored the possibility of better object manipulation performance by connecting the visual recognition of objects to their physical attributes, such as weight and center of gravity (CoG). To develop and test this idea, an object manipulation platform is built comprising a robotic arm, a depth camera fixed at the top center of the workspace, embedded encoders in the robotic arm mechanism, and microcontrollers for position and force control. Since both the visual recognition and force estimation algorithms use deep learning principles, the test set-up was named as Deep-Table. The objects in the manipulation tests are selected from everyday life and are common to be seen on modern office desktops. The visual object localization and recognition processes are performed from two distinct branches by deep convolutional neural network architectures. We present five of the possible cases, having different levels of information availability on the object weight and CoG in the experiments. The results confirm that using our algorithm, the robotic arm can move different types of objects successfully varying from several grams (empty bottle) to around 250 g (ceramic cup) without failure or tipping. The proposed method also shows that connecting the object recognition with load estimation and contact point further improves the performance characterized by a smoother motion.

This thesis presents the design and fabrication of an autonomous mobile robot with dual six-axis robotic arms, enhanced by augmented reality (AR) and powered by artificial intelligence (AI). The innovation aims to advance object manipulation tasks in various complex and dynamic environments, potentially for industrial applications. The system's Artificial Intelligence feature enhances the robot's ability to navigate and interact with its surroundings more accurately. The AR interface also allows human operators to intuitively control and guide the robot's actions in real time when needed, bridging the gap between autonomous operation and human oversight. The dual six-axis robotic arms enable dexterous and precise object manipulation. These arms mimic the versatility and adaptability of human arms, opening new possibilities for tasks that were previously challenging for traditional robotic systems. In developing this advanced mobile robot, this work embodies a pioneering step in autonomous robotics, showcasing a blend of AI, AR, and mechanical design. The robot's capabilities in object manipulation present a leap forward in robotics, promising a future where robots can take on more complex roles in our societies, while also offering a practical solution to labor-intensive or high-risk tasks. The research concludes with extensive testing and validation of the robot, demonstrating its effectiveness and efficiency in performing complex object manipulation tasks autonomously in various controlled and real-world scenarios. The results suggest promising implications for the future of robotics and automation, reinforcing the importance of interdisciplinary approaches in technological advancement.

Tactile control based on Gaussian images and its application in bi-manual manipulation of deformable objects

Individual and sex differences in learning abilities of ravens

Prisma Hand II is an under-actuated prosthetic hand developed at the University of Naples, Federico II to study in-hand manipulations during grasping activities. 3 motors equipped on the robotic hand drive 19 joints using elastic tendons. The operations of the hand are achieved by combining tactile hand sensing with under-actuation capabilities. The hand has the potential to be employed in both industrial and prosthetic applications due to its dexterous motion capabilities. However, currently there are no commercially available tactile sensors with compatible dimensions suitable for the prosthetic hand. Hence, in this work, we develop a novel tactile sensor designed based on an opto-electronic technology for the Prisma Hand II. The optimised dimensions of the proposed sensor made it possible to be integrated with the fingertips of the prosthetic hand. The output voltage obtained from the novel tactile sensor is used to determine optimum grasping forces and torques during in-hand manipulation tasks employing Neural Networks (NNs). The grasping force values obtained using a Convolutional Neural Network (CNN) and an Artificial Neural Network (ANN) are compared based on Mean Square Error (MSE) values to find out a better training network for the tasks. The tactile sensing capabilities of the proposed novel sensing method are presented and compared in simulation studies and experimental validations using various hand manipulation tasks. The developed tactile sensor is found to be showcasing a better performance compared to previous version of the sensor used in the hand.

This paper presents a robotic system based on a serial manipulator. The robot is a vertical articulated arm with 5 revolute joints having 6 Degree Of Freedom. Actuated with six precise servo motors, the system offers positional accuracy of ±0.5mm with a movement speed of 100mm/s. Forward and Inverse Kinematic model of the robot has been developed and its workspace has been analyzed to facilitate the use of robotic arm as a simulated industrial manipulator. Image processing has been done to make system more autonomous. Followed by a user's commands, the system acquires image of the environment using on-board camera. This image is processed to extract information about object's coordinates. Based on these coordinates, Inverse Kinematic model computes the required joint angles for the end-effector to reach at desired position and orientation thus enabling it to manipulate the object. The proposed system can be used in wide range of industrial applications involving pick and place, sorting and other object manipulation tasks. The system can also be potentially useful for heavy and ‘giant’ industrial applications after scaling up i.e. using huge robotic arm, employing multiple and better cameras and optimizing algorithms.

A robot hand is used to transport the glass substrate in TFT-LCD manufacturing process. Carbon/epoxy composite is one of the best materials for this kind of robot hand application, due to their lightweight, high stiffness, and good damping characteristics. Major requirement of the robot hand is given as allowable deflection under weight loading of glass substrate and robot hand itself. In this thesis, a carbon/epoxy robot hand was analyzed using finite element method and beam theory to determine the deflection of the hand under the loading that is equivalent to actual weight. Because natural frequency is one of the major interests in robot hand design for TFT-LCD manufacturing process, modal analysis is also conducted using finite element method and beam theory. A robot hand was manufactured, and actual deflection and natural frequency were measured to verify the analysis results and compliance to requirement. The test results showed good agreement with analysis results.

The control of multi-fingered robot hands has been the subject of recent interest. To manipulate some object with it, it is necessary to determine the contact points and grasping forces at the fingertips, but it is difficult to optimize such manipulation parameters for performing effective manipulation. In this paper, we propose an acquisition method for re-grasping motion for a four-fingered robot hand using evolutionary programming (EP). In this case, EP determines not only when and where the fingers are moved, but also which finger is operated at the next contact point according to the attitude changes of object grasping with the grasping stability. Evolutionary programming is generally able to find an optimal solution without a supervisor after much iteration, which is almost impractical using a real robot. Therefore we control the real robot hand with the optimal controller that has been obtained in computer simulations. We show the effectiveness of the proposed acquisition method for the re-grasping motion with experimental results.

To model manipulation tasks, we propose a novel method for learning manipulation skills based on the degree of motion granularity. Even though manipulation tasks usually consist of a mixture of fine-grained and coarse-grained movements, to the best of our knowledge, manipulation skills have so far been modeled without considering their motion granularity. To model such a manipulation skill, Gaussian mixture models (GMMs) have been represented using several well-known techniques such as principal component analysis, k-means, Bayesian information criterion, and expectation-maximization (EM) algorithms. However, in this GMM, there is a problem in that when a mixture of fine-grained and coarse-grained movements is modeled as a GMM, fine-grained movements tend to be poorly represented. To resolve this issue, we measure a continuous degree of motion granularity for every time step of a manipulation task from a GMM. Then, we remodel the GMM by weighting a conventional k-means algorithm with motion granularity. Finally, we also estimate the parameters of the GMM by weighting the conventional EM with motion granularity. To validate our proposed method, we evaluate the GMM estimated using our proposed method by comparing it with those estimated by different GMMs in terms of inference, regression, and generalization using a robot arm that performs two daily tasks, namely decorating a very small area and passing through a narrow tunnel.