Motion capture with constrained inverse kinematics for real-time hand tracking

In this article we present a technique that enhances an inverse kinematics (IK) solver such that when the results are applied to a computer character, we can generate a level of individualization tailored to both the character and the environment, e.g., a walking motion can become "stiffer" or can be turned into a limping motion. Since the technique is based on an IK solver, we also have the desirable effect of solving retargetting issues when mapping motion data between characters. As the individualization aspect of our technique is very tightly coupled with the inverse kinematics solver, we can achieve both the individualization and retargetting of characters in real time.

The Inverse Kinematics (IK) algorithms implemented in the open-source Orocos Kinematics and Dynamics Library (KDL) are arguably the most widely-used generic IK solvers worldwide. However, KDL's only joint-limit-constrained IK implementation, a pseudoinverse Jacobian IK solver, repeatedly exhibits false-negative failures on various humanoid platforms. In order to find a better IK solver for generic manipulator chains, a variety of open-source, drop-in alternatives have been implemented and evaluated for this paper. This article provides quantitative comparisons, using multiple humanoid platforms, between an improved implementation of the KDL inverse Jacobian algorithm, a set of sequential quadratic programming (SQP) IK algorithms that use a variety of quadratic error metrics, and a combined algorithm that concurrently runs the best performing SQP algorithm and the improved inverse Jacobian implementation. The best alternative IK implementation finds solutions much more often than KDL, is faster on average than KDL for typical manipulation chains, and (when desired) allows tolerances on each Cartesian dimension, further improving speed and convergence when an exact Cartesian pose is not possible and/or necessary.

In recent years, a plan has been developed to conduct an investigation of an unknown environment by remotely controlling an exploration robot system sent to the Moon. For the robot to successfully perform sophisticated tasks, it must implement multiple degrees of freedom in not only its arms and legs but also its hands and fingers. Moreover, the robot has to be a humanoid type so that it can use tools used by astronauts, with no modification. On the other hand, to manipulate the multiple-degrees-of-freedom robot without learning skills necessary for manipulation and to minimize the psychological burden on operators, employing a method that utilizes everyday movements of operators as input to the robot, rather than a special controller, is ideal. In this paper, the authors propose a compact wearable device that allows for the estimation of the hand pose (hand motion capture) of the subject. The device has a miniature wireless RGB camera that is placed on the back of the user’s hand, rather than on the palm. Attachment of the small camera to the back of the hand may make it possible to minimize the restraint on the subject’s motions during motion capture. In the conventional techniques, the camera is attached on the palm because images of fingertips always need to be captured by the camera before images of any other part of the hand can be captured. In contrast, the image processing algorithm proposed herein is capable of estimating the hand pose with no need for first capturing the fingertips.

Simple and effective deep hand shape and pose regression from a single depth image

Having a virtual body can increase embodiment in virtual reality (VR) applications. However, comsumer-grade VR falls short of delivering sufficient sensory information for full-body motion capture. Consequently, most current VR applications do not even show arms, although they are often in the field of view. We address this shortcoming with a novel human upper-body inverse kinematics algorithm specifically targeted at tracking from head and hand sensors only. We present heuristics for elbow positioning depending on the shoulder-to-hand distance and for avoiding reaching unnatural joint limits. Our results show that our method increases the accuracy compared to general inverse kinematics applied to human arms with the same tracking input. In a user study, participants preferred our method over displaying disembodied hands without arms, but also over a more expensive motion capture system. In particular, our study shows that virtual arms animated with our inverse kinematics system can be used for applications involving heavy arm movement. We demonstrate that our method can not only be used to increase embodiment, but can also support interaction involving arms or shoulders, such as holding up a shield.

The Electromyography (EMG) signal is a nonstationary bio-signal based on the measurement of the electrical activity of the muscles. EMG based recognition systems play an important role in many fields such as diagnosis of neuromuscular diseases, human-computer interactions, console games, sign language detection, virtual reality applications, and amputee device controls. In this study, a novel approach based on deep learning has been proposed to improve the accuracy rate in the prediction of hand movements. Firstly, 4-channel surface EMG (sEMG) signals have been measured while simulating 7 different hand gestures (Extension, Flexion, Open Hand, Punch, Radial Deviation, Rest, and Ulnar Deviation) from 30 participants. The obtained sEMG signals have been segmented into sections where each movement was found. Then, spectrogram images of the segmented sEMG signals have been created by means of ShortTime Fourier Transform (STFT). The created colored spectrogram images have trained with 50-layer Convolutional Neural Network (CNN) based on Residual Networks (ResNet) architecture. Owing to the proposed method, test accuracy of 99.59% and F1 Score of 99.57% have achieved for 7 different hand gesture classifications.

Robotic manipulation in surgical applications often demands the surgical instrument to pivot around a fixed point, known as remote center of motion (RCM). The RCM constraint ensures that the pivot point of the surgical tool remains stationary at the incision port, preventing tissue damage and bleeding. Precisely and efficiently controlling tool positioning and orientation under this constraint poses a complex Inverse Kinematics (IK) problem that must be solved in real-time to ensure patient safety and minimize complications. To address this problem, we propose PivotIK, an efficient IK solver that combines efficient evolutionary exploration with multi-objective Jacobian-based optimization. PivotIK can track desired tool poses accurately while satisfying the RCM constraint in real-time. We evaluated PivotIK through simulations and real-world experiments using redundant robotic manipulators with multi-degree-of-freedom surgical instruments. We compare PivotIK with other IK solvers in terms of solve rates, computation times, tracking errors, and RCM errors under various scenarios, including unconstrained and RCM-constrained trajectories. Our results show that PivotIK achieves superior performance, solving the IK problem in less than 1 ms with errors below 0.01 and 0.1 mm for tracking and RCM, respectively. Our real-world experiments confirm the effectiveness of PivotIK in ensuring smooth trajectory tracking and RCM compliance. PivotIK offers a promising solution for real-time IK for robotic manipulation under RCM constraints in surgical applications.

Hand pose estimation based on 2D RGB images has drawn increasing research interest due to its many practical applications, such as Human-Computer Interaction (HCI) and Virtual Reality (VR). However, most existing methods focus on learning hand structure and key point representations, which cannot well exploit the joint interdependency of 2D occluded hand pose estimation. In this paper, we propose an adaptive joint interdependency learning network (AJIL) for 2D occluded hand pose estimation by adaptively learning hand joint interdependency, including three sub-networks. First, a cascade multi-task mask-learning subnetwork is used to learn hand pose structure. Then, a modified transformer encoder is designed to exploit the high spatial relationship between the hand joints. Lastly, the joint correlation is obtained from the multi-view hand pose images via 21 long short-term memory (LSTM). Extensive studies on three widely used datasets including the CMU Panoptic Hand, Large-Scale Multiview Hand Pose, and also our newly established pen-holding hand pose (PHHP) images dataset which is conducted to evaluate our proposed method. Experimental results show that our proposed method can achieve a very competitive 2D hand pose estimation performance when compared with the baseline models.KeywordsHand pose estimation2D occluded hand pose estimationHand joint interdependency learning

Hand motion capture is an important yet challenging topic for biomechanics and human computer interaction. We proposed a novel electrical sensing technology for capturing the finger angles from the variation of the wrist shape. The proposed device detects the signal related to the wrist-electrode contact resistances, which change according to the variation of the wrist shape accompanying finger movements. The developed sensing device consists of a wrist band, sixteen electrodes and a sensing circuit of contact resistances. We investigated the relationships between the finger angles and the system outputs by using a glove-type joint angle sensor. As a result, we confirmed high correlations of the system outputs with the finger angles for several electrodes. Therefore, we conclude that the proposed system can be used for the estimation of the finger joint angles.

In the framework of virtual prototyping, a virtual human is a useful tool to evaluate what one can reach or see. However because engineers and industrial designers are not necessarily skilled animators, it is important to provide intuitive or high-level interfaces to control virtual characters. With those considerations in mind, in this thesis we investigate three interfaces to control virtual humans for reaching tasks. First we propose an online full-body control using motion capture technology. The posture is reconstructed with an inverse kinematics solver. Coupled with a collision avoidance module, it is a powerful interface to control interactively virtual humans in cluttered environment. Secondly, we propose a novel interface for the full-body online control of differing-height characters. Instead of scaling the markers to the size of the virtual human, we suggest inversely scaling the virtual world. We conduct a user-study to evaluate if such interface is susceptible to help the user experiences the efforts involved by the differing-height character to complete a reaching task. Finally, we investigate motion graph technology for navigation coupled with reaching motions.

The use of the human hand as a natural interface device serves as a motivating force for research in the modeling, analysis and capture of the motion of an articulated hand. Model-based hand motion capture can be formulated as a large nonlinear programming problem, but this approach is plagued by local minima. An alternative way is to use analysis-by-synthesis by searching a huge space, but the results are rough and the computation expensive. In this paper, articulated hand motion is decoupled, a new two-step iterative model-based algorithm is proposed to capture articulated human hand motion, and a proof of convergence of this iterative algorithm is also given. In our proposed work, the decoupled global hand motion and local finger motion are parameterized by the 3D hand pose and the state of the hand respectively. Hand pose determination is formulated as a least-median-of-squares (LMS) problem rather than the nonrobust least-squares (LS) problem, so that 3D hand pose can be reliably calculated even if there are outliers. Local finger motion is formulated as an inverse kinematics problem. A genetic algorithm-based method is proposed to find a sub-optimal solution of the inverse kinematics effectively. Our algorithm and the LS-based algorithm are compared in several experiments. Both algorithms converge when local finger motion between consecutive frames is small. When large finger motion is present, the LS-based method fails, but our algorithm can still estimate the global and local finger motion well.

This paper presents a method for vision based estimation of the pose of human hands in interaction with objects. Despite the fact that most robotics applications of human hand tracking involve grasping and manipulation of objects, the majority of methods in the literature assume a free hand, isolated from the surrounding environment. Our hand tracking method is non-parametric, performing a nearest neighbor search in a large database (100000 entries) of hand poses with and without grasped objects. The system operates in real time, it is robust to self occlusions, object occlusions and segmentation errors, and provides full hand pose reconstruction from markerless video. Temporal consistency in hand pose is taken into account, without explicitly tracking the hand in the high dimensional pose space.

Hands-free is very important in the design of human-computer interaction (HCI) in the application of virtual reality or augmented reality (AR). Head tracker with six-degrees-of-freedom (6-DOF) is common used in AR, without additional devices, this paper presents a new and practical approach to design HCI by virtue of it. The design includes two steps: the first is the calibration or initialization, in this stage the rotate coordinate planes or axes for every HCI actions should be calculated; the second is the recognition of HCI, by analyzing the motion trajectory of user's head to judge whether it matches one of the interactions. For the effect of this design, this paper uses a test program and a practical application to demonstrate it.

Feedback Control—Based Inverse Kinematics Solvers for a Nuclear Decommissioning Robot

In order to grasp an object, we need to solve the inverse kinematics problem, i.e., the coordinate transformation from the visual coordinates to the joint angle vector coordinates of the arm. In human motion control, the learning of the hand position error feedback controller in the inverse kinematics solver is important. Although several models of coordinate transformation learning have been proposed, they suffer from a number of drawbacks. This paper proposes a novel model of the coordinate transformation learning of the human visual feedback controller by using disturbance noise and feedback error signal. The feasibility of the proposed model is illustrated using numerical simulations.