Hand pose estimation and motion recognition using egocentric RGB-D video

The latest advances in smart sensor technology, e.g., Leap Motion Sensor, has increased the precision in tracking fully articulated human hand and finger movements, without the need for placing electrical or optical markers. A remaining challenge is finger occlusion, which can affect tracking accuracy. In this paper, we introduce a spatio-temporal optimization technique for motion data generated from multiple sensors. We demonstrate that our algorithm can produce a fused stream of probabilistic optimal hand poses, by improving local spatial domain analysis and proposing a fast and effective flow analysis technique in the temporal domain, which computes how well the hand pose estimation in the current frame fits the movement flow within a time segment. By using an artificial hand to represent the hand pose ground truth at selected time steps, experimental results demonstrate that our spatio-temporal optimization algorithm increases the estimation accuracy by 6% compared to the reference method, achieving an overall accuracy of 91.29%. Our proposed method can be used offline or in real-time, and can benefit a wide range of applications, including surgical planning and training, where hand motion is the focus of performance efficiency and assessment.

Surface electromyography (sEMG) signals are an indirect measurement of muscle activity, and their applications range from biomechanics to control and rehabilitation. Hand movement recognition is a very difficult endeavor due to forearm anatomy. Hence, a multichannel approach for signal acquisition and processing is required. Conventional electrodes can limit the ease-of-use and repeatability of multi-channel sEMG recordings. New techniques have been proposed in this regard, with dry electrodes being one of them. Dry electrode technology has enabled the design of better donning and doffing procedures for multichannel sEMG recording, particularly for rehabilitation and prosthetic applications. However, there is a debate about the quality of the signals recorded with them and their usefulness for the recognition of multiple hand movements. To mitigate these quality issues, this work proposes an array of reusable stainless steel dry electrodes for multichannel sEMG recording with a design that facilitates its positioning on the forearm. The dry electrodes were characterized through electrical impedance measures and a Bland-Altman test. They were found to have similar characteristics to standard, disposable sEMG pre-gelled electrodes. For placement repeatability and application feasibility, an anatomy-based electrode positioning protocol was implemented with 17 healthy subjects and six hand movements. To evaluate the application feasibility of the electrode array, a feed-forward artificial neural network was trained to classify signals from the six movements, with a 97,86±0,58% accuracy. The amplitude of the sEMG signals for two antagonist movements was compared, finding a 24,81% variation. The dry electrode array showed feasibility in acquiring and classifying sEMG signals of hand movements with high accuracy.

Hand pose estimation from 3D data is a key challenge in computer vision as well as an essential step for human–computer interaction. A lot of deep learning-based hand pose estimation methods have made significant progress but give less consideration to the inner interactions of input data, especially when consuming hand point clouds. Therefore, this paper proposes an end-to-end capsule-based hand pose estimation network (Capsule-HandNet), which processes hand point clouds directly with the consideration of structural relationships among local parts, including symmetry, junction, relative location, etc. Firstly, an encoder is adopted in Capsule-HandNet to extract multi-level features into the latent capsule by dynamic routing. The latent capsule represents the structural relationship information of the hand point cloud explicitly. Then, a decoder recovers a point cloud to fit the input hand point cloud via a latent capsule. This auto-encoder procedure is designed to ensure the effectiveness of the latent capsule. Finally, the hand pose is regressed from the combined feature, which consists of the global feature and the latent capsule. The Capsule-HandNet is evaluated on public hand pose datasets under the metrics of the mean error and the fraction of frames. The mean joint errors of Capsule-HandNet on MSRA and ICVL datasets reach 8.85 mm and 7.49 mm, respectively, and Capsule-HandNet outperforms the state-of-the-art methods on most thresholds under the fraction of frames metric. The experimental results demonstrate the effectiveness of Capsule-HandNet for 3D hand pose estimation.

Capturing the human hand motion from video involves the estimation of the rigid global hand pose as well as the nonrigid finger articulation. The complexity induced by the high degrees of freedom of the articulated hand challenges many visual tracking techniques. For example, the particle filtering technique is plagued by the demanding requirement of a huge number of particles and the phenomenon of particle degeneracy. This paper presents a novel approach to tracking the articulated hand in video by learning and integrating natural hand motion priors. To cope with the finger articulation, this paper proposes a powerful sequential Monte Carlo tracking algorithm based on importance sampling techniques, where the importance function is based on an initial manifold model of the articulation configuration space learned from motion-captured data. In addition, this paper presents a divide-and-conquer strategy that decouples the hand poses and finger articulations and integrates them in an iterative framework to reduce the complexity of the problem. Our experiments show that this approach is effective and efficient for tracking the articulated hand. This approach can be extended to track other articulated targets.

Background Stroke is a leading cause of disability, often accompanied by unilateral spatial neglect (USN), which severely impairs recovery. Traditional assessments like paper-pencil tests provide limited insights into behaviors and eye–hand coordination during real-world tasks. Advances in hand pose estimation and eye tracking in combination with augmented reality (AR) offer potential for data-driven assessments of naturalistic interactions. Objective This proof-of-concept study presents and evaluates a multimodal behavioral tracking system that captures gaze, body, and hand movements during interactions within an AR environment. Our primary goals are to (1) validate that this system can achieve robust and accurate interaction data capture in clinical settings, (2) show that the system can reliably detect known USN behavioral patterns, and (3) explore how comprehensive data can provide new understanding of eye–hand coordination deficits in USN. Methods We developed an AR-based assessment system using Microsoft HoloLens 2 and an external body-tracking camera to capture real-time gaze, hand, and body movements in an interactive environment. Multimodal data streams were temporally synchronized, fused, and filtered to enhance spatial accuracy and availability. Tracking performance was benchmarked against a traditional optical motion-capture system to validate reliability. In a study, 7 patients with right-brain lesions with mild to moderate USN and 8 healthy controls participated. Each performed a designed reaching task, stamping virtual sheets of paper that appeared randomly on a table. We analyzed participants’ search behavior patterns to assess attentional biases and examined gaze anchoring timing during targeted reaching motions to explore potential eye–hand coordination deficits. Results The fusion of hand-tracking data from the HoloLens 2 and external system reduced tracking loss from 25.7% to 2.4%, with an absolute trajectory error of 3.27 cm. The system demonstrated high usability and was well accepted by patients. Data from the control group confirmed the absence of intrinsic lateral biases in the system and task design. The USN group displayed typical search behavior through ipsilesional biases in gaze direction during visual exploration (median deviation 7.46 [1.61-9.48] deg, P <.05) and longer times to find contralesional targets (median difference 1.08 [0.20-1.80] s, P =.02). Additionally, the eye–hand coordination analysis revealed lateral differences in gaze anchoring during targeted reaching motions in the USN group, with earlier fixation on contralesional targets (median difference 112 [71-146] ms, P =.02). Conclusions The proposed AR framework provides a novel, comprehensive data-driven method for capturing interaction behavior in a controlled, yet naturalistic environment. Our results demonstrated the system’s effectiveness in measuring hallmark USN symptoms, such as gaze and head orientation biases, and highlighted its potential to complement traditional assessments by offering deep insights into torso rotation and eye–hand coordination with a high resolution and accuracy. This data-driven approach shows promise for enhancing current USN assessment practices and gaining new insights into patients’ behaviors.

\n Hand pose estimation is a challenging problem in computer vision with a wide range of applications, especially in human-computer interface. With the development of inexpensive consumer-level depth cameras and the evolution on deep learning techniques, the current state-of-art in the problem is continuously developing and several new methods have been proposed in recent years. Those methods are mostly data-driven and reach good results in standard datasets such as NYU, ICVL and HANDS17. An application that would benefit from the use of computer vision techniques is hand occupational therapy. In chronic diseases like rheumatoid arthritis (RA), the evaluation of the hand functional state is fundamental for the treatment and prevention of finger deformities. One of the procedures for deformity diagnosis is the measurement of movement angles i.e. flexion/extension and abduction/addution, made using goniometers in a process that can be time-consuming and invasive for the patient. The main proposal of this PhD is to fill a gap in the literature by proposing and evaluating the viability of using a framework composed of a 3D low-cost sensor and a 3D hand pose estimation state-of-art method for automatic assessment of rheumatoid arthritis patients. Given depth maps as input, our framework estimates 3D hand joint positions using a deep convolutional neural network. The proposed pose estimation algorithm can be executed in real-time, allowing users to visualise 3D skeleton tracking results at the same time as the depth images are acquired. Once 3D joint poses are obtained, our framework estimates flexion/extension and abduction/adduction angles by applying computational geometry oper- ations. The absence of public datasets with RA patients in the literature makes the estimation of hand poses of patients a challenge for computer vision data-driven methods. We therefore proposed a protocol to acquire new data from groups of patients and control. We performed experiments of identification of RA patients and control sets and also performed comparison with goniometer data. Results show that a method based on Fourier descriptors is able to perform automatic discrimination of hands with Rheumatoid Arthritis (RA) and healthy pa- tients. The angle between joints can be used as an indicative of current movement capabilities and function. The acquisition is much easier, non-invasive and patient-friendly, significantly reducing the evaluation time and offering real-time data for the dynamic movement.\n

Upper limb prosthesis control with pattern recognition (PR)- based on surface Electromyogram (EMG) signals has been heavily investigated in the literature. However, challenges related to signal non-stationarity and its impact by changing force levels, limb positions and many other factors all lead to poor system stability, prevent the widespread adoption of this technology. In this study, we propose an alternative modality based on Acoustic Myography (AMG), a recording of muscle vibrations at low frequencies, as a control signal to decipher the intended hand movements. A custom-built AMG armband, consisting of 4 microphones, has been developed in this study and evaluated on intact-limbed subjects performing six classes of hand and finger movements. Time domain and auto-regression (TD-AR) features, in addition to Bi long-short term memory (BiLSTM) deep learning classifier were utilized to perform the classification. An average classification accuracy of 85% was obtained, which shows the potential of using the developed AMG armband, with PR system for prostheses control.

Hand pose estimation based on 2D RGB images has drawn increasing research interest due to its many practical applications, such as Human-Computer Interaction (HCI) and Virtual Reality (VR). However, most existing methods focus on learning hand structure and key point representations, which cannot well exploit the joint interdependency of 2D occluded hand pose estimation. In this paper, we propose an adaptive joint interdependency learning network (AJIL) for 2D occluded hand pose estimation by adaptively learning hand joint interdependency, including three sub-networks. First, a cascade multi-task mask-learning subnetwork is used to learn hand pose structure. Then, a modified transformer encoder is designed to exploit the high spatial relationship between the hand joints. Lastly, the joint correlation is obtained from the multi-view hand pose images via 21 long short-term memory (LSTM). Extensive studies on three widely used datasets including the CMU Panoptic Hand, Large-Scale Multiview Hand Pose, and also our newly established pen-holding hand pose (PHHP) images dataset which is conducted to evaluate our proposed method. Experimental results show that our proposed method can achieve a very competitive 2D hand pose estimation performance when compared with the baseline models.KeywordsHand pose estimation2D occluded hand pose estimationHand joint interdependency learning

Hand motion recognition is pivotal for enhancing skill learning in endovascular surgery. However, catheterization procedures typically involve multimodal sources that introduce data dissimilarities, leading to unstable and poor performance in learning-based methods used for recognizing surgical tasks. This study proposes a resolution-based dynamic time warping (rDTW) method for aligning multimodal signals to improve hand motion recognition in endovascular surgery. The rDTW model is designed to align multimodal signals from wearable sensors, referencing the resolution of a unique source. The reference signal traces the events, creating a consistent progression that can localize a surgeon’s hand motions across all signals. Other signals are stretched or folded to match the reference timeline, facilitating event localization relative to the reference signal. By preprocessing multimodal signals from 33 catheterization trials, we show that the proposed rDTW method enhances hand motion recognition performance in two neural network models. When the data was used directly for the motion recognition task without signal warping, a performance decrease of 2.59% was observed. Also, the rDTW method exhibits higher data similarity index and lower warping time compared to three existing DTW techniques, showing its effectiveness in aligning multimodal signals for surgeons’ hand motion recognition.

With the advantages of comfortable wearing and outdoor usage, the myoelectric gesture recognition techniques have gained much attention in the field of human-machine interaction (HMI). The purpose of this study is to optimize model structure and transfer generalized features to improve the robustness of myoelectric hand motion decoding. We derived the hand motion recognition framework from the muscle synergy theory, which is formulated as a temporal convolutional (TC) model of array sEMG signals, then a hierarchical myoelectric decoding model was proposed to predict simultaneous and continuous hand motion. The model was trained by the methods of unsupervised low-level feature learning and automated data labeling to minimize training supervision. Extensive experiments on the public sEMG database (17 subjects in Biopatrec) show that the TC model can extract muscle synergy features with higher fidelity ( R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.85±0.23) than the traditional instantaneous mixture model, the results of online test demonstrate robust myoelectric decoding on multiple simultaneous and continuous hand motions. More importantly, the analysis of weights visualization shows that the low-level feature representation layer of TC model can be migrated across the individuals, which provides a transferrable feature extraction layer for generalized hand motion decoding.

In this paper we are focusing on the discrimination and recognition of arm movements described in terms of the evolution of the joint angles during the trajectory of the movement. Considering this as a classification problem, we could use a three layered feedforward network trained by the backpropagation learning algorithm to discriminate between arm movements described in the representation proposed by Marr and Vaina (1982) and Vaina and Bennour (1985). We discuss the contribution of the sampling rate for the value of the joint angles in the input layers, the number of angles necessary for a good recognition of the movement. We introduce a new method for determining the size of the network at input stage, which eliminates the unnecessary data in the training set. The effect of changing the number of frames used to train the network for each movement and the properties of the backpropagation algorithm in this application are discussed. We demonstrate that a description of movements at multiple scales of resolution organized from general to particular is conducive to an efficient learning scheme in the network. The main result obtained is that the discrimination between different arm movements is most efficiently obtained at the appropriate and most meaningful scale of resolution, as predicted by the Marr and Vaina's model.

The use of the human hand as a natural interface device serves as a motivating force for research in the modeling, analysis and capture of the motion of an articulated hand. Model-based hand motion capture can be formulated as a large nonlinear programming problem, but this approach is plagued by local minima. An alternative way is to use analysis-by-synthesis by searching a huge space, but the results are rough and the computation expensive. In this paper, articulated hand motion is decoupled, a new two-step iterative model-based algorithm is proposed to capture articulated human hand motion, and a proof of convergence of this iterative algorithm is also given. In our proposed work, the decoupled global hand motion and local finger motion are parameterized by the 3D hand pose and the state of the hand respectively. Hand pose determination is formulated as a least-median-of-squares (LMS) problem rather than the nonrobust least-squares (LS) problem, so that 3D hand pose can be reliably calculated even if there are outliers. Local finger motion is formulated as an inverse kinematics problem. A genetic algorithm-based method is proposed to find a sub-optimal solution of the inverse kinematics effectively. Our algorithm and the LS-based algorithm are compared in several experiments. Both algorithms converge when local finger motion between consecutive frames is small. When large finger motion is present, the LS-based method fails, but our algorithm can still estimate the global and local finger motion well.

Recognition of the Surgeon's Motions During Endoscopic Operation by Statistics Based Algorithm and Neural Networks Based ANARX Models

Because handicapped people account for a large percentage of our community, we should make an effort to interact with them in order to exchange knowledge, perspectives, and ideas. To that aim, we wish to establish a means of contact. Individuals who are deaf or hard of hearing can communicate with one another using sign language. A handicapped person can communicate without using acoustic noises when they use sign language. The objective of this article is to explain the design and development of a hand gesture-based sign language recognition system. To aid handicapped individuals, particularly those who are unable to communicate verbally, sign language is translated into text and subsequently into speech. The solution is based on a web camera as the major component, which is used to record a live stream video using a proprietary MATLAB algorithm. Recognition of hand movements is possible with the technology. Recognizing hand gestures is a straightforward technique of providing a meaningful, highly flexible interaction between robots and their users. There is no physical communication between the user and the devices. A deep learning system that is efficient at picture recognition is used to locate the dynamically recorded hand movements. Convolutional neural networks are used to optimize performance. A static image of a hand gesture is used to train the model. Without relying on a pre-trained model, the CNN is constructed.KeywordsHuman–computer interactionGesture recognitionWeb cameraCNNMATLAB

The recognition of hand movements is an important method for human-computer interaction (HCI) technology, and it is widely used in virtual reality and other HCI areas. While many valuable efforts have been made, efficient ways to capture over 20 types of hand movements with high accuracy by one data glove are still lacking. This paper addresses a new classification framework for 52 hand movements. This classification framework includes the following two parts: the movement detection algorithm and the movement classification algorithm. The fine K-nearest neighbor (Fine KNN) is the core of the movement detection algorithm. The movement classification algorithm is composed of downsampling in data preparation and a new deep learning network named the DBDF network. Bidirectional Long Short-Term Memory (BiLSTM) is the main part of the DBDF network. The results of experiments using the Ninapro DB1 dataset demonstrate that our work can classify more types of hand movements than related algorithms with a precision of 93.15%.