Hand Gestures Classification Using Electrical Impedance Tomography Images

A human gesture prediction system can be used to estimate human gestures in advance of the actual action to reduce delays in interactive systems. Hand gestures are particularly necessary for human–computer interaction. Therefore, the gesture prediction system must be able to capture hand movements that are both complex and quick. We have already reported a method that allows strain sensors and wearable devices to be fabricated in a simple and easy manner using pyrolytic graphite sheets (PGSs). The wearable electronics could detect various types of human gestures with high sensitivity, high durability, and fast response. In this study, we demonstrated hand gesture prediction by artificial neural networks (ANNs) using gesture data obtained from data gloves based on PGSs. Our experiments entailed measuring the hand gestures of subjects for learning purposes and we used these data to create four-layered ANNs, which enabled the proposed system to successfully predict hand gestures in real time. A comparison of the proposed method with other algorithms using temporal data analysis suggested that the hand gesture prediction system using ANNs would be able to forecast various types of hand gestures using resistance data obtained from wearable devices based on PGSs.

Recently, human–robot interaction technology has been considered as a key solution for smart factories. Surface electromyography signals obtained from hand gestures are often used to enable users to control robots through hand gestures. In this paper, we propose a dynamic hand-gesture-based industrial robot control system using the edge AI platform. The proposed system can perform both robot operating-system-based control and edge AI control through an embedded board without requiring an external personal computer. Systems on a mobile edge AI platform must be lightweight, robust, and fast. In the context of a smart factory, classifying a given hand gesture is important for ensuring correct operation. In this study, we collected electromyography signal data from hand gestures and used them to train a convolutional recurrent neural network. The trained classifier model achieved 96% accuracy for 10 gestures in real time. We also verified the universality of the classifier by testing it on 11 different participants.

The relationship between humans and computers is called human computer interaction (HCI). HCI is a major research topic in the area of assistive technology. In the field of medical assistive technology specifically, a hand gesture is considered a suitable method to convey information. It can help elderly people, who are unable to walk or speak, communicate with caregivers whenever they need help. This paper proposes a system to recognize the hand gestures of elderly people using an inexpensive Raspberry Pi. A vision-based algorithm is developed to detect and classify dynamic hand gestures in real time on the Raspberry Pi embedded platform. There are three main procedures; contour detection, convex extraction, and rule-based classification. The system can detect six different gestures on both hands in various orientations. The experiment showed good results in detecting and classifying their meanings as lingual descriptions.

Since the 1970s, the field of gesture recognition and its applications has been at the centre of considerable research in human–computer interaction. Researchers have been able to construct strong models that can recognize gestures in real time thanks to recent advances in deep learning and computer vision, but they face hurdles when it comes to classifying gestures in variable lighting conditions. In this paper, we train a 3D convolutional neural network to recognize dynamic hand gestures in real time. Our focus is on ensuring that gesture recognition systems can perform well under varying light conditions. We use a huge training set consisting of numerous clips of people performing specific gestures in varying lighting conditions to train the model. We were able to attain an accuracy of 76.40% on the training set and 66.56% on the validation set with minimal pre-processing applied to the data set. The trained model was able to successfully recognize hand gestures recorded from a Webcam in real time. We were then able to use the model’s predictions to control video playback on the VLC media player such as increasing/decreasing volume and pausing video. These experimental results show the effectiveness and efficiency of the proposed framework to recognize gestures in both bright and dim lighting conditions.KeywordsDeep learningConvolutional neural networksPyTorch3D CNN

Any human–computer interaction application needs to be able to recognize gestures. Hand gesture detection systems that recognize gestures in real time can improve human–computer interaction by making it more intuitive and natural. Colour gloves and skin colour detection are two prominent hand segmentation and detection technologies, but each has its own set of benefits and drawbacks. For physically challenged people, gesture identification is a crucial technique of sharing information. Support vector machine algorithm with principal component analysis, hidden Markov model, superposed network with multiple restricted Boltzmann machines, growing neural gas algorithm, convolutional neural network, double channel convolutional neural network, artificial neural network, and linear support vector machine with gesture dataset matches the interaction of gestures for various postures in real time. Although this method can recognize a huge number of gestures, it does have certain downsides, such as missing movements due to the accuracy of the categorization algorithms. Furthermore, matching the vast dataset takes a longer time. The main contribution of this work lies in a conceptual framework based on the findings of a systematic literature review that provides fruitful implications based on recent research findings and insights that can be used to direct and initiate future research initiatives in the field of hand gesture recognition and artificial intelligence. As a result, a novel study based on a hybrid recurrent neural network (RNN) with chaos game optimization may reduce classification mistakes, increase stability, maximize resilience, and efficiently use the hand gestures recognition system.

In recent years, technologies such as Augmented Reality (AR) and Virtual Reality (VR) have become more popular and available to broader audiences, leading to the research and development of a myriad of extended applications. A modality for end-users to interact with these applications is through hand gestures, hence the importance of detecting the different gestures in real time. This paper presents an interval type-2 Fuzzy Rule-based System (FRBS) optimised by the Big Bang-Big Crunch (BB-BC) algorithm that uses the fingers’ position from the hand-tracking technology in extended reality (XR) headsets (namely HoloLens 2 and Oculus Quest 2) to classify the user’s hand gestures. This approach achieved an accuracy of 96.4%, and it is an interpretable model that can be understood and adjusted by end-users. The interval type-2 FRBS was tested against a type-1 FRBS and a k-nearest neighbours (KNN) model. It outperformed the type-1 FRBS and was close to the 98.9% accuracy performance of the KNN model, making our suggested approach a competitive alternative to opaque models.

The sign language digits based on hand gestures have been utilized in various applications such as human-computer interaction, robotics, health and medical systems, health assistive technologies, automotive user interfaces, crisis management and disaster relief, entertainment, and contactless communication in smart devices. The color and depth cameras are commonly deployed for hand gesture recognition, but the robust classification of hand gestures under varying illumination is still a challenging task. This work presents the design and deployment of a complete end-to-end edge computing system that can accurately provide the classification of hand gestures captured from thermal images. A thermal dataset of 3200 images was created with each sign language digit having 320 thermal images. The solution presented here utilizes live images taken from a low-resolution thermal camera of 32×32 pixels, feeding into a novel light weight deep learning model based on bottleneck motivated from deep residual learning for classification of hand gestures. The edge computing system presented here utilizes Raspberry pi with a thermal camera making it highly portable. The designed system achieves an accuracy of 99.52% on the test data set with an added advantage of accuracy being invariable to background lighting conditions as it is based on thermal imaging.

Classification of hand gesture is crucial for the development of hand gesture based system for human machine interaction. Gesture recognition system consists of hand gesture acquisition, segmentation, morphological filtering, contour extraction, and classification. This paper aims at classification of hand gesture as a similarity measure using Dynamic Time Warping and Piecewise Dynamic Time Warping. Experiments and evaluation on a subset of American Sign Language (ASL) hand gesture show that, by using Dynamic Time Warping hand gesture can be classified. Additionally, it is also estimated that Piecewise DTW can be efficiently used to speed up the computations of DTW.

The virtual mouse is an important research topic in HCI (Human Computer Interaction) technology, and the most popular method is based on hand gestures. In this paper, a six-degree-of-freedom virtual mouse based on hand gestures is presented using an inexpensive USB webcam. The hand tracking and gesture recognition framework includes the motion, skin color and finger information of the hand. It can even provide computer access for the disabled people who cannot bend their fingers at all with severe disabilities. The experiments show that the system has good interactive input functions in real time.

We present a pipeline for recognizing dynamic freehand gestures on mobile devices based on extracting depth information coming from a single Time-of-Flight sensor. Hand gestures are recorded with a mobile 3D sensor, transformed frame by frame into an appropriate 3D descriptor and fed into a deep LSTM network for recognition purposes. LSTM being a recurrent neural model, it is uniquely suited for classifying explicitly time-dependent data such as hand gestures. For training and testing purposes, we create a small database of four hand gesture classes, each comprising 40 \(\times \) 150 3D frames. We conduct experiments concerning execution speed on a mobile device, generalization capability as a function of network topology, and classification ability ‘ahead of time’, i.e., when the gesture is not yet completed. Recognition rates are high (>95%) and maintainable in real-time as a single classification step requires less than 1 ms computation time, introducing freehand gestures for mobile systems.

Real-time monitoring of upper airway collapse during sleep could be instrumental for studies in biomechanics of obstructive sleep apnea (OSA) and selecting individualized treatment modalities. Although some imaging techniques are used under sedated sleep, none are available during the entire natural sleep process. We hypothesized that electrical impedance tomography (EIT) can be used for noninvasive continuous imaging of the upper airway during natural sleep and quantifying upper airway collapse in terms of its size. After determining surface landmarks to attach the electrodes for monitoring the retroglossal airway, EIT was conducted in 10 healthy participants. As a feasibility test of EIT in detecting upper airway collapse, transient airway closure was induced by the swallowing maneuver. These EIT images were confirmed by simultaneous magnetic resonance imaging (MRI) scans. Subsequently, EIT scans were conducted in 7 healthy participants and 10 patients with OSA under nonsedated sleep to determine whether it could identify upper airway narrowing or collapse. Respiratory events were identified by concurrent polysomnography (PSG). Swallowing-induced airway closure was identified successfully in all 10 participants on simultaneous EIT and MRI scans. Sizes and positions of the upper airway closures in reconstructed EIT images were well correlated with those in magnetic resonance images. Obstructive hypopnea and apnea were detected successfully by EIT in 10 patients with OSA, and no significant changes in EIT data were observed in 7 healthy participants during concurrent EIT and PSG tests. Additionally, conductivity changes in the airway were greater during obstructive apnea than during hypopnea (64.3% versus 26.3%, respectively; P < .001) compared with those during baseline respiration. EIT could be a useful real-time monitoring device for detecting upper airway narrowing or collapse during natural sleep in patients with OSA. Currently, changes in the upper airway size can be estimated with good accuracy, but shape estimation needs future improvements in the EIT image quality.

We read with interest the recent article by Zarantonello et al.1 on the physiologic effects of prone positioning on ventilation and perfusion distribution, oxygenation, and lung mechanics in COVID-19 acute respiratory distress syndrome patients. We would like to first congratulate the authors for the nicely conducted study that once again proved the clinical applicability of electrical impedance tomography for the assessment of ventilation–perfusion matching at the bedside. We acknowledge the clinical significance of the findings, but we wish to point out that several parts of the electrical impedance tomography methodology were incorrectly described in the article.In the Methods section, the authors described how the lung areas that were (1) perfused but not ventilated, (2) ventilated but not perfused, and (3) both ventilated and perfused were “divided by the sum of ventilated and perfused pixels” to calculate the corresponding relative fractions of the lung area in percentages, i.e., (1) “% shunt area,” (2) “% dead space area,” and (3) “% matched area,” respectively. However, the denominator must be the number of pixels that were ventilated or perfused, i.e., using the logical operator “or,” otherwise the calculated relative areas would be incorrect. The authors also determined “the percentage of total ventilated lung area that was located in the dorsal half of the thorax,” which they inaccurately termed as “dorsal ventilation.” According to the consensus electrical impedance tomography terminology and definitions,2 regional ventilation is determined as the sum of all pixel values of tidal impedance variation not as the pixel count, which defines just the ventilated area.The authors further introduced a new electrical impedance tomography parameter, the “homogeneity index,” which they “calculated as 1 – global inhomogeneity index,” based on the assumption of the global inhomogeneity index ranging from 1 to 0. However, the global inhomogeneity index values are commonly higher than 1 in patients with pronounced ventilation inhomogeneity, as shown already in some of the examined mechanically ventilated subjects in the original description of the global inhomogeneity index3 or recently in patients with exacerbated chronic lung disease.4 An example patient case is presented in figure 1, in which a global inhomogeneity index value of 2.59 was noted in the presence of pendelluft, which is a frequent cause of out-of-phase impedance variation both in spontaneously breathing and ventilated patients.5 The “homogeneity index” proposed by the authors would be negative in this case and assume the confusing value of –1.59. We therefore strongly advise against using the “homogeneity index” as a surrogate for the well established global inhomogeneity index.The authors pointed out that electrical impedance tomography perfusion assessment with saline bolus injection had been validated only in two animal studies. In fact, this technique of contrast-enhanced electrical impedance tomography has been studied in several other animal studies as well.6–9 Last, we agree with the authors that electrical impedance tomography images do not cover the whole lungs, but as summarized in the consensus paper on chest electrical impedance tomography,2 an electrical impedance tomography image does not originate from “only the area of the lung surrounded by the belt.”Electrical impedance tomography receives increasing attention due to its noninvasive bedside monitoring features. Correct description and implementation of the electrical impedance tomography technique and analysis procedures are crucial for further development and clinical use.Dr. Zhao receives a consulting fee from Dräger Medical (Lübeck, Germany). Dr. Frerichs reports funding from the European Union’s Framework program for Research and Innovation Horizon 2020 (WELMO, grant No. 825572).

Industrial robots and cobots are widely deployed in most industrial sectors. However, robotic programming still needs a lot of time and effort in small batch sizes, and it demands specific expertise and special training, especially when various robotic platforms are required. Actual low-code or no-code robotic programming solutions are exorbitant and meager. This work proposes a novel approach for no-code robotic programming for end-users with adequate or no expertise in industrial robotic. The proposed method ensures intuitive and fast robotic programming by utilizing a finite state machine with three layers of natural interactions based on hand gesture, finger gesture, and voice recognition. The implemented system combines intelligent computer vision and voice control capabilities. Using a vision system, the human could transfer spatial information of a 3D point, lines, and trajectories using hand and finger gestures. The voice recognition system will assist the user in parametrizing robot parameters and interacting with the robot’s state machine. Furthermore, the proposed method will be validated and compared with state-of-the-art “Hand-Guiding” cobot devices within real-world experiments. The results obtained are auspicious, and indicate the capability of this novel approach for real-world deployment in an industrial context.

Conventional pattern-recognition algorithms for surface electromyography (sEMG)-based hand-gesture classification have difficulties in capturing the complexity and variability of sEMG. The deep structures of deep learning enable the method to learn high-level features of data to improve both accuracy and robustness of a classification. However, the features learned through deep learning are incomprehensible, and this issue has precluded the use of deep learning in clinical applications where model comprehension is required. In this paper, a generative flow model (GFM), which is a recent flourishing branch of deep learning, is used with a SoftMax classifier for hand-gesture classification. The proposed approach achieves accuracy in classifying 53 different hand gestures from the NinaPro database 5. The distribution of all 53 hand gestures is modelled by the GFM, and each dimension of the feature learned by the GFM is comprehensible using the reverse flow of the GFM. Moreover, the feature appears to be related to muscle synergy to some extent.

Myoelectric prostheses are a viable solution for people with amputations. The challenge in implementing a usable myoelectric prosthesis lies in accurately recognizing different hand gestures. The current myoelectric devices usually implement very few hand gestures. In order to approximate a real hand functionality, a myoelectric prosthesis should implement a large number of hand and finger gestures. However, increasing number of gestures can lead to a decrease in recognition accuracy. In this work a Myo arm band device is used to recognize fourteen gestures (five build in gestures of Myo armband in addition to nine new gestures). The data in this research is collected from three body-able subjects for a period of 7 seconds per gesture. The proposed method uses a pattern recognition technique based on Multi-Layer Perceptron Neural Network (MLPNN). The results show an average accuracy of 90.5% in recognizing the proposed fourteen gestures.