Deep Learning-Based Sign Language Digits Recognition From Thermal Images With Edge Computing System

Classification of hand gesture is crucial for the development of hand gesture based system for human machine interaction. Gesture recognition system consists of hand gesture acquisition, segmentation, morphological filtering, contour extraction, and classification. This paper aims at classification of hand gesture as a similarity measure using Dynamic Time Warping and Piecewise Dynamic Time Warping. Experiments and evaluation on a subset of American Sign Language (ASL) hand gesture show that, by using Dynamic Time Warping hand gesture can be classified. Additionally, it is also estimated that Piecewise DTW can be efficiently used to speed up the computations of DTW.

In this paper, a multimodal hand gesture detection and recognition system using differential Pyroelectric Infrared (PIR) sensors and a regular camera is described. Any movement within the viewing range of the differential PIR sensors are first detected by the sensors and then checked if it is due to a hand gesture or not by video analysis. If the movement is due to a hand, one-dimensional continuous-time signals extracted from the PIR sensors are used to classify/recognize the hand movements in real-time. Classification of different hand gestures by using the differential PIR sensors is carried out by a new winner-take-all (WTA) hash based recognition method. Jaccard distance is used to compare the WTA hash codes extracted from 1-D differential infrared sensor signals. It is experimentally shown that the multimodal system achieves higher recognition rates than the system based on only the on/off decisions of the analog circuitry of the PIR sensors(1).

Human hands can perform far more gestures than the number of muscles controlling them, as most gestures result from coordinated combinations of muscle activations and relaxations. This complexity poses a key challenge for human-machine interfaces performing gesture classification based on electromyography (EMG). Rather than identifying all conceivable gestures, it may be simpler to instead identify the activity of the individual muscles which generate a variety of complicated gestures. Here we suggest a three-dimensional model with volume representations of individual digit extensor muscles, averaged across multiple individuals, and evaluate its application and performance in hand gesture classification. Time-domain peaks in high-density surface EMG data from different hand gestures were extracted and localized within the model, from which a gesture classification scheme was generated for both single and multi-label cases. The model was created and tested on a publicly available dataset with 19 participants, leveraging a leave-one-out approach to assess inter-subject generalizability, and multi-label data to assess generalizability to gestures not included in the creation of the model. For single-label classification performance, true positive rates were between 61.9 and 95.1%, with false positive rates between 0 and 24.1%, for different single-digit extensions. The multi-label test demonstrated some degree of generalizability in identifying completely new gesture compositions, while simultaneously maintaining the leave-one-out approach for inter-subject generalizability. A model generated with this approach could be used for gesture classification by anyone, without individual modelling data, with the potential to generalize to any number of gesture compositions.

Conventional pattern-recognition algorithms for surface electromyography (sEMG)-based hand-gesture classification have difficulties in capturing the complexity and variability of sEMG. The deep structures of deep learning enable the method to learn high-level features of data to improve both accuracy and robustness of a classification. However, the features learned through deep learning are incomprehensible, and this issue has precluded the use of deep learning in clinical applications where model comprehension is required. In this paper, a generative flow model (GFM), which is a recent flourishing branch of deep learning, is used with a SoftMax classifier for hand-gesture classification. The proposed approach achieves accuracy in classifying 53 different hand gestures from the NinaPro database 5. The distribution of all 53 hand gestures is modelled by the GFM, and each dimension of the feature learned by the GFM is comprehensible using the reverse flow of the GFM. Moreover, the feature appears to be related to muscle synergy to some extent.

Hand gesture for communication has proven effective for humans, and active research is ongoing in replicating the same success in computer vision systems. Human–computer interaction can be significantly improved from advances in systems that are capable of recognizing different hand gestures. In contrast to many earlier works, which consider the recognition of significantly differentiable hand gestures, and therefore often selecting a few gestures from the American Sign Language (ASL) for recognition, we propose applying deep learning to the problem of hand gesture recognition for the whole 24 hand gestures obtained from the Thomas Moeslund’s gesture recognition database. We show that more biologically inspired and deep neural networks such as convolutional neural network and stacked denoising autoencoder are capable of learning the complex hand gesture classification task with lower error rates. The considered networks are trained and tested on data obtained from the above-mentioned public database; results comparison is then made against earlier works in which only small subsets of the ASL hand gestures are considered for recognition.

Human computer interaction is becoming more integrated in daily life with the proliferation of mobile devices and virtual reality technology. Hand gesture recognition is a potentially promising mechanism to facilitate human computer interaction, however wrist-mounted surface electromyography (sEMG) hand gesture classification is particularly challenging given the relatively small sEMG signals as compared to traditional forearm-based sEMG sensing. This paper introduces the development of a wristband for detecting eight air gestures and four surface gestures at two different force levels through sEMG and inertial measurement unit (IMU) sensor fusion. To validate the wrist-worn device, ten healthy subjects performed hand gesture recognition experiments resulting in a total average recognition rate of 92.6% for air gestures and 88.8% for surface gestures. This paper demonstrates the potential of wrist-worn devices for accurate hand gesture recognition applications.

Human–computer communication using hand gestures has always been difficult. More than half a century ago, people used differentways of interactionwith computers from the early mediums such as perforated game cards. Nowadays, if a richer lexicon of gestures is given, people can communicate more effectively with computers. Machine learning is now used to recognize and classify hand gestures in amore preciseway. In order to increase the communication between computers and humans, we proposed a technique, which uses a wearable low-cost device to generate the electrical impedance tomography (EIT) images to recover the inner impedance structure of a user’s wrist. This is done by measuring the transverse impedance between all the 16 pairs of electrodesofwrist band that lie on the skin of the user hand. The proposedtechnique is enough to integrate the technology into the prototypewrist band tomonitor and classify gestures in real time. We have conducted a study of 16 gestures with a focus on gross hand and pinch finger gestures. The results evaluation shows that the gross hand gestures achieved 90% accuracy inwrist position, while pinch gestures achieved 93% accuracy.

This paper presents head and hand gestures recognition system for Human Computer Interaction (HCI). Head and Hand gestures are an important modality for human computer interaction. Vision based recognition system can give computers the capability of understanding and responding to the hand and head gestures. The aim of this paper is the proposal of real time vision system for its application within a multimedia interaction environment. This recognition system consists of four modules, i.e. capturing the image, image extraction, pattern matching and command determination. If hand and head gestures are shown in front of the camera, hardware will perform respective action. Gestures are matched with the stored database of gestures using pattern matching. Corresponding to matched gesture, the hardware is moved in left, right, forward and backward directions. An algorithm for optimizing connected component in gesture recognition is proposed, which makes use of segmentation in two images. Connected component algorithm scans an image and group its pixels into component based on pixel connectivity i.e. all pixels in connected component share similar pixel intensity values and are in some way connected with each other. Once all groups have been determined, each pixel is labeled with a color according to component it was assigned to.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. As the technology advances and communication between humans and machines becomes more complex, HCI systems should also be scaled accordingly in order to accommodate the introduced complexities. In this paper, we propose a methodology to scale hand gestures by forming them with predefined gesture-phonemes, and a convolutional neural network (CNN) based framework to recognize hand gestures by learning only their constituents of gesture-phonemes. The total number of possible hand gestures can be increased exponentially by increasing the number of used gesture-phonemes. For this objective, we introduce a new benchmark dataset named Scaled Hand Gestures Dataset (SHGD) with only gesture-phonemes in its training set and 3-tuples gestures in the test set. In our experimental analysis, we achieve to recognize hand gestures containing one and three gesture-phonemes with an accuracy of 98.47% (in 15 classes) and 94.69% (in 810 classes), respectively. Our dataset, code and pretrained models are publicly available.

Tangible Acoustic Interfaces (TAIs) exploit the propagation of sound in physical objects in order to localize touching positions and to analyse user's gesture on the object. Designing and developing TAIs consists of exploring how physical objects, augmented surfaces, and spaces can be transformed into tangible-acoustic embodiments of natural seamless unrestricted interfaces. Our research focuses on Expressive TAIs , i.e., TAIs able at processing expressive user's gesture and providing users with natural multimodal interfaces that fully exploit expressive, emotional content. This paper presents a concrete example of analysis of expressive gesture in TAIs: hand gestures on a TAI surface are classified according to the Space and Time dimensions of Rudolf Laban's Theory of Effort. Research started in the EU-IST Project TAI-CHI (Tangible Acoustic Interfaces for Computer-Human Interaction) and is currently going on in the EU-ICT Project SAME (Sound and Music for Everyone, Everyday, Everywhere, Every way, www.sameproject.eu). Expressive gesture analysis and multimodal and cross-modal processing are achieved in the new EyesWeb XMI open platform (available at www.eyesweb.org) by means of a new version of the EyesWeb Expressive Gesture Processing Library.

We present a pipeline for recognizing dynamic freehand gestures on mobile devices based on extracting depth information coming from a single Time-of-Flight sensor. Hand gestures are recorded with a mobile 3D sensor, transformed frame by frame into an appropriate 3D descriptor and fed into a deep LSTM network for recognition purposes. LSTM being a recurrent neural model, it is uniquely suited for classifying explicitly time-dependent data such as hand gestures. For training and testing purposes, we create a small database of four hand gesture classes, each comprising 40 \(\times \) 150 3D frames. We conduct experiments concerning execution speed on a mobile device, generalization capability as a function of network topology, and classification ability ‘ahead of time’, i.e., when the gesture is not yet completed. Recognition rates are high (>95%) and maintainable in real-time as a single classification step requires less than 1 ms computation time, introducing freehand gestures for mobile systems.

Hand gestures aid the user in unspoken communication and help interpret sign language. Modern nonverbal communication detection employs gesture recognition systems and allows us to recognize shapes and signals made by hand movements using numerous machine learning techniques. Gestures have a very high value in human-computer interaction and sign language detection systems. To implement the gesture recognition and detection the existing methodologies rely on the application of bodily sensors. Due to this, the effectiveness in communication becomes challenging and cumbersome. Hence targeting on the major fallacies of the existing approaches, the primary aspiration of this research work is to recognize hand gestures in real-time using depth sensors that support natural communication avoiding on body sensor dependency. A novel approach is presented to detect hand gestures using minimalistic training and the efficiency of other algorithms concerning this proposed technique has been compared. Experimental investigations on the proposed methodology have successfully confirmed that, the research exertion has obtained an overall gesture detection accuracy of 90.86% and a high precision of 98.25%.

We developed an augmented reality tool for vision-based hand gesture recognition in a camera-projector system. Our recognition method uses modified Fourier descriptors for the classification of static hand gestures. Hand segmentation is based on a background subtraction method, which is improved to handle background changes. Most of the recognition methods are trained and tested by the same service-person, and training phase occurs only preceding the interaction. However, there are numerous situations when several untrained users would like to use gestures for the interaction. In our new practical approach the correction of faulty detected gestures is done during the recognition itself. Our main result is the quick on-line adaptation to the gestures of a new user to achieve user-independent gesture recognition.

People suffering from speech impairment can't communicate using hearing and speech methods, they believe signing for communication. Sign language is employed among everybody who is speech impaired, but they find a tough time in communicating with people which are non-signers (people aren’t proficient in sign language). So, requirement of a symbol language interpreter may be a must for speech impaired people. There has been favourable progress within the field of gesture recognition and motion recognition with current advancements in deep learning. There has been quite a significant development in computer vision which would enable us to easily track the hand gestures. The proposed system tries to try to a true time translation of hand gestures into equivalent English text. This system takes hand gestures as input through video and translates it text which might be understood by a non-signer. There will be use of CNN for classification of hand gestures. By deploying this technique, the communication gap between signers and non-signers will be reduced and they will be easily able to communicate with normal people.

Human-computer interaction for virtual-real fusion