Hand Gesture Detection Based on Non-audible Sound

Hand gestures are a form of non-verbal communication. Apart from traditional input devices, hand gestures are used for interaction with computers too. Communication via hand gestures finds many applications in the real world. Different people have hands with different shapes and orientations, which is termed as nonlinearity. The nonlinearity affects the performance of hand gesture models. A convolutional neural network (CNN) is an approach of neural networks, specifically known as deep learning. CNN is used to recognize and classify images. Sometimes, CNN could not correctly understand the hand gesture due to nonlinearity. Data augmentation helps CNN to understand the nonlinearity and complexity of images better. Data augmentation generates enormous data from lesser data, thus increasing the data adversity. Data augmentation uses various operations like zooming, rotating, shifting, shearing, and scaling to generate more data from the existing data. This article executes a CNN model using augmented data for recognition of static hand gestures. The dataset consists of 10 different hand gestures. The experimented CNN model has been trained using 10000 images and tested using 1000 images. The changes in the output of CNN with and without data augmentation have been highlighted. The CNN model employing data augmentation achieved an accuracy of 98.10%, whereas the CNN model excluding the data augmentation process attained an accuracy of 94.90% only.

Hand gesture for communication has proven effective for humans, and active research is ongoing in replicating the same success in computer vision systems. Human–computer interaction can be significantly improved from advances in systems that are capable of recognizing different hand gestures. In contrast to many earlier works, which consider the recognition of significantly differentiable hand gestures, and therefore often selecting a few gestures from the American Sign Language (ASL) for recognition, we propose applying deep learning to the problem of hand gesture recognition for the whole 24 hand gestures obtained from the Thomas Moeslund’s gesture recognition database. We show that more biologically inspired and deep neural networks such as convolutional neural network and stacked denoising autoencoder are capable of learning the complex hand gesture classification task with lower error rates. The considered networks are trained and tested on data obtained from the above-mentioned public database; results comparison is then made against earlier works in which only small subsets of the ASL hand gestures are considered for recognition.

The inertial measurement unit (IMU) is a popular sensor device, which is mainly employed to acquire body or hand gesture action information for performing specific recognition tasks. We present a dual-channel artificial neural network (ANN) recognition decision hybridization scheme incorporated with deep leaning of IMU-based spectrogram images for cognition of several common hand gesture intention categorization actions focused on the 6-axis IMU sensing data (containing 3-axis accelerometer and 3-axis gyroscope information) and the 6-axis IMU derived spectrogram images. In this hand gesture intention cognition approach, both symmetric and asymmetric ANN structures are considered for intention action classifications. The proposed dual-channel ANN decision fusion framework contains one ANN recognition channel with inputs of “6-axis IMU raw data” and the other ANN recognition channel with inputs of “IMU spectrogram image derived-critical deep learning features”. Recognition decisions estimated from either of these two ANN recognition channels form the fusion framework. Three fusion schemes on dual-channel ANN recognition decisions are presented in this study, channel output layer accumulation, same channel candidate output and same-or-dual channel candidate output. In this study, the well-known deep learning neural network, visual geometry group- convolution neural network (VGG-CNN), is employed to carry out deep learning computations on IMU-based spectrogram images, from which, the critical deep learning feature of each spectrogram image can then be extracted and used as an input for the dual-channel ANN. For recognition performance comparisons, hand gesture intention recognition by the traditional VGG-CNN deep neural network approach (i.e. recognition of IMU spectrogram images using typical deep learning of the CNN model) is also performed. Experiments on classifications of six hand gesture intention actions show that the presented dual-channel ANN decision fusion incorporated with deep learning of IMU spectrum images has competitive performances, reaching better recognition accuracy than traditional CNN deep learning.

Hand gestures are a type of non-verbal communication that uses visible body movementsto convey important messages. This paper presents a much better approach of hand gestureprediction. Image Identification is an important step in most of the modern hand gestureprediction system. A convolutional neural network are used for improving the accuracy of thesystem. Proposed system tested for large number of hand gesture images using Tensor flowtool. . The convolutional neural network (ConvNet) is a deep learning algorithm for learningand classifying hand gestures and achieved accuracy 93.61%.

In this paper, we investigate the optimal structure of deep convolutional neural networks for classifying human hand gestures using Doppler radar. When hand motions are captured by Doppler radar, the unique micro-Doppler signatures can be observed in the spectrogram. If the signature is distinguishable by a classifier, then the hand gesture can be used for controlling electronics and as an input modality for a human-computer interface. To classiiy signatures in the spectrogram, we propose the use of a deep convolutional neural network (DCNN) as a classifier. DCNN is a powerful classifier that extracts features as well as class boundaries through a training process. We measured seven hand gestures performed in front of Doppler radar while generating spectrograms. To identify an optimal structure, we trained several DCNNs by changing hyperparameters, such as the number of convolutional layers, the number of filters, and the filter size. The classification accuracy obtained from the optimal DCNN structure was approximately 87%.

The Electromyography (EMG) signal is a nonstationary bio-signal based on the measurement of the electrical activity of the muscles. EMG based recognition systems play an important role in many fields such as diagnosis of neuromuscular diseases, human-computer interactions, console games, sign language detection, virtual reality applications, and amputee device controls. In this study, a novel approach based on deep learning has been proposed to improve the accuracy rate in the prediction of hand movements. Firstly, 4-channel surface EMG (sEMG) signals have been measured while simulating 7 different hand gestures (Extension, Flexion, Open Hand, Punch, Radial Deviation, Rest, and Ulnar Deviation) from 30 participants. The obtained sEMG signals have been segmented into sections where each movement was found. Then, spectrogram images of the segmented sEMG signals have been created by means of ShortTime Fourier Transform (STFT). The created colored spectrogram images have trained with 50-layer Convolutional Neural Network (CNN) based on Residual Networks (ResNet) architecture. Owing to the proposed method, test accuracy of 99.59% and F1 Score of 99.57% have achieved for 7 different hand gesture classifications.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Predicting and accurately classifying intentions for human hand gestures can be used not only for active prosthetic hands, rehabilitation robots and entertainment robots but also for artificial intelligence robots in general. In this paper, first of all, source data of three hand gestures of grasping and three hand gestures of sign language are acquired by using the armband combined with eight sEMG (surface Electromyography) sensors. To classify these hand gestures, basically simple CNN (convolutional neural network) models with raw data, short-time Fourier transform (STFT), wavelet transform (WT), and scale average wavelet transform (SAWT) are applied, and their performances are compared. Finally, it is shown that by using a CNN with SAWT images, the accuracy can be improved up to 94.6% for selected hand gestures with higher accuracy and lower computational burden than conventional multi-channel STFT or WT.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. As the technology advances and communication between humans and machines becomes more complex, HCI systems should also be scaled accordingly in order to accommodate the introduced complexities. In this paper, we propose a methodology to scale hand gestures by forming them with predefined gesture-phonemes, and a convolutional neural network (CNN) based framework to recognize hand gestures by learning only their constituents of gesture-phonemes. The total number of possible hand gestures can be increased exponentially by increasing the number of used gesture-phonemes. For this objective, we introduce a new benchmark dataset named Scaled Hand Gestures Dataset (SHGD) with only gesture-phonemes in its training set and 3-tuples gestures in the test set. In our experimental analysis, we achieve to recognize hand gestures containing one and three gesture-phonemes with an accuracy of 98.47% (in 15 classes) and 94.69% (in 810 classes), respectively. Our dataset, code and pretrained models are publicly available.

Any human–computer interaction application needs to be able to recognize gestures. Hand gesture detection systems that recognize gestures in real time can improve human–computer interaction by making it more intuitive and natural. Colour gloves and skin colour detection are two prominent hand segmentation and detection technologies, but each has its own set of benefits and drawbacks. For physically challenged people, gesture identification is a crucial technique of sharing information. Support vector machine algorithm with principal component analysis, hidden Markov model, superposed network with multiple restricted Boltzmann machines, growing neural gas algorithm, convolutional neural network, double channel convolutional neural network, artificial neural network, and linear support vector machine with gesture dataset matches the interaction of gestures for various postures in real time. Although this method can recognize a huge number of gestures, it does have certain downsides, such as missing movements due to the accuracy of the categorization algorithms. Furthermore, matching the vast dataset takes a longer time. The main contribution of this work lies in a conceptual framework based on the findings of a systematic literature review that provides fruitful implications based on recent research findings and insights that can be used to direct and initiate future research initiatives in the field of hand gesture recognition and artificial intelligence. As a result, a novel study based on a hybrid recurrent neural network (RNN) with chaos game optimization may reduce classification mistakes, increase stability, maximize resilience, and efficiently use the hand gestures recognition system.

Recently, deep learning has become a promising technique for constructing gesture recognition classifiers from surface electromyography (sEMG) signals in human–computer interaction. In this paper, we propose a gesture recognition method with sEMG signals based on a deep multi-parallel convolutional neural network (CNN), which solves the problem that traditional machine learning methods may lose too much useful information during feature extraction. CNNs provide an efficient way to constrain the complexity of feedforward neural networks by weight sharing and restriction to local connections. Sophisticated feature extraction is to be avoided and hand gestures are to be classified directly. A multi-parallel and multi-convolution layer convolution structure is proposed to classify hand gestures. Experiment results show that in comparison with five traditional machine learning methods, the proposed method could achieve higher accuracy.

The use of computer vision along with machine learning techniques resulted in a breakthrough in medical and social fields. One such intervention is the sign language detection, enabling communication for differently-abled people. Hand gestures are the common method used for this. This paper deals with using image processing techniques like RGB to binary image conversion, skin detection, edge detection to extract the important features of the gesture and provide them as inputs to Convolution Neural Networks (CNN) to improve the accuracy in predicting American Sign Language alphabets (ASL). Using the preprocessed image data set of the hand gesture in our proposed mechanism has predicted the alphabets with higher accuracy up to 99% as compared to the model trained with original data set of images. KeywordsImage processingEdge detectionConvolution neural netsAmerican sign language

Gesture Mouse is a real-time hand gesture recognition system that uses deep convolutional neural networks to mimic mouse functionalities for controlling computers. The system requires only a standard webcam, and it can detect and recognize various hand movements and gestures, such as clicking, scrolling, zooming, movements, and hovering. The aim of this project is to improve and ease the human-computer interaction, especially for users who are physically challenged or have limited mobility. The Gesture Mouse system consists of three main components: hand detection, hand landmark extraction, and gesture recognition. The hand detection component is based on the Single Shot Multi Box Detector (SSD) model, which is used to detect the hand region in each frame of the webcam stream. The hand landmark extraction component uses a convolutional neural network (CNN) to predict the 3D coordinates of 21 key hand landmarks, including fingertips, palm canter, and wrist. The gesture recognition component is based on a CNN classifier that is trained on a large dataset of hand gesture images. The system performs well in real-time and can be used to control a computer with a high degree of accuracy and reliability. The proposed system has the potential to provide a more natural and intuitive way of interacting with computers, especially for individuals with motor impairments and also in technologies such as Virtual Reality and Augmented Reality.

Remote controls are commonly used today as an interface for almost all consumer electronic equipment. Different home appliances use a multitude of remote controllers, offering a variety of problems: confusion in the different button layout, spending time in looking for the right remote, problems in replacement when destroyed and so on. Hand gestures have been one of the oldest and natural methods of human interaction. It is considered as an interaction medium that can possibly deliver more intuitive, creative and natural method for communication between human and a machine. The researchers choose to create an android based hand gesture controlled interface system for home appliances that serves as an alternative to existing remote controllers. The camera of the android device will be used to detect static hand gestures for processing. The gathered data will be compared to a database of recognizable static hand gestures to execute commands on certain home appliances using infrared light. After series of testing and evaluation, the system is able to execute control functions to the home appliances indicated by specific hand gestures. The system required correct execution of the corresponding static gesture to properly execute the command functions of the appliances with proper lighting and illuminance for proper recognition else it would result into loss of accuracy and false detection. For the system to be able to communicate or receive information the transmission of the sender (IR) should be in line of sight to the receivers (IR) with six (6) meters range between the appliances and the mobile phone. A 100% success rate of gesture recognition was observed in Samsung Galaxy S4 with 2.26451 seconds recognition time while there were only three failures of hand gesture detection using Samsung Galaxy Note 3 with 2.6991 seconds recognition time.