A real-time human-robot interaction framework with robust background invariant hand gesture detection

Hand gesture for communication has proven effective for humans, and active research is ongoing in replicating the same success in computer vision systems. Human–computer interaction can be significantly improved from advances in systems that are capable of recognizing different hand gestures. In contrast to many earlier works, which consider the recognition of significantly differentiable hand gestures, and therefore often selecting a few gestures from the American Sign Language (ASL) for recognition, we propose applying deep learning to the problem of hand gesture recognition for the whole 24 hand gestures obtained from the Thomas Moeslund’s gesture recognition database. We show that more biologically inspired and deep neural networks such as convolutional neural network and stacked denoising autoencoder are capable of learning the complex hand gesture classification task with lower error rates. The considered networks are trained and tested on data obtained from the above-mentioned public database; results comparison is then made against earlier works in which only small subsets of the ASL hand gestures are considered for recognition.

Hand gesture acquisition refers to the process whereby a computer is able to detect and record human hand movements. Vision-based hand gesture detection and a data glove-based approach can do this. In the realm of vision-based methods, we find both static hand and real-time hand gesture detection approaches. Five flex sensors are sewn onto the glove for the data glove method. Compared to the data glove-based method, the vision-based method is more consistent and robust. Designing a user-friendly vision-based interface, however, is a formidable challenge. The collected hand motions will need to be manually categorised by the user once again. So, it's no surprise that deciphering data is a laborious process that eats up plenty of time. This analysis should ideally be done mechanically. Relating the electrical nature of the human nervous system to computers presents a significant challenge in the field of Human Computer Interaction (HCI). Electromyographic (EMG) impulses, which change depending on the sort of hand motion utilised, are used to establish a brain link with computers. Because of this, an EMG-based system for recognising hand gestures can be created. The goal of this research is to create signal processing methods and Artificial Neural Networks (ANN) that can automatically recognise hand motions. The recorded Electromyographic (EMG) signals for a wide variety of hand motions are analysed using signal processing techniques, from which a number of features are extracted. Characteristics are extracted using Wavelet Transform (WT) based techniques including Discrete Wavelet Transform (DWT) and Wavelet Packet Transform (WPT). Better performance was achieved with the recommended methods, as shown by the simulation results.

Remote controls are commonly used today as an interface for almost all consumer electronic equipment. Different home appliances use a multitude of remote controllers, offering a variety of problems: confusion in the different button layout, spending time in looking for the right remote, problems in replacement when destroyed and so on. Hand gestures have been one of the oldest and natural methods of human interaction. It is considered as an interaction medium that can possibly deliver more intuitive, creative and natural method for communication between human and a machine. The researchers choose to create an android based hand gesture controlled interface system for home appliances that serves as an alternative to existing remote controllers. The camera of the android device will be used to detect static hand gestures for processing. The gathered data will be compared to a database of recognizable static hand gestures to execute commands on certain home appliances using infrared light. After series of testing and evaluation, the system is able to execute control functions to the home appliances indicated by specific hand gestures. The system required correct execution of the corresponding static gesture to properly execute the command functions of the appliances with proper lighting and illuminance for proper recognition else it would result into loss of accuracy and false detection. For the system to be able to communicate or receive information the transmission of the sender (IR) should be in line of sight to the receivers (IR) with six (6) meters range between the appliances and the mobile phone. A 100% success rate of gesture recognition was observed in Samsung Galaxy S4 with 2.26451 seconds recognition time while there were only three failures of hand gesture detection using Samsung Galaxy Note 3 with 2.6991 seconds recognition time.

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Comprehending and replicating human hand gestures is crucial for advancements in robotics, sign language interpretation and human-computer interactions. While extensive research has focused on improving hand gesture recognition, the therapeutic benefits of dance movements have often been overlooked. This study introduces a novel approach to understanding hand gestures through structured movement primitives derived from hand gestures (mudras) used in Indian classical dance form known as Bharatanatyam. Hand gesture synergies were extracted using Gaussian-modeled joint angular velocities and represented as fundamental syllables of motion. These syllables were then employed to reconstruct 75 diverse hand gestures, including American Sign Language (ASL) postures, a dataset of natural hand grasps and traditional mudras. Comparative analysis between mudra-derived synergies achieved superior reconstruction accuracy (95.78% for natural grasps and 92.99% for mudras) compared to synergies derived from natural grasps (88.92% for natural grasps and 82.51% for mudras). The results suggest that the structured and intentional nature of Bharatanatyam mudras leads to much stronger representation of syllables of movements that have superior generalizability and precision. Additionally, the reconstructed gestures were successfully mapped onto Mitra, a humanoid robot with five degree of freedom hand using a continuous joint-mapping approach. This research highlights the potential of dance inspired structured learning in enhancing dexterity, rehabilitation, and motor control, paving the way for more efficient gesture-based interaction models in robotics, prosthetics and rehabilitation.

In this article, a novel method for continuous hand gesture detection and recognition is proposed based on a frequency modulated continuous wave (FMCW) radar. Firstly, we adopt the 2-Dimensional Fast Fourier Transform (2D-FFT) to estimate the range and Doppler parameters of the hand gesture raw data, and construct the range-time map (RTM) and Doppler-time map (DTM). Meanwhile, we apply the Multiple Signal Classification (MUSIC) algorithm to calculate the angle and construct the angle-time map (ATM). Secondly, a hand gesture detection method is proposed to segment the continuous hand gestures using a decision threshold. Thirdly, the central time-frequency trajectory of each hand gesture spectrogram is clustered using the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -means algorithm, and then the Fusion Dynamic Time Warping (FDTW) algorithm is presented to recognize the hand gestures. Finally, experiments show that the accuracy of the proposed hand gesture detection method can reach 96.17%. The hand gesture average recognition accuracy of the proposed FDTW algorithm is 95.83%, while its time complexity is reduced by more than 50%.

Many companies and even public institutions for civil servants currently use photo-taking for the attendance. However, this strategy is still considered ineffective since the employees still can hack the attendance by making their own photos and put them in their desks. Therefore, an alternative that can complement the current face detection method is highly needed so that the employee’s attendance can be directly monitored. One of the methods that can be used to detect the attendance is hand gesture detection. This research aims to detect hand gestures made by the employees to ensure whether they really come to work or not. This research make the chance for manipulation using photo or fake GPS is quite small. For the purpose of hand gesture recognition, this study utilized Local Binary Pattern Histogram algorithm. The hand gesture image was first taken using a raspberry pi camera and then processed by the device to examine whether it matches the registered ID or not. The results showed that ID recognition by using hand gestures is detectable. The number recognition in hand gestures includes numbers 1 to 10. The test results showed that for 5 trials, the average time required for reading hand gestures using a laptop was 9.2 seconds, while that of using raspberry was 14.2 seconds. The results of this research show that the system has not been able to distinguish which hand is read first, so numbers that have the same number are considered the same, such as 81 and 18. So, the motion reading using a raspberry takes longer than that of using a laptop because the laptop's performance is higher than that of a raspberry and system cannot distinguish between numbers consisting of the same number.

Stationary hand gesture recognition can be applied in a variety of domains such as human-computer interaction (HCI), remote control, virtual reality etc. Hand gesture recognition is mainly the study of detection and recognition of various hand gestures like Sign Language hand gestures, by a computer. This work is focused on three main issues in developing a gesture recognition system. These are (i) Threshold (ii) illumination normalization (iii) user independent gesture recognition based on fusion of Moments. It is very demanding to segment hand regions from the static hand gesture color images, due to unreliable light conditions and composite background. Since skin pixels can vary with different explanation condition, to find the range of skin pixels, becomes a hard task in case of color space based skin color segmentation. These methods propose a learning algorithm based on customized clustering expanse to remove person skin color regions from the stationary hand gesture color descriptions. To improve the performance of classification, two feature synthesis strategies have been proposed in this work.

The goal of static hand gesture recognition is to classify the given hand gesture data represented by some features into some prede ned nite number of gesture classes. The main objective of this e ort is to explore the utility of two feature extraction methods, namely, hand contour and complex moments to solve the hand gesture recognition problem by identifying the primary advantages and disadvantages of each method. Arti cial neural network is built for the purpose of classi cation by using the back-propagation learning algorithm. The proposed system presents a recognition algorithm to recognize a set of six speci c static hand gestures, namely: Open, Close, Cut, Paste, Maximize, and Minimize. The hand gesture image is passed through three stages, namely, pre-processing, feature extraction, and classi cation. In the pre-processing stage some operations are applied to extract the hand gesture from its background and prepare the hand gesture image for the feature extraction stage. In the rst method, the hand contour is used as a feature which treats scaling and translation problems (in some cases). The complex moments algorithm is, however,

In this research paper, an effort has been placed to convert 24 American Sign Language (ASL) signer independent, real time hand gesture alphabets into human or machine recognizable English text. In the proposed work, the ASL hand gestures used for cognition and recognition process is completely invariant to scale, luminance, gender, and distance in the complex background of indoor location. The Viola-Jones algorithm, CIE Lab color model and canny approximation to the derivative are used for proper hand segmentation. In both the cognition and recognition process, the various features such as boundary, centroid, entropy, Hu moments, Zernike moments, Gabor filters, Histogram of Oriented Gradients (HOG) and Local Phase Quantization (LPQ) are extracted from the hand gestures. The K-Nearest Neighbor (KNN), Multiclass-Support Vector Machines (M-SVM) and Decision Tree (DT) classifiers are used for classifying the hand gestures. In recognition task, these classifiers are applied independently on the same set of hand gestures to check the optimality of recognition rate and recognition time. With the detailed experimentation, it is found that, the KNN classifier achieved an average recognition rate and average recognition time of 92.71% and 0.48 s per gestures. This recognition rate and time is better and optimal compared to M-SVM and DT classifiers. Also it is an inspiring result compared to state of art techniques in real time environment by considering various invariants.

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

The rapid spread of the novel corona virus disease (COVID-19) has disrupted the traditional clinical services all over the world. Hospitals and healthcare centers have taken extreme care to minimize the risk of exposure to the virus by restricting the visitors and relatives of the patients. The dramatic changes happened in the healthcare norms have made it hard for the deaf patients to communicate and receive appropriate care. This paper reports a work on automatic sign language recognition that can mitigate the communication barrier between the deaf patients and the healthcare workers in India. Since hand gestures are the most expressive components of a sign language vocabulary, a novel dataset of dynamic hand gestures for the Indian sign language (ISL) words commonly used for emergency communication by deaf COVID-19 positive patients is proposed. A hybrid model of deep convolutional long short-term memory network has been utilized for the recognition of the proposed hand gestures and achieved an average accuracy of 83.36%. The model performance has been further validated on an alternative ISL dataset as well as a benchmarking hand gesture dataset and obtained average accuracies of 97% and 99.34pm 0.66%, respectively.

Technology becomes a part of human lives for decades, especially in human—computer interaction (HCI) that considered as the important research area involving with an assistive technology and a medical system. Hand gesture is classified as an intuitive method for human to interact with the computer. It is useful for elderly people who cannot express their feelings by words. This paper proposed the hand gesture recognition technique for the elderly by using contour detection with convex hull feature extraction and rule-based classification. Vision-based hand gesture recognition is considered in classifying six hand gestures as lingual description. From the experimental results, the hand gesture system provides good detection and classification results for all the six static hand gestures.

As part of the development of an educational tool that can help students achieve fluency in American Sign Language (ASL) through independent and interactive practice with immediate feedback, this paper introduces a near real-time system to recognize grammatical errors in continuous signing videos without necessarily identifying the entire sequence of signs. Our system automatically recognizes if a performance of ASL sentences contains grammatical errors made by ASL students. We first recognize the ASL grammatical elements including both manual gestures and nonmanual signals independently from multiple modalities (i.e. hand gestures, facial expressions, and head movements) by 3D-ResNet networks. Then the temporal boundaries of grammatical elements from different modalities are examined to detect ASL grammatical mistakes by using a sliding window-based approach. We have collected a dataset of continuous sign language, ASL-HW-RGBD, covering different aspects of ASL grammars for training and testing. Our system is able to recognize grammatical elements on ASL-HW-RGBD from manual gestures, facial expressions, and head movements and successfully detect 8 ASL grammatical mistakes.

Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.