Gesture Prediction Using Wearable Sensing Systems with Neural Networks for Temporal Data Analysis

The relationship between humans and computers is called human computer interaction (HCI). HCI is a major research topic in the area of assistive technology. In the field of medical assistive technology specifically, a hand gesture is considered a suitable method to convey information. It can help elderly people, who are unable to walk or speak, communicate with caregivers whenever they need help. This paper proposes a system to recognize the hand gestures of elderly people using an inexpensive Raspberry Pi. A vision-based algorithm is developed to detect and classify dynamic hand gestures in real time on the Raspberry Pi embedded platform. There are three main procedures; contour detection, convex extraction, and rule-based classification. The system can detect six different gestures on both hands in various orientations. The experiment showed good results in detecting and classifying their meanings as lingual descriptions.

Since the 1970s, the field of gesture recognition and its applications has been at the centre of considerable research in human–computer interaction. Researchers have been able to construct strong models that can recognize gestures in real time thanks to recent advances in deep learning and computer vision, but they face hurdles when it comes to classifying gestures in variable lighting conditions. In this paper, we train a 3D convolutional neural network to recognize dynamic hand gestures in real time. Our focus is on ensuring that gesture recognition systems can perform well under varying light conditions. We use a huge training set consisting of numerous clips of people performing specific gestures in varying lighting conditions to train the model. We were able to attain an accuracy of 76.40% on the training set and 66.56% on the validation set with minimal pre-processing applied to the data set. The trained model was able to successfully recognize hand gestures recorded from a Webcam in real time. We were then able to use the model’s predictions to control video playback on the VLC media player such as increasing/decreasing volume and pausing video. These experimental results show the effectiveness and efficiency of the proposed framework to recognize gestures in both bright and dim lighting conditions.KeywordsDeep learningConvolutional neural networksPyTorch3D CNN

This paper presents a hand gesture interaction system in virtual environment, in which mixed means static and dynamic hand gestures are combined for both navigation and object manipulation. Firstly, a simple average background model and skin color are used for hand area segmentation. Then a state-based spotting algorithm is employed to automatically identify two types of hand gestures. A voting-based method is used for quick classification of static gestures. And we use the hidden Markov model (HMM) to recognize dynamic gestures. Since the training of HMM requires the consistency of the training data, outputted by the feature extraction, a data aligning algorithm is raised. Through our hand gesture system, users can perform complicated operating commands in a natural way. The experimental results demonstrate that our methods are effective and accurate.

The LSTM based sign language detection system combined with the use of mediapipe can recognize hand gestures in real time with high accuracy. Alphabet based sign language can use this model to collect temporal patterns of hand gestures. The data used in this study are 30 sample for each alphabet based on American Sign Language (ASL). The data is processed through landmark detection on the hand using mediapipe and opencv, keypoints extraction, folder creation and pre – processing with 80% data divisior for training data and 20 % data for testing data. Using Adam optimization, categorical crossentropy loss function and evaluation metric. The model was trained with 100 epochs and evaluated using confusion matrix. The evaluation results showed that the LSTM model performed well with an accuracy of 98%.

Glove Puppetry is popular in Chinese societies. Glove puppet shows have been performed in a traditional (manual) way for over 300 years. Due to the advances in information and communications technology (ICT), we can extend glove puppet art in a very different dimension. However, existing ICT solutions for puppet control have not demonstrated the time complexities for their control mechanisms nor how they can effectively control the puppets. This paper proposes PuppetTalk, an Internet of Things (IoT) application platform for puppet control. We allow multiple input IoT devices such as smart gloves to control a puppet robot, and smartphones can be used to control the sequences of actions on that robot at the same time. In particular, the PuppetTalk Graphical User Interface (GUI) allows users to quickly configure multiple distributed devices to control a puppet robot. Also, controlling puppet legs using a second glove and puppet movements with hand gestures in real time through smart phones are both innovations that have not appeared in the literature so far. When multiple input devices are used to control a puppet robot, their instructions must be received by the robot in a correct sequence. We have conducted measurements, simulation and analytic modeling to show that PuppetTalk can correctly control a puppet robot by multiple input devices with the probability higher than 99.7%.

When acquiring a foreign language, the first challenge is to break into the speech stream to identify basic linguistic units. The present study tested the hypothesis that hand gestures facilitate this process by alerting auditory cortices to attend to and identify meaningful phonemic information. During fMRI data acquisition, participants watched videos of an actor speaking in Russian under three conditions. Sentences were produced with just speech alone or were accompanied by two types of hand gestures: 1) metaphoric gesture and 2) free gesture. The main finding was that there was increased auditory cortex activation when both types of gestures accompanied speech compared to speech alone, but there were no differences between the two speech + gesture conditions (or gesture alone conditions). These results suggest that hand gestures may play a role in focusing attention to auditory processing to increase capacity when listening to novel speech in a foreign language.

Human–computer communication using hand gestures has always been difficult. More than half a century ago, people used differentways of interactionwith computers from the early mediums such as perforated game cards. Nowadays, if a richer lexicon of gestures is given, people can communicate more effectively with computers. Machine learning is now used to recognize and classify hand gestures in amore preciseway. In order to increase the communication between computers and humans, we proposed a technique, which uses a wearable low-cost device to generate the electrical impedance tomography (EIT) images to recover the inner impedance structure of a user’s wrist. This is done by measuring the transverse impedance between all the 16 pairs of electrodesofwrist band that lie on the skin of the user hand. The proposedtechnique is enough to integrate the technology into the prototypewrist band tomonitor and classify gestures in real time. We have conducted a study of 16 gestures with a focus on gross hand and pinch finger gestures. The results evaluation shows that the gross hand gestures achieved 90% accuracy inwrist position, while pinch gestures achieved 93% accuracy.

Political oratory is a diverse genre which effectively juxtaposes speech and non-verbal cues. Politicians from diverse backgrounds and cultures often incorporate non-verbal elements, such as hand gestures, into their speeches. These hand gestures possess high utility and universality; thus, researchers believe that hand gestures and speech work together as a unified system for production and comprehension. This research explores the complex relationship between non-verbal and verbal communication within the dynamic realm of political public speech, drawing on McNeill’s (354) Gesture theory. McNeill’s theory classifies gestures into four categories: iconic, metaphoric, beat, and deictic. Employing a qualitative approach, the study provides an in-depth analysis of Imran Khan's gestures and relationship with spoken words. Iconic and metaphoric gestures were found to reinforce spoken content, while deictic and beat gestures directed attention to specific subjects and enhanced overall communication of meaning. The study found that Imran Khan effectively employed all four types of hand gestures and demonstrated a clear correlation between these gestures and his spoken words, thereby revealing his underlying thought processes. This study confirms the intricate correlation between gestures and language, emphasising their importance in influential public speaking. This resource provides unique perspectives on effective communication tactics, emphasising the significance of non-verbal clues in global conversations.

How hand gestures influence the enjoyment in gamified mobile marketing

Any human–computer interaction application needs to be able to recognize gestures. Hand gesture detection systems that recognize gestures in real time can improve human–computer interaction by making it more intuitive and natural. Colour gloves and skin colour detection are two prominent hand segmentation and detection technologies, but each has its own set of benefits and drawbacks. For physically challenged people, gesture identification is a crucial technique of sharing information. Support vector machine algorithm with principal component analysis, hidden Markov model, superposed network with multiple restricted Boltzmann machines, growing neural gas algorithm, convolutional neural network, double channel convolutional neural network, artificial neural network, and linear support vector machine with gesture dataset matches the interaction of gestures for various postures in real time. Although this method can recognize a huge number of gestures, it does have certain downsides, such as missing movements due to the accuracy of the categorization algorithms. Furthermore, matching the vast dataset takes a longer time. The main contribution of this work lies in a conceptual framework based on the findings of a systematic literature review that provides fruitful implications based on recent research findings and insights that can be used to direct and initiate future research initiatives in the field of hand gesture recognition and artificial intelligence. As a result, a novel study based on a hybrid recurrent neural network (RNN) with chaos game optimization may reduce classification mistakes, increase stability, maximize resilience, and efficiently use the hand gestures recognition system.

The natural user interface (NUI) is used for the natural motion interface without using device or tool such as mice, keyboards, pens and markers. In this paper, we develop natural user interface framework based on two recognition module. First module is real-time head pose estimation module using random forests and second module is hand gesture recognition module, named Hand gesture Key Emulation Toolkit (HandGKET). Using the head pose estimation module, we can know where the user is looking and what the user’s focus of attention is. Moreover, using the hand gesture recognition module, we can also control the computer using the user’s hand gesture without mouse and keyboard. In proposed framework, the user’s head direction and hand gesture are mapped into mouse and keyboard event, respectively.

Gesture recognition is a topic in computer science and language technology that aims to interpret human gestures with computer programs and many different algorithms. It can be seen as the way computers can understand human body language. Today, the main interaction tools between computers and humans are still the keyboard and mouse. Gesture recognition can be used as a tool for communication with the machine and interaction without any mechanical device such as a keyboard or mouse. In this paper, we present the results of a comparison of eight different machine learning (ML) classifiers in the task of human hand gesture recognition and classification to explore how to efficiently implement one or more tested ML algorithms on an 8-bit AVR microcontroller for on-line human gesture recognition with the intention to gesturally control the mobile robot. The 8-bit AVR microcontrollers are still widely used in the industry, but due to their lack of computational power and limited memory, it is a challenging task to efficiently implement ML algorithms on them for on-line classification. Gestures were recorded by using inertial sensors, gyroscopes, and accelerometers placed at the wrist and index finger. One thousand and eight hundred (1800) hand gestures were recorded and labelled. Six important features were defined for the identification of nine different hand gestures using eight different machine learning classifiers: Decision Tree (DT), Random Forests (RF), Logistic Regression (LR), Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) with linear kernel, Naïve Bayes classifier (NB), K-Nearest Neighbours (KNN), and Stochastic Gradient Descent (SGD). All tested algorithms were ranged according to Precision, Recall, and F1-score (abb.: P-R-F1). The best algorithms were SVM (P-R-F1: 0.9865, 0.9861, and 0.0863), and RF (P-R-F1: 0.9863, 0.9861, and 0.0862), but their main disadvantage is their unusability for on-line implementations in 8-bit AVR microcontrollers, as proven in the paper. The next best algorithms have had only slightly poorer performance than SVM and RF: KNN (P-R-F1: 0.9835, 0.9833, and 0.9834) and LR (P-R-F1: 0.9810, 0.9810, and 0.9810). Regarding the implementation on 8-bit microcontrollers, KNN has proven to be inadequate, like SVM and RF. However, the analysis for LR has proved that this classifier could be efficiently implemented on targeted microcontrollers. Having in mind its high F1-score (comparable to SVM, RF, and KNN), this leads to the conclusion that the LR is the most suitable classifier among tested for on-line applications in resource-constrained environments, such as embedded devices based on 8-bit AVR microcontrollers, due to its lower computational complexity in comparison with other tested algorithms.

Parent-child interactions, particularly storytelling practices, play a critical role in children's language development, yet the influence of cultural variation in these practices remains underexplored. This quantitative study aims to examine differences between English and Zimbabwean storytelling practices and their impact on children's story recall. To address limitations in previous research, including the reliance on artificial settings and insufficient attention to cultural diversity, we will conduct a naturalistic study in participants' homes. Parent-child dyads will be observed in three contexts: playtime, story time using a storybook and oral narration. Data will be collected on verbal behaviour, hand gestures and visual attention, while children's recall will be measured following each type of storytelling. This study will address four key research questions: (1) How do cultural groups differ in their storytelling mode and pedagogic orientation? (2) How do multimodal dyadic behaviours differ across cultures during storytelling and play? (3) Are there cross-cultural differences in the frequency and types of hand gestures used in storytelling? (4) Is greater parental experience with oral narration associated with enhanced child recall? The study will involve English dyads in England and Zimbabwean dyads in Zimbabwe. By examining these interactions in participants' home environments and by including both Zimbabwean families in Zimbabwe and English families in England, this research will shed light on how culturally rooted storytelling practices influence cognitive and language development and inform our understanding of culturally responsive parenting approaches.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

With the escalating role of computers in educational system, human computer interaction, is becoming gradually more important part of it. The general believe is that with the progress in computing speed, communication technologies, and display techniques the existing HCI techniques may become a constraint in the effectual utilization of the existing information flow. The development of user interfaces influences the changes in the Human-Computer Interaction (HCI). Human hand gestures have been a mode of non verbal interaction widely used. The vocabulary of hand gesture communication has many variations. It ranges from simple action of using our finger to point at and using hands to move objects around for more complex expressions for the feelings and communicating with others. Also the hand gestures play a prominent role in teaching considering the explanations and exemplifications being highly dependent on hand gestures. Naturalistic and intuitiveness of the hand gesture has been an immense motivating aspect for the researchers in the field of Human Computer Interaction to put their efforts to research and develop the more promising means of interaction involving human and computers. The pursuance for the Human Computer Interaction research is moved by the central dogma of removing the complex and cumbersome interaction devices and replacing them with more obvious and expressive means of interaction which easily comes to the users with least cognitive burden like hand gestures. This paper designs a simple, natural system for gestural interaction between the user and computer for providing a dynamic user interface. The gesture recognition system uses image processing techniques for detection, segmentation, tracking and recognition of hand gestures for converting it to a meaningful command. This hand gesture recognition system has been proposed, designed and developed with the intensions to make it a substitute for mouse while making dynamic user interface between human and machine. Hence instead of making effort to develop a new vocabulary of hand gesture we have matched control instruction set of mouse to subset of most discriminating hand gestures, so that we get a robust interface. The interface being proposed here can be substantially applied towards different applications like image browser, games etc.