Hand Gesture Virtual Car Driving Simulator

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

This article explores the application of YOLOv11 algorithm in gesture recognition field to evaluate its performance in human-computer interaction (HCI). By introducing the YOLOv8 model and corresponding dataset for training, we obtained the confusion matrix predicted by the model, which shows that the model can accurately recognize most gestures, although there are a few cases of misidentification. When the IoU threshold is 0.5, the average accuracy (mAP) of the model steadily improves with the progress of training, indicating that the overall performance of the model in gesture detection tasks has been enhanced. In addition, even under stricter evaluation conditions where the IoU threshold was increased from 0.5 to 0.95, mAP still showed an upward trend, although the growth rate was not as significant as mAP50, which still demonstrated the improvement in model performance. Through the detection of the test set images, we found that the YOLOv11 model can effectively recognize gestures and accurately interpret their meanings, demonstrating high accuracy. This study not only demonstrates the potential of YOLOv11 in gesture recognition tasks, but also provides a new technological path for the future development of HCI field. Overall, the YOLOv11 algorithm has demonstrated strong performance and accuracy in gesture recognition, providing a more natural and intuitive way for interaction between smart devices and humans.

Hand joints-based gesture recognition using a neural network provides excellent performance in hand gesture recognition. However, during the collection of sequential skeletal datasets, joints identification through hand pose estimations usually includes noise and even errors, which often diminish the accuracy of gesture recognition. To promote the availability of hand gesture recognition for such noisy datasets, this paper presents a nested interval unscented Kalman filter (UKF) with long short-term memory (NIUKF-LSTM) network to improve the accuracy of hand gesture recognition from noisy datasets. This nested interval method with the UKF changes the distribution of the sigma points based on two sampling intervals. By considering the information of previous frames, the nested interval method helps the NIUKF-LSTM network revise the noise in the sequential hand skeletal data and improve the recognition accuracy. The experimental results showing the removal of noisy skeletal data from the dynamic hand gesture dataset demonstrate the effectiveness of our NIUKF-LSTM network, which achieves better performance than do other state-of-the-art methods.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

Stationary hand gesture recognition can be applied in a variety of domains such as human-computer interaction (HCI), remote control, virtual reality etc. Hand gesture recognition is mainly the study of detection and recognition of various hand gestures like Sign Language hand gestures, by a computer. This work is focused on three main issues in developing a gesture recognition system. These are (i) Threshold (ii) illumination normalization (iii) user independent gesture recognition based on fusion of Moments. It is very demanding to segment hand regions from the static hand gesture color images, due to unreliable light conditions and composite background. Since skin pixels can vary with different explanation condition, to find the range of skin pixels, becomes a hard task in case of color space based skin color segmentation. These methods propose a learning algorithm based on customized clustering expanse to remove person skin color regions from the stationary hand gesture color descriptions. To improve the performance of classification, two feature synthesis strategies have been proposed in this work.

Hand gestures are a form of non-verbal communication. Apart from traditional input devices, hand gestures are used for interaction with computers too. Communication via hand gestures finds many applications in the real world. Different people have hands with different shapes and orientations, which is termed as nonlinearity. The nonlinearity affects the performance of hand gesture models. A convolutional neural network (CNN) is an approach of neural networks, specifically known as deep learning. CNN is used to recognize and classify images. Sometimes, CNN could not correctly understand the hand gesture due to nonlinearity. Data augmentation helps CNN to understand the nonlinearity and complexity of images better. Data augmentation generates enormous data from lesser data, thus increasing the data adversity. Data augmentation uses various operations like zooming, rotating, shifting, shearing, and scaling to generate more data from the existing data. This article executes a CNN model using augmented data for recognition of static hand gestures. The dataset consists of 10 different hand gestures. The experimented CNN model has been trained using 10000 images and tested using 1000 images. The changes in the output of CNN with and without data augmentation have been highlighted. The CNN model employing data augmentation achieved an accuracy of 98.10%, whereas the CNN model excluding the data augmentation process attained an accuracy of 94.90% only.

Human-computer interaction (HCI) is an important feature of augmented reality (AR) technology. The naturalness is the inevitable trend of HCI. Gesture is the most natural and frequently used body auxiliary interaction mode in daily interactions except for language. However, there are often meaningless, subconscious gesture intervals between the two adjacent dynamic gestures. So, continuous dynamic gesture spotting is the premise and basis of dynamic gesture recognition, but there is no mature and unified algorithm to solve this problem. In order to realize the natural HCI based on gesture recognition entirely, a general AR application development platform is presented in this paper. According to the position and pose tracking data of the user's hand, the dynamic gesture spotting algorithm based on evidence theory is proposed. Firstly, Through analysis of the speed change of hand motion during the dynamic gestures, three knowledge rules are summed up. Then, accurate dynamic gesture spotting is realized with the application of evidence reasoning. Moreover, this algorithm first detects the starting point of gesture in the rising trend of hand motion speed, eliminates the delay between spotting and recognition, and thus ensures real-time performance. Finally, the algorithm is verified in several AR applications developed on the platform. There are two main experimental results. First, there are six users participating in the dynamic gesture spotting experiment, and the gesture spotting accuracy can meet the demand. Second, The accuracy of recognition after spotting is higher than that of the simultaneous recognition and spotting. So, It can be concluded that the proposed continuous dynamic gesture spotting algorithm based on Dempster-Shafer theory can extract almost all the effective dynamic gestures in the HCI of our AR platform, and on this basis, it can effectively improve the accuracy of the subsequent dynamic gesture recognition.

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.

We present a method for robust and accurate hand pose recognition from egocentric depth cameras. Our method combines CNN based hand pose estimation and joint locations based hand gesture recognition. In pose estimation stage, we use a hand geometry prior network to estimate the hand pose. In gesture recognition stage, we defined a hand language which based on a set of pre-define basic propositions, obtained by applying four predicate types to the fingers and palm states. The hand language is used to convert the estimated joint location to hand gesture. Our experimental results indicate that the method enables robust and accurate gesture recognition in self-occlusion environment.

As a new type of human-robot interaction (HRI), hand gesture has many advantages such as natural operation, rich expression and not subject to environ-mental constraints. So it is very suitable for space human-robot interaction tasks in special and harsh environment. Considering that static hand gesture is one of the main gesture expressions in human-computer interaction, so a parallel convolution neural networks (CNNs) is designed to improve the accuracy of static hand gesture recognition in the conditions of complex background and changing illumination. In addition, the method is applied to the operation of space human-robot system with hand gesture control. Various space HRI hand gestures from different subjects are evaluated and tested, and experimental results demonstrate that the proposed method outperforms the single-channel CNN methods and other popular methods with a higher accuracy.

Myoformer: sEMG missing signal recovery for gesture recognition based on multi-channel self-attention mechanism

Electrode shift of a prosthetic device is one of most challengeable problems in surface Electromyography (sEMG) based hand gesture recognition. Electrode shift is usually caused by repositioning, donning or doffing of a prosthetic device. Accuracy of gesture recognition may significantly drop since a pattern of collected signals may change after electrode shift. Although re-training a recognition system after every reposition is able to maintain accurate recognition, collecting labeled samples is inconvenient to users. In this paper, we apply an online semi-supervised learning in which a classifier is trained with a small amount of labeled samples and then is updated with unlabeled samples online to hand gesture recognition. A well-known online semi-supervised learning algorithm, online multi-channel semi-supervised growing neural gas (OSSMGNG) algorithm, is used in this preliminary study. OSSMGNG is compared with an intuitive method which learns from the initial label training set only in experiments. The data is collected from able-bodied individuals across three days for experiments. The results indicate OSSMGNG achieves a higher classification accuracy than others. It suggests that the online semi-supervised learning algorithm enhances robustness of hand gesture identification against electrode shift.

In this paper, a wireless hand gesture recognition glove is proposed for real-time translation of Taiwanese sign language. To discriminate between different hand gestures, we have flex and inertial sensors embedded into the glove so that the three most important parameters, i.e., the posture of fingers, orientation of the palm, and motion of the hand, defined in Taiwanese Sign Language can be recognized without ambiguity. The finger flexion postures acquired by flex sensors, the palm orientation acquired by G-sensor, and the motion trajectory acquired by gyroscope are used as the input signals of the proposed system. The input signals will be acquired and examined periodically to see if it is a legal sign language gesture or not. Once the sampled signal can last longer than a predefined clock cycles, it is regarded as a valid gesture and will be sent to cell phone via Bluetooth for gesture discrimination and speech translation. With the proposed architecture and algorithm, the accuracy for gesture recognition is quite satisfactory. As we can see in experiments that an accuracy rate up to 94% on sensitivity for gesture recognition can be achieved which justifies the superiority of the proposed architecture.

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

The hand gesture provides a natural and intuitive communication medium for the human and machine interaction. Because, it can use in virtual reality, language detection, computer games, and other human-computer or human-machine instruction applications. Currently, the sensor and camera-based application is a field of interest for many researchers. This paper proposes a new hand gesture recognition system using the Kinect sensor's skeleton data, which works in an environment where people do not touch devices or communicate verbally. The proposed model focuses on mainly two modules, namely, hand area and fingertip detection, and hand gesture recognition. The hand area and fingertip are detected by positioning the palm point and find extreme of contour. And, the hand gesture is recognized by measuring the distance between different body indexes of skeleton information. Here, six gestures instructions are considered such as move right to left, move left to right, move up to down, move down to up, open and closed, and also recognize the numeric number using the fingertip. This system is able to detect the presence of hand area and fingers and to recognize different hand gestures. As a result, the average recognition accuracy of different hand gestures and stretched fingers numbers are 95.91% and 96%, respectively.