Real-time hand gesture recognition based on feature points extraction

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

With the escalating role of computers in educational system, human computer interaction, is becoming gradually more important part of it. The general believe is that with the progress in computing speed, communication technologies, and display techniques the existing HCI techniques may become a constraint in the effectual utilization of the existing information flow. The development of user interfaces influences the changes in the Human-Computer Interaction (HCI). Human hand gestures have been a mode of non verbal interaction widely used. The vocabulary of hand gesture communication has many variations. It ranges from simple action of using our finger to point at and using hands to move objects around for more complex expressions for the feelings and communicating with others. Also the hand gestures play a prominent role in teaching considering the explanations and exemplifications being highly dependent on hand gestures. Naturalistic and intuitiveness of the hand gesture has been an immense motivating aspect for the researchers in the field of Human Computer Interaction to put their efforts to research and develop the more promising means of interaction involving human and computers. The pursuance for the Human Computer Interaction research is moved by the central dogma of removing the complex and cumbersome interaction devices and replacing them with more obvious and expressive means of interaction which easily comes to the users with least cognitive burden like hand gestures. This paper designs a simple, natural system for gestural interaction between the user and computer for providing a dynamic user interface. The gesture recognition system uses image processing techniques for detection, segmentation, tracking and recognition of hand gestures for converting it to a meaningful command. This hand gesture recognition system has been proposed, designed and developed with the intensions to make it a substitute for mouse while making dynamic user interface between human and machine. Hence instead of making effort to develop a new vocabulary of hand gesture we have matched control instruction set of mouse to subset of most discriminating hand gestures, so that we get a robust interface. The interface being proposed here can be substantially applied towards different applications like image browser, games etc.

The hand gestures provide a natural and intuitive communication modality for human-computer interaction. Efficient human computer interfaces (HCIs) have to be developed to allow computers to visually recognize in real time hand gestures. This project deals with real time hand gesture recognition using AVR Microcontroller. The existing system uses Digital camera for hand gesture recognition. The camera was used as an input device and the object need to present in front of the camera for capturing gesture. It limits mobility and cannot use camera in dark area as well as cost is very high. The camera specifications significantly affect GRS (Gesture Recognition System) characteristics. The proposed system, based on the input signal from the MEMS (Micro Electro Mechanical System) accelerometer sensors measure the acceleration such as tilt, shock & vibrate. It is reliably and accurate detect. Here, the tilt motion information produced by the human subjects and transmitted to the AVR microcontroller i.e. ATmega32A The sign sequence and template matching algorithm developed to identify the gesture in a sequence. Based on the algorithm developed the microcontroller display the position of the hand in a display unit. We conclude that the recognition algorithm based on sign sequence and template matching as presents in this project can be used for non-specific users hand gesture recognition.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

Hand motion tracking and gesture recognition are a fundamental technology in the field of proactive computing for a better human computer interaction system. In this paper, we have developed a 3-D hand motion tracking and gesture recognition system via a data glove (namely the KHU-1 data glove consisting of three tri-axis accelerometer sensors, one controller, and one Bluetooth). The KHU-1 data glove is capable of transmitting hand motion signals to a PC through wireless communication via Bluetooth. Also we have implemented a 3-D digital hand model for hand motion tracking and recognition. The implemented 3-D digital hand model is based on the kinematic chain theory utilizing ellipsoids and joints. Finally, we have utilized a rule-based algorithm to recognize simple hand gestures namely scissor, rock, and paper using the 3-D digital hand model and the KHU-1 data glove. Some preliminary experimental results are presented in this paper.

Human-computer interaction introduces critical open door with the proceeds with improvement of wearable gadgets. Gesture recognition through smart devices is becoming a popular research direction. This paper proposes a hand gesture recognition and real-time game control system that is capable of continues human-computer interaction in view of an off-the-rack business wearable wristband. We utilize three-axis accelerator and gyroscope sensors embedded in smart band to collect hand motion information and use Kinect camera capture video information for manual segmentation during the training model phase. A continuous gesture segmentation algorithm based on sliding window and DTW algorithm is developed to detect meaningful gestures in the real-time game control stage. In addition, an android game named Fly Birds which is controlled by gesture recognition result is presented to simulate real-time human-computer interaction. Then, we classify the data in the window using common classifiers. Finally, our experimental results show that, we can accurately identify the designed gestures during the stage of static gesture recognition, and we also achieve a perfect interactive effect in the process of dynamic real-time game control. The experiment outcomes will advance the ascent of human-PC cooperation in view of hand gesture recognition and related applications will rise in vast numbers.

Hand gesture recognition (HGR) systems using electromyography (EMG) bracelet-type sensors are currently largely used over other HGR technologies. However, bracelets are susceptible to electrode rotation, causing a decrease in HGR performance. In this work, HGR systems with an algorithm for orientation correction are proposed. The proposed orientation correction method is based on the computation of the maximum energy channel using a synchronization gesture. Then, the channels of the EMG are rearranged in a new sequence which starts with the maximum energy channel. This new sequence of channels is used for both training and testing. After the EMG channels are rearranged, this signal passes through the following stages: pre-processing, feature extraction, classification, and post-processing. We implemented user-specific and user-general HGR models based on a common architecture which is robust to rotations of the EMG bracelet. Four experiments were performed, taking into account two different metrics which are the classification and recognition accuracy for both models implemented in this work, where each model was evaluated with and without rotation of the bracelet. The classification accuracy measures how well a model predicted which gesture is contained somewhere in a given EMG, whereas recognition accuracy measures how well a model predicted when it occurred, how long it lasted, and which gesture is contained in a given EMG. The results of the experiments (without and with orientation correction) executed show an increase in performance from 44.5% to 81.2% for classification and from 43.3% to 81.3% for recognition in user-general models, while in user-specific models, the results show an increase in performance from 39.8% to 94.9% for classification and from 38.8% to 94.2% for recognition. The results obtained in this work evidence that the proposed method for orientation correction makes the performance of an HGR robust to rotations of the EMG bracelet.

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

Hand gesture recognition is a quickly developing field with many uses in human-computer interaction, sign language recognition, virtual reality, gaming, and robotics. This paper reviews different ways to model hands, such as vision-based, sensor-based, and data glove-based techniques. It emphasizes the importance of accurate hand modeling and feature extraction for capturing and analyzing gestures. Key features like motion, depth, color, shape, and pixel values and their relevance in gesture recognition are discussed. Challenges faced in hand gesture recognition include lighting variations, complex backgrounds, noise, and real-time performance. Machine learning algorithms are used to classify and recognize gestures based on extracted features. The paper emphasizes the need for further research and advancements to improve hand gesture recognition systems' robustness, accuracy, and usability. This review offers valuable insights into the current state of hand gesture recognition, its applications, and its potential to revolutionize human-computer interaction and enable natural and intuitive interactions between humans and machines. In simpler terms, hand gesture recognition is a way for computers to understand what people are saying with their hands. It has many potential applications, such as allowing people to control computers without touching them or helping people with disabilities communicate. The paper reviews different ways to develop hand gesture recognition systems and discusses the challenges and opportunities in this area.

Considerable effort has been put toward the development of intelligent and natural interfaces between users and computer systems. In line with this endeavor, several modes of information (e.g., visual, audio, and pen) that are used either individually or in combination have been proposed. The use of gestures to convey information is an important part of human communication. Hand gesture recognition is widely used in many applications, such as in computer games, machinery control (e.g., crane), and thorough mouse replacement. Computer recognition of hand gestures may provide a natural computer interface that allows people to point at or to rotate a computer-aided design model by rotating their hands. Hand gestures can be classified into two categories: static and dynamic. The use of hand gestures as a natural interface serves as a motivating force for research on gesture taxonomy, its representations, and recognition techniques. This paper summarizes the surveys carried out in human--computer interaction (HCI) studies and focuses on different application domains that use hand gestures for efficient interaction. This exploratory survey aims to provide a progress report on static and dynamic hand gesture recognition (i.e., gesture taxonomies, representations, and recognition techniques) in HCI and to identify future directions on this topic.

Hand gesture recognition is used in human–computer interaction, with multiple applications in assistive technologies, virtual reality, and smart systems. While vision-based methods are commonly employed, they are often computationally intensive, sensitive to environmental conditions, and raise privacy concerns. This work proposes a hardware/software co-optimized system for real-time hand gesture recognition using accelerometer data, designed for a portable, low-cost platform. A Convolutional Neural Network from TinyML is implemented on a Xilinx Zynq-7000 SoC-FPGA, utilizing fixed-point arithmetic to minimize computational complexity while maintaining classification accuracy. Additionally, combined architectural optimizations, including pipelining and loop unrolling, are applied to enhance processing efficiency. The final system achieves a 62× speedup over an unoptimized floating-point implementation while reducing power consumption, making it suitable for embedded and battery-powered applications.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded system. In this work, we propose an HCI system for dynamic recognition of driver micro hand gestures, which can have a crucial impact in automotive sector especially for safety related issues. For this purpose, we initially collected a dataset named Driver Micro Hand Gestures (DriverMHG), which consists of RGB, depth and infrared modalities. The challenges for dynamic recognition of micro hand gestures have been addressed by proposing a lightweight convolutional neural network (CNN) based architecture which operates online efficiently with a sliding window approach. For the CNN model, several 3-dimensional resource efficient networks are applied and their performances are analyzed. Online recognition of gestures has been performed with 3D-MobileNetV2, which provided the best offline accuracy among the applied networks with similar computational complexities. The final architecture is deployed on a driver simulator operating in real-time. We make DriverMHG dataset and our source code publicly available 1.1https://www.mmk.ei.tum.de/DriverMHG/

Human Computer Interaction (HCI) has become very important in today's world. Existing HCI techniques like keyboard, mouse, joysticks etc. limit the speed and naturalness of our interaction. Hand gestures provide a more natural way of communicating with the computer. Hand segmentation is the most crucial step in every hand gesture recognition system. All subsequent steps greatly depend on the quality of segmentation. If data is lost due to improper segmentation, it can hardly be recovered in the later stages of gesture recognition process. In this paper, we present a robust and effective method of hand segmentation which overcomes problems such as skin color detection, complex background removal, complexity from multiple gesturers in front of the camera and variable lighting condition.

Hand gesture recognition is a relevant study topic for a reason that sometimes we may not be in position to communicate verbally. There is need to design Hand gesture recognition systems in order to help people adopt to nonverbal communication mainly sign language. However, there is no clue to understand the meaning of gesture through the computers directly. So this calls for definitions that generalize models in a computer. That is why the machine-learning approaches are implemented in recognition systems. There are generally two types of hand gestures recognition systems which researches have concentrated on. These include static and dynamic Hand gesture recognition systems. However, in building Hand gesture recognition systems, various machine learning approaches have been used. For implementing the proposed system, MS Kinect depth sensor was used as a hardware. The Kinect depth sensor is composed of an infrared camera. This is an advantage to the systems that are designed basing on the depth sensing because factors like color, clothing and background have less effect on the performance. So Kinect based depth sensor systems have a high accuracy and performance making them relevant and applicable in our daily lives. In this paper, we propose a static hand gesture recognition system in real time using two machine learning methods namely Support Vector Machine and Artificial Neural Networks. We use of the newly launched Microsoft Kinect sensor for image extraction. The sensor helps us to extract the hand images. We implement the system on a Matlab platform for reasons that Matlab is widely used by researchers in different fields and that can handle complex computations. In the training of the model, we collect a hundred depth-based Histogram of Oriented Gradient features per alphabet from the hand gesture images which we trained, tested and validated using Artificial Neural Networks (ANN) and Support Vector Machine (SVM). From this dataset, we can generate the generalized gesture model for each alphabet image. For the proposed system, the classification with ANN proves a higher performance then SVM.

In today’s ever-evolving healthcare landscape, our objective is to bring technology closer to human experience. In this paper, a vision-based hand gesture recognition (HGR) system is introduced as a human computer interface (HCI) that transforms natural movements into intuitive medium for device and application manipulation. This innovative approach is designed to empower the elderly and enhance patient monitoring by offering a seamless, nonintrusive interface. By bridging compassion with cutting-edge technology, this approach aims to redefine care and create a more connected, responsive healthcare environment. The study begins by reviewing both traditional and contemporary HGR methodologies, with a focus on vision-based systems, which have shown greater potential for practical applications compared to conventional sensor-based systems. Traditional approaches, such as contour analysis, colour segmentation, and template matching, have demonstrated limitations. In contrast, deep learning approaches have gained mass adaptation in vision-based systems for their ability to learn hierarchical features and better handling complex, dynamic scenarios. However, existing models struggle in real-time performance due to computational inefficiencies and environmental variations. This study introduces a deep learning-based detection framework which enables the practical application of HGR. For example, it supports the development of gesture-based virtual assistants for interacting with digital devices and allows elderly residents in care homes to give instructions to remote caregivers. The framework consists of two deep learning models, one for region of interest (ROI) detection and another for classification. The recognition process begins with capturing RGB images, chosen for versatility, affordability, and compatibility, which then passes through YOLOv8 (You Only Look Once version 8) for detection. The YOLOv8 model is trained on a subset, only 4.5%, of the HaGRID dataset, comprises over 550,000 RGB images, to accurately locate hand regions. Once detection is complete, several image processing techniques are applied to overcome typical HGR constraints. These include ROI extraction and gray scaling for faster computation, resizing for model optimization and distance viability, HSV conversion for lighting independence, and histogram equalization for enhanced feature extraction. The processed image is then fed into a convolutional neural network (CNN) for gesture classification. The network is designed with several convolutional layers, followed by dense layers with ReLU activation to introduce non-linearity, and SoftMax activation for classification. To improve the model's robustness, data augmentation techniques such as random flipping and rotation are applied. The HGR system demonstrates a staggering accuracy of 97% for training & validation. The proposed framework effectively addresses the limitations of conventional HGR systems—many of which have been developed and tested in controlled environments—demonstrating its applicability in real-life situations.