Mediapipe-based keypoint extraction with optimized neural models for offline smart control

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Human hand gestures as a natural way of interaction and communicating with computers is becoming an emerging and demanding field of research due to its various applications like sign language recognition, human computer interaction, gaming, virtual reality etc. Hand gesture recognition under 2D environment has some limitations as the information about the other dimension (z-axis) is missed. So, hand gesture recognition under 3D environment is becoming a growing field of research. In this paper, we have implemented the realtime disparity computation and proposed a novel technique to detect gesture of Front and Back along with forward and backward movement towards and from the camera respectively. Our technique is based on stereo-vision and we have used the disparity map-based intensity measure of segmented hand and changing of its intensity as feature to classify and recognize the gesture. Stereo calibration followed by rectification is done to get the rectified images and correspondence gives the depth map. Finally three dimensional reprojection is performed. Our technique works well for the gestures and the results are promising.

DeReFNet: Dual-stream Dense Residual Fusion Network for static hand gesture recognition

Interaction with a computer has been the center of innovation ever since the advent of input devices. From simple punch cards to keyboards, there are number of novel ways of interaction with computers which influence the user experience. Communicating using gestures is perhaps one of the most natural ways of interaction. Gesture recognition as a tool for interpreting signs constitutes a pivotal area in gesture recognition research where accuracy of the algorithm and the ease of usability determine the effectiveness of the algorithm or system. Introducing gesture based interaction in Virtual reality applications has not only helped solve problems which were commonly reported in traditional Virtual Reality systems, but also gives user a more natural and enriching experience. This paper concentrates on comparison of different systems and identifying their similarities, differences, advantages and demerits which can play a key role in designing a system using such technologies.

Researchers have recently focused their attention on vision-based hand gesture recognition. However, due to several constraints, achieving an effective vision-driven hand gesture recognition system in real time has remained a challenge. This paper aims to uncover the limitations faced in image acquisition through the use of cameras, image segmentation and tracking, feature extraction, and gesture classification stages of vision-driven hand gesture recognition in various camera orientations. This paper looked at research on vision-based hand gesture recognition systems from 2012 to 2022. Its goal is to find areas that are getting better and those that need more work. We used specific keywords to find 108 articles in well-known online databases. In this article, we put together a collection of the most notable research works related to gesture recognition. We suggest different categories for gesture recognition-related research with subcategories to create a valuable resource in this domain. We summarize and analyze the methodologies in tabular form. After comparing similar types of methodologies in the gesture recognition field, we have drawn conclusions based on our findings. Our research also looked at how well the vision-based system recognized hand gestures in terms of recognition accuracy. There is a wide variation in identification accuracy, from 68% to 97%, with the average being 86.6 percent. The limitations considered comprise multiple text and interpretations of gestures and complex non-rigid hand characteristics. In comparison to current research, this paper is unique in that it discusses all types of gesture recognition techniques.

Hand gestures have been widely applied to interface as the way of interaction between human and computers. Since a human hand can express various shapes of gestures, previous models for recognizing them cannot distinguish them accurately since they use only single model for recognition. For efficient hand gesture recognition with its enhanced performance, we propose the fuzzy c-means clustering based mixture-of-experts (FME). The proposed method uses multiple local experts obtained via fuzzy c-means clustering and decisions from them are combined with the gating network. To evaluate the performance of the proposed method, we conduct experiments including comparisons with alternative models for hand gesture recognition. As the result of experiments, the proposed model shows improved gesture recognition performance, especially performance on similar hand gesture recognition.

Human-computer interaction introduces critical open door with the proceeds with improvement of wearable gadgets. Gesture recognition through smart devices is becoming a popular research direction. This paper proposes a hand gesture recognition and real-time game control system that is capable of continues human-computer interaction in view of an off-the-rack business wearable wristband. We utilize three-axis accelerator and gyroscope sensors embedded in smart band to collect hand motion information and use Kinect camera capture video information for manual segmentation during the training model phase. A continuous gesture segmentation algorithm based on sliding window and DTW algorithm is developed to detect meaningful gestures in the real-time game control stage. In addition, an android game named Fly Birds which is controlled by gesture recognition result is presented to simulate real-time human-computer interaction. Then, we classify the data in the window using common classifiers. Finally, our experimental results show that, we can accurately identify the designed gestures during the stage of static gesture recognition, and we also achieve a perfect interactive effect in the process of dynamic real-time game control. The experiment outcomes will advance the ascent of human-PC cooperation in view of hand gesture recognition and related applications will rise in vast numbers.

This paper aims to develop a real time system for deaf and dump people for achieving better way of communication by using hand gesture recognition. Existing way of communication is not much helpful for the deaf and dump people, since existing approach depends upon translator person, who can understand the sign language of the dump and they manually decode their signs and convey the message. The translator may commit some mistakes due to wrong recognition and that limitation could be significantly overcome by using an image acquisition device and processing unit to process the gestures. The personal computer will correctly display the message which was communicated.

Gesture recognition technology is rapidly growing in the recent years due to the demands of many application such as computer game and sport, human robot interaction, assistant systems, sign language interpretation and e-commerce. One of the most important of gesture recognition is hand-gesture recognition. For example, it can be used to control all devices (television, radio, air-condition, and doors) by just hand gestures for smart home application. The HGM-4 dataset is built for hand gesture recognition (the full dataset is available from: https://data.mendeley.com/datasets/jzy8zngkbg/4) which contains total 4,160 color images (1280 × 700 pixels) of 26 hand gestures captured by four cameras at different position. The training and testing set are defined to create a benchmark framework for comparing the experimental results.

For hand gesture recognition (HGR) sensing based on frequency modulated continuous wave (FMCW) radar, it is challenging to cope with the ambiguity and variability of gesture motions performed by different peoples due to the private habits or position, which inevitably reduce the robustness of HGR sensing. To solve this problem, we propose a novel dual-flow fusion deformable residual network (DFDRN). More detailed gesture information, time frequency maps and spectrum map videos, are fed into DFDRN, which improves the generalization and motion modeling ability of HGR through the extensive feature space. To further improve the robustness of HGR sensing, the deformable block is exploited to replace the convolution. To focus on the important features of dual-flow fusion, the convolutional block attention module (CBAM) is embedded in the backbone of DFDRN. Experimental results show that the accuracy of DFDRN, which can reach 97.5%, is obviously better than the state-of-the-art HGR methods.

Hand gesture recognition is a relevant study topic for a reason that sometimes we may not be in position to communicate verbally. There is need to design Hand gesture recognition systems in order to help people adopt to nonverbal communication mainly sign language. However, there is no clue to understand the meaning of gesture through the computers directly. So this calls for definitions that generalize models in a computer. That is why the machine-learning approaches are implemented in recognition systems. There are generally two types of hand gestures recognition systems which researches have concentrated on. These include static and dynamic Hand gesture recognition systems. However, in building Hand gesture recognition systems, various machine learning approaches have been used. For implementing the proposed system, MS Kinect depth sensor was used as a hardware. The Kinect depth sensor is composed of an infrared camera. This is an advantage to the systems that are designed basing on the depth sensing because factors like color, clothing and background have less effect on the performance. So Kinect based depth sensor systems have a high accuracy and performance making them relevant and applicable in our daily lives. In this paper, we propose a static hand gesture recognition system in real time using two machine learning methods namely Support Vector Machine and Artificial Neural Networks. We use of the newly launched Microsoft Kinect sensor for image extraction. The sensor helps us to extract the hand images. We implement the system on a Matlab platform for reasons that Matlab is widely used by researchers in different fields and that can handle complex computations. In the training of the model, we collect a hundred depth-based Histogram of Oriented Gradient features per alphabet from the hand gesture images which we trained, tested and validated using Artificial Neural Networks (ANN) and Support Vector Machine (SVM). From this dataset, we can generate the generalized gesture model for each alphabet image. For the proposed system, the classification with ANN proves a higher performance then SVM.

Cloud computing and big data: Technologies and applications

- Stroke patients often face challenges in communication and independent medication management, necessitating innovative assistive technologies for better healthcare monitoring. This project introduces a real-time hand gesture-based system to monitor tablet intake in stroke patients using computer vision and machine learning techniques. The solution integrates OpenCV for video processing, Media Pipe for accurate hand landmark detection, and Twilio for sending automated SMS alerts to caregivers. The system operates by detecting specific hand gestures-particularly the movement of the index finger toward the mouth-which signifies an attempt to consume medication. When this gesture is confirmed, the system logs the event, updates the tablet count, and triggers both a visual alert using Tkinter and an SMS notification via Twilio. This ensures real-time tracking and caregiver awareness without physical intervention. This low-cost, non-invasive solution can be implemented using a standard webcam and basic hardware, making it especially suitable for home healthcare environments. The methodology enhances patient autonomy while maintaining safety and accountability. Future developments may incorporate deep learning for more robust gesture recognition and extend the system’s functionality to cover broader health monitoring scenarios. Key Words: Hand Gesture Recognition, Stroke Patient Monitoring, Computer Vision, Real-time Alert System. .

Interacting with mobile devices can be challenging in adverse working environments. Using hand gestures for interaction can overcome severe usability issues that users face when working with mobile devices in industrial settings. This chapter is dedicated to the design, implementation, and evaluation of mobile information systems with hand gesture recognition as means for human computer interaction. The chapter provides a concise theoretical background on human gestural interaction and gesture recognition, guidelines for the design of gesture vocabularies, recommendations for proper implementation, and parameterization of robust and reliable recognition algorithms on energy-efficient 8-bit architectures. It additionally presents an elaborated process for participatory design and evaluation of gesture vocabularies to ensure high usability in the actual context of use. The chapter concludes with a case study that proves the suitability of the proposed framework for the design of efficient and reliable hand gesture-based user interfaces.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded system. In this work, we propose an HCI system for dynamic recognition of driver micro hand gestures, which can have a crucial impact in automotive sector especially for safety related issues. For this purpose, we initially collected a dataset named Driver Micro Hand Gestures (DriverMHG), which consists of RGB, depth and infrared modalities. The challenges for dynamic recognition of micro hand gestures have been addressed by proposing a lightweight convolutional neural network (CNN) based architecture which operates online efficiently with a sliding window approach. For the CNN model, several 3-dimensional resource efficient networks are applied and their performances are analyzed. Online recognition of gestures has been performed with 3D-MobileNetV2, which provided the best offline accuracy among the applied networks with similar computational complexities. The final architecture is deployed on a driver simulator operating in real-time. We make DriverMHG dataset and our source code publicly available 1.1https://www.mmk.ei.tum.de/DriverMHG/