SSEC: Semantic Segmentation and Ensemble Classification Framework for Static Hand Gesture Recognition using RGB-D Data

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

The interaction between human and machine can be achieved by implementing human gesture applications which are known as human computer interaction (HCI). It has been developed several years ago, and now we can easily find it in many applications in real world e.g. in virtual reality, games, robotics, sign language recognition, etc. Gesture recognition has the potential to offer a natural way of communication between humans and machines. The most popular methods in human gesture application are hand gesture recognition. Several of hand gesture recognition methods still have the same problem when trying to recognize hand gesture accurately because of the existence of the uncontrolled environment problem, especially complex background. This study offered a method that can handle complex background in hand gesture recognition by analyzing pattern in spatio temporal domain. The system could reduce irregular noise in complex background significantly that affected to recognize the hand gesture. The experiment had been conducted using 180 videos with a total of 15 gestures. The result showed that the proposed method could recognize 15 hand gestures with accuracy of 98.33%.

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Hand gesture recognition is a quickly developing field with many uses in human-computer interaction, sign language recognition, virtual reality, gaming, and robotics. This paper reviews different ways to model hands, such as vision-based, sensor-based, and data glove-based techniques. It emphasizes the importance of accurate hand modeling and feature extraction for capturing and analyzing gestures. Key features like motion, depth, color, shape, and pixel values and their relevance in gesture recognition are discussed. Challenges faced in hand gesture recognition include lighting variations, complex backgrounds, noise, and real-time performance. Machine learning algorithms are used to classify and recognize gestures based on extracted features. The paper emphasizes the need for further research and advancements to improve hand gesture recognition systems' robustness, accuracy, and usability. This review offers valuable insights into the current state of hand gesture recognition, its applications, and its potential to revolutionize human-computer interaction and enable natural and intuitive interactions between humans and machines. In simpler terms, hand gesture recognition is a way for computers to understand what people are saying with their hands. It has many potential applications, such as allowing people to control computers without touching them or helping people with disabilities communicate. The paper reviews different ways to develop hand gesture recognition systems and discusses the challenges and opportunities in this area.

Recently, human - machine interaction (HMI) becomes a hot research topic because of its wide applications, ranging from automatic device control to designing and development of assistant robot or even smart building at sparser scale. One of the most important questions in this research field is finding out an efficient and natural method of HMI. Among several channels of communication, hand gestures have been shown to be an intuitive and efficient mean to express an idea or to control something. For a successful hand gesture based interaction between human and robot, a vocabulary of hand gestures needs to be defined. To resolve this problem, in this paper, we propose a framework to study the behavior of Vietnamese in using of hand gesture in communication with robot. This study allows designing a hand gesture vocabulary for human - robot interaction (HRI) applications. In the literature, there are no works similar to ours. This makes our twofold contributions: (1) a proposed framework for hand gesture recognition: hand gesture vocabulary design, feature extraction for hand posture representation, hand posture classification, and hand gesture database construction; (2) some experimentations are realized to evaluate the defined hand gesture set that can be used in general situation of HRI. The experiment results show that the defined hand gesture set satisfies the both criteria: intuitiveness and recognisability.

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

There has been growing interest in the development of new approaches and technologies for bridging the human-computer barrier. Hand gesture recognition is considered as an interaction technique having potential to communicate with machines. Human computer interaction (HCI) was never an easy task and lots of approaches are available to build such systems. Hand gesture recognition (HGR) using wearable data glove provides a solution to build a HCI system, but lags in terms of its computational time and poor interface. Pattern matching is one more solution which uses vision based techniques and provides strong interface to build HCI systems. But again, it requires complex algorithms which takes lots of computational time and hence limits its use in real time HCI applications. In this paper, we presented an eccentric approach for hand gesture recognition which is simple, fast and user independent and can be used to develop real time HCI applications. Based on proposed algorithm we built a system for Indian Sign Language recognition which converts Indian Sign numbers into text. The algorithm first captures the image of single handed gesture of speech/hearing impaired person using simple webcam and then using our proposed algorithm it classifies the gesture into its appropriate class. It uses simple logical conditions for gesture classification which make its use in real time HCI applications.

Semantic segmentation is a fundamental task in image processing and computer vision domains. Therefore, it is a core component of most emerging industry applications such as medical imaging, autonomous driving, and agriculture. The last five years have witnessed a huge growth in the computer vision domain with the introduction of Generative Adversarial Networks (GANs) due to their ability to learn deep representations by a competitive process between two neural networks called the Generator the Discriminator. Neural Architecture Search (NAS) is undergoing a revolution aimed at automating the architectural engineering process. This paper discusses implementing a novel framework for semantic image segmentation called iSegmentor using NAS and GANs. The proposed approach is the first framework that utilizes NAS in GANs for semantic image segmentation to the best of our knowledge. The iSegmentor framework was evaluated using mean Intersection over Union (mIoU) and Mean Pixel Accuracy (MPA) metrics. The iSegmentor was mainly tested and evaluated for the autonomous driving domain. With the analysis of the past five-year literature, Cityscapes and CamVid datasets were chosen. The iSegmentor achieved 81.73 MPA and 71.90 mIoU for the Cityscapes dataset, whereas CamVid achieved 80.41 MPA and 70.63 mIoU. Further, we present future avenues that we identified with the conducted experiments to assist other researchers to extend the research area of NAS in GANs in semantic segmentation.

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Human hand gestures as a natural way of interaction and communicating with computers is becoming an emerging and demanding field of research due to its various applications like sign language recognition, human computer interaction, gaming, virtual reality etc. Hand gesture recognition under 2D environment has some limitations as the information about the other dimension (z-axis) is missed. So, hand gesture recognition under 3D environment is becoming a growing field of research. In this paper, we have implemented the realtime disparity computation and proposed a novel technique to detect gesture of Front and Back along with forward and backward movement towards and from the camera respectively. Our technique is based on stereo-vision and we have used the disparity map-based intensity measure of segmented hand and changing of its intensity as feature to classify and recognize the gesture. Stereo calibration followed by rectification is done to get the rectified images and correspondence gives the depth map. Finally three dimensional reprojection is performed. Our technique works well for the gestures and the results are promising.

Any human–computer interaction application needs to be able to recognize gestures. Hand gesture detection systems that recognize gestures in real time can improve human–computer interaction by making it more intuitive and natural. Colour gloves and skin colour detection are two prominent hand segmentation and detection technologies, but each has its own set of benefits and drawbacks. For physically challenged people, gesture identification is a crucial technique of sharing information. Support vector machine algorithm with principal component analysis, hidden Markov model, superposed network with multiple restricted Boltzmann machines, growing neural gas algorithm, convolutional neural network, double channel convolutional neural network, artificial neural network, and linear support vector machine with gesture dataset matches the interaction of gestures for various postures in real time. Although this method can recognize a huge number of gestures, it does have certain downsides, such as missing movements due to the accuracy of the categorization algorithms. Furthermore, matching the vast dataset takes a longer time. The main contribution of this work lies in a conceptual framework based on the findings of a systematic literature review that provides fruitful implications based on recent research findings and insights that can be used to direct and initiate future research initiatives in the field of hand gesture recognition and artificial intelligence. As a result, a novel study based on a hybrid recurrent neural network (RNN) with chaos game optimization may reduce classification mistakes, increase stability, maximize resilience, and efficiently use the hand gestures recognition system.

Hand gesture recognition has great applications in human-computer interaction (HCI), human-robot interaction (HRI), and supporting the deaf and mute. To build a hand gesture recognition model using deep learning (DL) with high results then needs to be trained on many data and in many different conditions and contexts. In this paper, we publish the TQU-HG dataset of large RGB images with low resolution (640×480) pixels, low light conditions, and fast speed (16 fps). TQU-HG dataset includes 60,000 images collected from 20 people (10 male, 10 female) with 15 gestures of both left and right hands. A comparative study with two branches: i) based on Mediapipe TML and ii) Based on convolutional neural networks (CNNs) (you only look once (YOLO); YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLO-Nas, single shot multiBox detector (SSD) VGG16, residual network (ResNet)18, ResNext50, ResNet152, ResNext50, MobileNet V3 small, and MobileNet V3 large), the architecture and operation of CNNs models are also introduced in detail. We especially fine-tune the model and evaluate it on TQU-HG and HaGRID datasets. The quantitative results of the training and testing are presented (F1-score of YOLOv8, YOLO-Nas, MobileNet V3 small, ResNet50 is 98.99%, 98.98%, 99.27%, 99.36%, respectively on the TQU-HG dataset and is 99.21%, 99.37%, 99.36%, 86.4%, 98.3%, respectively on the HaGRID dataset). The computation time of YOLOv8 is 6.19 fps on the CPU and 18.28 fps on the GPU.

With rapid developments in biometric recognition, a great deal of attention is being paid to robots which interact smartly with humans and communicate certain types of biometrical information. Such human–machine interaction (HMI), also well-known as human–robot interaction (HRI), will, in the future, prove an important development when it comes to automotive manufacturing applications. Currently, hand gesture recognition-based HRI designs are being practically used in various areas of automotive manufacturing, assembly lines, supply chains, and collaborative inspection. However, very few studies are focused on material-handling robot interactions combined with hand gesture communication of the operator. The current work develops a depth sensor-based dynamic hand gesture recognition scheme for continuous-time operations with material-handling robots. The proposed approach properly employs the Kinect depth sensor to extract features of Hu moment invariants from depth data, through which feature-based template match hand gesture recognition is developed. In order to construct continuous-time robot operations using dynamic hand gestures with concatenations of a series of hand gesture actions, the wake-up reminder scheme using fingertip detection calculations is established to accurately denote the starting, ending, and switching timestamps of a series of gesture actions. To be able to perform typical template match on continuous-time dynamic hand gesture recognition with the ability of real-time recognition, representative frame estimates using centroid, middle, and middle-region voting approaches are also presented and combined with template match computations. Experimental results show that, in certain continuous-time periods, the proposed complete hand gesture recognition framework can provide a smooth operation for the material-handling robot when compared with robots controlled using only extractions of full frames; presented representative frames estimated by middle-region voting will maintain fast computations and still reach the competitive recognition accuracy of 90.8%. The method proposed in this study can facilitate the smart assembly line and human–robot collaborations in automotive manufacturing.

Considerable effort has been put toward the development of intelligent and natural interfaces between users and computer systems. In line with this endeavor, several modes of information (e.g., visual, audio, and pen) that are used either individually or in combination have been proposed. The use of gestures to convey information is an important part of human communication. Hand gesture recognition is widely used in many applications, such as in computer games, machinery control (e.g., crane), and thorough mouse replacement. Computer recognition of hand gestures may provide a natural computer interface that allows people to point at or to rotate a computer-aided design model by rotating their hands. Hand gestures can be classified into two categories: static and dynamic. The use of hand gestures as a natural interface serves as a motivating force for research on gesture taxonomy, its representations, and recognition techniques. This paper summarizes the surveys carried out in human--computer interaction (HCI) studies and focuses on different application domains that use hand gestures for efficient interaction. This exploratory survey aims to provide a progress report on static and dynamic hand gesture recognition (i.e., gesture taxonomies, representations, and recognition techniques) in HCI and to identify future directions on this topic.