Integrating Human Hand Gestures with Vision Based Feedback Controller to Navigate a Virtual Robotic Arm

Real world virtual world communication and interaction will be a cornerstone of future intelligent manufacturing ecosystems. Human robotic interaction is considered to be the basic element of factories of the future. Despite the advancement of different technologies such as wearables and Augmented Reality (AR), human-robot interaction (HRI) is still extremely challenging. Whilst progress has been made in the development of different mechanisms to support HRI, there are issues with cost, naturalistic and intuitive interaction, and communication across heterogeneous systems. To mitigate these limitations, RoSTAR is proposed. RoSTAR is a novel open-source HRI system based on the Robot Operating System (ROS) and Augmented Reality. An AR Head Mounted Display (HMD) is deployed. It enables the user to interact and communicate through a ROS powered robotic arm. A model of the robot arm is imported directly into the Unity Game engine, and any interactions with this virtual robotic arm are communicated to the ROS robotic arm. This system has the potential to be used for different process tasks, such as robotic gluing, dispensing and arc welding as part of an interoperable, low cost, portable and naturalistically interactive experience.

Significance: The ability of robotic arm system to become teleoperational is essential for advancing remote work. Existing solutions often require complex setups, specialized training, and expensive hardware. Developing a simplified wearable device that can be rapidly integrated and deployed can significantly enhance accessibility where remote control is paramount, including remote surgeries in healthcare and hazardous environments. Aim and approach: This work presents the design and development of a wireless glove-based control system for remotely operating a robotic arm. A reliable, responsive, and user-friendly control system offers scalability, customization options, intuitive interaction, and ergonomic comfort. The approach combines advanced motion sensing using a 9 Degrees of Freedom (DoF) Inertial Measurement Unit (IMU) and real-time data processing. The IMU estimates roll, pitch, and yaw, directly controlling the translational movements (x, y, z) of the robotic arm’s end effector. Depending on button states, these inputs can also switch to control the end-effector rotations. Results: The developed glove demonstrated user compliance. The system is rapid to deploy, and the wireless device successfully captured real-time gesture control data and redirected it to the robotic system with an average latency of 185 ms, enabling the system to perform a range of manipulations, including pick and place, reaching, and grasping objects. Conclusions: This research presents a robust and efficient teleoperation system, offering a viable solution for advancements in remote robotic arm control. This innovation bridges the gap between the full potential of complex robotic systems and user interaction, unlocking new possibilities across various fields.

We have developed a hybrid model of motor control for a brain-machine interface that is based on a large-scale model of primary motor cortex (M1) based on several mammalian studies (Figure ​(Figure1).1). The M1 model consisted of 10,000 spiking Izhikevich neurons of four types: regular-firing and bursting pyramidal neurons, and fast-spiking and low-threshold-spiking interneurons. Within the M1 population, cell proportions, locations, connectivity and delays were drawn primarily from mouse experimental data. Properties were based on cell body cortical depth (distance from pia to white matter). Synapses included four different receptors: AMPA, NMDA, GABAA and GABAB. The model exhibited realistic physiological properties, including firing rates and local field potential spectra. Figure 1 Model of primary motor cortex showing inputs from PMd biological neurons, and closed-loop interaction with virtual musculoskeletal arm. Cell types are shown with different colors. The output connections of a single L5A neuron are shown to illustrate connectivity. ... Pyramidal tract-projecting neurons in layer 5B were connected to a descending spinal cord neural population, which provided excitation to the muscles of a realistic virtual musculoskeletal arm. Proprioceptive feedback from the arm was encoded in an ascending spinal cord population which then projected to M1 layer 2/3. The virtual arm movements were also followed by a robotic arm [1]. An additional population, which reproduced the spiking patterns recorded from 92 neurons of macaque dorsal premotor cortex (PMd) during a center-out reaching task, was connected to M1 layer 5A providing a modulatory input [2]. The network was trained to drive the virtual arm to reach multiple targets, by combining arm exploratory movements with reinforcement learning and spike-timing dependent plasticity (STDP). Synaptic plasticity occurred between multiple populations, including between the PMd inputs and layer 5A neurons. Tuning of learning metaparameters was achieved by employing parallel evolutionary algorithms in a high-performance computing cluster. This work moves towards a new generation of neuroprosthetic systems where biological brain circuits interact directly with biomimetic cortical models, and employ co-adaptation and learning to achieve a functional task. Such a multiscale approach, ranging from the cellular to the behavioral level, will provide deeper insights into brain dynamics and have applications for the diagnosis and restoration of brain disorders.

Kinect is a technology developed by Microsoft that perceives human movements and plays with these movements. On the other hand, virtual reality creates the feeling that we are in an environment that we are not actually in. Purpose of this work is to develop a robotic arm that repeats nonsensitive movement of the person through Kinect and through the camera, it is aimed to see the environment in which the robotic arm is placed virtually. With the combination of Kinect technology and VR technology, full control and realism are provided to the user from a distance. This feature distinguishes the proposed approach from other studies. When human health and work accidents are considered, this system is intended to increase safety and protect human health in dangerous environment.

Towards establishing future-oriented relations between humans and robots, not only verbal, but also nonverbal robot functions have been studied. However, it is not clear how these different functions should be combined and implemented in robots in order to achieve ‘natural’ human-robot communication or interaction. In this paper, we focus on the timing for the coordination between utterance as a verbal function and body motion as a nonverbal function. To begin with, we investigated the temporal order and correlation between utterance and gesture in an interpersonal (human-human) interaction scenario as a preliminary experiment (Experiment 1). Then, we controlled the robot's response timing based on findings of the first experiment and evaluated the effectiveness of the robot's timing control in human-robot interaction (Experiment 2). It was shown that elderly participants significantly preferred the controlled robot's response. In addition, the timing of utterance and gestures affected the impression of the robot and interpretation of the verbal message. These results are important findings for designing socially interactive robots for the aging society.

As population aging intensifies and automation technology advances, robotic massage has emerged as a significant healthcare innovation. However, participants often experience discomfort and fear when facing cold robot arms. This study proposes and validates a Mixed Reality (MR) solution integrating high-fidelity virtual massage therapists with robotic massage systems to improve participant experience. We evaluated three conditions: real robot arm, virtual robot arm, and virtual massage therapist. Results demonstrate that virtual anthropomorphic representation significantly improves participant trust, positive affect, and engagement while reducing psychological barriers. Despite challenges like increased oculomotor strain, our research confirms MR technology’s potential in enhancing psychological acceptance of robotic massage, opening new avenues for robotic massage services that may extend to other intimate human-robot interaction scenarios.

This paper presents the development of Robot Technology Components (RTCs) using RT-Middleware (RTM) for designing assistive robotic arms and human-machine interfaces (HMIs) that can be used by the disabled. RTM is an open software platform for a component-oriented robot system. An RTC is the basic functional unit for constructing a robot system using RTM. The interface definition for the basic functionality of RTCs is standardized by the Object Management Group (OMG). Therefore, software modules written as RTCs can be shared and reused in other robot systems. One feature of an assistive robotic arm is that the arm is controlled by manual operation. Various HMIs for manually operating the robotic arms have been developed according to the abilities of each user and the requirements regarding the actions that the robot can perform. These HMIs can be interchangeably connected to the assistive robotic arms by implementing the software modules for the robotic arm and the HMI as RTCs with a common software interface.

Robot arm technology has revolutionized the manufacturing industry, offering worker safety and productivity benefits. However, issues like unintended actions and difficulty in intuitive operation remain. This study proposes a remote-control system for robot arms that combines augmented reality (AR) and a multi-sensor interactive smart ball (PALL0) from Ai2Ai Corp. The AR application allows operators to synchronize the control of a virtual and physical robot arm using their hand and the PALL0 held in the hand.The AR application operates the robot arm by matching the position of its head with the operator's hand and the posture of the head with the acceleration and angular velocity of the PALL0. A button on the PALL0 controls the opening and closing of the gripper, while the physical robot arm also operates. The start and end of the operation, as well as the return of the robot arm to its initial posture, can be performed using buttons in the AR application.Future research should compare the method of controlling robot arms using PALL0 with other methods, such as those that use the movement of the body's upper or lower extremities, and invest resources in building the robot's digital twin to improve system portability.

In the rapid development of the world, robots have become involved in different aspects of human lives. A robotic arm is a type of machine that is able to manipulate its end effector in two or more axes, thus more mobile than a human arm would be. The most common restraint of a robotic arm is how it needs to be connected to the user through wires, which limits the overall capabilities of the robotic arm. Thus, the study aims to design, fabricate, and test a prototype of a robotic arm which can be controlled wirelessly through joystick module. The robotic arm controlled by a wireless joystick can change the position of its end effector within the three axes; X,Y, and Z axis. The prototype was then tested on its accuracy and on the maximum distance for safe performance when it is unobstructed and when it is obstructed by a glass window. The controller for the robotic arm was able to transmit data wirelessly through a radio frequency module while the robotic arm was able to receive the data using the same module. The robotic arm can be controlled at a maximum distance of 25ft with a glass obstruction, and 29 ft when unobstructed. The accuracy of the robotic arm is then tested wherein the researchers were operating the robotic arm. It was proven that it was accurate enough to be able to move its end effector to the desired point while being controlled remotely using a joystick controller. The ability of the robotic arm to carry load depends on the type of servo motor used in the robotic arm, however, the researchers opted to focus more on the accuracy of the robotic arm and not on the load carrying capacity.

Background: With the rapid development of digital twinning technology, the compatibility of digital twinning technology to other technologies is continuously enhanced. It is because of this that the application of image recognition technology in digital twinning technology becomes a reality. However, the key technology of digital twin, the virtual-actual mutual control technology, is not mature enough, and the image recognition technology applied in digital twin technology also has the problem of coordinate system transformation, which becomes an important link of image recognition technology in digital twin. Methods: Based on the above two problems, we take the tangram splicing project as an example to realize a virtual-actual mutual control method, so that the digital twinning technology can be well presented. Furthermore, we implement an image recognition applied to the conversion of digital twinning technology, so that digital twinning technology and image recognition have a seamless connection, allowing the application range of digital twinning technology to be further expanded. Results: In this paper, image recognition technology is successfully applied to digital twin technology by adopting the conversion between different coordinate systems and the real and virtual real time control, which makes the applicability of digital twin technology in high-tech fields such as smart factories and smart manufacturing to a higher level. Conclusions: Finally, through the tangram splicing project, the motion trajectories of the robotic arm and the tangram are consistent with the virtual robotic arm and the tangram in the computer. It is proved that our method can well combine digital twinning technology and image recognition technology.

After introductory considerations on the main functional and design differences between anthropomorphic hands conceived as robotic end effectors or as prostheses, this chapter presents two topics related to advances in robotic hand design that seem transferable to prosthetic hands, in order to increase their functional capability yet coping with specific constraints like simplicity, lightweight, cost effectiveness, robustness, etc. The development of a bio-inspired robotic hand, called UB Hand IV, based on an endoskeletal articulated structure, actuated by tendons and covered by a soft dermal-epidermal layer is briefly illustrated, in order to show the potential of its design solutions to be transferred into prosthetic hands. The first part of the chapter presents alternative design approaches for articulated joints and finger structures based on purposely designed compliant hinges. The basic problem of compliant hinges adoption in robotic structures, that is the limitation of secondary compliance effects, is analyzed and considerations about comparative metrics are proposed. Two hinge morphologies which show promising features are critically compared and pros and cons the production of fully integral fingers with compliant joints are discussed. The second part reports on the development of thin soft covers for robotic (and prosthetic) hands capable of strictly mimicking the actual compliance of human finger pulps. A design method, called by the authors Differentiated Layer Design (DLD), is reviewed and its potential for application on both robotic and prosthetic devices is underlined. Conclusions summarize the main aspects that encourage the transfer of the described results from the world of robots to that of human portable devices.

ABSTRACTThis study presents a novel tri‐manual interaction framework that enables users to control two physical hands via VR controllers and a virtual robotic arm through a hybrid brain‐computer interface (32‐channel EEG and eye‐tracking). The virtual robotic arm, implemented as a 6‐DOF industrial manipulator in Unity, is controlled through simplified BCI commands: forward/backward movement along the z‐axis based on motor imagery strength, with automatic grasping triggered by sustained attention thresholds. Twenty‐five participants completed 30 trials, each following a 60‐s protocol with five phases: rest, target presentation, preparation, execution, and feedback. Results demonstrated that the hybrid BCI system achieved superior performance compared to EEG‐only control: 8.5% improvement in task success rate, 34.5% increase in positioning accuracy, and 46.2% reduction in cognitive load. The CNN‐LSTM architecture achieved 86.3% motor imagery classification accuracy. Learning effects were observed within trials, with performance plateauing after 3.8 ± 0.7 attempts. Time‐frequency analysis revealed hierarchical neural coordination mechanisms underlying the tri‐manual control. This research validates the feasibility of augmented manipulation in virtual environments, establishing a foundation for advanced human‐robot collaboration in animation and VR applications.

We present an intuitive, real-time control system for a 6-axis robotic arm, designed to support assistive applications in biomedical contexts. Using four MARG-based inertial sensors placed on the arm and hand, human motion is captured and mapped to robot kinematics via quaternion-based sensor fusion and a two-step calibration process. The resulting end-effector pose is scaled to match the robot’s workspace, while EMG signals enable control of a robotic gripper through thumb-index gripping gestures. The system enables direct, gesture-based control without extensive training, making it accessible for users with limited mobility but preserved upper limb function. Evaluation results confirm stable, low-latency operation during pick-and-place tasks, with users intuitively compensating for minor sensor inaccuracies through visual feedback. This approach demonstrates strong potential for assistive robotics in home or care environments, particularly where conventional control methods are unsuitable.

Assistive robot solutions are mostly designed as robot helpers with robotic arms and aim to assist disabled and elderly people with carrying out basic activities of daily life such as reaching household objects, i.e. cups, feeding with spoon, opening a drawer/fridge doors, etc., However, commercial assistive robotic arms with joystick control require extensive and tiring hand motor skill training that limits robot’s practical usage by patients with disabilities. The main objective of this work is to present the methodology for designing an intelligent human-machine interface for a commercial joystick controlled assistive robotic arm realizing shared autonomy and supervisory control modes. Preliminary results on a RGB-D based object detection and position estimation system development using publicly available YOLOv3 and CenterNet deep learning models implementation of an autonomous object grasping mode by the Kinova Jaco robotic arm are described in detail and experimentally demonstrated.

Recent developments in the non-muscular human–robot interface (HRI) and shared control strategies have shown potential for controlling the assistive robotic arm by people with no residual movement or muscular activity in upper limbs. However, most non-muscular HRIs only produce discrete-valued commands, resulting in non-intuitive and less effective control of the dexterous assistive robotic arm. Furthermore, the user commands and the robot autonomy commands usually switch in the shared control strategies of such applications. This characteristic has been found to yield a reduced sense of agency as well as frustration for the user according to previous user studies. In this study, we firstly propose an intuitive and easy-to-learn-and-use hybrid HRI by combing the Brain–machine interface (BMI) and the gaze-tracking interface. For the proposed hybrid gaze-BMI, the continuous modulation of the movement speed via the motor intention occurs seamlessly and simultaneously to the unconstrained movement direction control with the gaze signals. We then propose a shared control paradigm that always combines user input and the autonomy with the dynamic combination regulation. The proposed hybrid gaze-BMI and shared control paradigm were validated for a robotic arm reaching task performed with healthy subjects. All the users were able to employ the hybrid gaze-BMI for moving the end-effector sequentially to reach the target across the horizontal plane while also avoiding collisions with obstacles. The shared control paradigm maintained as much volitional control as possible, while providing the assistance for the most difficult parts of the task. The presented semi-autonomous robotic system yielded continuous, smooth, and collision-free motion trajectories for the end effector approaching the target. Compared to a system without assistances from robot autonomy, it significantly reduces the rate of failure as well as the time and effort spent by the user to complete the tasks.