Human Motion Recognition for Industrial Human-Robot Collaboration based on a Novel Skeleton Descriptor

Trust has been identified as a key element for the successful cooperation between humans and robots. However, little research has been directed at understanding trust development in industrial human-robot collaboration (HRC). With industrial robots becoming increasingly integrated into production lines as a means for enhancing productivity and quality, it will not be long before close proximity industrial HRC becomes a viable concept. Since trust is a multidimensional construct and heavily dependent on the context, it is vital to understand how trust develops when shop floor workers interact with industrial robots. To this end, in this study a trust measurement scale suitable for industrial HRC was developed in two phases. In phase one, an exploratory study was conducted to collect participants’ opinions qualitatively. This led to the identification of trust related themes relevant to the industrial context and a related pool of questionnaire items was generated. In the second phase, three human-robot trials were carried out in which the questionnaire items were applied to participants using three different types of industrial robots. The results were statistically analysed to identify the key factors impacting trust and from these generate a trust measurement scale for industrial HRC.

In complex industrial human-robot collaboration (HRC) environment, obstacles in the shared working space will occlude the operator, and the industrial robot will threaten the safety of the operator if it is unable to get the complete human spatial point cloud. This paper proposes a real-time human point cloud inpainting method based on the deep generative model. The method can recover the human point cloud occluded by obstacles in the shared working space to ensure the safety of the operator. The method proposed in this paper can be mainly divided into three parts: (i) real-time obstacles detection. This process can detect obstacle locations in real time and generate the image of obstacles. (ii) the application of the deep generative model algorithm. It is a complete convolutional neural network (CNN) structure and introduces advanced generative adversarial loss. The model can generate the missing depth data of operators at arbitrary position in the human depth image. (iii) spatial mapping of the depth image. The depth image will be mapped to point cloud by coordinate system conversion. The effectiveness of the method is verified by filling hole of the human point cloud occluded by obstacles in industrial HRC environment. The experiment results show that the proposed method can accurately generate the occluded human point cloud in real time and ensure the safety of the operator.

Simulation Platform to Investigate Safe Operation of Human-Robot Collaboration Systems

The development and evaluation of Robot Light Skin: A novel robot signalling system to improve communication in industrial human–robot collaboration

The inherent variability in human performance introduces stochastic perturbations into manufacturing environments, undermining the seamless coordination required for effective human-robot collaboration (HRC) systems. While human cognitive flexibility enhances adaptability, it simultaneously acts as a source of operational uncertainty, complicating the modeling and optimization of integrated robotic systems. Given these challenges, there is an urgent need to substantially expand the adaptability of robotic systems through real-time detection, algorithmic analysis and dynamic behavioral adjustments in response to human performance fluctuations. The systematic development of such systems capable of precisely detecting task-specific variations, analyzing them via advanced AI algorithms and adapting their behavior accordingly remains a critical focus of contemporary research. To evaluate progress in this domain, this study conducts a systematic literature review, synthesizing advancements across 124 publications and identifying underexplored research frontiers. The findings reveal a persistent misalignment between current technical capabilities and the requirements of adaptive collaboration in dynamic industrial environments. Key gaps include the absence of explainable AI frameworks for transparent decision-making, limited generalizability of adaptive control architectures and a lack of proactive strategies that anticipate rather than merely react to performance deviations.

Language is an indispensable for humanoid robot to be integrated into daily life. This paper proposes a novel approach to construct a space of motion labels from their mapping to human whole body motions. The motions are abstracted by Hidden Markov Models, which are referred to as motion symbols. The human motions are automatically partitioned into motion segments, and recognized as sequences of the motion symbols. Sequences of motion labels are also assigned to these motions. The referential relationship between the motion symbols and the motion labels is extracted by stochastic translation model, and distances among the labels are calculated from the association probability of the motion symbols being generated by the labels. The labels are located in a multidimensional space so that the distances are satisfied, and it results in a label space. The label space encapsulates relations among the motion labels such as their similarities. The label space also allows motion recognition. The validity of the constructed label space is demonstrated on a motion capture data-set.

To address the problems of the traditional human motion gesture tracking and recognition methods, such as poor tracking effect, low recognition accuracy, high frame loss rate, and long-time cost, a dynamic human motion gesture tracking and recognition algorithm using multimode deep learning was proposed. Firstly, the collected human motion images are repaired in the three-dimensional (3D) environment, and the multimodal 3D human motion model is reconstructed using the processed images. Secondly, according to the results of model reconstruction, the camera gesture and other parameters of the keyframe are used to construct the target tracking optimization function so as to achieve the purpose of accurate tracking of human motion. Finally, for multimodal human motion gesture learning, a convolutional neural network (CNN) is developed. The trained CNN is utilized to complete dynamic human motion recognition after convolutional and pooling calculations. The results demonstrate that the proposed algorithm is effective in tracking human motion gestures. The average recognition accuracy is 96%, the average frame loss rate is 8.8%, the time cost is low, and the proposed algorithm has a high F-measure and much lower power consumption than other algorithms, indicating that the proposed algorithm is effective.

We propose a skeleton-based motion recognition system focusing on local parts of the human body closely related to a target motion. In this system, a skeleton feature is composed of a sequence of relative positions between paired joints calculated by Inverse Kinematics. Several joints of skeleton model are connected as a Local Skeleton Feature. The temporal sequence is modeled as human motion model by using Hidden Markov Model. Motion features are represented as Fisher vectors parameterized by the human motion models, and weighted and integrated by using Multiple Kernel Learning. This system makes it possible for robots to recognize human actions in our daily life. The experimental results based on two datasets show an improvement in performance of classification rate, which shows that the design of motion feature is effective for motion recognition.

In this paper, a nonlinear optimization solver is proposed to establish the human upper body motion capture system, including the orientation of each bone joint and the global position of the hip bone. 5 IMUs and one Kinect are used, the rotation data and acceleration data of IMUs, Kinect skeleton position data and human pose prior are fused in a non-linear optimization way. The method proposed in this paper combines the characteristics of Kinect data with high accuracy and IMU data with stability, which can make up for the accuracy of human pose capture in dynamic, magnetically disturbed and occlusion conditions. In order to verify the practicability of the human motion capture system proposed in this paper, the system is firstly applied to real-time human motion capture, then a simple human-robot collaboration (HRC) case is implemented, which is very common in modern collaborative medical field. The experimental results show that the proposed human motion capture system can achieve stable human motion measurement, which can meet the needs of HRC scenarios.

The recognition accuracy of human motion visual feature recognition method is low, and the recognition information integrity is low. Based on 3D skeleton, a new visual feature recognition method of human motion in sequence images is proposed. The 3D skeleton information is used to represent human actions, and then human actions are treated as manifolds. Based on 3D skeleton information, main pose features and main action trend features are proposed to represent human actions and measure the distance between actions; Based on the local linearity measurement of human motion manifold, the decomposition of human action is realized, which is conducive to the representation and recognition of classification tasks. The experimental results show that the method based on 3D skeleton can effectively improve the recognition accuracy and integrity, which proves the effectiveness of the proposed method.

Trends on Human Factors in Industrial Human-Robot Collaboration: from Design to Implementation

In the Industry 5.0 era, the focus shifts from basic automation to fostering collaboration between humans and robots. Trust is crucial in this new paradigm, enabling smooth interaction, especially for users with limited robotics knowledge. This study presents a novel framework that uses human hand gestures and voice commands to control robot movements, aiming to enhance trust, reduce cognitive workload, and minimize task execution time-key for efficient manufacturing. In automated systems, swift completion of micromanagement tasks is essential to prevent process disruption. To evaluate this framework, we devised a testbed scenario within an automated carbon fiber transportation and draping process, focusing on a maintenance task as the micromanagement challenge. Participants inspected the gripper, guided the robot along a defined path, and performed maintenance, such as attaching cables. Two conditions were tested: gestures and voice commands versus a smartPAD. The results showed that gestures and voice commands increased trust, lowered cognitive load, and shortened execution times, improving overall manufacturing efficiency.

Human-robot collaboration (HRC) on assembly lines enhances productivity and human operators' well-being by enabling humans and robots to perform tasks independently or collaboratively in shared workspaces. Aligning with the human-centric vision of Industry 5.0, this study addresses the human-centered assembly line balancing problem with human-robot collaboration (HCALBP-HRC). A multi-objective mixed-integer linear programming (MILP) model is proposed to minimize cycle time, cognitive load, and ergonomic risk through effective task allocation and resource management. Experiment results of three groups of instances with different production scenarios verified the feasibility of the MILP model. The analysis of trade-offs among the three objectives under varying weight configurations highlights the conflicts between production efficiency and human-centered considerations, underscoring the importance of balanced decision-making in HRC-based assembly systems.

Human motion recognition based on wearable devices plays a vital role in pervasive computing. Smartphones have built-in motion sensors that measure the motion of the device with high precision. In this paper, we propose a human lower limb motion capture and recognition approach based on a Smartphone. We design a motion logger to record five categories of limb activities (standing up, sitting down, walking, going upstairs, and going downstairs) using two motion sensors (tri-axial accelerometer, tri-axial gyroscope). We extract the motion features and select a subset of features as a feature vector from the frequency domain of the sensing data using Fast Fourier Transform (FFT). We classify and predict human lower limb motion using three supervised learning algorithms: Naïve Bayes (NB), K-Nearest Neighbor (KNN), and Artificial Neural Networks (ANNs). We use 670 lower limb motion samples to train and verify these classifiers using the 10-folder cross-validation technique. Finally, we design and implement a live detection system to validate our motion detection approach. The experimental results show that our low-cost approach can recognize human lower limb activities with acceptable accuracy. On average, the recognition rate of NB, KNN, and ANNs are 97.01%, 96.12%, and 98.21%, respectively.

Human-Robot Collaboration (HRC) in industry is a promising research direction that has potential to expand robotics to previously unthinkable application areas. Orchestration of large hybrid human-robot teams carrying out many tasks in parallel within a shop floor faces new challenges due to unique aspects introduced by HRC. This paper presents a new approach to topological and temporal orchestration of hybrid human-robot workforce, considering the capabilities of robot agents as well as the potentially new roles that human operators may acquire in an HRC setting. We propose a two-stage approach to orchestrating large HRC teams: First, an abstract topological and task assignment problem is solved, which does not consider the precise sequence of tasks. Second, the result of the first step is used to initialize a constrained search for an efficient HRC schedule. Initial application of the proposed approach in problems of varying complexity demonstrates encouraging results.