What and How? Jointly Forecasting Human Action and Pose

Adaptive assistance is needed to support mobility of disabled person due to weakened muscles and loss of joint control. Assistive robot requires coordinated motion trajectory of its joint, similar to human joint trajectory. A robot trajectory planning mechanism, based on comparative analysis between robot trajectory and human joint trajectory data is important to find coordinated trajectory for assistive robot joint. To perform the comparative analysis, human motion trajectory is measured by using motion capture data, and robot trajectory is generated by extracting via points from human motion trajectory. Further, polynomial methods (cubic, quintic and linear segment with parabolic blends) are used to generate robot trajectory between via-points. A four-link Sit to Stand (STS) model is used for comparative analysis of human and robot trajectories. To generate the robot trajectory using via-point method, estimation of maximum number of via-point is well known problem. Analysis is aimed to find the robot trajectory, that is close to human motion trajectory, by finding minimum energy and minimum error trajectory and to find the maximum number of via-point for trajectory with minimum energy and maximum coordination. The paper proposed robot trajectory generation method that uses via-points of human trajectory and Principle Component Analysis (PCA) based synergy calculation method. The main research contributions are towards finding robot trajectory with minimum energy and minimum error and to estimate maximum number of via-point based on minimum error and maximum coordination.

This paper proposes a human movement trajectory recording method by a thermopile array sensor. In the system, the sensor is attached to the ceiling and it acquires place-dependent temperatures, which is called thermal distribution. The system obtains 4 × 4 pixels thermal distributions from the sensor. The distributions are analyzed to extract human movement trajectory. First, human candidate pixels are detected by background removal with three fuzzy rules on temperature. Second, those pixels are separated by Connected Component Labeling. Then, each human area is identified. Third, human centroid is calculated. Human movement trajectory is defined as a path of those centroids and recorded at all times. In the experiment, subjects performed 15 motion patterns such as standing and walking for an adult male. The system measured each motion during 1~3 minutes. As the experimental result, the system successfully recorded accurate human movement trajectories in the all.

Human-powered lower exoskeletons have gained considerable interests from both academia and industry over the past few decades, and thus have seen increasing applications in areas of human locomotion assistance and strength augmentation. One of the most important aspects in those applications is to achieve robust control of lower exoskeletons, which, in the first place, requires the proactive modeling of human movement trajectories through physical Human-Robot Interaction (pHRI). As a powerful representation tool for motion trajectories, Dynamic Movement Primitive (DMP) has been used extensively to model human movement trajectories. However, canonical DMPs only offers a general offline representation of human movement trajectory and neglects the real-time interaction term, therefore it cannot be directly applied to lower exoskeletons which need to model human motion trajectories online since different pilots have different trajectories and even one pilot might change his/her intended trajectory during walking. This paper presents a novel Coupled Cooperative Primitives (CCPs) scheme, which models the motion trajectories online. Besides maintaining canonical motion primitives, we also model the interaction term between the pilot and exoskeletons through impedance models and apply a reinforcement learning method based on Policy Improvement and Path Integrals (PI <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to learn the parameters online. Experimental results on both a single Degree-Of-Freedom (DOF) platform and a HUman-powered Augmentation Lower EXoskeleton (HUALEX) system demonstrate the advantages of our proposed CCP scheme.

Ensemble human movement sequence prediction model with Apriori based Probability Tree Classifier (APTC) and Bagged J48 on Machine learning

The traditional 3D human motion capture methods require operators to wear tracking sensors on their body. However, these devices would bring operators much inconvenience. Currently, vision based motion capture systems provide us an alternative approach for motion capture technology. In this paper, based on human kinematics and function approximation technique (FAT), a novel method is presented for the trajectory control of Baxter robot. Each arm of the Baxter robot has seven degrees of freedom. The geometry vector approach is applied to capture human motion trajectory by using Microsoft Kinect sensor. A FAT control system is employed to make the robot follow the trajectory of human motion. The UDP communication protocol is employed to send the reference human joint angle data to the robot. We carry out preliminary experiments on Baxter robot to verify the validity of the control approach and the results demonstrate that the method has achieved satisfactory performance.

A unified spatio-temporal human body region tracking approach to action recognition

The intricacies of human motion and action offer a rich vein of information crucial for enhancing the interaction between humans and machines. Human action understanding serves as the linchpin in this dynamic, bridging advanced computational methodologies with real-world applications across various sectors. This dissertation delves into the nuanced realm of action recognition and pose estimation, underscoring the transition from theoretical frameworks to practical applications. Central to the presented research is the development of the Graph Convolution Network (GCN)-based InfoGCN model for human action recognition, which introduces an informa tion bottleneck principle to optimize learning efficiency and accuracy. Additionally, the Pose Relation Transformer (PORT) is crafted to address challenges in hand and human pose esti mation, particularly under conditions of occlusion, by employing strategies akin to Masked Language Modeling in natural language processing. Further advancements are manifested in InfoGCN++, designed to facilitate real-time ac tion recognition, thereby catering to immediate analytical needs in dynamic environments. Concurrently, CARING-AI emerges as a novel framework for generating context-aware hu man motion, capturing both temporal and spatial dimensions to produce more realistic and accurate motion representations. Anticipating future needs, the dissertation proposes CARING-AI++, focusing on the enhancement of hand-object interaction within human motion generation. This progression signifies a deliberate move towards solving user-specific problems, illustrating a profound understanding of the necessary integration of technology and practicality. The collective work outlined in this dissertation not only propels the understanding of human action but also crystallizes its application in real-world scenarios, thereby bridging the gap between theoretical research and practical utility. With implications for healthcare, autonomous vehicles, and interactive technologies, the research underscores the potential of these advancements to revolutionize how we interact with and leverage technology for societal benefit, paving the way for future interdisciplinary collaboration and the ethical use of action recognition systems.

Transfer learning is a method for improving generalization performance by training a model for a different task first and then additionally training the pre-learned weights for the target task. However, transferability—the ease with which source task can be effectively transferred to which target task—is often unknown. Existing works proposed methods of measuring the transferability between classification tasks using images and discrete labels, but it cannot be applied to regression tasks. In this work, we investigate transferability among classification and regression tasks, and propose a method for predicting transferability by extending the optimal transport theory. Our transferability prediction model also can be applied to subjective tasks (e.g., aesthetics and memorability), which are usually regression tasks. We show that the appropriate source (pre-training) tasks can be predicted for the chosen target task without conducting actual pre-training and transferring trials. Experimental results demonstrated high prediction accuracy (correlation coefficient of ρ=0.791\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\rho =0.791$$\\end{document}) and a speed improvement of approximately 300 times compared with the above-mentioned greedy approach.

As the ubiquity of such wearable devices as smart-phones continues to deepen its presence in modern societies, it has become possible to analyze and visualize people who are moving as part of a trajectory of big data. In this study, we cluster human movement trajectories using time-series distributional representations. For the clustering, we calculated the distance of the representation vectors derived from neural network models. Previous work leveraged the Long short-term memory (LSTM) network to train the next mesh prediction. In this study, we propose using the Bi-directional LSTM (Bi-LSTM) network and the integrated additional geographical coordinates (latitude and longitude information) in models to accurately predict the next mesh and construct user clusters. As a result, we improved the accuracy of the next mesh prediction and obtained and visualized clusters of human movement trajectories.

In this work, we present an algorithm for regression and classification tasks on big datasets using augmented tree models. Partitioning a big dataset using a tree model permits us to apply a divide and conquer strategy to classification and regression tasks. Experiments conducted as part of this study illustrate that such an approach has an important benefit. Methods associated with good accuracies on learning tasks on big datasets such as ensemble tree methods or neural networks produce models that are not interpretable. The models produced by the proposed algorithm are interpretable while being as accurate as ensemble methods such as random forests or gradient boosted trees. Model interpretation can be performed at coarse and fine granularity. This permits us to extract insights that characterize the entire dataset or a particular subset of the data. Models that are accurate and interpretable are highly desirable in many application settings. The partitions created by the algorithm also permit a divide and conquer approach to model analysis. Analysis of performance by partition helped identify problems such as possible data errors and model overfitting.

&lt;sec&gt;This study aims to predict the thermodynamic stability of rare-earth compounds by using machine learning (ML) models, providing crucial data support for designing advanced materials and facilitating the discovery of new rare-earth compounds.&lt;/sec&gt;&lt;sec&gt;In terms of methods, this study is based on a dataset consisting of 280569 compounds. The formation energies of these compounds are calculated by density functional theory (DFT). A system consisting of 145 feature descriptors is constructed, covering stoichiometric properties, statistical properties of elements, electronic structure properties, and properties of ionic compounds, comprehensively describing the characteristics of rare-earth compounds. Two ML models, i.e. random forest (RF) and neural network (NN), are selected to perform classification and regression tasks respectively. The 5-fold cross-validation is used to improve the reliability of the models. The min-max scaling technique is used for preprocessing data, and an ensemble learning architecture is constructed to address the limitations of single model.&lt;/sec&gt;&lt;sec&gt;In the classification task, the RF and NN algorithms perform remarkably well. With 5-fold cross-validation, the accuracy reaches approximately 0.97, and the F1 score is around 0.98, enabling the precise classification of compounds into stable or unstable categories. In the regression task, the mean absolute errors (MAEs) of the formation energy predictions by the RF and NN models are 0.055 eV/atom and 0.071 eV/atom, respectively. This indicates that the model predictions are highly accurate and can replace complete DFT calculations to a certain extent. In the predictive analysis of system outside the test set, six representative components are selected from the material project database, covering binary, ternary, and quaternary systems. The prediction errors of all compositions are controlled within 0.5 eV/atom, with an error percentage of lower than 25%, indicating that the model has strong ability of extrapolation and prediction. When predicting the binary phase diagrams of rare-earth compounds La-Al and Ce-H by using the trained models, the convex hull phase diagrams constructed through the ensemble learning architecture, which combines the prediction results of the RF and NN models, are highly consistent with those constructed from the open quantum materials database. The models successfully capture several metastable phases that are not present in multiple databases. Moreover, the convex hull distances of the predicted phases are mostly less than 0.1 eV/atom, with the maximum not exceeding 0.2 eV/atom.&lt;/sec&gt;&lt;sec&gt;In conclusion, this study successfully uses ML models to predict the thermodynamic stability of rare-earth compounds. The constructed models demonstrate strong capabilities in classification and regression tasks. The ensemble learning architecture effectively improves the model performance, providing a promising tool for discovering materials in the field of rare-earth science, contributing to the research and development of new rare-earth compounds, and designing advanced materials.&lt;/sec&gt;

Assessing military trainee in an obstacle crossing competition requires an instructor to go along with participants or be strategically placed. These assessments sometimes suffer from fatigue or biasedness on the part of instructors. There is the need to have a system that can easily recognize various human actions involved in obstacle crossing and also give a fair assessment of the whole process. In this paper, VGG16 model features with neural network classifier is used to recognize human actions in a military obstacle-crossing competition video sequence involving multiple participants performing different activities. The dataset used was captured locally during military trainees’ obstacle-crossing exercises at a military training institution to achieve the objective. Images were segmented into background and foreground using a Grabcut-based segmentation algorithm. On the foreground masked images, features were extracted and used for classification with neural network. This method used the VGG16 model to automatically extract deep learned features at the max-pooling layer and the input presented to neural network classifier for classification into the various classes of human actions achieving 90% recognition accuracy which is at training time of 104.91secs. The accuracy obtained showed 3.6% performance improvement when compared to selected state-of-the-art model. The model also achieved 90.1% precision value and recall of 90.2%. Although many studies have focused on human recognition action recognition in several application areas, this study introduced a novel model for real time recognition of fifteen different classes of complex actions involving multiple participants during obstacle crossing competition in a military environment leveraging on the strength of deep learning and neural network classifier. This study will be of immense unbiased benefit to the military in the assessment of a trainee’s performance during training exercises or competitions.

The upper-limb exoskeleton is capable of enhancing human arm strength beyond normal levels, whereas deriving the operator’s desired action straightforward turns out to be one of the significant difficulties facing human-robot interaction research. In the study, the human-robot interface was presented to regulate the exoskeleton tracking human elbow motion trajectory that employed the contact force signals between the exoskeleton and its operator as the primary means of information transportation. The signals were recorded by adopting the novel soft skin sensors attached to the bracket on the exoskeleton linkage, which could reflect the human arm motion intention through the virtual admittance model and adaptive control. Subsequently, a 1-DOF upper-limb exoskeleton was designed to illustrate the performance of the proposed sensor and the interaction control method in the human-robot cooperation experiment.

In human–robot collaboration (HRC), safe and efficient robotic control requires a deep understanding and realistic modeling of human motion behavior. However, current human motion data collection is inefficient in real HRC and constrained by safety considerations, resulting in limited data coverage of HRC processes. This paper proposes a method for generating human trajectories in collaborative assembly based on a digital human model and reinforcement learning. A muscle‐driven musculoskeletal dynamic model of the digital human is developed, and reinforcement learning is employed to generate synthetic human motion trajectories that conform to biomechanical constraints. Furthermore, these trajectories are used to train collaborative robot control policies by reinforcement learning, enabling adaptive trajectory planning and real‐time obstacle avoidance. The synthetic trajectories are evaluated using Fitts’ law and compared with experimental data. The experimental results show that the synthetic trajectories exhibit consistency with real human motion. The proposed robotic control method can effectively improve the efficiency and safety of human–robot collaborative assembly.

With the wide adoption of multi-community structure in many popular online platforms, human mobility across online communities has drawn increasing attention from both academia and industry. In this work, we study the statistical patterns that characterize human movements in cyberspace. Inspired by previous work on human mobility in physical space, we decompose human online activities into return and exploration - two complementary types of movements. We then study how people perform these two movements, respectively. We first propose a preferential return model that uncovers the preferential properties of people returning to multiple online communities. Interestingly, this model echos the previous findings on human mobility in physical space. We then present a preferential exploration model that characterizes exploration movements from a novel online community-group perspective. Our experiments quantitatively reveal the patterns of people exploring new communities, which share striking similarities with online return movements in terms of underlying principles. By combining the mechanisms of both return and exploration together, we are able to obtain an overall model that characterizes human mobility patterns in cyberspace at the individual level. We further investigate human online activities using our models, and discover valuable insights on the mobility patterns across online communities. Our models explain the empirically observed human online movement trajectories remarkably well, and more importantly, sheds better light on the understanding of human cyberspace dynamics.