SEMG-based Static Force Estimation for Human-Robot Interaction using Deep Learning

Investigating the learning approaches of students in nursing education

A human motion intent estimation algorithm based on laser ranger finder and force sensors is designed for a walking-aid robot. By applying the proposed algorithm to the robot, the user can operate the walking-aid robot more fluently, comfortably and safely. A walking-aid or rehabilitation robot normally estimates the user's motion intention by measuring the interactive forces between human and robot. Whereas, the human motion should be represented as a coordination between the arms and the legs movements. This study therefore proposes a human motion intent estimation algorithm by fusing the motion intentions detected by the force sensors and the laser ranger finder, which are used to monitor the movements of arms and legs respectively. A special arrangement of push-pull force sensors is implemented for estimating the human motion intention I from user's operations on the handles. Meanwhile the human motion intention II is extracted by a laser ranger finder observing the user's movement of legs. A multi-sensor fusion algorithm is employed to fuse these two estimates. Experimental results are presented to show the validity of the proposed human motion intent estimation algorithm.

Intention recognition in Human–Robot Interaction (HRI) is critical for enabling robots to anticipate and respond to human actions effectively. This study explores the application of deep learning techniques for the classification of human intentions in HRI, utilizing data collected from Virtual Reality (VR) environments. By leveraging VR, a controlled and immersive space is created, where human behaviors can be closely monitored and recorded. Ensemble deep learning models, particularly Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Transformers, are trained on this rich dataset to recognize and predict human intentions with high accuracy. While CNN and CNN-LSTM models yielded high accuracy rates, they encountered difficulties in accurately identifying certain intentions (e.g., standing and walking). In contrast, the CNN-Transformer model outshone its counterparts, achieving near-perfect precision, recall, and F1-scores. The proposed approach demonstrates the potential for enhancing HRI by providing robots with the ability to anticipate and act on human intentions in real time, leading to more intuitive and effective collaboration between humans and robots. Experimental results highlight the effectiveness of VR as a data collection tool and the promise of deep learning in advancing intention recognition in HRI.

The sensation of touch and force is generated or conducted through the phenomenon of contact. This fact make us aware of a new force feedback method, which was named the concept of Surface Display. In this concept, we are remarking not on the force between the user and the object but on the surface and existence of the object. The force feedback device based on this concept presents the surface of an object, the user interact with the surface. Force is caused only as a result and it is measured by the device for the simulation of the behavior of the object. In this paper, the application of this device into a shape forming task is discussed. We have developed a kind of the virtual clay modeling environment, in which the shape of the surface can be changed by pushing directly on the surface. Sensation of making contact with the surface and the sensation of force caused by the elasticity of the clay is fedback by using the device. From an experiment in the environment, the importance and effectiveness of force feedback was made clear.

Deep learning (DL) is now generally acknowledged as the benchmark and evolution in machine learning (ML) fields. Further more, it has steadily suited the most extensively in computational techniques for ML, delivering excellent outcomes in several challenging intellectual works that have equal or still outperformed human ability. DL has the advantage of learning from enormous amounts of data. Here, the DL approach has been applied to classify thunderstorms and non-thunderstorms with a meteorological dataset. Daily observational hourly data sets from 2016 to 2021 have been used to classify the incidence of thunderstorms and non-thunderstorms. Three DL approaches are used: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM) to classify thunderstorm incidences. These DL techniques are compared with conventional ML techniques, showing that DL approaches outperformed ML. The CNN method provides the highest performance with 92.01% accuracy among all DL and ML approaches.

In recent years, research of haptic interface has been attracting attentions of researchers. By using haptic devices, people can easily handle 3D objects. Therefore, it is expected to be used for applications such as simulation in medical fields or remote control of robots. Falcon [1] and PHANToM [2] are one of the famous haptic devices. These devices have controller or pen to let user touch virtual objects. Furthermore, these devices can also provide a sense of force by each point of virtual objects for user. These haptic devices are classified as one point contact type haptic device. Users can experience as if he pokes virtual objects by using these haptic devices. However, point contact type haptic devices cannot perform a sense of force and a sense of touch at the same time. We define this sense of touch as the sense of friction caused by different materials. In order to perform more realistic sense of touch, we attempt to perform both a sense of force and a sense of touch at the same time. To realize this purpose, we developed a novel haptic device.

Probabilistic graphical modeling (PGM) provides a framework for formulating an interpretable generative process of data and expressing uncertainty about unknowns. This makes PGM very useful for understanding phenomena underlying data and for decision making. PGM has seen great success in domains where interpretable inferences are key, e.g. marketing, medicine, neuroscience, and social science. However, PGM tends to lack flexibility, which has hindered its use when it comes to modeling large scale high-dimensional complex data and performing tasks that require flexibility (e.g. in vision and language applications.)\n\nDeep learning (DL) is another framework for modeling and learning from data that has seen great empirical success in recent years. DL is very powerful and offers great flexibility, but it lacks the interpretability and calibration of PGM. \n\nThis thesis develops deep probabilistic graphical modeling (DPGM). DPGM consists in leveraging DL to make PGM more flexible. DPGM brings about new methods for learning from data that exhibit the advantages of both PGM and DL. \n\nWe use DL within PGM to build flexible models endowed with an interpretable latent structure. One family of models we develop extends exponential family principal component analysis (EF-PCA) using neural networks to improve predictive performance while enforcing the interpretability of the latent factors. Another model class we introduce enables accounting for long-term dependencies when modeling sequential data, which is a challenge when using purely DL or PGM approaches. This model class for sequential data was successfully applied to language modeling, unsupervised document representation learning for sentiment analysis, conversation modeling, and patient representation learning for hospital readmission prediction. Finally, DPGM successfully solves several outstanding problems of probabilistic topic models. \n\nLeveraging DL within PGM also brings about new algorithms for learning with complex data. For example, we develop entropy-regularized adversarial learning, a learning paradigm that deviates from the traditional maximum likelihood approach used in PGM. From the DL perspective, entropy-regularized adversarial learning provides a solution to the long-standing mode collapse problem of generative adversarial networks.

The number of hemiplegic patients with a unilateral-disabled limb has been increasing globally, this stimulate the development of rehabilitation robots that assist patients in motor function recovery and strength enhancement. However, as yet, few robots have achieved self-controlled rehabilitation training, and force feedback is generally realized with force sensors. This paper introduces a self-controlled master-slave system implementing force sensing without any force sensor for training patients with limited upper extremity function. The system contains two identical motors that have a directly wired connection. One limb controls the master motor to generate electrical energy, which is recycled to power the slave motor. The slave motor in turn works in electromotive state and supports the other limb to mirror the movement of the contralateral limb. Thus, self-controlled rehabilitation training is realized. An experimental prototype has been developed and the test results confirm that the system can achieve force sensing without a force sensor and realize motion tracking with a kind of energy recycling. Overall, the system has great potential for self-controlled rehabilitation training in the home environment.

Rehabilitation robots play an important role in the rehabilitation field, and effective human-robot interaction contributes to promoting the development of the rehabilitation robots. Though many studies about the human-robot interaction have been carried out, there are still several limitations in the flexibility and stability of the control system. Therefore, we proposed an advanced adaptive control method for lower limb rehabilitation robot. The method was devised with a dual closed loop control strategy based on the surface electromyography (sEMG) and plantar pressure to improve the robustness of the adaptive control for the rehabilitation robots. First, in the outer loop control, an advanced variable impedance controller based on the sEMG and plantar pressure was designed to correct robot's reference trajectory. Then, in the inner loop control, a sliding mode iterative learning controller (SMILC) based on the variable boundary saturation function was designed to achieve the tracking of the reference trajectory. The experiment results showed that, in the designed dual closed loop control strategy, a variable impedance controller can effectively reduce trajectory tracking errors and adaptively modify the reference trajectory synchronizing with the motion intention of patients; the designed sliding mode iterative learning controller can effectively reduce chattering in sliding mode control and excellently achieve the tracking of rehabilitation robot's reference trajectory. This study can improve the performance of the human-robot interaction of the rehabilitation robot system, and expand the application to the rehabilitation field.

Research in Brain-Computer Interface (BCI) aims to understand human intent with the goal to enhance Human-Robot Interaction (HRI) especially in the field of assistive robotics. The goal of this research is to develop a behavioral sequence based framework to help persons with upper limb disabilities to maintain self-dependence. The framework aims to operate in stages and links multiple functional components to identify human intent and control a robotic arm. The development, operation, and evaluation of the framework and the linked functional components to acquire, process, evaluate, and map BCI signals generated using facial expressions and head movements to predefined actions will be introduced.The framework will integrate multiple functional components such as a non-invasive BCI control device, a vibro-tactile haptic feedback device, a visual feedback environment, the evaluation and training platform, and a robotic arm. The robot pick, move and place actions are mapped to different facial expressions and presented using haptic and visual feedback to the user for classified action verification before performing the process using a robotic arm. The initial evaluation of the developed framework was 100% successful with two volunteers who also provided constructive feedback. The initial successful evaluation provides confidence to further test the framework with more volunteers to identify limitations and/or areas of improvement and its application for further research in HRI as it applies to assistive robotic systems.

Respiratory motion induces artifacts in reconstructed cardiac perfusion SPECT images. Correction for respiratory motion often relies on a respiratory signal describing the heart displacements during breathing. However, using external tracking devices to estimate respiratory signals can add cost and operational complications in a clinical setting. We aim to develop a deep learning (DL) approach that uses only SPECT projection data for respiratory signal estimation. A modified U-Net was implemented that takes temporally finely sampled SPECT sub-projection data (100ms) as input. These sub-projections are obtained by reframing the 20-s list-mode data, resulting in 200 sub-projections, at each projection angle for each SPECT camera head. The network outputs a 200-time-point motion signal for each projection angle, which was later aggregated over all angles to give a full respiratory signal. The target signal for DL model training was from an external stereo-camera visual tracking system (VTS). In addition to comparing DL and VTS, we also included a data-driven approach based on the center-of-mass (CoM) strategy. This CoM method estimates respiratory signals by monitoring the axial changes of CoM for counts in the heart region of the sub-projections. We utilized 900 subjects with stress cardiac perfusion SPECT studies, with 302 subjects for testing and the remaining 598 subjects for training and validation. The Pearson's correlation coefficient between the DL respiratory signal and the reference VTS signal was 0.90, compared to 0.70 between the CoM signal and the reference. For respiratory motion correction on SPECT images, all VTS, DL, and CoM approaches partially de-blured the heart wall, resulting in a thinner wall thickness and increased recovered maximal image intensity within the wall, with VTS reducing blurring the most followed by the DL approach. Uptake quantification for the combined anterior and inferior segments of polar maps showed a mean absolute difference from the reference VTS of 1.7% for the DL method for patients with motion >12mm, compared to 2.6% for the CoM method and 8.5% for no correction. We demonstrate the capability of a DL approach to estimate respiratory signal from SPECT projection data for cardiac perfusion imaging. Our results show that the DL based respiratory motion correction reduces artefacts and achieves similar regional quantification to that obtained using the stereo-camera VTS signals. This may enable fully automatic data-driven respiratory motion correction without relying on external motion tracking devices.

The upper-limb rehabilitation robots can be developed as an efficient tool for motor function assessments. Circle-drawing has been used as a specific task for robot-based motor function measurement. The upper-limb movement-related kinematic and kinetic parameters measured by motion and force sensors embedded in the rehabilitation robots have been widely studied. However, the muscle synergies characterized by multiple surface electromyographic (sEMG) signals in upper limbs during human-robot interaction (HRI) with circle-drawing movements are rarely investigated. In this research, the robot-assisted and constrained circle-drawing movements for upper limb were used to increase the consistency of muscle synergy features. Both clockwise and counterclockwise circle-drawing tasks were implemented by all healthy subjects using right hands. The sEMG signals were recorded from six muscles in upper limb, and nonnegative matrix factorization (NMF) analysis was utilized to obtain muscle synergy information. Both synergy pattern and activation coefficient were calculated to represent the spatial and temporal features of muscle synergies, respectively. The results obtained from the experimental study confirmed that high structural similarity of muscle synergies was found among the subjects during HRI with circle-drawing movement by healthy subjects, which indicates healthy people may share a common underlying muscle control mechanism during constrained upper-limb circle-drawing movement. This study indicates the muscle synergy analysis during the HRI with constrained circle-drawing movement could be considered as a task for upper-limb motor function assessment.

Modeling interaction dynamics to generate robot trajectories that enable a robot to adapt and react to a human's actions and intentions is critical for efficient and effective collaborative Human-Robot Interactions (HRI). Learning from Demonstration (LfD) methods from Human-Human Interactions (HHI) have shown promising results, especially when coupled with representation learning techniques. However, such methods for learning HRI either do not scale well to high dimensional data or cannot accurately adapt to changing via-poses of the interacting partner. We propose Multimodal Interactive Latent Dynamics (MILD), a method that couples deep representation learning and probabilistic machine learning to address the problem of two-party physical HRIs. We learn the interaction dynamics from demonstrations, using Hidden Semi-Markov Models (HSMMs) to model the joint distribution of the interacting agents in the latent space of a Variational Autoencoder (VAE). Our experimental evaluations for learning HRI from HHI demonstrations show that MILD effectively captures the multimodality in the latent representations of HRI tasks, allowing us to decode the varying dynamics occurring in such tasks. Compared to related work, MILD generates more accurate trajectories for the controlled agent (robot) when conditioned on the observed agent's (human) trajectory. Notably, MILD can learn directly from camera-based pose estimations to generate trajectories, which we then map to a humanoid robot without the need for any additional training.

The goal of this research is to develop a new haptic interface device, which can improve operability and expressiveness of personal computer (PC). A new mouse type device named active mouse is developed. It consists of one omni-traveling mechanism, force sensor, position sensor and communication and control system. The omni-traveling mechanism consists of one sphere ball, two differential mechanisms, and two motors. This mechanism realizes rotation of one motor that will not affect the other motor. Therefore, output of the two motors can be composed as output to the ball. As a result, this omni-traveling mechanism can achieve 4.8[N] force at 5[V] and 500 [mAJ in compact size. The two motors of omni-traveling mechanism, force sensor and position sensor are controlled with communication and control modules developed in this research. The communication system for Active Mouse equips with USB interface. The USB interface supplies DC voltage and support communication with PC. Therefore this device can be used for general PC just connecting USB interface. To confirm function ability of the Active Mouse, a graphical program is developed. Using this program, Active Mouse can input user's force and position data to PC, and output force to user. From here, Active Mouse realizes functions of general mouse and also works as haptic device.

Cervical cancer is the second most common cancer affecting women worldwide but is curable if diagnosed early. Routinely, expert pathologists visually examine histology slides for assessing cervix tissue abnormalities. A localized, fusion-based, hybrid imaging and deep learning approach is explored to classify squamous epithelium into cervical intraepithelial neoplasia (CIN) grades for a dataset of 83 digitized histology images. Partitioning the epithelium region into 10 vertical segments, 27 handcrafted image features and rectangular patch, sliding window-based convolutional neural network features are computed for each segment. The imaging and deep learning patch features are combined and used as inputs to a secondary classifier for individual segment and whole epithelium classification. The hybrid method achieved a 15.51% and 11.66% improvement over the deep learning and imaging approaches alone, respectively, with a 80.72% whole epithelium CIN classification accuracy, showing the enhanced epithelium CIN classification potential of fusing image and deep learning features.