Motion Macro Programming on Assistive Robotic Manipulators: Three Skill Types for Everyday Tasks

The research and development of assistive robotic manipulators (ARMs) aims to enhance the upper-extremity daily functioning of individuals with disability. Resources continue to be invested, yet the field still lacks a standard framework to serve as a tool for the functional assessment and performance evaluation of ARMs. A review of the literature lends several suggestions from research in occupational therapy, rehabilitation robotics, and human-robot interaction. Performance assessments are often used during rehabilitation intervention by occupational therapists to evaluate a client’s functional performance. Similarly, such assessments should be developed to make predictions regarding how ARM performance in a clinical setting may generalize to task execution throughout daily living. However, ergonomics and environmental differences have largely been ignored in past research. Additional insights from the literature provide suggestions for a common set of coding definitions, and a framework to organize the ad hoc performance measures observed across ARM studies.

User and usability studies, as well as technical analysis, revealed that some aspects of the gripper (end-effector) of the assistive robotic manipulator (ARM) need improvement. This paper proposes, discusses and presents the results of three redesigns of the gripper.

One of the major issues for the success of assistive robotics concerns the question whether patients not only accept the technology and profit from it, but also whether they can effectively use it. This is especially relevant when patients are highly impaired and present several functional limitations. Therefore, it is important to enable patients to control robots with alternative methods during their activities of daily living. This work deals with the development of a voice control system based on the ROS middleware framework. The voice control was customized on the JACO2(Kinova Technology, Montreal, QC, Canada), a 6 degree-of-freedom assistive robotic manipulator. Two subjects with different impairments due to a neurodegenerative disease tested the robot for usability controlling it through the JACO2joystick and the developed voice control system. Both subjects used the voice control system successfully and scored highly its usability. The most impaired subject preferred the voice control while, by contrast, the less impaired one preferred to use the joystick. Preliminary results showed good usability of the system, which could be an important aid for highly impaired people.

We propose a generalizable natural language interface that allows users to provide corrective instructions to an assistive robotic manipulator in real-time. This work is motivated by the desire to improve collaboration between humans and robots in a home environment. Allowing human operators to modify properties of how their robotic counterpart achieves a goal on-the-fly increases the utility of the system by incorporating the strengths of the human partner (e.g. visual acuity and environmental knowledge). This work is applicable to users with and without disability. Our natural language interface is based on the distributed correspondence graph, a probabilistic graphical model that assigns semantic meaning to user utterances in the context of the robot’s environment and current behavior. We then use the desired corrections to alter the behavior of the robotic manipulator by treating the modifications as constraints on the motion generation (planning) paradigm. In this paper, we highlight four dimensions along which a user may wish to correct the behavior of his or her assistive manipulator. We develop our language model using data collected from Amazon Mechanical Turk to capture a comprehensive sample of terminology that people use to describe desired corrections. We then develop an end-to-end system using open-source speech-to-text software and a Kinova Robotics MICO robotic arm. To demonstrate the efficacy of our approach, we run a pilot study with users unfamiliar with robotic systems and analyze points of failure and future directions.

The increasing number of individuals with disabilities—over 61 million adults in the United States alone—underscores the urgent need for technologies that enhance autonomy and independence. Among these individuals, millions rely on wheelchairs and often require assistance from another person with activities of daily living (ADLs), such as eating, grooming, and dressing. Wheelchair-mounted assistive robotic arms offer a promising solution to enhance independence, but their complex control interfaces can be challenging for users. Automating control through deep learning-based object detection models presents a viable pathway to simplify operation, yet progress is impeded by the absence of specialized datasets tailored for ADL objects suitable for robotic manipulation in home environments. To bridge this gap, we present a novel ADL object dataset explicitly designed for training deep learning models in assistive robotic applications. We curated over 112,000 high-quality images from four major open-source datasets—COCO, Open Images, LVIS, and Roboflow Universe—focusing on objects pertinent to daily living tasks. Annotations were standardized to the YOLO Darknet format, and data quality was enhanced through a rigorous filtering process involving a pre-trained YOLOv5x model and manual validation. Our dataset provides a valuable resource that facilitates the development of more effective and user-friendly semi-autonomous control systems for assistive robots. By offering a focused collection of ADL-related objects, we aim to advance assistive technologies that empower individuals with mobility impairments, addressing a pressing societal need and laying the foundation for future innovations in human–robot interaction within home settings.

Wheelchair-mounted robotic manipulators have the potential to help the elderly and individuals living with disabilities carry out their activities of daily living (ADLs) independently. Robotics researchers focus on assistive tasks from the perspective of various control schemes and motion types, whereas, health research focuses on clinical assessment and rehabilitation, arguably leaving important differences between the two domains. In particular, there have been many studies on which activities are relevant to functional independence, but little is known quantitatively about the frequencies of ADLs that are typically carried out in everyday life. Understanding what activities are frequently carried out during the day can help guide the development and prioritization of robotic technology for in-home assistive robotic deployment. Robotics and health care communities have differing terms and taxonomies for representing tasks and motions; we aim to ameliorate taxonomic differences by consolidating quantitative task data with prior results from subjective task priority surveys. This study targets lifelogging databases, where we compute (i) daily activity task frequency from long-term low sampling frequency video and Internet of Things sensor data, and (ii) short term arm and hand movement data from video data of domestic tasks. In this work, we aim to provide deeper insights and meaningful guidelines to focus research and future developments in the field of assistive robotic manipulation that support the needs and performance requirements of the target population.

Researchers have investigated the dynamics of oscillation of a single-link assistive manipulator and carried out vibration analysis of the same due to its great application in real life. Thus, studying the dynamics of flexible assistive robots and developing the model is indeed an interesting research topic. It is possible that the exact solution to the equations associated with such a single-link manipulator does not always exist. We used a numerical method, viz., Galerkin’s method to solve the governing differential equations of the problem, and the corresponding frequency of oscillation was calculated. This is known as the forward problem. On the other hand, the inverse problem is to compute values of the unknowns when the frequency is known by experiment, but the values of the remaining parameters are not known. With this backdrop, the purpose of this chapter is to investigate forward and inverse problems related to single-link robotic manipulators using Euler’s beam theory to define the system dynamics. Firstly, the frequencies of the first four modes of vibration have been calculated. Furthermore, the frequency is assumed to be known (experimental outcome), and the inverse case is studied using Young’s modulus and density of the material of manufacturing of the manipulator as unknown parameters. Besides, the system of single-link manipulators has been studied in uncertain environments using fuzzy theory for forward and inverse problems.

We report our learning experience in simulated and real surgical tasks with a new voice-controlled robotic endoscope manipulator: an industrial robot with the tool-holder arm modified to support the optic and camera. The manipulator control-card programs have been rewritten to meet the needs of endoscopic surgeons. For this preliminary work, systems engineers with an additional monitor monitored, recorded, and compared the percentage effectiveness and precision of the responses of the robotic and human assistant to successive oral commands during the several different experimental surgical tasks. Simultaneously, to help develop this voice-commanded system for future, more precise robotic manipulation of surgical instruments, they measured the cartesian and spherical coordinates of successive positions of the optic. In unexpectedly difficult experimental conditions, the tireless robot proved more precise and effective than the demonstrably fatigable human: the steadier screen images of the robotic manipulations helped the surgeon tie knots in 7-0 sutures.

Our goal is to create an interface that can ease the use of an assistive device by patients who have lost their ability and control in one or more of their limbs. This interface will make the robotic arm called Manus mounted on a wheel chair much easier to use and also be more affordable. Our team is currently working on a proposal to create an interface which will be more user-friendly, less costly, easily replaceable and upgradeable in the event of failure of existing technologies.

The UCF-MANUS, a vision-based 6DOF assistive robotic arm, has been designed to aid individuals with arm function limitations to complete tasks of daily living that they would otherwise be unable to complete themselves. This paper reports a small dual cohort pilot study with traumatic spinal cord injured (SCI) subjects designed to investigate the utility of the UCF-MANUS for these subjects. Pick-and-place ADL tasks were de÷ned and users trained and tested with the system for three weeks during which they controlled the robot either through a manual or an autonomous (supervised) mode of operation. Baseline characteristics (pre-study), quantitative performance metrics (during study) and psychometrics (post-study) were obtained and statistically analyzed to test a set of hypotheses related to performance and satisfaction with the two control modes. It was seen that manual interaction showed more variability and inef÷ciency in performance metrics as compared to autonomous operation. Suprisingly the latter mode, however, did not lead to better measures for user satisfaction. A discussion is provided to explain the results. Based on qualitative feedback and quantitative results, possible directions for system design are presented in order to concurrently achieve better performance and satisfaction outcomes.

Daily Activity Performance Evaluation of Commercial Assistive Robotic Manipulator User Interfaces

A particle filter based tracking scheme is proposed to robustify visual servoing of objects in the UCF-MANUS camera-in-hand vision setup. Instead of simply fusing global and local information, a concatenation of the two sources of information is proposed here which enables the combination of the two independent measurements with a synergistic collaboration between them. A novel overlap metric to encode the degree and quality of overlap between two arbitrarily shaped Regions of Interest (ROIs) is defined to facilitate the prior and posterior pdfs in the particle filter setup.A sub-ROI is defined and utilized in the observation step to facilitate the global target detection. Based on extensive experimental results under a variety of scenarios obtained by using the UCF-MANUS assistive robotic testbed, it is seen that the proposed particle filter based fusion approach is superior to other non-fused global detection or local tracking approaches. The efficacy of the proposed approach has also been verified using standard data sets. Finally, robustification of a hybrid visual servoing technique is shown by implementing the proposed particle-filter based tracker during closed-loop operation in real-time.

Socially assistive robots help the human in everyday tasks. In particular, they can also provide assistance with regard to health and wellbeing, which is an important opportunity to support the independence of the elderly. Since spoken language often carries the essential information, the robot's verbal communication style is of special interest. In order to explore individual user's preferences with regard to the robot's linguistic style we built an autonomous, assistive health companion. It is operated with a custom hardware control panel and offers games, jokes, health-related recommendations, applications for information retrieval and communication. This paper reports on insights from the in-situ user study with regard to the system's usability and participants' feedback.

The design of user interfaces (UIs) for assistive robot systems can be improved through the use of a set of design guidelines presented in this article. As an example, the article presents two different UI designs for an assistive manipulation robot system. We explore the design considerations from these two contrasting UIs. The first is referred to as the graphical user interface (GUI), which the user operates entirely through a touchscreen as a representation of the state of the art. The second is a type of novel UI referred to as the tangible user interface (TUI). The TUI makes use of devices in the real world, such as laser pointers and a projector–camera system that enables augmented reality. Each of these interfaces is designed to allow the system to be operated by an untrained user in an open environment such as a grocery store. Our goal is for these guidelines to aid researchers in the design of human–robot interaction for assistive robot systems, particularly when designing multiple interaction methods for direct comparison.

[abstFig src='/00290002/11.jpg' width='300' text='User-assisted pick and place task' ] Despite remarkable progress of service robotics in recent years, it seems that a fully autonomous robot which would be able to solve everyday household tasks in a safe and reliable manner is still unachievable. Under certain circumstances, a robot’s abilities might be supported by a remote operator. In order to allow such support, we present a user interface for a semi-autonomous assistive robot allowing a non-expert user to quickly asses the situation on a remote site and carry out subtasks which cannot be finished automatically. The user interface is based on a mixed reality 3D environment and fused sensor data, which provides a high level of situational and spatial awareness for teleoperation as well as for telemanipulation. Robot control is based on low-cost commodity hardware, optionally including a 3D mouse and stereoscopic display. The user interface was developed in a human-centered design process and continuously improved based on the results of five evaluations with a total of 81 novice users.