Closed-Loop Control of Grasp Force With Biorealistic Hand Prosthesis

In this study we compared tactile and visual feedbacks for the motor imagery-based brain–computer interface (BCI) in five healthy subjects. A vertical green bar from the center of the fixing cross to the edge of the screen was used as visual feedback. Vibration motors that were placed on the forearms of the right and the left hands and on the back of the subject’s neck were used as tactile feedback. A vibration signal was used to confirm the correct classification of the EEG patterns of the motor imagery of right and left hand movements and the rest task. The accuracy of recognition in the classification of the three states (right hand movement, left hand movement, and rest) in the BCI without feedback exceeded the random level (33% for the three states) for all the subjects and was rather high (67.8% ± 13.4% (mean ± standard deviation)). Including the visual and tactile feedback in the BCI did not significantly change the mean accuracy of recognition of mental states for all the subjects (70.5% ± 14.8% for the visual feedback and 65.9% ± 12.4% for the tactile feedback). The analysis of the dynamics of the movement imagery skill in BCI users with the tactile and visual feedback showed no significant differences between these types of feedback. Thus, it has been found that the tactile feedback can be used in the motor imagery-based BCI instead of the commonly used visual feedback, which greatly expands the possibilities of the practical application of the BCI.

Fitts’ law models the relationship between amplitude, precision, and speed of rapid movements. It is widely used to quantify performance in pointing tasks, study human-computer interaction, and generally to understand perceptual-motor information processes, including research to model performance in isometric force production tasks. Applying Fitts’ law to an isometric grip force task would allow for quantifying grasp performance in rehabilitative medicine and may aid research on prosthetic control and design. We examined whether Fitts’ law would hold when participants attempted to accurately produce their intended force output while grasping a manipulandum when presented with images of various everyday objects (we termed this the implicit task). Although our main interest was the implicit task, to benchmark it and establish validity, we examined performance against a more standard visual feedback condition via a digital force-feedback meter on a video monitor (explicit task). Next, we progressed from visual force feedback with force meter targets to the same targets without visual force feedback (operating largely on feedforward control with tactile feedback). This provided an opportunity to see if Fitts’ law would hold without vision, and allowed us to progress toward the more naturalistic implicit task (which does not include visual feedback). Finally, we changed the nature of the targets from requiring explicit force values presented as arrows on a force-feedback meter (explicit targets) to the more naturalistic and intuitive target forces implied by images of objects (implicit targets). With visual force feedback the relation between task difficulty and the time to produce the target grip force was predicted by Fitts’ law (average r2 = 0.82). Without vision, average grip force scaled accurately although force variability was insensitive to the target presented. In contrast, images of everyday objects generated more reliable grip forces without the visualized force meter. In sum, population means were well-described by Fitts’ law for explicit targets with vision (r2 = 0.96) and implicit targets (r2 = 0.89), but not as well-described for explicit targets without vision (r2 = 0.54). Implicit targets should provide a realistic see-object-squeeze-object test using Fitts’ law to quantify the relative speed-accuracy relationship of any given grasper.

A virtual reality (VR) study was conducted both in our lab setting and remotely to investigate the effects of virtual hand appearance and tactile feedback on ownership, external appearance, and tactile sensation embodiment dimensions when participants were instructed to perform an assembly task in a virtual environment. Six experimental conditions that combine hand appearances (abstract, mannequin, and realistic) and tactile feedback (tactile and no tactile) levels were examined. The results of our study indicated that: (1) the more realistic hand had a stronger effect on tactile sensation and external appearance, while the mannequin hand was associated with a greater ownership effect; (2) tactile feedback was associated with a stronger effect on external appearance compared to no tactile feedback; (3) the realistic hand model in conjunction with tactile feedback significantly affected the tactile sensation of the participants; (4) the participants in the lab setting rated the external appearance of the realistic hand model higher than the remote participants; (5) the participants exposed for less than one hour per week to VR rated the tactile sensation in the presence of tactile feedback higher, while they similarly rated the external appearance of the virtual hand model in the presence of a realistic hand model higher when compared to those who were exposed for more than one hour per week to VR; and finally, (6) the younger participants rated the tactile sensation higher than the older participants in the presence of the realistic hand model. In this paper, we discuss our findings and provide design considerations for virtual reality applications that consider virtual hands and tactile feedback.

Virtual reality can help to learn basic laparoscopic tasks. However, no haptic feedback, which alerts for tissue slippage, is provided by most simulators, although, it might be of influence for the decrease of errors. This study explored whether visual or tactile feedback can be used to alert the surgeon of tissue slippage. Twenty-four participants performed a laparoscopic grasping task and where provided with either visual or tactile feedback about tissue slippage. The reaction time with the visual feedback was compared to the reaction time with tactile feedback signal. The results showed that when tissue slippage is simulated, tactile feedback shows significant faster reaction times (269ms) than visual feedback signals (398ms).

We examined changes in variability, accuracy, frequency composition, and temporal regularity of force signal from vision-guided to memory-guided force-matching tasks in 17 subacute stroke and 17 age-matched healthy subjects. Subjects performed a unilateral isometric knee extension at 10, 30, and 50% of peak torque [maximum voluntary contraction (MVC)] for 10 s (3 trials each). Visual feedback was removed at the 5-s mark in the first two trials (feedback withdrawal), and 30 s after the second trial the subjects were asked to produce the target force without visual feedback (force recall). The coefficient of variation and constant error were used to quantify force variability and accuracy. Force structure was assessed by the median frequency, relative spectral power in the 0-3-Hz band, and sample entropy of the force signal. At 10% MVC, the force signal in subacute stroke subjects became steadier, more broadband, and temporally more irregular after the withdrawal of visual feedback, with progressively larger error at higher contraction levels. Also, the lack of modulation in the spectral frequency at higher force levels with visual feedback persisted in both the withdrawal and recall conditions. In terms of changes from the visual feedback condition, the feedback withdrawal produced a greater difference between the paretic, nonparetic, and control legs than the force recall. The overall results suggest improvements in force variability and structure from vision- to memory-guided force control in subacute stroke despite decreased accuracy. Different sensory-motor memory retrieval mechanisms seem to be involved in the feedback withdrawal and force recall conditions, which deserves further study. NEW & NOTEWORTHY We demonstrate that in the subacute phase of stroke, force signals during a low-level isometric knee extension become steadier, more broadband in spectral power, and more complex after removal of visual feedback. Larger force errors are produced when recalling target forces than immediately after withdrawing visual feedback. Although visual feedback offers better accuracy, it worsens force variability and structure in subacute stroke. The feedback withdrawal and force recall conditions seem to involve different memory retrieval mechanisms.

This paper describes development of an activity based, multimodal balance rehabilitation training device. Various sensors can be used, including a force plate, inertial sensors, and depth sensing cameras, and various combinations of visual, auditory and tactile feedback can be configured depending on the rehabilitation task and activity. Tactile feedback is presented via a lightweight belt that is worn on the torso. Generally, visual feedback is only needed at the start of rehabilitation training (task orientation) while tactile feedback may be used to augment balance control. Tactile feedback can be configured as a cue that certain movement targets or limits have been reached or as an immediate indicator of the variance in postural sway. Tactile feedback allows the subject to naturally concentrate on the functional rehabilitation task, and is less reliant on visual or verbal cues.

We examined aging-related differences in the contribution of visuomotor correction to force fluctuations during index finger abduction via the analysis of two datasets from similar subjects. Study (1) Young (N = 27, 23 ± 8 years) and older adults (N = 14, 72 ± 9 years) underwent assessment of maximum voluntary contraction force (MVC) and force steadiness during constant-force (CF) index finger abduction (2.5, 30, 65% MVC). For each trial, visual feedback of the force (VIS) was provided for 8–10 s and removed for 8–10 s (NOVIS). Visual gain of the force feedback at 2.5% MVC was high; 12- and 26-fold greater than the 30 and 65% MVC targets. Mean force, standard deviation (SD) of force, and coefficient of variation (CV) of force was calculated for detrended (<0.5 Hz drift removed) VIS and NOVIS data segments. Study (2) A similar group of 14 older adults performed discrete, randomly-ordered VIS or NOVIS trials at low target forces (1–3% MVC) and high visual gain. Study (1) For young adults the CV of force was similar between VIS and NOVIS for the 2.5% (4.8 vs. 4.3%), 30% (3.2 vs. 3.2%) and 65% (3.5 vs. 4.2%) target forces. In contrast, for older adults the CV of force was greater for VIS than NOVIS for 2.5% MVC (6.6 vs. 4.2%, p < 0.001), but not for the 30% (2.4 vs. 2.4%) and 65% (3.1 vs. 3.3%) target forces. At 2.5% MVC, the increase in CV of force for VIS compared with NOVIS was significantly greater (age × visual condition p = 0.008) for older than young adults. Study (2) Similarly, for older adults performing discrete, randomly ordered trials the CV of force was greater for VIS than NOVIS (6.04 vs. 3.81%, p = 0.01). When visual force feedback was a dominant source of information at low forces, normalized force variability was ~58% greater for older adults, but only 11% greater for young adults. The significant effect of visual feedback for older adults was not dependent on the order of presentation of visual conditions. The results indicate that impaired processing of visuomotor information underlies the greater motor variability observed in older adults during lab-based isometric contractions of a hand muscle.

Lower limb amputees can use electrical activity from their residual muscles for myoelectric control of a powered prosthesis. The most common approach for myoelectric control is a finite state controller that identifies behavioral states and discrete changes in motor tasks. An alternative approach to state-based myoelectric control is continuous proportional myoelectric control where ongoing electrical activity has a proportional relationship to the prosthetic joint torque or power. To test the potential of continuous proportional myoelectric control for powered lower limb prostheses, we recruited five unilateral transtibial amputees to walk on a treadmill with an experimental powered prosthesis. Subjects walked using the powered prosthesis with and without visual feedback of their control signal in real time. Amputee subjects were able to adapt their residual muscle activation patterns to alter prosthetic ankle mechanics when we provided visual feedback of their myoelectric control signal in real time. During walking with visual feedback, subjects significantly increased their peak prosthetic ankle power ( p = 0.02, ANOVA) and positive work ( p = 0.02, ANOVA) during gait above their prescribed prosthesis values. However, without visual feedback, the subjects did not increase their peak ankle power during push off. These results show that amputee users were able to volitionally alter their prosthesis mechanics during walking, but only when given an explicit goal for their residual muscle motor commands. Future studies that examine the motor and learning capabilities of lower limb amputees using their residual muscles for continuous proportional myoelectric control are needed to determine the viability of integrating continuous high-level control with existing finite state prosthetic controllers.

BackgroundIndividuals with autism spectrum disorder (ASD) show deficits processing sensory feedback to reactively adjust ongoing motor behaviors. Atypical reliance on visual and somatosensory feedback each have been reported during motor behaviors in ASD suggesting that impairments are not specific to one sensory domain but may instead reflect a deficit in multisensory processing, resulting in reliance on unimodal feedback. The present study tested this hypothesis by examining motor behavior across different visual and somatosensory feedback conditions during a visually guided precision grip force test.MethodsParticipants with ASD (N = 43) and age-matched typically developing (TD) controls (N = 23), ages 10–20 years, completed a test of precision gripping. They pressed on force transducers with their index finger and thumb while receiving visual feedback on a computer screen in the form of a horizontal bar that moved upwards with increased force. They were instructed to press so that the bar reached the level of a static target bar and then to hold their grip force as steadily as possible. Visual feedback was manipulated by changing the gain of the force bar. Somatosensory feedback was manipulated by applying 80 Hz tendon vibration at the wrist to disrupt the somatosensory percept. Force variability (standard deviation) and irregularity (sample entropy) were examined using multilevel linear models.ResultsWhile TD controls showed increased force variability with the tendon vibration on compared to off, individuals with ASD showed similar levels of force variability across tendon vibration conditions. Individuals with ASD showed stronger age-associated reductions in force variability relative to controls across conditions. The ASD group also showed greater age-associated increases in force irregularity relative to controls, especially at higher gain levels and when the tendon vibrator was turned on.ConclusionsOur findings that disrupting somatosensory feedback did not contribute to changes in force variability or regularity among individuals with ASD suggests a reduced ability to integrate somatosensory feedback information to guide ongoing precision manual motor behavior. We also document stronger age-associated gains in force control in ASD relative to TD suggesting delayed development of multisensory feedback control of motor behavior.

Humans systematically adapt their strategies for closed-loop control based on visual feedback according to the dynamics of the system. Tactile feedback is a key element in many human–machine interfaces, but it is not known if and how well human control adapts to changes in system dynamics when information about the system state is provided using this type of feedback. In this study, 11 participants tracked a pseudorandom trajectory with a virtual, position- or velocity-controlled plant using a joystick. Visual or electrotactile feedback provided the instantaneous error between the target and generated trajectory. Frequency-domain system identification indicated that human control adapted in similar ways to the different control modes (i.e., position/velocity control) for both feedback modalities. For the plant dynamics modeled as gain and integrator, the human controller behaved as a low-pass filter and gain, respectively (under the assumption of quasi-linear behavior). However, while tracking quality was largely similar for both control modes with visual feedback, velocity control enabled substantially worse control with electrotactile feedback compared to position control. Furthermore, for both control modes, the crossover frequency of open-loop transfer functions was lower for electrotactile feedback (0.9 and 1.1 rad/s) than for visual feedback (1.5 and 1.7 rad/s) indicating limited control bandwidth. To summarize, closed-loop control based on electrotactile feedback enables natural adaptations in human control strategy, which is encouraging for tactile feedback-controlled human–machine interfaces, but the lower control bandwidth and lower tracking quality with velocity control may impose functional limitations.

Tactile feedback plays an important role in hand manipulation, especially in the grasping process which is one of the major functions of the hand. However, few commercially available prosthetic hands or hand motor function rehabilitation systems are equipped with tactile feedback. The absence of suitable tactile feedback modules leads to an inferior rehabilitation performance with a large burden on user training and compromised usability. Thus, it is challenging and essential to integrate a proper tactile feedback module with the existing hand rehabilitation systems to achieve a better control performance and accelerate the rehabilitation process. This paper focuses on the implementation and evaluation of the electrotactile feedback (EF) enhanced rehabilitation system. A virtual hand rehabilitation platform is proposed comprising an surface electromyography (sEMG) acquisition module, an electrotactile stimulation module, a virtual environment with sEMG-driven humanlike hand and numerical feedbacks of grasping force and fingertip deformation, where a closed-loop control is formed. Three different feedback conditions including visual feedback (VF), EF, and no feedback (NF) are compared based on the proposed platform. Experiments were conducted on 10 able-bodied subjects, and multiple quantitative metrics for the rehabilitation performance evaluation including training burden estimation and success rate (SR) of tasks were adopted. Results indicate that the integration of EF is helpful to both reduce the rehabilitation duration and improve the virtual grasping SR in comparison with the NF condition while possessing a better practicality over VF. Note to Practitioners —This paper is motivated by the problem of hand grasp control for rehabilitation purposes, but it also applies to other hand motor function rehabilitation process. Existing hand motor function rehabilitation approaches generally lack a proper feedback and rely on the heavy burden during user training and the users experience. This paper suggests incorporating EF to improve the efficiency and efficacy of the rehabilitation process. The electrical stimulation is driven by the myoelectric-sensing-based force estimation and encoded in a manual scheme to fit each individual involved. In our work, a virtual hand rehabilitation platform is implemented to verify the feasibility of EF in reducing the burden of user training and improving the rehabilitation performance, which allows the expanding of the current system into a broader spectrum of motor function rehabilitation applications. Experiments on able-bodied subjects suggest that the EF in the proposed virtual hand rehabilitation platform is feasible, but it has not been tested on the limb-impaired subjects and confined to a manual encoding of electrical stimulation. In the future research, the design of a general EF enhanced hand rehabilitation platform with a standardized stimulation parameter optimization will be addressed and further validated on the subjects with limb impairments and amputation.

Influence of aging and visual feedback on the stability of hand grip control in elderly adults.

Tactile feedback is pivotal for grasping and manipulation in humans. Providing functionally effective sensory feedback to prostheses users is an open challenge. Past paradigms were mostly based on vibro- or electrotactile stimulations. However, the tactile sensitivity on the targeted body parts (usually the forearm) is greatly less than that of the hand/fingertips, restricting the amount of information that can be provided through this channel. Visual feedback is the most investigated technique in motor learning studies, where it showed positive effects in learning both simple and complex tasks; however, it was not exploited in prosthetics due to technological limitations. Here, we investigated if visual information provided in the form of augmented reality (AR) feedback can be integrated by able-bodied participants in their sensorimotor control of a pick-and-lift task while controlling a robotic hand. For this purpose, we provided visual continuous feedback related to grip force and hand closure to the participants. Each variable was mapped to the length of one of the two ellipse axes visualized on the screen of wearable single-eye display AR glasses. We observed changes in behavior when subtle (i.e., not announced to the participants) manipulation of the AR feedback was introduced, which indicated that the participants integrated the artificial feedback within the sensorimotor control of the task. These results demonstrate that it is possible to deliver effective information through AR feedback in a compact and wearable fashion. This feedback modality may be exploited for delivering sensory feedback to amputees in a clinical scenario.

Vitreo-retinal surgery encompasses the surgical procedures performed on the vitreous humor and the retina. A procedure typically consists of the removal of the vitreous humor, the peeling of a membrane and/or the repair of a retinal detachment. Operations are performed with needle shaped instruments which enter the eye through surgeon made scleral openings. An instrument is moved by hand in four degrees of freedom (three rotations and one translation) through this opening. Two rotations (? and ? ) are for a lateral instrument tip movement. The other two DoFs (z and ?) are the translation and rotation along the instrument axis. Actuation of for example a forceps can be considered as a fifth DoF. Characteristically, the manipulation of delicate, micrometer range thick intraocular tissue is required. Today, eye surgery is performed with a maximum of two instruments simultaneously. The surgeon relies on visual feedback only, since instrument forces are below the human detection limit. A microscope provides the visual feedback. It forces the surgeon to work in a static and non ergonomic body posture. Although the surgeon’s proficiency improves throughout his career, hand tremor may become a problem around his mid-fifties. Robotically assisted surgery with a master-slave system enhances dexterity. The slave with instrument manipulators is placed over the eye. The surgeon controls the instrument manipulators via haptic interfaces at the master. The master and slave are connected by electronic hardware and control software. Implementation of tremor filtering in the control software and downscaling of the hand motion allow prolongation of the surgeon’s career. Furthermore, it becomes possible to do tasks like intraocular cannulation which can not be done by manually performed surgery. This thesis focusses on the master console. Eye surgery procedures are observed in the operating room of different hospitals to gain insight in the requirements for the master. The master console as designed has an adjustable frame, a 3D display and two haptic interfaces with a coarse adjustment arm each. The console is mounted at the head of the operating table and is combined with the slave. It is compact, easy to place and allows the surgeon to have a direct view on and a physical contact with the patient. Furthermore, it fits in today’s manual surgery arrangement. Each haptic interface has the same five degrees of freedom as the instrument inside the eye. Through these interfaces, the surgeon can feel the augmented instrument forces. Downscaling of the hand motion results in a more accurate instrument movement compared to manually performed surgery. Together with the visual feedback, it is like the surgeon grasps the instrument near the tip inside the eye. The similarity between hand motion and motion of the instrument tip as seen on the display results in an intuitive manipulation. Pre-adjustment of the interface is done via the coarse adjustment arm. Mode switching enables to control three or more instruments manipulators with only two interfaces. Two one degree of freedom master-slave systems with force feedback are built to derive the requirements for the haptic interface. Hardware in the loop testing provides valuable insights and shows the possibility of force feedback without the use of force sensors. Two five DoF haptic interfaces are realized for bimanual operation. Each DoF has a position encoder and a force feedback motor. A correct representation of the upscaled instrument forces is only possible if the disturbance forces are low. Actuators are therefore mounted to the fixed world or in the neighborhood of the pivoting point for a low contribution to the inertia. The use of direct drive for ' and and low geared, backdriveable transmissions for the other three DoFs gives a minimum of friction. Disturbance forces are further minimized by a proper cable layout and actuator-amplifier combinations without torque ripple. The similarity in DoFs between vitreo-retinal eye surgery and minimally invasive surgery (MIS) enables the system to be used for MIS as well. Experiments in combination with a slave robot for laparoscopic and thoracoscopic surgery show that an instrument can be manipulated in a comfortable and intuitive way. User experience of surgeons and others is utilized to improve the haptic interface further. A parallel instead of a serial actuation concept for the ' and DoFs reduces the inertia, eliminates the flexible cable connection between frame and motor and allows that the heat of the motor is transferred directly to the frame. A newly designed z-?? module combines the actuation and suspension of the hand held part of the interface and has a three times larger z range than in the first design of the haptic interface.

Haptic technologies and applications have received enormous attention in the last decade. The incorporation of haptic modality into multimedia applications adds excitement and enjoyment to an application. It also adds a more natural feel to multimedia applications, that otherwise would be limited to vision and audition, by engaging as well the user's sense of touch, giving a more intrinsic feel essential for ambient intelligent applications. However, the improvement of an application's Quality of Experience (QoE) by the addition of haptic feedback is still not completely understood. The research presented in this paper focuses on the effect of haptic feedback and what it potentially adds to the experience of the user as opposed to the traditional visual and auditory feedback. In essence, it investigates certain issues regarding stylus-based haptic education applications and haptic-enhanced entertainment videos. To this end, we used two haptic applications: the haptic handwriting learning tool to experiment with force feedback haptic interaction and the tactile YouTube application for tactile haptic feedback. In both applications, our analysis shows that the addition of haptic feedback will increase the QoE in the absence of fatigue or discomfort for this category of applications. This implies that the incorporation of haptic modality (both force feedback as well as tactile feedback) has positively contributed to the overall QoE for the users.