Quantifying the Value of Visual and Haptic Position Feedback During Force-Based Motion Control

This research evaluated visual and haptic feedback for enhancing operator situational awareness during multitasking drone operations, investigating how task complexity influenced feedback effectiveness. Two virtual reality experiments were conducted: Study 1 assessed three types of visual feedback (Static, Blink, and Shake), while Study 2 compared visual, haptic, and visual-haptic multimodal feedback across varying task complexity levels determined by target density. Participants performed drone operation tasks requiring simultaneous obstacle avoidance and target counting. Results revealed that dynamic visual feedback (Shake) significantly improved primary task accuracy, reduced cognitive load, and was the preferred visual feedback type. Furthermore, haptic and visual-haptic feedback significantly shortened obstacle avoidance reaction times and decreased cognitive workload compared to visual feedback. Task complexity proved to be a critical factor: while visual feedback sufficed for low complexity scenarios, visual-haptic feedback offered distinct advantages in medium and high complexity tasks, providing insights for complexity-responsive drone interface design.

We routinely have physical interactions with others, whether it be handing someone a glass of water or jointly moving a heavy object together. These sensorimotor interactions between humans typically rely on visual feedback and haptic feedback. Recent single-participant studies have highlighted that the unique noise and time delays of each sense must be considered to estimate the state, such as the position and velocity, of one's own movement. However, we know little about how visual feedback and haptic feedback are used to estimate the state of another person. Here, we tested how humans utilize visual feedback and haptic feedback to estimate the state of their partner during a collaborative sensorimotor task. Across two experiments, we show that visual feedback dominated haptic feedback during collaboration. Specifically, we found that visual feedback led to comparatively lower task-relevant movement variability, smoother collaborative movements, and faster trial completion times. We also developed an optimal feedback controller that considered the noise and time delays of both visual feedback and haptic feedback to estimate the state of a partner. This model was able to capture both lower task-relevant movement variability and smoother collaborative movements. Taken together, our empirical and modeling results support the idea that visual accuracy is more important than haptic speed to perform state estimation of a partner during collaboration.NEW & NOTEWORTHY Physical collaboration between two or more individuals involves both visual and haptic feedback. Here, we investigated how visual and haptic feedback is used to estimate the movements of a partner during a collaboration task. Our experimental and computational modeling results parsimoniously support the notion that greater visual accuracy is more important than faster yet noisier haptic feedback when estimating the state of a partner.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

Most prosthetic hand users are limited to visual feedback of movement performance. To characterize the benefit of vibrotactile feedback for a task that lacks haptic feedback, a virtual environment was used to experimentally manipulate visual, task-relevant haptic, and remote vibrotactile feedback on simple object manipulation for unimpaired subjects. The combination of visual and remote vibrotactile feedback was compared to visual feedback alone, and to simultaneous visual and direct haptic feedback to represent ideal performance. Visual and vibrotactile feedback resulted in improvement of most performance variables including difficulty ratings relative to visual feedback alone. However addition of sensory cues to visual feedback increased trial times and the increase was steeper for vibrotactile than for haptic feedback. Specifically, during vibrotactile feedback the velocity did not change, but the duration of execution increased due to improved performance, resulting in increased trial times. This result suggests future exploration of performance improvement and execution speed for augmented sensory feedback.

The focus of this research is on the perception and control of forces produced by the hand. It is known that the change in force that subjects can reliably discriminate is approximately 6%, and that subjects are able to match forces generated by two corresponding muscle groups with an accuracy of approximately 15%. The present study was concerned with measuring the ability to control finger force using only haptic feedback and both haptic and visual feedback. It was found that subjects were surprising good at controlling forces for up to 120 s using only haptic feedback with mean absolute errors of less than 0.3 N over a force range of 2–6 N. The variation in force averaged 10.5% when haptic feedback was available and decreased to 6% when visual feedback was also provided. These results provide a framework for specifying the characteristics of force-reflecting haptic interfaces that should optimize human perception.

The ability to control index finger and elbow flexion forces was measured while subjects used either haptic feedback or both haptic and visual feedback to control the forces exerted. Over a 120-s time period subjects were able to control the finger forces ranging from 2 to 6 N to within 1 N using only haptic feedback, and elbow flexion forces to within 4.5 N over a force range of 10-30 N. At the same force amplitude there was no significant difference between the two muscle groups in the precision or accuracy with which the force could be controlled, suggesting that there is not a proximal to distal gradient in force control as has been found for the control of limb movement and position.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

Virtual reality can help to learn basic laparoscopic tasks. However, no haptic feedback, which alerts for tissue slippage, is provided by most simulators, although, it might be of influence for the decrease of errors. This study explored whether visual or tactile feedback can be used to alert the surgeon of tissue slippage. Twenty-four participants performed a laparoscopic grasping task and where provided with either visual or tactile feedback about tissue slippage. The reaction time with the visual feedback was compared to the reaction time with tactile feedback signal. The results showed that when tissue slippage is simulated, tactile feedback shows significant faster reaction times (269ms) than visual feedback signals (398ms).

We introduce a novel posture guidance office chair and evaluate the effectiveness of vibrotactile and visual feedback methods for guiding seated postures. For visually dominant office work such as typing on the computer, it is possible that delivering posture feedback visually may overload the visual sense while haptic feedback may be a viable alternative. We performed two experiments to compare vibrotactile and visual feedback--posture compliance and dual-task cognitive workload assessment. In the first experiment, our results showed no statistically significant difference in effectiveness between using vibrotactile and visual feedback to obtain postural compliance to a reference posture. In the second experiment, participants experienced typing performance and response time degradations from both types of feedback. However the differences in performance degradation were not statistically significant between the two feedback methods. We conclude that vibrotactile and visual feedback are similarly effective for guiding quasistatic postures in routine tasks such as seated office work.

The perceptual deadband (PD) data reduction scheme allows for a significant reduction of network load in real-time haptic communication scenarios. Aiming to ascertain how the optimum PD data reduction parameter k is best determined, an experiment was conducted in which a possible interaction of PD data reduction and visual feedback on haptic feedback quality and task performance was investigated. The results show that when haptic feedback is complemented with visual feedback, haptic data reduction affects haptic feedback quality at a lower data reduction rate than it affects task performance. When visual feedback is missing, however, this effect is reversed. These results imply that the perception of haptic feedback quality is markedly influenced by visual sensory input as well as task requirements; hence, it is recommended to consider feedback quality and task performance in the optimization of PD reduction before such schemes may be applied to industrial telepresence and teleaction systems.

BackgroundThe development of wearable solutions for tracking upper limb motion has gained research interest over the past decade. This paper provides a systematic review of related research on the type, feasibility, signal processing techniques, and feedback of wearable systems for tracking upper limb motion, mostly in rehabilitation applications, to understand and monitor human movement.ObjectiveThe aim of this article is to investigate how wearables are used to capture upper limb functions, especially related to clinical and rehabilitation applications.MethodsA systematic literature search identified 27 relevant studies published in English from 2011 to 2023, across 4 databases: ACM Digital Library, IEEE Xplore, PubMed, and ScienceDirect. We included papers focusing on motion or posture tracking for the upper limbs, wearable devices, feedback given to end users, and systems having clinical or rehabilitation purposes. We excluded papers focusing on exoskeletons, robotics, prosthetics, orthoses, or activity recognition systems; reviews; and books.ResultsThe results from this research focus on wearable devices that are designed to monitor upper limb movement. More specifically, studies were divided into 2 distinct categories: clinical motion tracking (15/27, 56%) and rehabilitation (12/27, 44%), involving healthy individuals and patients, with a total of 439 participants. Among the 27 studies, the majority (19/27) used inertial measurement units to track upper limb movement or smart textiles embedded with sensors. These devices were attached to the body with straps (mostly Velcro), providing flexibility and stability. The developed wearable devices positively influenced user motivation through the provided feedback, with visual feedback being the most common owing to the high level of independence provided. Moreover, a variety of signal processing techniques, such as Kalman and Butterworth filters, were applied to ensure data accuracy. However, limitations persist and include sensor positioning, calibration, and battery life, as well as a lack of clinical data on the effectiveness of these systems. The sampling rate of the data collection ranged from 50 Hz to 2000 Hz, which notably affected data quality and battery life. In addition, several findings were inconclusive, and thus, further future research is needed to understand and improve upper limb posture to develop progressive wearable systems.ConclusionsThis paper offers a comprehensive overview of wearable monitoring systems, with a focus on upper limb motion tracking and rehabilitation. It emphasizes the various types of available solutions; their efficacy, wearability, and feasibility; and proposed processing techniques. Finally, it presents robust findings regarding feedback accuracy derived from experiments and outlines potential future research directions.

The Simulation-Based Open Surgery Training System (SOSTS) is a virtual reality simulator designed to provide training on a simulated orthopedic bone pinning procedure. The present version of SOSTS offers five distinct combinations of multi-sensory feedback during training. The purpose of this study was to investigate the effects of visual and haptic feedback on participants' ability to place a pin through a fractured tibia on the simulator. Enhanced visual feedback resulted in a significant reduction of pin alignment error, although participants in these conditions took five times longer to achieve proficiency than those in the moderate visual feedback conditions. Any form of haptic feedback resulted in significantly greater pin placement error than no haptic feedback. The results provide partial support for the proximity compatibility principle for display design.

This chapter examines the use of haptic feedback in a multimedia simulation as a means of conveying information about physical science concepts. The case study presented herein investigates the effects of force feedback on children’s conceptions of gravity, mass, and related concepts following experimentation with a force-feedback-enabled simulation. Two groups of 17 children conducted experiments with the simulation; the experimental group used both visual and force feedback, and the control group used visual feedback only. Evidence of positive gains by the experimental group who used the simulation with force feedback is presented. Guidelines for applying these technologies effectively for educational purposes are discussed. This chapter adds to the limited research on the application of haptic feedback for conceptual learning and provides a basis for further research into the effects of computer-based haptic feedback on children’s cognition.

Robotic-assisted surgery requires an intuitive and effective human-machine interface. Providing haptic feedback for pinching and dissecting motions of bipolar forceps, a tool commonly used in neurosurgery, could potentially improve the surgeon's experience. Current haptic hand controllers have limited actuation and feedback capability, requiring surgeons to hold the handle differently compared to a conventional tool. This paper presents a new master design that provides 1-DOF force feedback by adding a Hall-effect sensor and a voice coil actuator directly onto a bipolar forceps. Twenty participants used this interface to perform a remote stiffness perception test that employed the method of constant stimuli. Ten participants pinched the samples, and the other ten dissected them. Each participant did two blocks of 35 trials with only visual feedback or with visual and haptic feedback in random order. Psychometric functions were created from the results to compare perceptual capabilities, metrics were calculated from the force and position data, and participant survey responses were analyzed. The results show that providing the force feedback made the task seem easier, increased the participant's confidence, and reduced the total tip distance traveled in the pinching task. The haptic feedback slightly improved stiffness perception in the pinching task but did not improve perception in the dissection task. These results support the utility of a force-feedback attachment to conventional forceps for pinching and motivate further investigation into the design for dissection.

Recently a number of TV manufacturers introduced TV remotes with a touchpad which is used for indirect control of TV UI. Users can navigate the UI by moving a finger across the touch pad. However, due to the latency in visual feedback, there is a disconnection between the finger movement on the touchpad and the visual perception in the TV UI, which often causes overshooting. In this paper, we investigate how haptic feedback affects the user experience of the touchpad-based TV remote. We described two haptic prototypes built on the smartphone and Samsung 2013 TV remote respectively. We conducted two user studies with two prototypes to evaluate how the user preference and the user performance been affected. The results show that there is overwhelming support of haptic feedback in terms of subjective user preference, though we didn't find significant difference in performance between with and without haptic feedback conditions.