Perception and Control of Finger Forces

The ability to control index finger and elbow flexion forces was measured while subjects used either haptic feedback or both haptic and visual feedback to control the forces exerted. Over a 120-s time period subjects were able to control the finger forces ranging from 2 to 6 N to within 1 N using only haptic feedback, and elbow flexion forces to within 4.5 N over a force range of 10-30 N. At the same force amplitude there was no significant difference between the two muscle groups in the precision or accuracy with which the force could be controlled, suggesting that there is not a proximal to distal gradient in force control as has been found for the control of limb movement and position.

We routinely have physical interactions with others, whether it be handing someone a glass of water or jointly moving a heavy object together. These sensorimotor interactions between humans typically rely on visual feedback and haptic feedback. Recent single-participant studies have highlighted that the unique noise and time delays of each sense must be considered to estimate the state, such as the position and velocity, of one's own movement. However, we know little about how visual feedback and haptic feedback are used to estimate the state of another person. Here, we tested how humans utilize visual feedback and haptic feedback to estimate the state of their partner during a collaborative sensorimotor task. Across two experiments, we show that visual feedback dominated haptic feedback during collaboration. Specifically, we found that visual feedback led to comparatively lower task-relevant movement variability, smoother collaborative movements, and faster trial completion times. We also developed an optimal feedback controller that considered the noise and time delays of both visual feedback and haptic feedback to estimate the state of a partner. This model was able to capture both lower task-relevant movement variability and smoother collaborative movements. Taken together, our empirical and modeling results support the idea that visual accuracy is more important than haptic speed to perform state estimation of a partner during collaboration.NEW & NOTEWORTHY Physical collaboration between two or more individuals involves both visual and haptic feedback. Here, we investigated how visual and haptic feedback is used to estimate the movements of a partner during a collaboration task. Our experimental and computational modeling results parsimoniously support the notion that greater visual accuracy is more important than faster yet noisier haptic feedback when estimating the state of a partner.

This research evaluated visual and haptic feedback for enhancing operator situational awareness during multitasking drone operations, investigating how task complexity influenced feedback effectiveness. Two virtual reality experiments were conducted: Study 1 assessed three types of visual feedback (Static, Blink, and Shake), while Study 2 compared visual, haptic, and visual-haptic multimodal feedback across varying task complexity levels determined by target density. Participants performed drone operation tasks requiring simultaneous obstacle avoidance and target counting. Results revealed that dynamic visual feedback (Shake) significantly improved primary task accuracy, reduced cognitive load, and was the preferred visual feedback type. Furthermore, haptic and visual-haptic feedback significantly shortened obstacle avoidance reaction times and decreased cognitive workload compared to visual feedback. Task complexity proved to be a critical factor: while visual feedback sufficed for low complexity scenarios, visual-haptic feedback offered distinct advantages in medium and high complexity tasks, providing insights for complexity-responsive drone interface design.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

Most prosthetic hand users are limited to visual feedback of movement performance. To characterize the benefit of vibrotactile feedback for a task that lacks haptic feedback, a virtual environment was used to experimentally manipulate visual, task-relevant haptic, and remote vibrotactile feedback on simple object manipulation for unimpaired subjects. The combination of visual and remote vibrotactile feedback was compared to visual feedback alone, and to simultaneous visual and direct haptic feedback to represent ideal performance. Visual and vibrotactile feedback resulted in improvement of most performance variables including difficulty ratings relative to visual feedback alone. However addition of sensory cues to visual feedback increased trial times and the increase was steeper for vibrotactile than for haptic feedback. Specifically, during vibrotactile feedback the velocity did not change, but the duration of execution increased due to improved performance, resulting in increased trial times. This result suggests future exploration of performance improvement and execution speed for augmented sensory feedback.

This study investigates the influence of shoe type (sneakers and safety boots), age, and gender on the perception of haptic pulse feedback provided by a prototype accelerator pedal in a running stationary vehicle. Haptic feedback can be a less distracting alternative to traditionally visual and auditory in-vehicle feedback. However, to be effective, the device delivering the haptic feedback needs to be in contact with the person. Factors such as shoe type vary naturally over the season and could render feedback that is perceived well in one situation, unnoticeable in another. In this study, we evaluate factors that can influence the subjective perception of haptic feedback in a stationary but running car: shoe type, age, and gender. Thirty-six drivers within three age groups (≤39, 40-59, and ≥60) took part. For each haptic feedback, participants rated intensity, urgency, and comfort via a questionnaire. The perception of the haptic feedback is significantly influenced by the interaction between the pulse's duration and force amplitude and the participant's age and gender but not shoe type. The results indicate that it is important to consider different age groups and gender in the evaluation of haptic feedback. Future research might also look into approaches to adapt haptic feedback to the individual driver's preferences. Findings from this study can be applied to the design of an accelerator pedal in a car, for example, for a nonvisual in-vehicle warning, but also to plan user studies with a haptic pedal in general.

The perceptual deadband (PD) data reduction scheme allows for a significant reduction of network load in real-time haptic communication scenarios. Aiming to ascertain how the optimum PD data reduction parameter k is best determined, an experiment was conducted in which a possible interaction of PD data reduction and visual feedback on haptic feedback quality and task performance was investigated. The results show that when haptic feedback is complemented with visual feedback, haptic data reduction affects haptic feedback quality at a lower data reduction rate than it affects task performance. When visual feedback is missing, however, this effect is reversed. These results imply that the perception of haptic feedback quality is markedly influenced by visual sensory input as well as task requirements; hence, it is recommended to consider feedback quality and task performance in the optimization of PD reduction before such schemes may be applied to industrial telepresence and teleaction systems.

Camera drones allow exploring remote scenes that are inaccessible or inappropriate to visit in person. However, these exploration experiences are often limited due to insufficient scene information provided by front cameras, where only 2D images or videos are supplied. Combining a camera drone vision with haptic feedback would augment users' spatial understandings of the remote environment. But such designs are usually difficult for users to learn and apply, due to the complexity of the system and unfluent UAV control. In this paper, we present a new telepresence system for remote environment exploration, with a drone agent controlled by a VR mid-air panel. The drone is capable of generating real-time location and landmark details using integrated Simultaneous Location and Mapping (SLAM). The SLAMs' point cloud generations are produced using RGB input, and the results are passed to a Generative Adversarial Network (GAN) to reconstruct 3D remote scenes in real-time. The reconstructed objects are taken advantage of by haptic devices which could improve user experience through haptic rendering. Capable of providing both visual and haptic feedback, our system allows users to examine and exploit remote areas without having to be physically present. An experiment has been conducted to verify the usability of 3D reconstruction result in haptic feedback rendering.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

Evaluation of Motor Training Performance in 3D Virtual Environment via Combining Brain-computer Interface and Haptic Feedback

Mastering ballroom dance requires learning motor skills such as synchronizing the body motion to a tempo as well as memorizing long sequences of dance steps. Traditionally, the acquisition of these skills is guided by an instructor, who adapts the training to the level of the trainee. However, due to the measures that are taken globally against the COVID19, a great number of training facilities have partially or completely restricted access to their trainers. To facilitate the practice of ballroom dance without the need of a human instructor nor human contact, this paper presents a mixed-reality human-machine interface that provides part practice of ballroom dance. It achieves it by fractionalizing the dance into only the footwork and conveying the tempo, position trajectory, and velocity trajectory with multimodal feedback. The multimodal feedback is generated with a floor projection, vibrotactile actuators, and a metronome, which provide visual, haptic, and auditory feedback. In an experiment with 15 participants, results suggest that the motor learning of dance skills occurs when visual and haptic feedback is present, whereas it is further improved when the three modalities (visual, haptic, and auditory) are employed during training.

Controlling the motion of a prosthetic upper limb without visual feedback is extremely difficult because the wearer does not know the prosthesis' configuration. This paper describes an experiment designed to determine the relative importance of visual and haptic position feedback during targeted force-based motion by non-amputee human subjects as an analogy to prosthetic use. Subjects control the angle of a virtual proxy through an admittance relationship by generating torque at the MCP joint of the right index finger. During successive repetitions of a target acquisition task, the proxy's state is selectively conveyed to the user through graphical display, finger motion, and tactile stimulation. Performance metrics for each feedback condition will provide insights on the role of haptic position feedback and may help guide the development of future upper-limb prostheses

Manual preferences for visually- and haptically-guided grasping

The Simulation-Based Open Surgery Training System (SOSTS) is a virtual reality simulator designed to provide training on a simulated orthopedic bone pinning procedure. The present version of SOSTS offers five distinct combinations of multi-sensory feedback during training. The purpose of this study was to investigate the effects of visual and haptic feedback on participants' ability to place a pin through a fractured tibia on the simulator. Enhanced visual feedback resulted in a significant reduction of pin alignment error, although participants in these conditions took five times longer to achieve proficiency than those in the moderate visual feedback conditions. Any form of haptic feedback resulted in significantly greater pin placement error than no haptic feedback. The results provide partial support for the proximity compatibility principle for display design.

This chapter examines the use of haptic feedback in a multimedia simulation as a means of conveying information about physical science concepts. The case study presented herein investigates the effects of force feedback on children’s conceptions of gravity, mass, and related concepts following experimentation with a force-feedback-enabled simulation. Two groups of 17 children conducted experiments with the simulation; the experimental group used both visual and force feedback, and the control group used visual feedback only. Evidence of positive gains by the experimental group who used the simulation with force feedback is presented. Guidelines for applying these technologies effectively for educational purposes are discussed. This chapter adds to the limited research on the application of haptic feedback for conceptual learning and provides a basis for further research into the effects of computer-based haptic feedback on children’s cognition.