Experimental evaluation of virtual needle insertion framework with enhanced haptic feedback

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

Haptic information in robotic surgery can significantly improve clinical outcomes and help detect hard soft-tissue inclusions that indicate potential abnormalities. Visual representation of tissue stiffness information is a cost-effective technique. Meanwhile, direct force feedback, although considerably more expensive than visual representation, is an intuitive method of conveying information regarding tissue stiffness to surgeons. In this study, real-time visual stiffness feedback by sliding indentation palpation is proposed, validated, and compared with force feedback involving human subjects. In an experimental tele-manipulation environment, a dynamically updated color map depicting the stiffness of probed soft tissue is presented via a graphical interface. The force feedback is provided, aided by a master haptic device. The haptic device uses data acquired from an F/T sensor attached to the end-effector of a tele-manipulated robot. Hard nodule detection performance is evaluated for 2 modes (force feedback and visual stiffness feedback) of stiffness feedback on an artificial organ containing buried stiff nodules. From this artificial organ, a virtual-environment tissue model is generated based on sliding indentation measurements. Employing this virtual-environment tissue model, we compare the performance of human participants in distinguishing differently sized hard nodules by force feedback and visual stiffness feedback. Results indicate that the proposed distributed visual representation of tissue stiffness can be used effectively for hard nodule identification. The representation can also be used as a sufficient substitute for force feedback in tissue palpation.

We routinely have physical interactions with others, whether it be handing someone a glass of water or jointly moving a heavy object together. These sensorimotor interactions between humans typically rely on visual feedback and haptic feedback. Recent single-participant studies have highlighted that the unique noise and time delays of each sense must be considered to estimate the state, such as the position and velocity, of one's own movement. However, we know little about how visual feedback and haptic feedback are used to estimate the state of another person. Here, we tested how humans utilize visual feedback and haptic feedback to estimate the state of their partner during a collaborative sensorimotor task. Across two experiments, we show that visual feedback dominated haptic feedback during collaboration. Specifically, we found that visual feedback led to comparatively lower task-relevant movement variability, smoother collaborative movements, and faster trial completion times. We also developed an optimal feedback controller that considered the noise and time delays of both visual feedback and haptic feedback to estimate the state of a partner. This model was able to capture both lower task-relevant movement variability and smoother collaborative movements. Taken together, our empirical and modeling results support the idea that visual accuracy is more important than haptic speed to perform state estimation of a partner during collaboration.NEW & NOTEWORTHY Physical collaboration between two or more individuals involves both visual and haptic feedback. Here, we investigated how visual and haptic feedback is used to estimate the movements of a partner during a collaboration task. Our experimental and computational modeling results parsimoniously support the notion that greater visual accuracy is more important than faster yet noisier haptic feedback when estimating the state of a partner.

This research evaluated visual and haptic feedback for enhancing operator situational awareness during multitasking drone operations, investigating how task complexity influenced feedback effectiveness. Two virtual reality experiments were conducted: Study 1 assessed three types of visual feedback (Static, Blink, and Shake), while Study 2 compared visual, haptic, and visual-haptic multimodal feedback across varying task complexity levels determined by target density. Participants performed drone operation tasks requiring simultaneous obstacle avoidance and target counting. Results revealed that dynamic visual feedback (Shake) significantly improved primary task accuracy, reduced cognitive load, and was the preferred visual feedback type. Furthermore, haptic and visual-haptic feedback significantly shortened obstacle avoidance reaction times and decreased cognitive workload compared to visual feedback. Task complexity proved to be a critical factor: while visual feedback sufficed for low complexity scenarios, visual-haptic feedback offered distinct advantages in medium and high complexity tasks, providing insights for complexity-responsive drone interface design.

This chapter examines the use of haptic feedback in a multimedia simulation as a means of conveying information about physical science concepts. The case study presented herein investigates the effects of force feedback on children’s conceptions of gravity, mass, and related concepts following experimentation with a force-feedback-enabled simulation. Two groups of 17 children conducted experiments with the simulation; the experimental group used both visual and force feedback, and the control group used visual feedback only. Evidence of positive gains by the experimental group who used the simulation with force feedback is presented. Guidelines for applying these technologies effectively for educational purposes are discussed. This chapter adds to the limited research on the application of haptic feedback for conceptual learning and provides a basis for further research into the effects of computer-based haptic feedback on children’s cognition.

Needle insertion is the most basic skill in medical care, and training has to be imparted not only for physicians but also for nurses and paramedics. In most needle insertion procedures, haptic feedback from the needle is the main stimulus in which novices need training. For better patient safety, the classical methods of training the haptic skills have to be replaced with simulators based on new robotic and graphics technologies. This paper reviews the current advances in needle insertion modeling, classified into three sections: needle insertion models, tissue deformation models, and needle-tissue interaction models. Although understated in the literature, the classical and dynamic friction models, which are critical for needle insertion modeling, are also discussed. The experimental setup or the needle simulators that have been developed to validate the models are described. The need of psychophysics for needle simulators and psychophysical parameter analysis of human perception in needle insertion are discussed, which are completely ignored in the literature.

Robotic-assisted surgery requires an intuitive and effective human-machine interface. Providing haptic feedback for pinching and dissecting motions of bipolar forceps, a tool commonly used in neurosurgery, could potentially improve the surgeon's experience. Current haptic hand controllers have limited actuation and feedback capability, requiring surgeons to hold the handle differently compared to a conventional tool. This paper presents a new master design that provides 1-DOF force feedback by adding a Hall-effect sensor and a voice coil actuator directly onto a bipolar forceps. Twenty participants used this interface to perform a remote stiffness perception test that employed the method of constant stimuli. Ten participants pinched the samples, and the other ten dissected them. Each participant did two blocks of 35 trials with only visual feedback or with visual and haptic feedback in random order. Psychometric functions were created from the results to compare perceptual capabilities, metrics were calculated from the force and position data, and participant survey responses were analyzed. The results show that providing the force feedback made the task seem easier, increased the participant's confidence, and reduced the total tip distance traveled in the pinching task. The haptic feedback slightly improved stiffness perception in the pinching task but did not improve perception in the dissection task. These results support the utility of a force-feedback attachment to conventional forceps for pinching and motivate further investigation into the design for dissection.

The universality of music among human cultures as well as our common experience of naturally responding to music with motion, seem to be widely recognized (Tramo 2001). Recent brain-imaging studies (Lahav et al. 2005; Lahav, Saltzman, and Schlaug 2007) show that humans, given appropriate auditory inputs, seem to be tuned to produce corresponding motor outputs. This unique auditorymotor interplay provides the conceptual basis for the use of music therapy, in particular, active music therapy, where a patient is physically involved in producing music rather than simply reacting to or accompanying music (Pacchetti et al. 2000; Paul and Ramsey 2000; Lahav 2005). Music therapy has benefited people with physical disabilities, mental health needs, developmental and learning disabilities, Alzheimer's and Parkinson's disease, autism, substance abuse problems, brain injuries, and acute and chronic pain (Pacchetti et al. 2000; Harris and Summa-Chadwick 2005; see also the American Music Therapy Association's Web site, www.musictherapy.org). Playing an instrument, such as piano, guitar, or drums, may be very difficult or even inf easible for patients with motor dysfunctions. As an alternative, easy-to-use tool for active music therapy, we designed Music M ker, a human-computer interface that converts body movements into musical and visual feedback in real time using the open software platform EyesWeb (Camurri et al. 2004). Music Maker is a non-obtrusive camera-based tool that allows physically impaired patients to naturally create music with no prior musical training. Patients simply move their hand or foot in space or on a surface, and Music Maker detects and interprets these movements. Detection is accomplished without any sensors or markers that must be attached to the patients' bodies. Music Maker relies only on the video input from a camera to observe patient motion and on computervision techniques to analyze the motion. Music Maker is an adaptive interface that can be adjusted to provide auditory and visual feedback based on the patient's needs and interests. Auditory feedback could range from a single piano note to a recording of the patient's favorite piece of music. Visual feedback is provided by a graphical display on a computer monitor or wall-mounted screen. Music Maker uses fun displays, for example, cartoon drawings or pictures of musical instruments. Its hardware setup can be adjusted according to patients' levels of impairment, their particular therapeutic goals, and the equipment available in hospitals or patients' homes. (See Figure 1 for sample setups.) Computer Music Journal, 31:2, pp. 39-53, Summer 2007 © 2007 Massachusetts Institute of Technology.

IntroductionVisual feedback is an effective method to enhance postural and balancecontrol in clinical and sports training. The aim of this study was toexplore the effect of real-time visual feedback provided by a video cameraon the performance of a dynamic balance test, which is the star excursionbalance test in healthy subjects.MethodsWe compared the performance of the star excursion balance test using themaximum reach distance in 20 healthy participants (10 male and 10 female,26.8 ± 3.7 years) under two conditions: without feedback and whilst theyviewed their movements in real-time on a screen in front of them via a videocamera.ResultsThe results showed that real-time visual feedback had a significant effect onmaximum reach distance of the star excursion balance test in theposterolateral direction (P < 0.001). There was a non-significantincrease in the maximum reach distance in the anterior and posteromedialdirections.ConclusionThe result indicates that the real-time visual feedback appears to be aneffective means for improving the performance of the star excursion balancetest in the posterolateral direction, and may be a promising tool forclinical rehabilitation and athlete training to enhance dynamic posturalcontrol.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

Acute and chronic pain after median sternotomy is common and often underestimated. The mechanical retractors used for median sternotomy exert significant forces on the skeletal cage. We hypothesized that instrumented retractors could be developed to enable real-time monitoring and control of retraction forces, functions that may provide equivalent exposure with significantly reduced forces and tissue damage, and thus, less postoperative pain. We developed a novel instrumented retractor designed to enable real-time force monitoring during surgical retraction and then tested it by performing median sternotomies on 16 mature sheep. For 8 of these median sternotomies, retraction was performed to 7.5 cm at a standard "clinical pace" of 7.25 +/- 0.97 minutes without real-time monitoring of retraction forces. For the other 8 median sternotomies, we performed retraction to the same exposure using real-time visual force feedback and, consequently, a more deliberate pace of 12.05 +/- 1.73 minutes (P <.001). Retraction forces, blood pressure, and heart rate were monitored throughout the procedure. Full retraction resulted in an average force of 102.99 +/- 40.68 N at the standard clinical pace, compared to 64.68 +/- 17.60 N with force feedback (a 37.2% reduction, P = .021). Standard retraction produced peak forces of 368.79 +/- 133.61 N, whereas force feedback yielded peak forces of 254.84 +/- 75.77 N (a 30.9% reduction, P = .1152). Heart rate was significantly higher during standard clinical retraction (P = .025). Use of the novel instrumented retractor resulted in lower average and peak retraction forces during median sternotomy. Moreover, these reduced retraction forces correlated to a reduction in animal stress, as documented by heart rate.

Camera drones allow exploring remote scenes that are inaccessible or inappropriate to visit in person. However, these exploration experiences are often limited due to insufficient scene information provided by front cameras, where only 2D images or videos are supplied. Combining a camera drone vision with haptic feedback would augment users' spatial understandings of the remote environment. But such designs are usually difficult for users to learn and apply, due to the complexity of the system and unfluent UAV control. In this paper, we present a new telepresence system for remote environment exploration, with a drone agent controlled by a VR mid-air panel. The drone is capable of generating real-time location and landmark details using integrated Simultaneous Location and Mapping (SLAM). The SLAMs' point cloud generations are produced using RGB input, and the results are passed to a Generative Adversarial Network (GAN) to reconstruct 3D remote scenes in real-time. The reconstructed objects are taken advantage of by haptic devices which could improve user experience through haptic rendering. Capable of providing both visual and haptic feedback, our system allows users to examine and exploit remote areas without having to be physically present. An experiment has been conducted to verify the usability of 3D reconstruction result in haptic feedback rendering.

This study examined the impact of adding vibrotactile feedback to existing force feedback in a VR-Haptic needle insertion simulation. A new programmable tactile feedback system using direct digital synthesis (DDS) to drive LRA and piezo actuators was designed and integrated into the VR needle simulation. A user study measured needle insertion performance, usability, and user preferences for realistic vein insertion. Results showed that vibrotactile feedback (via piezo and LRA) combined with force feedback improved needle insertion performance over LRA and realism over Force Feedback alone. Participants preferred the combination of force and vibrotactile feedback. While vibrotactile feedback cannot replace force feedback, it serves as a valuable supplemental cue, enhancing overall realism. This suggests that dual-modal haptic feedback, combining force (kinesthetic) and tactile (cutaneous) cues, can address the limitations of single-modal force feedback.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

A new method of investigating the ergonomics of visual cardiopulmonary resuscitation (CPR) feedback delivery