Haptic Interactions Using Virtual Manipulator Coupling With Applications to Underactuated Systems

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Automatic performance assessment in Virtual Reality medical simulators: A model based on procedure trajectories and machine learning

In this study, a haptic device is proposed for virtual reality (VR) applications. The proposed haptic device is linked to the bundle controller of a VR device. The device proposed in this study is a force-feedback device that focuses on the sensation felt by the hand rather than the feeling of the tool colliding with the object. The device provides sensations of centrifugal or inertial forces due to the weight of the tool when the user swings the tool and the sensation of a reaction force when the user stabs or pokes an object with the tool. The force generated by a haptic device of the linked type was measured and examined to determine whether a sensation of actually using a tool in the virtual environment was provided. In addition, we conducted two types of experiments to evaluate the proposed haptic device. First, VR content was created for the user experience, and a survey was conducted with 30 experimenters to compare the bundle controllers and a haptic device, called HapTug, in terms of realism, immersion, and enjoyment. The survey results demonstrated that the realism, immersion, and enjoyment increased compared to using the bundle controller alone. A total of 41 experimenters were allowed to play freely in a virtual environment for six sessions of 30 s each. They pressed the reset session button when they felt 30 s had passed, and the device they were wearing turned on and off randomly every session. During the experience, their playback time, number of interactions, and range were measured. As a result, when the device was activated, experimenters felt the time was shorter and showed widespread movement. Thus, it was proven that the proposed linked-type haptic device effectively delivered the feeling of wielding a tool, allowing the user to feel the virtual object coming in and going out of the hand when interacting with it. Moreover, three conclusions were drawn from the results of the experiments: previous VR experience times have little to do with satisfaction related to haptic device use, users have low expectations for haptic realism, and users interact more actively and exaggeratedly with a haptic device.

The goal of this research is to develop a new haptic interface device, which can improve operability and expressiveness of personal computer (PC). A new mouse type device named active mouse is developed. It consists of one omni-traveling mechanism, force sensor, position sensor and communication and control system. The omni-traveling mechanism consists of one sphere ball, two differential mechanisms, and two motors. This mechanism realizes rotation of one motor that will not affect the other motor. Therefore, output of the two motors can be composed as output to the ball. As a result, this omni-traveling mechanism can achieve 4.8[N] force at 5[V] and 500 [mAJ in compact size. The two motors of omni-traveling mechanism, force sensor and position sensor are controlled with communication and control modules developed in this research. The communication system for Active Mouse equips with USB interface. The USB interface supplies DC voltage and support communication with PC. Therefore this device can be used for general PC just connecting USB interface. To confirm function ability of the Active Mouse, a graphical program is developed. Using this program, Active Mouse can input user's force and position data to PC, and output force to user. From here, Active Mouse realizes functions of general mouse and also works as haptic device.

Currently, in virtual reality simulations and laboratories much attention is focused on the development of a user interaction controller for better immersion and student experience. Of course, the visual experience is a base for the laboratory study of any students, but direct contact with the visualized environment is also important. This paper presents a virtual scalpel technique that simulates the experience of medical discipline student education using virtual reality (VR) and augmented reality (AR) laboratory software. User's hands and a pencil with a tracker marker are the main tracking system components. Due to the high sensitivity of hand sign recognition, the Leap Motion camera was chosen as the base device for tracking the interaction of the user's virtual scalpel and hand models. We used Unreal Engine to create and visualize such virtual laboratory. During the recognition hand-pencil system, the 3D scalpel model is activated in our VR and scene, with a collision in the proposed virtual blade area. The user manipulates such “hand-scalpel” system in VR and AR simulation process, where the collision area of blade interacts with an imitating an organic 3D object. In this paper we presented the sensitivity and efficiency of “hand-pencil” system inside the virtual scene. In addition, the comparison and application of the methodology were highlighted for VR and AR prototypes of the laboratory scene.

Minimally Invasive Surgeries (MIS) such as laparoscopic procedures are increasingly preferred over conventional surgeries due to many different advantages. Laparoscopic surgical procedures are very complex compared to open surgeries and require high level of experience and expertise. Hybrid surgery simulators available for training using physical phantoms are expensive and not readily available in majority of health care facilities around the world. Therefore, computer simulation or Virtual Reality (VR) is a better way to obtain skills for MIS. A VR simulator incorporated with haptic feedback provides a comprehensive training closer to real world experience. In this paper, we present a novel approach to incorporate force feedback to VR laparoscopic surgery training. The proposed interface incorporates force feedback in all three axes to provide three levels of force feedback. Computational models of abdomen organs were generated using the cryosection data of Visible Human Project of the National Library of Medicine, USA. The organ models were developed with three basic force categories: soft, mild and hard. A hardware interface is developed to provide the force feedback for the interaction of virtual tools with the said organ models while generating the tool navigation information for the VR simulator.

Can virtual reality simulation be used for advanced bariatric surgical training?

In this paper, we present a new image-based method to display haptic texture information extracted from a static two dimensional image. The three dimensional forces allow the user to feel the contours and textures of the image with sufficient realism using haptic devices. The texture force is decomposed into a normal force component and a tangential force component at the haptic interaction point. The magnitude of the normal force is determined either by the color temperature (i.e. the emotions of warmth/coolness evoked by colors) or the luminance values in images. Warmer/brighter colors determine stronger normal forces to generate bumps, and cooler/darker colors determine weaker normal forces to produce depressions. The tangential force describes the relative color variation between haptic interaction point and its neighbors, thus, can be considered as a descriptor of the local texture feature. Because it is oriented to the pixels with cooler color in the neighborhood, when the user is exploring the bump by a haptic device, tangential resistive forces are sent to the user until the top of the bump is reached and, after the top, the virtual probe is pulled in the other direction. Our force model is by means of simple image processing techniques. The calculation is totally done by local, independent and direct operation. In other words, global operation such as recovering height maps is not necessary.

Haptic devices need to render accurate force to a human operator in order to create a virtual environment from a remote real environment. Existing end-effector type haptic devices employ parallel mechanisms to achieve force interaction in multiple degrees-of-freedom (DoF). Those parallel mechanisms have non-constant Jacobian matrices that create difficulties in the actuator and controller design for accurate haptic force interaction. This paper presents a 3-DoF translational parallel manipulator (TPM) with a constant Jacobian matrix to facilitate the force and impedance controller design. Instead of using external force sensors, series elastic actuators (SEAs) are used for the proposed device to realize the force rendering. SEAs can render more accurate output force and stiffness while the inertia and friction problems of using a gearhead can be ignored. In this paper, the modeling, design, and control of the haptic device are established to show the merits of the proposed TPM. Initial force control experiments demonstrate the accuracy and speed of SEA-based force control. We expect that this novel haptic device can serve as an alternative for robotic teleoperations requiring accurate force sensing and control.

To investigate the effect on final-product performance of a distributed, virtual reality (VR) simulation training program on cadaveric dissection performance and learning curves compared with standard VR simulation training during a temporal bone course. Educational interventional cohort study. The national Danish temporal bone courses of 2016 and 2017. Postgraduate year 2 to 5 residents in otorhinolaryngology. Nine participants volunteered for additional VR simulation training (intervention) before the temporal bone course, with training blocks distributed (i.e., separated). The remaining 28 participants received standard VR simulation training during the temporal bone course (control). VR simulation and cadaveric dissection final-product performances were analyzed by blinded raters using a 26-item modified Welling Scale. Distributed VR simulation training before the temporal bone course (intervention) significantly increased dissection final-product performance by 25% compared with standard VR simulation training during the course (control) (mean scores 12.8 points versus 10.3 points, p < 0.01). Distributed and repeated VR simulation practice markedly decreased drilling time. Guidance by the simulator-integrated tutor-function significantly increased final-product performance by 2.3 points compared with nontutored procedures but at the cost of increased drilling time. Skills acquired in a VR simulation environment translate to cadaveric dissection skills and repeated and distributed VR simulation can be used to further increase performance compared with standard VR simulation training during a temporal bone course. Further dissemination of inexpensive VR simulators would allow all future temporal bone course participants to train locally before attending future centralized courses.

Development of three-dimensional visualisation technology of the aerodynamic environment in a greenhouse using CFD and VR technology, Part 2: Development of an educational VR simulator

Virtual reality (VR) applications for surgical planning and mentoring are currently becoming more widespread. Within these fields of applications, haptic systems are crucial to help the understanding of the feedback provided by the tissue during the interaction with the surgical tool. The accuracy of the VR scenario is related to many aspects, among them the ability of the haptic device to provide the force feedback and thus the adoption of proper characterization and modeling of the soft tissue behavior during deformation. This paper presents the current development concerning a VR environment to be equipped with a force feedback haptic device to simulate and experience clamping in colorectal surgery. The reaction at the clamp handle is provided starting from a FEA of the contact between the surgical tool and the tissue, coupled with the kinematic analysis of the mechanism. Connecting the virtual scenario to a microcontroller board, the reaction force is applied to the the haptic device, shaped as a clamping tool handle, so that the user may experience the tissue impedence during clamping. By simulations made through Simscape, a concept is evaluated according to some simulation paths that involve common scenarios of the virtual reality experience (such as collision, partial or total clamping operation). The novelty of the research regards the system design oriented to increase the accuracy of the soft tissue material behaviour through a preliminary off-line dedicated FEA investigation. Through Simulink simulations, the system design is verified so that the VR system set-up may be carried out. The results highlighted: (a) a good correspondence among the theoretical study of the reference kinematism that the haptic device must reproduce; (b) the implementation of a bilinear material model, derived off-line from FEA, is suitable to capture hyperelastic and geometric non-linearities of the soft tissue; (c) the feasibility of the open loop haptic control in accordance to the explored kinematism. The final goal of the research is to propose a system design approach able to tune the VR experince for teaching and mentoring through a in-house set-up easy to be tailored.

In haptic interaction, friction caused by slip on the fingertip is a key factor for manual manipulation as well as exploration of texture and shape. From the moment of contact, the friction contains vertical and tangential skin deformations and vibrations, not all of which have been simultaneously supported by previous portable/wearable haptic devices. We propose a portable haptic device that has the ability to present skin deformation and vibration with two degrees of freedom by using two types of motors: a voice coil motor (VCM) for vertical motion and vibration, and direct current motors for tangential skin stretch. The VCM also achieves encounter-type haptic interactions. A combination of these motions encompasses most cutaneous cues for realistic friction.

Simulation is a key component of surgical training, enabling trainees to develop their skills in a safe environment. With simulators broadly grouped into physical models and virtual-reality (VR) simulators, it is important to evaluate the comparative effectiveness of the simulator types in terms of validity as well as cost. The review aims to compare the benefits and drawbacks of novel VR and physical simulators within the broader themes of endourology, laparoscopic and robotic operations, and other urological procedures. Key benefits of bench models include their comparatively lower cost, easy access and provision of haptic feedback, whereas VR simulators are generally self-sufficient, reusable and enable skills of haemostasis to be practised. The advent of perfused 3D printed simulators across a range of urological procedures may replace cadavers as the traditional gold-standard simulation modality. Although possessing differing strengths and downsides, VR and physical simulators when used together can have an additive effect due to skill transferability across the platforms. Further comparative studies are required to directly quantify the differences between physical models and VR simulators in terms of performance metrics and cost-effectiveness. There is lack of validated VR simulators for open and reconstructive procedures.