Evaluating Force-Based Haptics for Immersive Tangible Interactions with Surface Visualizations.

This chapter discusses the applications and solutions of emerging Virtual Reality (VR) and video games technologies in the healthcare sector, e.g. physical therapy for motor rehabilitation, exposure therapy for psychological phobias, and pain relief. Section 2 reviews state-of-the-art interactive devices used in current VR systems and high-end games such as sensor-based and camera-based tracking devices, data gloves, and haptic force feedback devices. Section 3 investigates recent advances and key concepts in games technology, including dynamic simulation, flow theory, adaptive games, and their possible implementation in serious games for healthcare. Various serious games are described in this section: some were designed and developed for specific healthcare purposes, e.g. BreakAway (2009)’s Free Dive, HopeLab (2006)’s Re-Mission, and Ma et al. (2007)’s VR game series, others were utilising off-the-shelf games such as Nintendo Wii sports for physiotherapy. A couple of experiments of using VR systems and games for stroke rehabilitation are highlighted in section 4 as examples to showcase the benefits and impacts of these technologies to conventional clinic practice. Finally, section 5 points some future directions of applying emerging games technologies in healthcare, such as augmented reality, Wii-mote motion control system, and even full body motion capture and controller free games technology demonstrated recently on E3 2009 which have great potentials to treat motor disorders, combat obesity, and other healthcare applications.

Camera drones allow exploring remote scenes that are inaccessible or inappropriate to visit in person. However, these exploration experiences are often limited due to insufficient scene information provided by front cameras, where only 2D images or videos are supplied. Combining a camera drone vision with haptic feedback would augment users' spatial understandings of the remote environment. But such designs are usually difficult for users to learn and apply, due to the complexity of the system and unfluent UAV control. In this paper, we present a new telepresence system for remote environment exploration, with a drone agent controlled by a VR mid-air panel. The drone is capable of generating real-time location and landmark details using integrated Simultaneous Location and Mapping (SLAM). The SLAMs' point cloud generations are produced using RGB input, and the results are passed to a Generative Adversarial Network (GAN) to reconstruct 3D remote scenes in real-time. The reconstructed objects are taken advantage of by haptic devices which could improve user experience through haptic rendering. Capable of providing both visual and haptic feedback, our system allows users to examine and exploit remote areas without having to be physically present. An experiment has been conducted to verify the usability of 3D reconstruction result in haptic feedback rendering.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

Visual representations are used in various digitization processes to display objects in immersive environments. In industrial environments, the self-representation can be very helpful when reviewing 3D models to conform realistic proportions, particularly in combi-nation with haptic feedback that goes beyond vibrating controllers. For instance, haptic feedback in combination with a virtual representation supports working on use cases where collision is an important part to maintain data quality. To understand the dependency of both, we conducted a pilot study with 15 users from the automotive sector to examine the influence of upper limb representations on haptic feedback in Virtual Reality. Each participant was assigned with one of three upper limb representations which was used in different scenarios using haptic feedback devices. Overall, we found that the realistic arm representation was rated highest in terms of perceived realism and achieved the best task performance.

Minimally invasive laparoscopic surgery requires intensive training due to challenges such as loss of haptic feedback and depth perception. Traditional training methods include box trainers (BT), which offer realistic haptic feedback but lack objective performance assessment, and virtual reality (VR) simulators, which provide automated feedback but lack haptic feedback. This review, conducted at the Barts Cancer Institute, Queen Mary University, examines the learning outcomes of VRsimulators with haptic feedback compared to BT. A systematic review and meta-analysis was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines from December 2023 to April 2024. Research databases, such as PubMed, EMBASE, CINAHL, and Web of Science, were searched for randomized controlled trials (RCTs) comparing VR simulators with haptic feedback to BT in training medical students. Seven RCTs met the inclusion criteria, and four were included in the meta-analysis. The primary outcomes were learning curves and learning effectswhile secondary outcomes included skill transfer to a surgical environment. The review analysed data from 125 participants across the studies. Results indicated that BTs demonstrated a superior learning curve, with participants achieving proficiency faster than those using VR. Both simulators showed significant learning effects; however, BTs resulted in greater improvements across more performance parameters. Regarding skill transfer to surgical environments, there was no significant difference between the two groups, suggesting both approaches effectively support surgical skill transfer. Overall, BT has a more effective learning curve and marginally better performance in skill acquisition. While VR with haptic feedback offers enhanced realism, it does not fully replicate the natural haptic feedback provided by BT. Further studies are needed to improve VR haptic feedback and its integration into training programs to enhance learning outcomes.

Teleoperated minimally invasive surgical robots can significantly enhance a surgeon's accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon's performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through "virtual fixtures" and the automatic acquisition of tissue properties.

The use of virtual reality technology has become popular in modern theme parks across the world. Attractions using virtual reality technologies offer highly immersive experiences as if people are really staying in fictional worlds that the theme parks like to create. Haptic feedback is an important piece in virtual reality, where it can increase the immersion and enjoyment, but most virtual reality attractions in theme parks do not offer enough haptic feedback. In this paper, we present HapticDaijya, which is a waist-worn robot capable of giving various haptic and tactile feedback on the torso, neck, face, arms and hands. We present the design and implementation of HapticDaijya. Then, we show a user study and analyze the results for investigating its feasibility. Also, we present preliminary evaluation results that gauged the user’s accuracy in distinguishing the locations of taps applied on the chest, as well as general usability and user acceptance.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

How does haptic feedback during human-virtual character interaction affect participant physiological responses in virtual reality? In this between-subjects study, haptic feedback and non-haptic feedback conditions in which virtual characters bump into the participant who is immersed in a virtual environment are compared. A questionnaire was developed to determine the influence of haptic feedback on presence, embodiment, positive and negative affect, interaction with virtual characters, and haptic feedback realism, among other more exploratory concepts. These exploratory variables include engagement, flow, comfort with virtual characters, comfort with virtual characters’ appearance, realism of virtual character interaction, realism of haptic feedback, and virtual reality sickness. Physiological data was collected using galvanic skin response (GSR) to investigate the influence of haptic feedback on physiological arousal during human-virtual character interaction. Five conditions were developed (no haptic feedback, full and half intensity, incorrect position, and delayed timing). Significant differences were found in embodiment, realism of virtual character interaction, haptic feedback realism, and GSR amplitude after the first interaction with the virtual character. These results may inform future virtual reality studies that investigate haptic feedback during human-virtual character interaction and/or arousal via GSR data, as well as advise studies that seek to correlate self-report responses with physiological data.

How does haptic feedback during a human-virtual human interaction affect emotional arousal in virtual reality? In this between-subjects study, we compare haptic feedback and no haptic feedback conditions in which a virtual human "bumps" into the participant in order to determine the influence of haptic feedback on emotional arousal, sense of presence, and embodiment in virtual reality, as well as compare self-report measures of emotional arousal to those objectively collected via event-related galvanic skin response (GSR) recordings. We plan to extend the current preliminary study by adding three more conditions as described in the future work section. Participants are students age 18-32 with at least moderate experience in virtual reality. Preliminary results indicate significant differences in presence and embodiment between haptic feedback and no haptic feedback groups. With our small sample size at the current time, GSR does not show significant differences between haptic and no haptic feedback conditions.

Haptic feedback is important to improve the operability of virtual objects. However, AR systems with haptic feedback are still not many. Moreover, users cannot perform tasks that need large workspaces in those systems because of physical constraints from haptic feedback devices. In this paper, we combined a wearable haptic device and a marker-less AR technology to develop an AR system in which users can perform activities that need large workspaces. Differences of the operability of virtual objects in an assembly task of large machines are examined to evaluate effects of the haptic feedback in this system. The result indicates that haptic feedback is effective and users can perform activities that need large space in this system.

Robot-assisted minimally invasive surgery (MIS) has countless benefits over open surgery, from shorter re- covery times and lower risk procedures for the patient to higher accuracy and broader capabilities for the surgeon [1]. However, a significant detriment to these procedures is that current systems lack haptic feedback. The lack of haptic feedback in MIS forces the surgeon to depend merely on visual cues, such as the deformation of tissue under load, to estimate the forces [1]. The likely outcome of misreading these cues is torn tissue, patient discomfort or broken sutures [2]. Moreover, haptic feed- back is specifically vital for robot-assisted endoscopy procedures. A recent study evaluating an Endoscopic Operation Robot (EOR) concluded that haptic feedback is beneficial in remote manipulation of flexible endo- scopes. When haptic feedback was absent there were more incidences of overstretching of sigmoid colon in a colonoscopy training model [3]. This work presents a soft robotic glove that provides haptic feedback for endoscopic procedures (Fig. 1, A). In our previous work, we introduced a soft robotic sleeve [4] that can detect forces between a colonoscope and colon walls during navigation. The glove receives force input from the soft robotic sleeve wrapped around the colonoscope (Fig. 1, B). Any incident force on the sleeve, during endoscopic navigation, is relayed to the surgeon as haptic feedback through proportional inflation of the glove’s pneumatic actuators.

Passing an object between two people is a common event that happens in various forms for example when giving someone a cup of coffee. An experimental study is presented where passing objects between two people in a virtual environment with haptic feedback was compared to passing objects in a nonhaptic virtual environment. The aim of the experiment was to investigate if and how added haptic feedback in such an environment affects perceived virtual presence, perceived social presence and perceived task performance. A within subject design was used, were nine pairs of subjects performed a hand off task with six differently sized cubes without audio communication. Results showed that haptic force feedback significantly improved perceived virtual presence, perceived social presence and perceived performance in this experiment.KeywordsHapticPresenceSocial presenceCollaborative

Rendering haptic feedback on the wrist is an effective solution for freeing users' hands during virtual interactions while still providing haptic feedback, mostly focusing on Virtual Reality (VR) environments. Unfortunately, whether these solutions can be extended to Augmented Reality (AR) interactions has not been investigated before. In this paper, we investigate the perceptual differences between VR and AR environments with a user study experiment based on a stiffness discrimination task. Our findings revealed similar task accuracy and sensitivity in VR and AR but different exploration behaviors (i.e. higher interaction time in AR) and user experience (i.e. higher enjoyment and less mental fatigue in AR). These differences may be explained by the increased cognitive load and the environmental immersion in AR, where users navigate both virtual and real elements simultaneously. This suggests that visuo-haptic psychophysical findings obtained in VR can be extended to AR, even though user experience and exploration behavior might differ.

Stroke is a worldwide health problem causing loss of sensory and motor functions. Biofeedback plays a significant role in the rehabilitation process of stroke patients where various haptic feedback methods and devices are used. Such methods and devices can greatly improve the biofeedback provided during the rehabilitation process. This work proposes a soft monolithic haptic feedback device that can be directly manufactured using a low-cost additive manufacturing technology known as fused deposition modelling (FDM) that employ soft and flexible materials with low elastic moduli. A single unit of the soft haptic feedback device proposed was analyzed using finite element modeling (FEM). The experimental mechanical deformation and force output of the device are presented. A conductive resistive material was used to directly 3D print the sensory component of the device that provides force generated by the soft haptic feedback device when it is in direct contact with the human skin. A thermoplastic poly(urethane) (TPU) was employed to 3D print the actuation unit of the soft haptic feedback device.