Study on the Influence of Upper Limb Representations and Haptic Feedback in Virtual Reality

This paper outlines a study into the effects of co-location (the term `co-location' is used throughout to refer to the co-location of haptic and visual sensory modes, except where otherwise specified) of haptic and visual sensory modes in VR simulations. The study hypothesis is that co-location of these sensory modes will lead to improved task performance within a VR environment. Technical challenges and technological limitations are outlined prior to a description of the implementation adopted for this study. Experiments were conducted to evaluate the effect on user performance of co-located haptics (force feedback) in a 3D virtual environment. Results show that co-location is an important factor, and when coupled with haptic feedback the performance of the user is greatly improved.

We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user's finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.

This paper presents the first results on a user study in which people with visual impairments (PVI) explored a virtual environment (VE) by walking in a virtual reality (VR) treadmill. As recently suggested, we have now acquired first results from our feasibility study investigating this walk-in-place interaction. This represents a new, more intuitive way of for example virtually exploring unknown spaces in advance. Our prototype consists of off-the-shelf VR components (i.e., treadmill, headphones, glasses, and controller) providing a simplified white cane simulation and was tested by six visually impaired subjects. Our results indicate that this interaction is yet difficult, but promising and an important step to make VR more and better usable for PVIs. As an impact on the CHI community, we would like to make this research field known to a wider audience by sharing our intermediate results and suggestions for improvements, on some of which we are already working on.

Haptic feedback provides an essential sensory stimulus crucial for interaction and analyzing three-dimensional spatio-temporal phenomena on surface visualizations. Given its ability to provide enhanced spatial perception and scene maneuverability, virtual reality (VR) catalyzes haptic interactions on surface visualizations. Various interaction modes, encompassing both mid-air and on-surface interactions-with or without the application of assisting force stimuli-have been explored using haptic force feedback devices. In this paper, we evaluate the use of on-surface and assisted on-surface haptic modes of interaction compared to a no-haptic interaction mode. A force-based haptic stylus is used for all three modalities; the on-surface mode uses collision based forces, whereas the assisted on-surface mode is accompanied by an additional snapping force. We conducted a within-subjects user study involving fundamental interaction tasks performed on surface visualizations. Keeping a consistent visual design across all three modes, our study incorporates tasks that require the localization of the highest, lowest, and random points on surfaces; and tasks that focus on brushing curves on surfaces with varying complexity and occlusion levels. Our findings show that participants took almost the same time to brush curves using all the interaction modes. They could draw smoother curves using the on-surface interaction modes compared to the no-haptic mode. However, the assisted on-surface mode provided better accuracy than the on-surface mode. The on-surface mode was slower in point localization, but the accuracy depended on the visual cues and occlusions associated with the tasks. Finally, we discuss participant feedback on using haptic force feedback as a tangible input modality and share takeaways to aid the design of haptics-based tangible interactions for surface visualizations.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Stroke is a worldwide health problem causing loss of sensory and motor functions. Biofeedback plays a significant role in the rehabilitation process of stroke patients where various haptic feedback methods and devices are used. Such methods and devices can greatly improve the biofeedback provided during the rehabilitation process. This work proposes a soft monolithic haptic feedback device that can be directly manufactured using a low-cost additive manufacturing technology known as fused deposition modelling (FDM) that employ soft and flexible materials with low elastic moduli. A single unit of the soft haptic feedback device proposed was analyzed using finite element modeling (FEM). The experimental mechanical deformation and force output of the device are presented. A conductive resistive material was used to directly 3D print the sensory component of the device that provides force generated by the soft haptic feedback device when it is in direct contact with the human skin. A thermoplastic poly(urethane) (TPU) was employed to 3D print the actuation unit of the soft haptic feedback device.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

Dysphagia, which results from various disorders, may increase the risk of aspiration pneumonia, dehydration, and malnutrition. The aim of this study was to develop an innovative evaluation and treatment system for swallowing therapy using virtual reality (VR) feedback and electrical stimulation (ES), and to make an initial evaluation of its potential. In this system, the activation of the submental muscle and acceleration of laryngeal movement are used as the evaluation and feedback information. Twenty-one patients with chronic dysphagia for an average of 26.3 months were recruited for the VR feedback study. Each participant underwent 16 treatment sessions. After therapy, the Functional Oral Intake Scale results changed from 3.3 ± 1.5 to 5.0 ± 1.6 with statistical significance (p = 0.000). Thirteen healthy subjects were enrolled in the ES study. ES was applied for more than 2 s while the subjects were swallowing. With and without ES, swallowing triggering times were 456.17 ± 106.92 and 552.13 ± 105.97 ms, respectively. These differences were statistically significant (p = 0.04). Accelerations of laryngeal movement were 0.23 ± 0.1 g (g = 9.8 m/s2) and 0.20 ± 0.08 g, respectively, with a significant statistical difference (p = 0.033). The feasibility of a prototype that combines swallowing evaluation, VR feedback therapy, and synchronized ES is demonstrated for further clinical trials. Further comprehensive clinical studies are needed to verify the clinical efficacy of the device.

How does haptic feedback during human-virtual character interaction affect participant physiological responses in virtual reality? In this between-subjects study, haptic feedback and non-haptic feedback conditions in which virtual characters bump into the participant who is immersed in a virtual environment are compared. A questionnaire was developed to determine the influence of haptic feedback on presence, embodiment, positive and negative affect, interaction with virtual characters, and haptic feedback realism, among other more exploratory concepts. These exploratory variables include engagement, flow, comfort with virtual characters, comfort with virtual characters’ appearance, realism of virtual character interaction, realism of haptic feedback, and virtual reality sickness. Physiological data was collected using galvanic skin response (GSR) to investigate the influence of haptic feedback on physiological arousal during human-virtual character interaction. Five conditions were developed (no haptic feedback, full and half intensity, incorrect position, and delayed timing). Significant differences were found in embodiment, realism of virtual character interaction, haptic feedback realism, and GSR amplitude after the first interaction with the virtual character. These results may inform future virtual reality studies that investigate haptic feedback during human-virtual character interaction and/or arousal via GSR data, as well as advise studies that seek to correlate self-report responses with physiological data.

Haptic feedback is important to improve the operability of virtual objects. However, AR systems with haptic feedback are still not many. Moreover, users cannot perform tasks that need large workspaces in those systems because of physical constraints from haptic feedback devices. In this paper, we combined a wearable haptic device and a marker-less AR technology to develop an AR system in which users can perform activities that need large workspaces. Differences of the operability of virtual objects in an assembly task of large machines are examined to evaluate effects of the haptic feedback in this system. The result indicates that haptic feedback is effective and users can perform activities that need large space in this system.

How does haptic feedback during a human-virtual human interaction affect emotional arousal in virtual reality? In this between-subjects study, we compare haptic feedback and no haptic feedback conditions in which a virtual human "bumps" into the participant in order to determine the influence of haptic feedback on emotional arousal, sense of presence, and embodiment in virtual reality, as well as compare self-report measures of emotional arousal to those objectively collected via event-related galvanic skin response (GSR) recordings. We plan to extend the current preliminary study by adding three more conditions as described in the future work section. Participants are students age 18-32 with at least moderate experience in virtual reality. Preliminary results indicate significant differences in presence and embodiment between haptic feedback and no haptic feedback groups. With our small sample size at the current time, GSR does not show significant differences between haptic and no haptic feedback conditions.

Typing is fundamental to our daily lives, particularly at work. Virtual reality (VR) technology has seen increasing interest in incorporating it into job training and rehabilitation programs. The concept of this study involves displaying the effect of virtual hands on a user's view in VR environment to mimic their real-time hand movements. The main objective of this study was to investigate muscle activity and task performance with and without virtual hand representation during VR typing tasks. As part of the study, wireless electromyography (EMG) sensors were used to measure the left and right neck splenius capitis muscles' activity (LSC and RSC), and typing performance was measured in terms of actual and net typing speed (WPM) and accuracy percentage (%). In addition, the NASA-TLX subjective workload scale was used to compare the users' mental workload with and without the virtual hand representations. This study involved 15 participants (7 females and 8 males), all over the age between 18 and 30 years who participated voluntarily and had no symptoms of musculoskeletal disorders or back/neck pain within 7 days of participation. During the study, participants were randomly assigned to two conditions: typing in VR with and without virtual hand representation. The Oculus Quest 2 VR headset and a Logitech k830 Bluetooth keyboard were utilized. A typing test was conducted using typingtest.com with five minutes of typing per condition. The results of the study indicated that virtual hand representation had a positive effect on typing performance, resulting in significantly higher typing speed (125.86% higher net speed) and accuracy (1.63% higher) compared to the condition without virtual hand representation. Additionally, the study found that typing in VR with virtual hand representation resulted in a significantly higher normalized muscle activities (% maximum voluntary contractions) of the neck muscles (17% higher for LSC and 19.58% higher for RSC) than in VR without virtual hand representation. Participants showed significantly lower NASA-TLX values with the virtual hand representation. In other words, lower mental demand was recorded during typing tasks with virtual hand representation. The findings of this study suggested that the presence of virtual hands for typing in VR resulted in more active typing behavior and improved performance. Increasing neck muscle activities during virtual hand representation could be associated with more active typing behavior and improved performance. Furthermore, participants perceived the virtual hand representation condition to be less mentally demanding due to the increased sensory feedback provided. Virtual hand representation was able to enhance typing efficiency, although further design configuration and improvement will be required to improve typing speed. With more realistic virtual hand representations, haptic feedback, and customizing virtual hand size and position to match the user's real hand size and position, virtual hand representation can be more effective in typing.

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state of the art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronized and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through, e.g., amplitude modulation (AM) or spatiotemporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

Minimally invasive laparoscopic surgery requires intensive training due to challenges such as loss of haptic feedback and depth perception. Traditional training methods include box trainers (BT), which offer realistic haptic feedback but lack objective performance assessment, and virtual reality (VR) simulators, which provide automated feedback but lack haptic feedback. This review, conducted at the Barts Cancer Institute, Queen Mary University, examines the learning outcomes of VRsimulators with haptic feedback compared to BT. A systematic review and meta-analysis was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines from December 2023 to April 2024. Research databases, such as PubMed, EMBASE, CINAHL, and Web of Science, were searched for randomized controlled trials (RCTs) comparing VR simulators with haptic feedback to BT in training medical students. Seven RCTs met the inclusion criteria, and four were included in the meta-analysis. The primary outcomes were learning curves and learning effectswhile secondary outcomes included skill transfer to a surgical environment. The review analysed data from 125 participants across the studies. Results indicated that BTs demonstrated a superior learning curve, with participants achieving proficiency faster than those using VR. Both simulators showed significant learning effects; however, BTs resulted in greater improvements across more performance parameters. Regarding skill transfer to surgical environments, there was no significant difference between the two groups, suggesting both approaches effectively support surgical skill transfer. Overall, BT has a more effective learning curve and marginally better performance in skill acquisition. While VR with haptic feedback offers enhanced realism, it does not fully replicate the natural haptic feedback provided by BT. Further studies are needed to improve VR haptic feedback and its integration into training programs to enhance learning outcomes.

The American Board of Orthopaedic Surgery has mandated dedicated skills training for first-year orthopedic surgical residents.1 Most residency programs address this requirement with training exercises with cadavers and plastic foam bones. Some programs incorporate one or more simulators in their skills training, including several sophisticated virtual reality simulators and a variety of low-tech simulators. Simulators are helpful because they can provide repeatable educational experiences and quantitative performance assessment. Unfortunately, few simulators have been developed for orthopedic trauma skills training. Even fewer simulators have been developed and validated with more advanced students, such as residents in their 3rd or 4th year of training, and for more complex surgeries. In contrast to the completely virtual surgical simulation using haptic feedback devices and sophisticated renderings of soft tissue deformation, our group has chosen to use physical models, real surgical instruments and position tracking in conjunction with virtual reality.2–4 The physical models provide experience with the surgical tools, and enable more realistic hand movements and haptic cue feedback.