The Movement-Related Cortical Activities under the Virtual Environment with Force Feedback

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Since [years] we hear about Virtual Reality as a discipline that could potentially provide benefits to many applications. Basically, the principle of Virtual Reality consists in stimulating user senses in order to give the impression to be in another place which they can discover and interact with. Today, most virtual reality systems create realistic visual and auditive environments. However the interaction with these environments can not be considered as natural. Indeed, we often use devices such as mouse, keyboard or joystick to move or manipulate them. These paradigms of interaction are in fact metaphors, which are not similar to reality. In some situations they are practical and efficient. However, the lack of intuitiveness sometimes makes them limited or simply ineffective. To overcome this, researchers can use Haptic Devices. They are designed to simulate what is commonly called the of touch, which includes more specifically, tactile, pain sense, thermal sense, and proprioception. Proprioception is knowledge gained by the perception of the relative member's position of the human body. In this thesis, we particularly focus on the simulation of proprioception. There are two advantages of such haptic devices. First, they can give the user more information on the nature of virtual objects (size, weight, finish, rigidity, etc.). Second, they can provide interaction paradigms that are closer to reality (three-dimensional interaction in a three-dimensional world). However, haptic device mechanics is complex. Moreover, proprioception is a sense that covers the entire body which is a rather large surface. For this reason, haptic devices usually apply force feedback on a very small portion of the body, such as fingertip. In addition to this hardware constraint, haptic research also faces software constraints. Indeed, a haptic application requires many computer resources in order to perform collision detection, dynamic animation of objects, and force feedback computation. Moreover, this should be done at a refresh rate that is much higher than the visualization for producing a convincing result. In the first part of this thesis, we propose to increase realism and complexity of haptic applications. To achieve this goal, we use a state-of-the-art commercial device which allows to acquire the posture and position of both hands, and to apply forces on the fingertips and wrists. We propose techniques to calibrate and improve the comfort of these kinds of devices in order to integrate them into Virtual Environments. However, a two-handed haptic device do not presents only advantages. Indeed, It is much more complicated to compute forces on two hand models, than on a single point or fingertip. For this reason, in this thesis, we propose a framework to optimize this computation. Thanks to it, we can create Virtual Environments in which an object is graspable and dynamically animated by the laws of physics. When the object is seized by both hands, the haptic rendering engine realistically computes the forces on both exoskeletons. The efficiency of our rendering permits to apply these techniques to complex environments that have a significant number of objects. But the existing visual Virtual Environments are much more detailed than the ones seen in common haptic applications. In this thesis, we aim at reducing this gap. One of the problems is that these quality environments usually do not include specific haptic object properties, such as mass or material. We thus propose a software allowing even non-professional to quickly and easily add this information to an environment. Our results show that this haptic rendering engine does not suffer from the large quantity of objects. They demonstrate that we have an efficient framework for integrating a two-handed haptic interface into a generic virtual environment. In the second part, we evaluate the potential of these kinds of Virtual Reality systems in more detail. While most applications can in theory take advantage of haptic devices, the practice shows that it is not always the case. Indeed, with experience, some metaphorical interaction paradigms remain more powerful than realistic ones. We thus present and study the integration of our two-handed haptic interface in a variety of applications. Evaluations show that depending on the application, it is not appropriate to reproduce reality: in teleoperation, for instance, simulating a virtual haptic steering wheel is less efficient than providing a force gesture interface. On the other hand, in virtual learning, the power of two-handed haptic manipulation is fully exploited and presents great advantages over standard techniques.

In this study, a haptic device is proposed for virtual reality (VR) applications. The proposed haptic device is linked to the bundle controller of a VR device. The device proposed in this study is a force-feedback device that focuses on the sensation felt by the hand rather than the feeling of the tool colliding with the object. The device provides sensations of centrifugal or inertial forces due to the weight of the tool when the user swings the tool and the sensation of a reaction force when the user stabs or pokes an object with the tool. The force generated by a haptic device of the linked type was measured and examined to determine whether a sensation of actually using a tool in the virtual environment was provided. In addition, we conducted two types of experiments to evaluate the proposed haptic device. First, VR content was created for the user experience, and a survey was conducted with 30 experimenters to compare the bundle controllers and a haptic device, called HapTug, in terms of realism, immersion, and enjoyment. The survey results demonstrated that the realism, immersion, and enjoyment increased compared to using the bundle controller alone. A total of 41 experimenters were allowed to play freely in a virtual environment for six sessions of 30 s each. They pressed the reset session button when they felt 30 s had passed, and the device they were wearing turned on and off randomly every session. During the experience, their playback time, number of interactions, and range were measured. As a result, when the device was activated, experimenters felt the time was shorter and showed widespread movement. Thus, it was proven that the proposed linked-type haptic device effectively delivered the feeling of wielding a tool, allowing the user to feel the virtual object coming in and going out of the hand when interacting with it. Moreover, three conclusions were drawn from the results of the experiments: previous VR experience times have little to do with satisfaction related to haptic device use, users have low expectations for haptic realism, and users interact more actively and exaggeratedly with a haptic device.

The motor function impairment resulting from a stroke injury has a negative impact on autonomy, the activities of daily living thus the individuals affected by a stroke need long-term rehabilitation. Several studies have demonstrated that learning new motor skills is important to induce neuroplasticity and functional recovery. Innovative technologies used in rehabilitation allow one the possibility to enhance training throughout generated feedback. It seems advantageous to combine traditional motor rehabilitation with innovative technology in order to promote motor re-learning and skill re-acquisition by means of enhanced training. An environment enriched by feedback involves multiple sensory modalities and could promote active patient participation. Exercises in a virtual environment contain elements necessary to maximize motor learning, such as repetitive and diffe-rentiated task practice and feedback on the performance and results. The recovery of the limbs motor function in post-stroke subjects is one of the main therapeutic aims for patients and physiotherapist alike. Virtual reality as well as robotic devices allow one to provide specific treatment based on the reinforced feedback in a virtual environment (RFVE), artificially augmenting the sensory information coherent with the real-world objects and events. Motor training based on RFVE is emerging as an effective motor learning based techniques for the treatment of the extremities.

This chapter examines virtual environments which usually represent computer simulated environments, otherwise called virtual worlds or virtual reality and its application in physical therapy. Virtual environments present computer-generated three-dimensional (3D) representation of a physical presence of places in the real world as well as in the imaginary world. Users mainly interact with virtual environments using virtual reality headsets. Many of the current virtual reality environments are primarily visual experiences display either on a computer screens or through other stereoscopic displays. Applications of virtual environments are mainly focused on artificial reality that projects users into a 3D space generated by the computer with some sort of tracking devices which may be the virtual reality headsets worn or another 3D tracking device attached to the body of the user such as the electromagnetic tracker system. Research in virtual environments has made significant impact in the entertainment industry and in the training field such as the flight simulator (Stern, 1995).

Mirror drawing is a motor learning task that is used to evaluate and improve eye-hand coordination of users and can be implemented in immersive Virtual Reality (VR) Head-Mounted Displays (HMDs) for training purposes. In this paper, we investigated the effect of color cues on user motor performance in a mirror-drawing task between Virtual Environment (VE) and Real World (RW), with three different colors. We conducted a 5-day user study with twelve participants. The results showed that the participants made fewer errors in RW compared to VR, except for pre-training, which indicated that hardware and software limitations have detrimental effects on the motor learning of the participants across different realities. Furthermore, participants made fewer errors with the colors close to green, which is usually associated with serenity, contentment, and relaxation. According to our findings, VR headsets can be used to evaluate participants' eye-hand coordination in mirror drawing tasks to evaluate the motor-learning of participants. VE and RW training applications could benefit from our findings in order to enhance their effectiveness.

Virtual reality aims to provide an immersive experience to a user, with the help of a virtual environment. This immersive experience requires two key components; one for capturing inputs from the real world, and the other for synthesizing real world outputs based on interactions with the virtual environment. However, a user in a real world environment experiences a greater set of feedback from real world inputs which relate directly to auditory, visual, and force feedback. As such, in a virtual environment, a dissociation is introduced between the user's inputs and the feedback from the virtual environment. This dissociation relates to the discomfort the user experiences with real world interaction. Our team has introduced a novel way of receiving synthesized feedback from the virtual environment through the use of a haptic wheelchair.

Virtual environments have been widely used in motor neuroscience and rehabilitation, as they afford tight control of sensorimotor conditions and readily afford visual and haptic manipulations. However, typically, studies have only examined performance in the virtual testbeds, without asking how the simplified and controlled movement in the virtual environment compares to behavior in the real world. To test whether performance in the virtual environment was a valid representation of corresponding behavior in the real world, this study compared throwing in a virtual set-up with realistic throwing, where the task parameters were precisely matched. Even though the virtual task only required a horizontal single-joint arm movement, similar to many simplified movement assays in motor neuroscience, throwing accuracy and precision were significantly worse than in the real task that involved all degrees of freedom of the arm; only after 3 practice days did success rate and error reach similar levels. To gain more insight into the structure of the learning process, movement variability was decomposed into deterministic and stochastic contributions. Using the tolerance-noise-covariation decomposition method, distinct stages of learning were revealed: While tolerance was optimized first in both environments, it was higher in the virtual environment, suggesting that more familiarization and exploration was needed in the virtual task. Covariation and noise showed more contributions in the real task, indicating that subjects reached the stage of fine-tuning of variability only in the real task. These results showed that while the tasks were precisely matched, the simplified movements in the virtual environment required more time to become successful. These findings resonate with the reported problems in transfer of therapeutic benefits from virtual to real environments and alert that the use of virtual environments in research and rehabilitation needs more caution.NEW & NOTEWORTHY This study compared human performance of the same throwing task in a real and a matched virtual environment. With 3 days' practice, subjects improved significantly faster in the real task, even though the arm and hand movements were more complex. Decomposing variability revealed that performance in the virtual environment, despite its simplified hand movements, required more exploration. Additionally, due to fewer constraints in the real task, subjects could modify the geometry of the solution manifold, by shifting the release position, and thereby simplify the task.

Force feedback is an interface based on the phenomenon of contact. In the implementation of a virtual force feedback environment, the object is defined in a computer and the user is in the real world. Therefore, an interface device is required to transmit touch sensation to the user's finger when he or she touches a virtual object in virtual space. In this paper, methodology to realize force feedback is categorized from this view point, and the idea of surface display is presented as a force feedback method in a virtual environment. >

The evolution of virtual reality (VR) has created the opportunity for a relatively low-cost and accessible method to practice motor skills. Previous studies have demonstrated how motor skill practice in non-immersive virtual environments transfers to physical environments. Though minimal research has investigated motor learning and transfer within immersive VR, multiple experiments provide empirical evidence of positive transfer effects. Most studies have used software engines and modified hardware to enhance the similarities between virtual and physical environments. However, many learners and practitioners currently use commercially available VR intending to enhance real-world performance, though there is very little evidence to support the notion of positive transfer for these systems. Therefore, this experiment aimed to investigate how motor skill practice using a commercially available VR system improved real-world performance and how that compared to physical practice. Participants (n = 68) were randomly selected into one of two groups: virtual reality (VR) practice (n = 33) or real-world (RW) practice (n = 35). The experiment occurred over two consecutive days, with a pretest, posttest, and practice phase occurring each day. The pre- and post-testing phases were identical for both groups and consisted of putting a golf ball 10 times on a carpeted surface towards the center of a target. The practice phases consisted of 60 total putts per day in the respective environment (VR or RW). Participants continuously alternated golf putting holes from three different distances until they accrued 60 total putts. Participants in the RW group performed golf putts to three targets. Participants in the VR group also performed golf putts on three different miniature golf putting holes, using the commercially available Oculus Rift and the Cloudlands VR Minigolf game. The VR putting targets were designed to replicate the putting holes in the physical environment. Separate 2 (group) x 4 (test) repeated measures ANOVAs were used to assess accuracy and club head kinematics. The results revealed a significant main effect for test, but not for group. Post hoc analyses revealed that both groups significantly improved their putting accuracy and club head kinematics at similar rates. The results from this study indicate that the transfer of learning that occurred as a result of practicing in a commercially available VR environment was similarly effective when compared to RW practice.

How can we increase awareness and understanding of other cultures using interactive online digital visualizations of past civilizations? In order to answer the above question, this chapter first questions the success of current virtual environments and asks whether they are capable of producing a platform that supports the experience and understanding of place-inscribed culture. To answer this question, a typology of successful virtual environment applications is suggested. These include online communities, virtual realityVirtual environmentVirtual Reality (VR) travel websites, game engines, virtual reality used for training, and virtual reality used as part of therapeutic means to either relax or divert patients, or to cure phobiasPhobias. Impeding the success of virtual environments in terms of technological constraints, lack of evaluation techniques and results, and a lack of content-specific applications that best utilize desktop computing capabilities and respond to user needs. It suggests that much more work needs to be done on not just usable but also useful content. Specifically, there is a large gap in knowledge on what might constitute useful and appropriate virtual travel environments, and how contextual interaction may affect our cultural understanding in these environments.

The effects of the presence of others during a rowing exercise in a virtual reality environment

Purpose/Hypothesis: Virtual Reality (VR) has been proposed as a strategy for rehabilitation. To measure the potential of new VR devices they need to be characterized and compared to their real world counterparts. The primary purpose of this study was to characterize and validate the gait of individuals who walked on a VR walking simulator (WS) by comparing it to the kinematics and velocity of overground walking. The secondary purpose was to determine if the presence of the virtual environment (VE) modified walking characteristics. We hypothesized that walking velocity would be greater overground and slowest without the VE. Number of Subjects: Ten healthy subjects (2 male and 8 female) ages 24–53 without musculoskeletal injuries or neuromuscular pathology participated. Materials/Methods: Gait step length and velocity were collected (three trials) as participants walked at their self selected speed on a Gait Rite mat. Participants wore a harness and refective markers on the foot, ankle, knee, hip and trunk. They stood on the WS with each foot secured on a robotic platform, a harness attached to an unweighing frame, facing a screen. Unweighed 40% of their body weight they practiced walking on the WS (2–8 min.) until achieving the criterion gait pattern. Data were collected for four street crossing trials, walk without VE (2), walk with VE (2). An auditory tone or the VE green light prompted participants to begin walking. Kinematics were acquired using a 6 camera Peak system at 60 Hz. Joint angles were calculated for push of and initial contact by averaging the peak angle of three steps. Velocity data were averaged from the x-translation plots of the ffth metatarsal. Joint excursions from push of to initial contact-loading response were compared to normative data. Differences between overgound walking velocity and step length relative to the mobility simulator with and without the VE were assessed using a repeated measures ANOVA (alpha level= .05). Results: Walking velocity and step length were significantly greater for overground walking (1.5 m/sec; .79 m) compared with the mobility simulator with (.22 m/sec; .17 m) and without (.21 m/s; .18 m) the VE. The average joint excursions on the mobility simulator for push of to initial contact-loading response were 34 (knee) and 18 (ankle) degrees; compared to a range from push of to initial contact-loading response of 22–40 (knee) and 10–20 (ankle) degrees in normal gait. Conclusions: Walking in the VE resulted in pronounced decreases in temporal distance measures of gait however joint excursions were comparable to normal gait. There were no differences in gait velocity, step length or joint excursions when walking on the mobility simulator with and without the VE. The mechanics of the robot limit step length however, changing street crossing time in the VE may increase walking velocity. Clinical Relevance: Selected aspects of gait were preserved in the WS while others were not. Transfer of training from the walking simulator to real world walking still needs to be determined.

Virtual Reality (VR) is a subset of computer graphics where computers generate simulated environments. VR allows users to experience a virtual three-dimensional world where scenes and objects seem real. Wearable human machine interfaces (HMI) like helmets, gloves, etc are essential in making the experience feel realistic and interactive. Gloves in particular stimulate the sense of touch, which in some ways, is equally as important as visual stimulation. Developing a glove is a multi-faceted problem. Accurately tracking hand movement and delivering contextually appropriate feedback to the user in response to the events in the VR environment are the two main challenges. This paper proposes a scalable and low-cost haptic VR glove design that can accurately track and convey hand movement and finger flexion data. The design also incorporates the haptic primary colors (HPC), i.e., the design can stimulate the full gamut of touch senses, namely force, vibration, and temperature. The design makes use of flex sensors to track finger flexion. Inertial measurement units (IMUs) were used to track hand movement and rotation. A Bluetooth module communicates with a microcontroller connected to the VR environment. A vibro-thermal unit delivers the sense of vibration as well as temperature change. Servo motors were used to restrict finger movement in order to simulate force feedback. A prototype was built which was based on the proposed design. The prototype was tested in an integrated game development environment. It accurately emulated hand movement and reliably delivered a sense of touch by means of vibration and force feedback from the dedicated motors.