Augmenting visual feedback with visualized interaction forces in haptic-assisted virtual-reality teleoperation

In digital interactive interfaces with high visual workloads, it is important for operators to allocate their limited attentional resources appropriately to ensure efficient information collection. The salience, effort, expectancy, value (SEEV) model, which combines top‐down and bottom‐up attention mechanisms for predicting attention allocation, has been validated in research areas such as piloting, driving, and surgical operations. However, the validity of the SEEV model in the field of robotic arm teleoperation has not yet been thoroughly studied. The primary purpose of this study was to confirm the feasibility of the SEEV model for operator visual attention allocation prediction in a robotic arm teleoperation scenario. The improved ITTI algorithm, distance‐measuring tool, Delphi method, and lowest ordinal algorithm were adopted to qualify the four factors of the SEEV model, which also contributed to salience and expectancy quantification methods. Accordingly, an attention allocation prediction model in a robotic arm teleoperation scene was constructed. To verify the validity of the prediction model, 20 participants were recruited to control the robotic arm using V‐REP simulation software, and their fixation durations were recorded using an eye tracker as an attention allocation indicator. Participants controlled the robotic arm according to the experimental requirements and operational tasks, such as grasping and placing the target. The results demonstrated that the theoretical data based on the SEEV prediction model are significantly related to the proportion of fixation durations. The experiment verifies the suitability of the SEEV prediction model, and it is anticipated to be utilized in the optimization of interactive interfaces for robotic arm teleoperation.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Virtual reality (VR) enables the study of visually-guided behavior in situations that are otherwise not amenable to laboratory research. An example is a fire evacuation scenario, in which building occupants have to find their way to safety through smoke and flames, identify safe egress routes, and avoid threats. First, however, it must be determined whether evacuation behavior in VR is comparable to that in a real environment. We are thus initially studying participant responses to a fire alarm in matched virtual and real environments. The present experiment investigates how the decision to evacuate is influenced by visual cues about a neighbor’s behavior. In the Control group, a lone participant is exposed to a fire alarm while performing a bogus task. In the Passive group, a participant performs the same task with a confederate who ignores the fire alarm. In the Active group, the confederate immediately leaves the room when the fire alarm sounds. The gender of the participant is crossed with the gender of the confederate. 75 participants experience the scenario in a real environment (25 per group), and 75 others in a matched virtual environment presented in a wireless head mounted display (HMD), with avatars of the confederates. In the real environment, two-thirds of participants in the Control group evacuated the room. In the Active group, nearly all did so, whereas in the Passive group all but one did. More female than male participants evacuated in the Control group, but there were no other gender effects. Data collection in the virtual environment is in progress, and will be completed by spring. The findings thus far demonstrate a very strong influence of neighbor behavior on a participant’s evacuation decision. The results will provide answers to the much-debated question of whether evacuation behavior is similar in virtual and real environments. Meeting abstract presented at VSS 2015

We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user's finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.

Visual and auditory feedback cues have been shown to improve gait and balance in patients with movement disorders, such as Parkinson's disease and multiple sclerosis. Subsequently, closed-loop virtual reality was used to enhance the gait improvement effect of such cues. Objectives: To study the effects of gait training with visual and auditory feedback cues on the walking abilities of patients with gait disorders due to cerebral palsy. Methods: Visual and auditory feedback cues were generated by a wearable accelerometry-driven device. Ten randomly selected patients with gait disorders due to cerebral palsy trained with visual feedback cues while ten such patients trained with auditory feedback cues. Baseline performance (walking speed and stride length along a 10 m straight track) was measured before device use. Following 20 min training with the device, performance without the device was measured again and compared to the baseline performance. Results: For patients training with visual feedback, the average improvement in walking speed was 21.70%+/-36.06% and in stride length 8.72%plusmn9.47%. For patients training with auditory feedback, average improvement in walking speed was 25.43%plusmn28.65% and in stride length 13.58%plusmn13.10%. Conclusion: Visual and auditory feedback cues can improve gait parameters in patients with gait disorders due to cerebral palsy.

Downstairs resident classification characteristics for upstairs walking vibration noise in an apartment building under virtual reality environment

Tele-manipulation of heavy loads typically requires the simultaneous use of two asymmetric slaves: a crane for vertical weight support and a robot for accurate lateral positioning. The industrial standard prescribes a pair of operators for such tasks (one operator to control each slave), although in principle one operator might control both slaves with a single, hybrid interface. Accurate and safe co-operative handling of the expensive and fragile heavy components is difficult, presumably due to problems in the coordination of the subtasks and the lack of mutual awareness between the two operators. This study proposes a novel haptic assistance system to improve subtask coordination and task performance. Its novelty consists of haptically linking operators/interfaces through the joint task environment. The system's efficacy is evaluated with 15 pairs of co-operators and 15 individual uni-manual operators who maneuvered a heavy load through a bounded path in Virtual Reality. Haptic assistance improves task completion time for both groups. It also reduces control activity and self-reported workload without affecting a number of critical errors made by the operators. Moreover, without haptic assistance, uni-manual operators perform worse than co-operators, but this difference between the interfaces was not found with haptic assistance.

Virtual reality (VR) is being used increasingly in many fields of medicine, including rehabilitation. Both 2D and 3D virtual environments (VEs) can be viewed either through a head mounted display (HMD) or on a screen (computer monitor and rear projection system, SPS). However, the question of whether the medium through which the environment is viewed affects motor performance has not been addressed. The objective of our study was to determine whether movement patterns were different when movements were performed in a 3D fully immersive VE viewed via an HMD or SPS. Two groups of subjects were recruited (stroke, healthy). They performed pointing movements to targets placed in the ipsilateral, central and contralateral arm workspaces in a VE. The VE, designed to resemble the interior of an elevator, was viewed via an HMD or a SPS. Arm motor impairment and spasticity were evaluated in both groups of subjects. The kinematics of the pointing movements were recorded using an optical tracking system (Optotrak Certus, 100 Hz, 6 markers). Arm motor performance (speed, precision and trajectory straightness) and movement quality outcomes (elbow and shoulder ranges of motion and trunk forward displacement) were analyzed using 2 way ANOVAs. Preliminary results suggest that the control group had straighter movements and used more shoulder flexion as compared to the stroke group. When the VE was viewed via both media, there were no differences in terms of endpoint precision and speed, elbow and shoulder ranges of motion and trunk forward displacement in both groups. Both groups reported that they completely enjoyed performing the movements when viewing them via both media. All subjects in the control group and 80% of subjects in the stroke group reported that VE was engaging, that it felt real and that the movements performed were similar to those made in the physical world. The results of this study have implications for the design of rehabilitation applications using VR aimed at improving arm motor activity and function.

With the proliferation of low-cost, consumer level, head-mounted displays (HMDs) such as Oculus VR or Sony's Morpheus, we are witnessing a reappearance of virtual reality. However, there are still important stumbling blocks that hinder the development of applications and reduce the visual quality of the results. Knowledge of human perception in virtual environments can help overcome these limitations. In this paper, within the much-studied area of perception in virtual environments, we chose to look into the less explored area of crossmodal perception, that is, the interaction of different senses when perceiving the environment. In particular, we looked at the influence of sound on visual motion perception in a virtual reality scenario. We first replicated a well-known crossmodal perception experiment, carried out on a conventional 2D display, and then extended it to a 3D head-mounted display (HMD). Next, we performed an additional experiment in which we increased the complexity of the stimuli of the previous experiment, to test whether the effects observed would hold in more realistic scenes. We found that the trend which was previously observed in 2D displays is maintained in HMDs, but with an observed reduction of the crossmodal effect. With more complex stimuli the trend holds, and the crossmodal effect is further reduced, possibly due to the presence of additional visual cues.

Research has shown that people are able to judge distances accurately in full-cue, real-world environments using visually directed actions. However, in virtual environments viewed with head-mounted display (HMD) systems, there is evidence that people act as though the virtual space is smaller than intended. This is a surprising result given how well people act in real environments. The behavior in the virtual setting may be linked to distortions in the available visual cues or to a person's ability to locomote without vision. Either could result from issues related to added mass, moments of inertia, and restricted field of view in HMDs. This article describes an experiment in which distance judgments based on normal real-world and HMD viewing are compared with judgments based on real-world viewing while wearing two specialized devices. One is a mock HMD, which replicated the mass, moments of inertia, and field of view of the HMD and the other an inertial headband designed to replicate the mass and moments of inertia of the HMD, but constructed to not restrict the field of view of the observer or otherwise feel like wearing a helmet. Distance judgments using the mock HMD showed a statistically significant underestimation relative to the no restriction condition but not of a magnitude sufficient to account for all the distance compression seen in the HMD. Indicated distances with the inertial headband were not significantly smaller than those made with no restrictions.

We tested whether humans can learn to sense and compensate for interaction forces in contact tasks. Many tasks, such as use of hand tools, involve significant interaction forces between hand and environment. One control strategy would be to use high hand impedance to reduce sensitivity to these forces. But an alternative would be to learn feedback compensation for the extrinsic dynamics and associated interaction forces, with the potential for lower control effort. We observed subjects as they learned control of a ball-and-beam system, a visuo-motor task where the goal was to quickly position a ball rolling atop a rotating beam, through manual rotation of the beam alone. We devised a ball-and-beam apparatus that could be operated in a real mode, where a physical ball was present; or in a virtual training mode, where the ball's dynamics were simulated in real time. The apparatus presented the same visual feedback in all cases, and optionally produced haptic feedback of the interaction forces associated with the ball's motion. Two healthy adult subject groups, vision-only and vision-haptics (each n= 10), both trained for 80 trials on the simulated system, and then were evaluated on the real system to test for skill transfer effects. If humans incorporate interaction forces in their learning, the vision-haptics group would be expected to exhibit a smoother transfer, as quantified by changes in completion time of a ball-positioning task. During training, both groups adapted well to the task, with reductions of 64%-70% in completion time. At skill transfer to the real system, the vision-only group had a significant 35% increase in completion time (p < 0.05). There was no significant change in the vision-haptics group, indicating that subjects had learned to compensate for interaction forces. These forces could potentially be incorporated in virtual environments to assist with motor training or rehabilitation.

It is known that systematic visual distortions using virtual reality technologies, prisms or mirrors, may have therapeutic effects for patients suffering from stroke or body image identity disorders. However, there are few studies which directly investigate neural activity changes during visual feedback manipulation. In the present study we created an experimental setup for investigating the effects of systematic virtual reality-mediated visual feedback manipulation of finger movements on cortical activity. We performed tests with two healthy female subjects who performed a line-tracking task under four conditions manipulating visual feedback of their own hand. To investigate hemodynamic responses in motor areas during the line tracking task we used functional near-infrared spectroscopy (fNIRS). We predicted that viewing larger or smaller virtual movements of fingers, compared to the real movements, would affect activity in motor areas and thus the hemodynamic response. Our preliminary results showed changes in the hemodynamic responses between stimulation period and baseline. There were indications of possible differences between conditions, and also of adaptation effects within conditions. However these effects were not significant in our preliminary data and we therefore need to collect additional data to draw further conclusions.

Ask the Expert: Anxiety Disorders Virtual Reality Therapy

Ultrasound mid-air haptic (UMH) devices are promising for tactile feedback in virtual reality (VR), as they do not require users to be tethered to, hold, or wear any device. This approach is less cumbersome, easy to set up, can simplify tracking, and leaves the hands free for concurrent interactions. This chapter explores work conducted at CNRS-IRISA dealing with the challenges arising from the integration of UMH interfaces in immersive VR through three main axes. These are discussed in the wider context of the state of the art on UMH for augmented and virtual reality, and illustrated through several VR use-cases. A first axis deals with device integration into the VR ecosystem. Interaction in immersive VR is based on the synergy between complex input devices allowing real-time tracking of the user and multimodal feedback devices delivering a coherent visual, auditory and haptic picture of a simulated virtual environment (VE). Using UMH in immersive VR therefore hinges on integrating UMH devices such that their operation does not interfere with other input and feedback devices. It is also critical to ensure that UMH feedback is adequately synchronized and co-located with respect to other stimuli, and delivered within a workspace that is compatible with that of VR interaction. Regarding this final point, we propose PUMAH, a robotic solution for increasing the usable workspace of UMH devices. The second and third axes, respectively, focus on stimulus perception and rendering of VE properties. Virtual object properties can be rendered in a variety of ways, through, e.g., amplitude modulation (AM) or spatiotemporal modulation (STM), with many parameters (modulation frequency, spatial sampling, etc.) coming into play, raising questions about the limits of the design space. To tackle this challenge, we begin by conducting psychophysical experimentation to understand the usable ranges for stimulus parameters and understand the perceptual implications of stimulus design choices. We propose an open-source software framework intended to facilitate UMH stimulus design and perceptual evaluation. These results in turn serve as the basis for the design and evaluation of rendering schemes for VR. Using amplitude variations along a focal point path in STM, we investigate the possibility of rendering geometric details and in a second step, sensations of stiffness in VR.

Deep into visual saliency for immersive VR environments rendered in real-time