Haptic and Visual Feedback for Manipulation Aid in a Virtual Environment

It is expected that See-Through Head-Mounted Display (STHMD), which superimposes the virtual environment generated by computer graphics (CG) on the real world, can vividly execute various simulations and designs by using both of the real and virtual environment around us. In STHMDs, information given as a virtual environment has to exactly match with the real environment, because both environments are visible. This is one of the problems to be solved for practical use. Particularly for matching of locations and size between real and virtual objects, disaccordance is likely to occur between the world coordinate of the real environment where the user of STHMD actually exists and that of the virtual environment described as parameters of CG, which directly causes displacement of locations where virtual objects are superimposed. This must be calibrated so that the virtual environment is superimposed properly. Among causes of such errors, we focused both on systematic errors of visual parameters caused in manufacturing process and differences between actual and supposed location of user's eye on STHMD when in use. The former is required to be calibrated only once after the fabrication of STHMDs, whereas the latter has to be calibrated every time users start using STHMDs. We have proposed calibration methods which are suitable to properties of these causes of errors. In the method, the direct fitting of the virtual cursor drawn in the virtual environment onto targets in the real environment is performed. Then, based on the result of fitting, the least square method identifies values of the visual parameters which minimize differences between locations of virtual cursor in the virutal environment and targets in the real environment. Application of the method to the STHMD which we have made is also repored. The differences between the virtal cursor and targets in the real environment due to systematic errors caused in the manufacturing process were reduced to about 1 [mm] per target, which was less than one-thirtieth of that before the calibration. The differences between the virtual cursor and targets in the real environment due to the location of user's eyes were also reduced about 2 [mm] per target, which was a half of that before the calibration. This result was well enough to prove the effectiveness of the calibration methods.

This research evaluated visual and haptic feedback for enhancing operator situational awareness during multitasking drone operations, investigating how task complexity influenced feedback effectiveness. Two virtual reality experiments were conducted: Study 1 assessed three types of visual feedback (Static, Blink, and Shake), while Study 2 compared visual, haptic, and visual-haptic multimodal feedback across varying task complexity levels determined by target density. Participants performed drone operation tasks requiring simultaneous obstacle avoidance and target counting. Results revealed that dynamic visual feedback (Shake) significantly improved primary task accuracy, reduced cognitive load, and was the preferred visual feedback type. Furthermore, haptic and visual-haptic feedback significantly shortened obstacle avoidance reaction times and decreased cognitive workload compared to visual feedback. Task complexity proved to be a critical factor: while visual feedback sufficed for low complexity scenarios, visual-haptic feedback offered distinct advantages in medium and high complexity tasks, providing insights for complexity-responsive drone interface design.

We routinely have physical interactions with others, whether it be handing someone a glass of water or jointly moving a heavy object together. These sensorimotor interactions between humans typically rely on visual feedback and haptic feedback. Recent single-participant studies have highlighted that the unique noise and time delays of each sense must be considered to estimate the state, such as the position and velocity, of one's own movement. However, we know little about how visual feedback and haptic feedback are used to estimate the state of another person. Here, we tested how humans utilize visual feedback and haptic feedback to estimate the state of their partner during a collaborative sensorimotor task. Across two experiments, we show that visual feedback dominated haptic feedback during collaboration. Specifically, we found that visual feedback led to comparatively lower task-relevant movement variability, smoother collaborative movements, and faster trial completion times. We also developed an optimal feedback controller that considered the noise and time delays of both visual feedback and haptic feedback to estimate the state of a partner. This model was able to capture both lower task-relevant movement variability and smoother collaborative movements. Taken together, our empirical and modeling results support the idea that visual accuracy is more important than haptic speed to perform state estimation of a partner during collaboration.NEW & NOTEWORTHY Physical collaboration between two or more individuals involves both visual and haptic feedback. Here, we investigated how visual and haptic feedback is used to estimate the movements of a partner during a collaboration task. Our experimental and computational modeling results parsimoniously support the notion that greater visual accuracy is more important than faster yet noisier haptic feedback when estimating the state of a partner.

Most prosthetic hand users are limited to visual feedback of movement performance. To characterize the benefit of vibrotactile feedback for a task that lacks haptic feedback, a virtual environment was used to experimentally manipulate visual, task-relevant haptic, and remote vibrotactile feedback on simple object manipulation for unimpaired subjects. The combination of visual and remote vibrotactile feedback was compared to visual feedback alone, and to simultaneous visual and direct haptic feedback to represent ideal performance. Visual and vibrotactile feedback resulted in improvement of most performance variables including difficulty ratings relative to visual feedback alone. However addition of sensory cues to visual feedback increased trial times and the increase was steeper for vibrotactile than for haptic feedback. Specifically, during vibrotactile feedback the velocity did not change, but the duration of execution increased due to improved performance, resulting in increased trial times. This result suggests future exploration of performance improvement and execution speed for augmented sensory feedback.

Even though it is recognized that vision plays an important role in grasping movements, it is not yet fully understood how the visual feedback of the hand contributes to the on-line control. Visual feedback could be used to shape the posture of the hand and fingers, to adjust the trajectory of the moving hand, or a combination of both. Here, we used a dynamic perturbation method that altered the position of the visual feedback relative to the actual position of the thumb and index finger to virtually increase or decrease the visually sensed grip aperture. Subjects grasped objects in a virtual 3D environment with haptic feedback and with visual feedback provided by small virtual spheres anchored to the their unseen fingertips. We found that the effects of the visually perturbed grip aperture arose preeminently late in the movement when the hand was in the object's proximity. The on-line visual feedback assisted both the scaling of the grip aperture to properly conform it to the object's dimension and the transport of the hand to correctly position the digits on the object's surface. However, the extent of these compensatory adjustments was contingent on the viewing geometry. The visual control of the actual grip aperture was mainly observed when the final grasp axis orientation was approximately perpendicular to the viewing direction. On the contrary, when the final grasp axis was aligned with the viewing direction, the visual control was predominantly concerned with the guidance of the digit toward the visible final contact point.

Though virtual reality (VR) has been advanced to certain levels of maturity in recent years, the general public, especially the population of the blind and visually impaired (BVI), still cannot enjoy the benefit provided by VR. Current VR accessibility applications have been developed either on expensive head-mounted displays or with extra accessories and mechanisms, which are either not accessible or inconvenient for BVI individuals. In this paper, we present a mobile VR app that enables BVI users to access a virtual environment on an iPhone in order to build their skills of perception and recognition of the virtual environment and the virtual objects in the environment. The app uses the iPhone on a selfie stick to simulate a long cane in VR, and applies Augmented Reality (AR) techniques to track the iPhone’s real-time poses in an empty space of the real world, which is then synchronized to the long cane in the VR environment. Due to the use of mixed reality (the integration of VR & AR), we call it the Mixed Reality cane (MR Cane), which provides BVI users auditory and vibrotactile feedback whenever the virtual cane comes in contact with objects in VR. Thus, the MR Cane allows BVI individuals to interact with the virtual objects and identify approximate sizes and locations of the objects in the virtual environment. We performed preliminary user studies with blind-folded participants to investigate the effectiveness of the proposed mobile approach and the results indicate that the proposed MR Cane could be effective to help BVI individuals in understanding the interaction with virtual objects and exploring 3D virtual environments. The MR Cane concept can be extended to new applications of navigation, training and entertainment for BVI individuals without more significant efforts.

This paper presents the creation of an assembly simulation environment with multisensory feedback (auditory and visual), and the evaluation of the effects of auditory and visual feedback on the task performance in the context of assembly simulation in a virtual environment (VE). This VE experimental system platform brings together complex technologies such as constraint-based assembly simulation, optical motion tracking technology, and real time 3D sound generation technology around a virtual reality workbench and a common software platform. A peg-in-a-hole and a Sener electronic box assembly task have been used as the task cases to conduct the human factor experiment, using sixteen participants. Both objective performance data (i.e., task completion time, TCT; and human performance error rate, HPER) and subjective opinions (i.e., questionnaires) on the utilization of auditory and visual feedback in a virtual assembly environment (VAE) have been gathered from the experiment. Results showed that the introduction of auditory and/or visual feedback into VAE did improve the assembly task performance. They also indicated that integrated feedback (auditory plus visual) offered better assembly task performance than either feedback used in isolation. Most participants preferred integrated feedback to either individual feedback (auditory or visual) or no feedback. The participants' comments demonstrated that nonrealistic or inappropriate feedback had a negative effect on the task performance, and easily made them frustrated.

To understand the nature of hemispatial attention allocation in virtual reality (VR), a line bisection task (LBT) was administered both in a real environment and a virtual environment to assess the rate of pseudoneglect. The mental construction of real and virtual environments was assumed to increase visuospatial activity in right hemisphere-related cognitive processes; an alteration in the activity that manifests in the direction and rate of line bisection lateral error. In the present study, fifty-one right-handed healthy college students were recruited. They performed a line bisection task in real and virtual environments. The obtained data showed that LBT errors in real and VR environments were correlated and individually consistent. Furthermore, a leftward LBT error was found in the physically real environment, however, in a VR the line bisection bias drifted towards the right hemispace. Participants with a lower right-handedness score showed a lower rate of left LBT bias in a real environment, but in VR, their LBT error showed a stronger rightwards error. Participants showed an individually consistent pattern in both real and VR environments, but VR-induced visuospatial reality construction was associated with rightward LBT bias in a virtual environment.

For educational purposes, virtual reality (VR) is often used to produce life-like experiences. However, the use of haptic feedback in educational practices for science and mathematics remained under-studied. Haptically-enabled science simulations (HESSs) enable students to physically experience the concepts being taught via haptic feedback. We present a study on the effect of a buoyancy HESS designed to aid in the understanding of basic physics concepts. We hypothesize that introducing both visual and haptic feedback of the underlying simulated forces will improve participants understanding. We investigate this hypothesis with a 2 (haptics: yes, no) × 2 (visuals: yes, no) between subjects design user study, where all participants were randomly assigned to one of the four conditions. Participants were given a pretest of buoyancy knowledge, then instructed to interact with the buoyancy simulation, then given a post-test of buoyancy knowledge. The present study is still in the process of data collection, with 40 out of 60 participants. Preliminary results highlight a significant improvement in performance of participants in the haptic-and-visual condition, while no significant differences were observed in other conditions.

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.

In an initial study, we characterized the properties of human reach motion in the presence or absence of visuo-haptic feedback in real and Immersive Virtual Environments (IVEs) or virtual reality within a participant’s maximum arm reach. Our goal is to understand how physical reaching actions to the perceived location of targets in the presence or absence of visuo-haptic feedback are different between real and virtual viewing conditions. Typically, participants reach to the perceived location of objects in the three-dimensional (3D) environment to perform selection and manipulation actions during 3D interaction in applications such as virtual assembly or rehabilitation. In these tasks, participants typically have distorted perceptual information in the IVE as compared to the real world, in part due to technological limitations such as minimal visual field of view, resolution, latency, and jitter. In an empirical evaluation, we asked the following questions: (i) how do the perceptual differences between virtual and real world affect our ability to accurately reach to the locations of 3D objects, and (ii) how do the motor responses of participants differ between the presence or absence of visual and haptic feedback? We examined factors such as velocity and distance of physical reaching behavior between the real world and IVE, both in the presence or absence of visuo-haptic information. The results suggest that physical reach responses vary systematically between real and virtual environments, especially in situations involving the presence or absence of visuo-haptic feedback. The implications of our study provide a methodological framework for the analysis of reaching motions for selection and manipulation with novel 3D interaction metaphors and to successfully characterize visuo-haptic versus non-visuo-haptic physical reaches in virtual and real-world situations.

Virtual reality (VR) enables the study of visually-guided behavior in situations that are otherwise not amenable to laboratory research. An example is a fire evacuation scenario, in which building occupants have to find their way to safety through smoke and flames, identify safe egress routes, and avoid threats. First, however, it must be determined whether evacuation behavior in VR is comparable to that in a real environment. We are thus initially studying participant responses to a fire alarm in matched virtual and real environments. The present experiment investigates how the decision to evacuate is influenced by visual cues about a neighbor’s behavior. In the Control group, a lone participant is exposed to a fire alarm while performing a bogus task. In the Passive group, a participant performs the same task with a confederate who ignores the fire alarm. In the Active group, the confederate immediately leaves the room when the fire alarm sounds. The gender of the participant is crossed with the gender of the confederate. 75 participants experience the scenario in a real environment (25 per group), and 75 others in a matched virtual environment presented in a wireless head mounted display (HMD), with avatars of the confederates. In the real environment, two-thirds of participants in the Control group evacuated the room. In the Active group, nearly all did so, whereas in the Passive group all but one did. More female than male participants evacuated in the Control group, but there were no other gender effects. Data collection in the virtual environment is in progress, and will be completed by spring. The findings thus far demonstrate a very strong influence of neighbor behavior on a participant’s evacuation decision. The results will provide answers to the much-debated question of whether evacuation behavior is similar in virtual and real environments. Meeting abstract presented at VSS 2015

Virtual environments are often used in pre-wearing training and assessment of prosthetic control abilities. Here, we developed a virtual prosthetic hand training platform for evaluation of closed-loop control of grasp force. Biorealistic controllers emulated a pair of antagonistic muscles that actuated the thumb and index fingers of the hand. Surface electromyographic (sEMG) signals from a pair of antagonistic residual muscles drove the biorealistic controllers. Tactile forces from fingertip sensors were conveyed to amputees through evoked tactile sensations (ETS) elicited at the projected finger map (PFM) areas of the stump. A forearm amputee subject participated in force tracking or holding tasks using the virtual hand with residual muscle EMGs, or the contralateral intact hand. Root-mean-square error (RMSE) was used as outcome measure of motor performance. Results in this subject showed that the biorealistic controller enabled the virtual hand to track and maintain grasping forces. The best performance in both tasks was achieved by the contralateral intact hand with visual feedback. The roles of visual or tactile feedback in force tracking or maintaining were also assessed with the virtual hand. For force holding task, hybrid tactile and visual feedback with biorealistic control had a better performance than single visual or tactile feedback in terms of RMSE, success rate, and force variability. While in the force pursuing task, tactile feedback did not seem to add visual feedback in following the target force. The study suggests that training may be required for a novel virtual hand user to perceive and integrate multiple modalities of feedback information, so as to optimize the closed-loop control ability.

In most applications of virtual environments (VEs), like training and design evaluation, a good sense of orientation is needed in the VE. Orientation performance when moving around in the real world relies on visual as well as proprioceptive feedback. However, the navigation metaphors which are used to move around the VE often lack proprioceptive feedback. Furthermore, the visual feedback in a VE is often relatively poor compared to the visual feedback available in the real world. Therefore, we have quantified the influence of visual and proprioceptive feedback on orientation performance in VEs. Subjects were immersed in a virtual forest and were asked to turn specific angles using three navigation metaphors, differing in the kind of proprioceptive feedback which is provided (no proprioceptive feedback, vestibular feedback, and vestibular and kinesthetic feedback). The results indicate that the most accurate turn performance is found when kinesthetic feedback is present, in a condition where subjects use their legs to turn around. This indicates that incorporating this kind of feedback in navigation metaphors is quite beneficial. Orientation on only the visual component is most inaccurate, leading to progressively larger undershoots for larger angles.

In virtual environments, perceived distances are frequently reported to be shorter than intended. One important parameter for spatial perception in a stereoscopic virtual environment is the stereo base—that is, the distance between the two viewing cameras. We systematically varied the stereo base relative to the interpupillary distance (IPD) and examined influences on distance and size perception. Furthermore, we tested whether an individual adjustment of the stereo base through an alignment task would reduce the errors in distance estimation. Participants performed reaching movements toward a virtual tennis ball either with closed eyes (blind reaches) or open eyes (sighted reaches). Using the participants' individual IPD, the stereo base was set to (a) the IPD, (b) proportionally smaller, (c) proportionally larger, or (d) adjusted according to the individual performance in an alignment task that was conducted beforehand. Overall, consistent with previous research, distances were underestimated. As expected, with a smaller stereo base, the virtual object was perceived as being farther away and bigger, in contrast to a larger stereo base, where the virtual object was perceived to be nearer and smaller. However, the manipulation of the stereo base influenced blind reaching estimates to a smaller extent than expected, which might be due to a combination of binocular disparity and pictorial depth cues. In sighted reaching, when visual feedback was available, presumably the use of disparity matching led to a larger effect of the stereo base. The use of an individually adjusted stereo base diminished the average underestimation but did not reduce interindividual variance. Interindividual differences were task specific and could not be explained through differences in stereo acuity or fixation disparity.