Face/On

Presentation of more haptic information to improve the users interactive capability is an important and challenging task in the virtual environment. As an emerging haptic feedback technology, electrovibration is an underlying solution to enhance the systems interactivity and improve user experience. However, such a solution is rarely studied in a virtual environment. In this work, we explore a new VR interaction method based on electrovibration technology with a touch screen and conduct evaluations about it. The key idea is to incorporate a set of manipulation gestures and three types of electrovibration in the VR interaction to help users acquire different kinds of tactile perception in the virtual manipulation. We present the evaluation in which we compare user performance measured first in a Fitts law task to evaluate different electrovibration types and then in a virtual office application to assess the interactive user interface. Our results show that the precision of interactions is significantly improved with the electrovibration haptic feedback. To the best of our knowledge, this is the first work to introduce electrovibration haptic feedback into the VR human-computer interaction and our work enlightens the potential of the electrovibration touchscreen-based interaction in virtual environments.

The ability to recognize and classify human gestures bears a huge potential of improving haptic feedback in virtual environments: when it is possible to predict how a user wants to explore or manipulate his environment both the visual and the auditive feedback may be adapted in order to enhance the immersiveness of haptic displays. Within this work a software approach is suggested that allows for such a real-time classification of gestures in a continuous data stream. A visually based tracking system is used to record the hand movements, while Hidden Markov models are applied to analyze the data. After presenting the methodological background and the software implementation, the outcome of an evaluation study is discussed. It reveals satisfying results of gesture classification and, what is particularly important, the recognition is fast enough to enable multi-modal haptic feedback.KeywordsGesture recognitionHidden Markov modelMulti-modal haptic feedbackVirtual reality

We present a system for sensing and reconstructing facial expressions of the virtual reality (VR) head-mounted display (HMD) user. The HMD occludes a large portion of the user's face, which makes most existing facial performance capturing techniques intractable. To tackle this problem, a novel hardware solution with electromyography (EMG) sensors being attached to the headset frame is applied to track facial muscle movements. For realistic facial expression recovery, we first reconstruct the user's 3D face from a single image and generate the personalized blendshapes associated with seven facial action units (AUs) on the most emotionally salient facial parts (ESFPs). We then utilize pre-processed EMG signals for measuring activations of AU-coded facial expressions to drive pre-built personalized blendshapes. Since facial expressions appear as important nonverbal cues of the subject's internal emotional states, we further investigate the relationship between six basic emotions - anger, disgust, fear, happiness, sadness and surprise, and detected AUs using a fern classifier. Experiments show the proposed system can accurately sense and reconstruct high-fidelity common facial expressions while providing useful information regarding the emotional state of the HMD user.

Background:Virtual reality (VR)-based simulation in hospital settings facilitates the acquisition of skills without compromising patient safety. Despite regular text-based training, a baseline survey of randomly selected healthcare providers revealed deficiencies in their knowledge, confidence, comfort, and care skills regarding tracheostomy. This prospective pre–post study compared the effectiveness of regular text- and VR-based intervention modules in training healthcare providers’ self-efficacy in tracheostomy care skills.Methods:Between January 2018 and January 2020, 60 healthcare providers, including physicians, nurses, and respiratory therapists, were enrolled. For the intervention, a newly developed head-mounted display (HMD) and web VR materials were implemented in training and clinical services. Subsequently, in-hospital healthcare providers were trained using either text or head-mounted display virtual reality (HMD-VR) materials in the regular and intervention modules, respectively. For tracheostomy care skills, preceptors directly audited the performance of trainees and provided feedback.Results:At baseline, the degree of trainees’ agreement with the self-efficacy-related statements, including the aspects of familiarity, confidence, and anxiety about tracheostomy-related knowledge and care skills, were not different between the control and intervention groups. At follow-up stage, compared with the regular group, a higher percentage of intervention group’ trainees reported that they are “strongly agree” or “somewhat agree” that the HMD-VR simulation increases their self-efficacy, including the aspects of familiarity and confidence, and reduced their anxiety about tracheostomy-related knowledge and care skills. After implementation, a higher degree of trainees’ average satisfaction with VR-based training and VR materials was observed in the intervention group than in the regular group. Most reported that VR materials enabled accurate messaging and decreased anxiety. The increasing trend of the average written test and hands-on tracheostomy care skills scores among the intervention group trainees was significant compared to those in the regular group. The benefits of HMD-VR simulations and web-VR material-based clinical services for in-hospital healthcare providers and patient families persisted until 3 to 4 weeks later.Conclusion:The current study suggests that VR materials significantly enhance trainees’ self-efficacy (increased familiarity, increased confidence, and reduced anxiety) and their satisfaction with the training, while motivating them to use acquired knowledge and skills in clinical practice.

Video assisted thoracoscopic surgery (VATS) has been adopted in pediatric age for the treatment of congenital lung malformations (CLM). The success of VATS in pediatrics largely depends on the surgeon’s skill ability to understand the airways, vascular system and lung parenchyma anatomy in CLM. In the last years, virtual reality (VR) and 3-dimensional (3D) printing of organ models and VR head mounted display (HMD) technologies have been introduced for completion of preoperative planning in adult patients. To date no reports about the use of VR HMD technologies in a pediatric setting are available. The aim of this report is to introduce a VR HMD model in VATS procedure to improve the quality of care in children with CLM. VR HMD set-up for planning thoracoscopic surgery was performed in a series of pediatric patients with diagnosis of CLM. The preoperative VR HMD evaluation allowed a navigation into the malformation with the aim to explore, interact, and make the surgeon more confident and skilled to answer to the traps. A development of surgical simulations models and teaching program dedicated to education and training in pediatric VATS is suitable among the pediatric surgery community. Further studies should demonstrate all the benefits of such technology in pediatric patients submitted to VATS procedure.

Haptic feedback has been frequently adopted as a more intuitive feedback method. Although vibration feedback is the most popular method for skill learning, thermal feedback has rarely been adopted. However, thermal feedback might be more appropriate than vibration feedback, depending on the target skill to learn. Therefore, this study aimed to find whether the feedback types (vibration and thermal feedback) and target skills (static and dynamic movements) affect individual differences in corrective action and performance. The results demonstrated that thermal feedback was more suitable than vibration feedback for reducing individual differences in the center of gravity variation when the user was required to maintain a specific posture (static movement). To select a suitable feedback type, we can provide feedback that reduces individual differences in users’ corrective actions.

Hand-based interaction, such as using a handheld controller or making hand gestures, has been widely adopted as the primary method for interacting with both virtual reality (VR) and augmented reality (AR) head-mounted displays (HMDs). In contrast, hands-free interaction avoids the need for users’ hands and although it can afford additional benefits, there has been limited research in exploring and evaluating hands-free techniques for these HMDs. As VR HMDs become ubiquitous, people will need to do text editing, which requires selecting text segments. Similar to hands-free interaction, text selection is underexplored. This research focuses on both, text selection via hands-free interaction. Our exploration involves a user study with 24 participants to investigate the performance, user experience, and workload of three hands-free selection mechanisms (Dwell, Blink, Voice) to complement head-based pointing. Results indicate that Blink outperforms Dwell and Voice in completion time. Users’ subjective feedback also shows that Blink is the preferred technique for text selection. This work is the first to explore handsfree interaction for text selection in VR HMDs. Our results provide a solid platform for further research in this important area.

Immersive virtual reality (VR) emerges as a promising research and clinical tool. However, several studies suggest that VR induced adverse symptoms and effects (VRISE) may undermine the health and safety standards, and the reliability of the scientific results. In the current literature review, the technical reasons for the adverse symptomatology are investigated to provide suggestions and technological knowledge for the implementation of VR head-mounted display (HMD) systems in cognitive neuroscience. The technological systematic literature indicated features pertinent to display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware that should be considered by the researchers. Subsequently, a meta-analysis of 44 neuroscientific or neuropsychological studies involving VR HMD systems was performed. The meta-analysis of the VR studies demonstrated that new generation HMDs induced significantly less VRISE and marginally fewer dropouts. Importantly, the commercial versions of the new generation HMDs with ergonomic interactions had zero incidents of adverse symptomatology and dropouts. HMDs equivalent to or greater than the commercial versions of contemporary HMDs accompanied with ergonomic interactions are suitable for implementation in cognitive neuroscience. In conclusion, researchers' technological competency, along with meticulous methods and reports pertinent to software, hardware, and VRISE, are paramount to ensure the health and safety standards and the reliability of neuroscientific results.

The usability of virtual reality (VR) controller interfaces are often complex and difficult for first time users. Most controllers provide minimal feedback which relegates the potential for heightened interaction and feedback within virtual experiences. This research explores how haptic technology systems partnered with VR can deliver immersive interactions between user and virtual environment (VE). This research involves the development of a haptic glove interface prototype that incorporates a force feedback and vibrotactile feedback system. It focuses on determining a workflow that communicates in real-time user interaction and environmental feedback using Unreal Engine and the produced haptic glove system. Testing and calibrating the prototype feedback system provided a baseline for developers to rationalise and improve accuracy of current real-time virtual feedback systems. The evaluation of this research in industry unfolds new technical knowledge for implementing a wider range of haptic technologies within VR. This further development would involve reviewing the usability and interaction standards for VR users in the design process.

Contrast ratio is the ratio of the brightest part to the darkest part of the image, which can reflect the picture hierarchy. In VR head‐mounted display (HMD), many scenes need to realistically simulate rooms or night environments. High contrast ratio helps to better display details in dark scenes, giving people the ultimate sense of immersion. There are many ways to measure contrast ratio, but still no effective standard to evaluate this indicator. In particular, many VR HMD adopt the emerging solution of folded optical path pancake. Due to the refraction and reflection of light on different interfaces, the pancake module will cause "ghosting", then affect the visual experience. Therefore, it becomes our focus to evaluate the contrast ratio of pancake VR HMD. We present a way to evaluate the contrast ratio of VR HMD by a near‐eye display measurement system, and compares the measurement results of different test methods. It can systematically and objectively evaluate the contrast of VR head‐mounted display devices and provide a useful reference for VR device evaluation.

Clinical robotic surgery systems do not currently provide haptic feedback because surgical instrument interactions are difficult to measure and display. Our laboratory recently developed a technology that allows surgeons to feel and/or hear the high-frequency vibrations of robotic instruments as they interact with patient tissue and other tools. Until now, this type of feedback had not been carefully evaluated by users. We conducted two human-subject studies to discover whether surgeons and non-surgeons value the addition of vibration feedback from surgical instruments during robotic surgery. In the first experiment, 10 surgeons and 10 non-surgeons (n = 20) used an augmented Intuitive da Vinci Standard robot to repeatedly perform up to four dry-lab tasks both with and without haptic and audio feedback. In the second experiment, 68 surgeons and 26 non-surgeons (n = 94) tested the same robot at a surgical conference: each participant spent approximately 5 min performing one or two tasks. Almost all subjects in both experiments (95 and 98 %, respectively) preferred receiving feedback of tool vibrations, and all subjects in the second experiment thought it would be useful for surgeons to have the option of such feedback. About half of the subjects (50, 60 %) preferred haptic and audio feedback together, and almost all the rest (45, 35 %) preferred haptic feedback alone. Subjects stated that the feedback made them more aware of tool contacts and did not interfere with use of the robot. There were no significant differences between the responses of different subject populations for any questions in either experiment. This study illustrates that both surgeons and non-surgeons prefer instrument vibration feedback during robotic surgery. Some participants found audio feedback useful but most preferred haptic feedback overall. This strong preference for tool vibration feedback indicates that this technology provides valuable tactile information to the surgeon.

Slip feedback is an important cue in everyday object manipulation, but it is generally missing in tele-operation systems. To test the usefulness of simple, abstract types of feedback that warn the user about slip events, we tested the effect of auditory and haptic vibration feedback in a tele-operation task. Participants were asked to hold an object in a remote robot hand, and the force profiles that they exerted in response to slip events were measured. Haptic feedback did not significantly change the response characteristics, but auditory feedback did significantly improve response latency. A small but significant difference between haptic and auditory reaction times (60 ms) found in our control experiment might explain the difference between the feedback types.

Virtual Reality (VR) has expanded from the training and high-end entertainment world into the homes of everyday users through mobile head mounted displays (HMDs). Currently, several different interaction methods for mobile VR HMDs exist, including remotes, external controllers, motion controllers, touch, and free form hand tracking. The goal of this study was to evaluate the impact that these interaction methods have on specifically mobile VR HMD user experience, including performance, user preference, and enjoyment. It was hypothesized that interaction methods would vary significantly in enjoyment, preference, and performance as many factors such as ease of use and familiarity can influence these factors. To evaluate the interaction methods, participants performed trials of a matching task utilizing menus in a VR environment in which the participants selected the features of an object (e.g., shape, color, pattern) from a menu to the right of the object. The Leap Motion had usability issues that impacted it’s performance and enjoyment scores resulting in it being the poorest performer of the group. Overall the gamepad was most preferred and performed the best due to its ease of use and familiarity, but the touchpad and leap motion offered immersive aspects. These results highlight the importance of ease of use in interaction methods and the impact it has on overall user experience. This paper will discuss these results and implications for design of interaction methods and associated VR HMD controllers.

In this paper we present an immersive brain computer interface (BCI) where we use a virtual reality head-mounted display (VRHMD) to invoke SSVEP responses. Compared to visual stimuli in monitor display, we demonstrate that visual stimuli in VRHMD indeed improve the user engagement for BCI. To this end, we validate our method with experiments on a VR maze game, the goal of which is to guide a ball into the destination in a 2D grid map in a 3D space, successively choosing one of four neighboring cells using SSVEP evoked by visual stimuli on neighboring cells. Experiments indicate that the averaged information transfer rate is improved by 10% for VRHMD, compared to the case in monitor display and the users feel easier to play the game with the proposed system.

Sound is an important, but often neglected, component for creating a self-motion illusion (vection) in Virtual Reality applications, for example, motion simulators. Apart from auditory motion cues, sound can provide contextual information representing self-motion in a virtual environment. In two experiments we investigated the benefits of hearing an engine sound when presenting auditory (Experiment 1) or auditory-vibrotactile (Experiment 2) virtual environments inducing linear vection. The addition of the engine sound to the auditory scene significantly enhanced subjective ratings of vection intensity in Experiment 1 and vection onset times but not subjective ratings in Experiment 2. Further analysis using individual imagery vividness scores showed that this disparity between vection measures was created by participants with higher kinesthetic imagery. On the other hand, for participants with lower kinesthetic imagery scores, the engine sound enhanced vection sensation in both experiments. A high correlation with participants' kinesthetic imagery vividness scores suggests the influence of a first person perspective in the perception of the engine sound. We hypothesize that self-motion sounds (e.g., the sound of footsteps, engine sound) represent a specific type of acoustic body-centered feedback in virtual environments. Therefore, the results may contribute to a better understanding of the role of self-representation sounds (sonic self-avatars), in virtual and augmented environments.