Force-Feedback Haptic Device for Representation of Tugs in Virtual Reality

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Since [years] we hear about Virtual Reality as a discipline that could potentially provide benefits to many applications. Basically, the principle of Virtual Reality consists in stimulating user senses in order to give the impression to be in another place which they can discover and interact with. Today, most virtual reality systems create realistic visual and auditive environments. However the interaction with these environments can not be considered as natural. Indeed, we often use devices such as mouse, keyboard or joystick to move or manipulate them. These paradigms of interaction are in fact metaphors, which are not similar to reality. In some situations they are practical and efficient. However, the lack of intuitiveness sometimes makes them limited or simply ineffective. To overcome this, researchers can use Haptic Devices. They are designed to simulate what is commonly called the of touch, which includes more specifically, tactile, pain sense, thermal sense, and proprioception. Proprioception is knowledge gained by the perception of the relative member's position of the human body. In this thesis, we particularly focus on the simulation of proprioception. There are two advantages of such haptic devices. First, they can give the user more information on the nature of virtual objects (size, weight, finish, rigidity, etc.). Second, they can provide interaction paradigms that are closer to reality (three-dimensional interaction in a three-dimensional world). However, haptic device mechanics is complex. Moreover, proprioception is a sense that covers the entire body which is a rather large surface. For this reason, haptic devices usually apply force feedback on a very small portion of the body, such as fingertip. In addition to this hardware constraint, haptic research also faces software constraints. Indeed, a haptic application requires many computer resources in order to perform collision detection, dynamic animation of objects, and force feedback computation. Moreover, this should be done at a refresh rate that is much higher than the visualization for producing a convincing result. In the first part of this thesis, we propose to increase realism and complexity of haptic applications. To achieve this goal, we use a state-of-the-art commercial device which allows to acquire the posture and position of both hands, and to apply forces on the fingertips and wrists. We propose techniques to calibrate and improve the comfort of these kinds of devices in order to integrate them into Virtual Environments. However, a two-handed haptic device do not presents only advantages. Indeed, It is much more complicated to compute forces on two hand models, than on a single point or fingertip. For this reason, in this thesis, we propose a framework to optimize this computation. Thanks to it, we can create Virtual Environments in which an object is graspable and dynamically animated by the laws of physics. When the object is seized by both hands, the haptic rendering engine realistically computes the forces on both exoskeletons. The efficiency of our rendering permits to apply these techniques to complex environments that have a significant number of objects. But the existing visual Virtual Environments are much more detailed than the ones seen in common haptic applications. In this thesis, we aim at reducing this gap. One of the problems is that these quality environments usually do not include specific haptic object properties, such as mass or material. We thus propose a software allowing even non-professional to quickly and easily add this information to an environment. Our results show that this haptic rendering engine does not suffer from the large quantity of objects. They demonstrate that we have an efficient framework for integrating a two-handed haptic interface into a generic virtual environment. In the second part, we evaluate the potential of these kinds of Virtual Reality systems in more detail. While most applications can in theory take advantage of haptic devices, the practice shows that it is not always the case. Indeed, with experience, some metaphorical interaction paradigms remain more powerful than realistic ones. We thus present and study the integration of our two-handed haptic interface in a variety of applications. Evaluations show that depending on the application, it is not appropriate to reproduce reality: in teleoperation, for instance, simulating a virtual haptic steering wheel is less efficient than providing a force gesture interface. On the other hand, in virtual learning, the power of two-handed haptic manipulation is fully exploited and presents great advantages over standard techniques.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

As an important part of virtual reality technology, haptic devices bring much better immersion and interactive to users. Haptic devices are continuously refined so they become more and more realistic with this technology's advancement. Multiple feedbacks, including the initial vibration feedback, advanced force feedback, electrotactile, and thermo-haptic feedback devices, are valuable because they help with complex surgical operations, create immersive gaming experiences, and provide practice-oriented education and training. This paper provides an in-depth analysis of the main applications of haptic devices into virtual reality. It is shown that while haptic feedback and user comfort have been widely applied in the field of highly realistic haptic education, many issues still have not been explored in other areas. Further exploration is needed in customization for specific user needs, the integration of multiple sensory feedback systems, and the reduction of latency to enhance the overall virtual experience. Research into lightweight, wearable haptic devices could also expand accessibility and usability across diverse fields.

Virtual reality (VR) is widely used in the fields of health care. We hypothesis that hand movement with VR simulated scene is similar to actual activity. Nine subjects participated in the study with brain wave monitoring, they used force feedback device to virtual ball catching. We found the brain wave patterns are similar to prior hand movement study, but pattern in the force feedback had longer latency than none force feedback. The results supported virtual reality paradigm can be the part of activity in the real world. We conclude that, movement in the virtual environment and real world had slight difference. Virtual reality can be used as a tool for the motor learning or skill re-acquisition.

This study proposes gaze-based hand interaction, which is helpful for improving the user’s immersion in the production process of virtual reality content for the mobile platform, and analyzes efficiency through an experiment using a questionnaire. First, three-dimensional interactive content is produced for use in the proposed interaction experiment while presenting an experiential environment that gives users a high sense of immersion in the mobile virtual reality environment. This is designed to induce the tension and concentration of users in line with the immersive virtual reality environment. Additionally, a hand interaction method based on gaze—which is mainly used for the entry of mobile virtual reality content—is proposed as a design method for immersive mobile virtual reality environment. The user satisfaction level of the immersive environment provided by the proposed gaze-based hand interaction is analyzed through experiments in comparison with the general method that uses gaze only. Furthermore, detailed analysis is conducted by dividing the effects of the proposed interaction method on user’s psychology into positive factors such as immersion and interest and negative factors such as virtual reality (VR) sickness and dizziness. In this process, a new direction is proposed for improving the immersion of users in the production of mobile platform virtual reality content.

Virtual reality (VR) applications for surgical planning and mentoring are currently becoming more widespread. Within these fields of applications, haptic systems are crucial to help the understanding of the feedback provided by the tissue during the interaction with the surgical tool. The accuracy of the VR scenario is related to many aspects, among them the ability of the haptic device to provide the force feedback and thus the adoption of proper characterization and modeling of the soft tissue behavior during deformation. This paper presents the current development concerning a VR environment to be equipped with a force feedback haptic device to simulate and experience clamping in colorectal surgery. The reaction at the clamp handle is provided starting from a FEA of the contact between the surgical tool and the tissue, coupled with the kinematic analysis of the mechanism. Connecting the virtual scenario to a microcontroller board, the reaction force is applied to the the haptic device, shaped as a clamping tool handle, so that the user may experience the tissue impedence during clamping. By simulations made through Simscape, a concept is evaluated according to some simulation paths that involve common scenarios of the virtual reality experience (such as collision, partial or total clamping operation). The novelty of the research regards the system design oriented to increase the accuracy of the soft tissue material behaviour through a preliminary off-line dedicated FEA investigation. Through Simulink simulations, the system design is verified so that the VR system set-up may be carried out. The results highlighted: (a) a good correspondence among the theoretical study of the reference kinematism that the haptic device must reproduce; (b) the implementation of a bilinear material model, derived off-line from FEA, is suitable to capture hyperelastic and geometric non-linearities of the soft tissue; (c) the feasibility of the open loop haptic control in accordance to the explored kinematism. The final goal of the research is to propose a system design approach able to tune the VR experince for teaching and mentoring through a in-house set-up easy to be tailored.

The rapid development of virtual and augmented reality technologies is currently taking place in almost all spheres of activity. Elements of virtual and augmented reality are used in such areas as education, medicine, transport, gaming, tourism and others. The active spread of these technologies causes the emergence of special competencies in the IT labor market and, as a result, the formation of new professions.Many Russian universities are training students in IT training areas. Specialization in the development of computer games and virtual reality applications has begun recently. The provision of practical classes is accompanied by specific tasks, which gives students the opportunity to improve the use of software and technical devices.The relevance of the research is determined by the current demand for the use of the latest technologies by IT developers in the field of creating computer games. Today, technologies that provide a player’s immersion in virtual reality are becoming more and more popular. One of these technologies is a suit with wearable sensors that track a person’s position in space in real time. However, there are quite a few real described projects in the literature and on the Internet. This study examines the process of developing a task for creating a game application using virtual reality technology: a suit with wearable sensors for teaching students.Materials and methods of research. Timely identification of the needs of the IT market in personnel training allows educational organizations to form new training programs of different levels of training. This approach makes it possible to target the educational and methodological materials being developed to use the latest achievements in the development of the field under study.Using a systematic approach, the study characterizes virtual reality suits and sensors for monitoring the position in the user’s space. Thus, the goal of the task was to ensure the immersiveness and convenience of interaction between the player and the game environment.Based on materials on software, position sensors in space, the approach of pedagogical design was applied and the procedure was formed for a practical task, reflecting the relevant competencies.Results. The study was conducted on the basis in the framework of laboratory and practical work of students, as well as at a real enterprise. Training in the new profile of the direction of training “Applied informatics” is fully equipped with all the latest technologies in this field. As a result of the work, the content of the practical task was developed.Real development of virtual and augmented reality applications is conducted jointly with students. Almost all projects used a suit with body sensors.Conclusion. Our study examines in detail the process of developing an application using a suit with wearable sensors for further training of students. Based on the results, work can be carried out on real projects for any field. Based on the research materials, it is planned to issue a textbook for students with the profile of developing computer games and virtual / augmented reality applications.

Now, the design of a Virtual Reality (VR) application is still a specialised and time consuming task. Many techniques and software tools have been created facilitating the development process. However, they require considerable knowledge of VR technology. Furthermore, modelling the behaviours of the objects in a Virtual Environment (VE) requires programming skills. For this reason, we have developed an approach called 'VR-WISE' that allows specifying a VR application at a conceptual level, free from implementation details and from the viewpoint of a domain expert, allowing non-VR experts to participate in the design of VEs. This approach uses ontologies, incorporating domain knowledge and high-level modelling concepts for describing the VE. In this paper, we explain how object-behaviour can be described at a conceptual level. We show how simple behaviours can be combined for composing complex behaviours. We illustrate the different modelling concepts by means of examples.

Virtual reality is rapidly expanding its capabilities and accessibility to consumers. The application of virtual reality in training for precision tasks has been limited to specialized equipment such as a haptic glove or a haptic stylus, but not studied for handheld controllers in consumer-grade systems such as the HTC Vive. A straight-line precision steadiness task was adopted in virtual reality to emulate basic linear movements in industrial operations and disability rehabilitation. This study collected the total time and the error time for the straight-line task in both virtual reality and a physical control experiment for 48 participants. The task was performed at four different gap widths, 4mm, 5mm, 6mm, and 7mm, to see the effects of virtual reality at different levels of precision. Average error ratios were then calculated and analyzed for strong associations to various factors. The results indicated that a combination of Environment x Gap Width factors significantly affected average error ratios, with a p-value of 0.000. This human factors study also collected participants’ ratings of user experience dimensions, such as difficulty, comfort, strain, reliability, and effectiveness, for both physical and virtual environments in a questionnaire. The results indicate that the ratings for difficulty, reliability, and effectiveness were significantly different, with virtual reality rating consistently rating worse than the physical environment. An analysis of questionnaire responses indicates a significant association of overall environment preference (physical or virtual) with performance data, with a p-value of 0.027. In general, virtual reality yielded higher error among participants. As the difficulty of the task increased, the performance in virtual reality degraded significantly. Virtual reality has great potential for a variety of precision applications, but the technology in consumer-grade hardware must improve significantly to enable these applications. Virtual reality is difficult to implement without previous experience or specialized knowledge in programming, which makes the technology currently inaccessible for many people. Future work is needed to investigate a larger variety of precision tasks and movements to expand the body of knowledge of virtual reality applications for training purposes.

Virtual reality (VR) applications are used in many areas of academic and industrial research areas including engineering, bio-medical & geo-sciences, among others. These applications generally focus on creating a VR environment to enhance user experience. One of the main challenges VR application developers face is to make objects within the environment move in a natural, realistic manner. Many commercial packages and programming libraries exist to help generate complex animations, including physics engines, game engines and modeling software such as Autodesk Maya. All of these tools are very useful, but have many disadvantages when applied to VR applications, as they were not designed for VR development. To address these issues, a VR application programming interface (API) was developed to help VR developers create and visualize natural, complex animations for VR-based systems utilizing OpenSceneGraph. This API, called the Animation Engine 2.0, was built in a manner animators and developers are already familiar with by integrating control points and keyframes for controlling animations. The system is time-based to scale to any size of VR system, which enabled the ability to support different time interpolations as well to incorporate acceleration into animations to create behavioral events such as a boing, bounce, or surge. In this paper, the Animation Engine API is presented along with its integration into a VR aircraft carrier application.

Immersive technologies offer an alternative means of presenting and communicating data derived from BIM modeling. Augmented, virtual, and mixed reality technologies allow architects to visualize and interact with digital data, providing a simulation of their physical presence in an architectural structure, and thus a more realistic and interactive approach to various stages of project activities. In fact, immersive technologies can be used throughout the entire life cycle of an architectural project: planning, design, construction, and operation. The aim of research is to identify and systematize data on computer tools using immersive technologies and the prospects for its development in architectural activities. The scientific paper describes three groups of software products based on immersive technologies with BIM modeling support. Each of the groups is based on a particular immersive technology: augmented reality (AR), virtual reality (VR), or mixed reality (MR). The main difference between these technologies is the level of immersion in the virtual environment: AR overlays computer-generated content on the real world; VR content is 100% digital; MR is a digital overlay that allows virtual 3D elements to interact with the real environment. Each group contains different software products: the first group is mostly characterized by mobile applications, since the physical world is supplemented with digital content through the screens of smartphones, tablets, and other compact equipment; the second group includes various plug-ins that integrate into professional tools and improve them in terms of visualization in a virtual environment and the possibility of its research; The third group with mixed reality support is smaller in number of programs, but it is the most promising for the future development of the architectural profession, as it provides an integrated approach to architectural activity by expanding the professional tools.

We developed The “Hapballoon”, a novel device that is worn on the fingertips. The device can present three types of sensations: force, warmth, and vibration. Force sensations are presented when the inflated balloons on individual devices contact one another. The device is easy to attach on the finger, and does not obstruct common optical finger tracking methods that track the back side of the hand in virtual reality (VR) applications. Each module weights approximately 6 grams, and the balloons are inflated via an air tube connected to a device on the user’s arm. This wearable haptic presentation device may improve the realism of various VR applications.

In accordance with the recent enhancement of display technology, users demand a higher quality of streaming service, which escalates the bandwidth requirement. Considering the recent advent of high FPS (frame per second) 4K and 8K resolution 360° videos, such bandwidth concern further intensifies in 360° Virtual Reality (VR) content streaming even at a larger scale. However, the currently available bandwidth in most of the developed countries can hardly support the bandwidth required to stream such a scale of content. To address the mismatch between the demand on higher quality of streaming service and the saturated network improvement, we propose SALI360 that practically solves the mismatch by utilizing the characteristics of the human vision system (HVS). By pre-rendering a set of regions - where viewers are expected to fixate - on 360° VR content in higher quality than the other regions, SALI360 improves viewers' quality of perception (QoP) while reducing content size with geometry-based 360° content encoding. In our user experiment, we compare the performance of SALI360 to the existing 360° content-encoding techniques based on 20 viewers' head movement and eye gaze traces. To evaluate viewers' QoP, we propose FoL (field of look) that captures viewers' quality perception area in the visual focal field (8°) rather than a wide (around 90°) field of view (FoV). Results of our experimental 360° VR video streaming show that SALI360 achieves 53.3% of PSNR improvement in FoL, while gaining 9.3% of PSNR improvement in FoV. In addition, our subjective study on 93 participants verifies that SALI360 improves viewers' QoP in the 360° VR streaming service.