Object Manipulation in Physically Constrained Workplaces: Remote Collaboration with Extended Reality

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

This study aims to systematically review the integration and clinical applications of 3D printing and extended reality technologies in medicine. This offers an in-depth review of technological advances and their application in different areas of medicine such as surgical planning, medical staff training, new ways of visualization, as well as the integration of tailor-made physical and digital elements. Factors that benefit patient safety and promote innovation through technological intervention. The study conducted a systematic review of the literature using the PRISMA methodology, with a selection of studies published between 2005 and 2023. The process included a comprehensive search in academic databases from which articles were extracted that included topics on 3D printing technologies, extended realities, and applied cases in which efficacy and application in clinical settings were proven. As a result, specific inclusion and exclusion filters were established to ensure relevance to the study's objective. The selected items were subjected to a detailed analysis, based on which the methodologies used, types of studies, areas of impact, population, and clinical impacts were evaluated. The synthesis of the results was carried out using the quantitative and qualitative approaches. This enabled a comprehensive understanding of current trends and potential future applications of these technologies in the field of medicine. The results indicate that the adoption of 3D printing and technologies such as augmented reality (AR) and mixed reality (MR) facilitates more precise and less invasive interventions. It also improves understanding of complex conditions and pathologies through enhanced and enriched interactions and visualizations. In addition, technologies are consolidated as tools for medical strengthening and training through more realistic simulations. Studies show that by having better understanding, visualization, and interaction with components and technologies, there is a reduction in surgical errors and optimized duration times in the surgeon and patient recovery; finally, the use of these technologies also improves the retention of information in training and education contexts. Furthermore, the study reveals that, although the associated costs may be considerable, the gains in accuracy and operational efficiency justify the investment. Among the challenges posed by the use and integration of 3D printing and augmented or MR is the need to further improve the accurate calibration of the 3D images with physical models, which turns out to be a key aspect for the success of these applications in the clinical context. Implementation of technologies such as 3D printing, AR, MR, and virtual reality in clinical practice has been shown to have great benefits in efficiency and safety in surgical procedures. These technologies profile themselves as solutions that promise to revolutionize approaches to learning, surgical planning, and surgery. This offers customization and accuracy features, significantly reducing error margins. Furthermore, medical training is enriched by more realistic simulations and detailed visualizations. However, the construction of innovative tools that consolidate the characteristics of these technologies requires continuous improvement and investment in training, development, and appropriate infrastructure to overcome the challenges associated with the integration of such technologies into clinical practice. This review provides a comprehensive synthesis of current applications and the transformative impacts of 3D printing technologies and extended realities in medicine. Unlike previous revisions, this study carries out a quantitative analysis of its benefits, supported by recent case studies and statistical data. This leads to valuable insights on future directions of medical technology integration.

In today’s digital era, the realms of virtual reality (VR), augmented reality (AR), and mixed reality (MR) collectively referred to as extended reality (XR) are reshaping human–computer interactions. XR technologies are poised to overcome geographical barriers, offering innovative solutions for enhancing emotional and social engagement in telecommunications and remote collaboration. This paper delves into the integration of (AI)-powered 3D talking heads within XR-based telecommunication systems. These avatars replicate human expressions, gestures, and speech, effectively minimizing physical constraints in remote communication. The contributions of this research encompass an extensive examination of audio-driven 3D head generation methods and the establishment of comprehensive evaluation criteria for 3D talking head algorithms within Shared Virtual Environments (SVEs). As XR technology evolves, AI-driven 3D talking heads promise to revolutionize remote collaboration and communication.

We present a Mixed Reality system for remote collaboration using Virtual Reality (VR) headsets with external depth cameras attached. By wirelessly sharing a 3D point-cloud data of a local workers' workspace with a remote helper, and sharing the remote helper's hand gestures back to the local worker, the remote helper is able to assist the worker to perform manual tasks.Displaying the point-cloud video in a conventional way, such as a static front view in VR headsets, does not provide helpers with sufficient understanding of the spatial relationships between their hands and the remote surroundings. In contrast, we propose a Mixed Reality (MR) system that shares with the remote helper, not only 3D captured environment data but also real-time orientation info of the worker's viewpoint. We conducted a pilot study to evaluate the usability of the system, and we found that extra synchronized orientation data can make collaborators feel more connected spatially and mentally.

We present a novel Mixed Reality (MR) remote collaboration system that integrates a telepresence drone to overcome viewpoint limitations in previous approaches. Current remote collaboration systems often restrict remote users to the local user's perspective or a fixed viewpoint, which may reduce spatial awareness and collaboration effectiveness. Our proposed system enables a remote user wearing a VR headset to independently navigate the local environment through a drone while guiding a local AR headset user. The drone represents the remote user physically and virtually with an AR avatar. The system provides the remote user with a 3D reconstruction of the local environment, the local user's avatar, and real-time drone camera video to facilitate drone operation and collaboration. A user study comparing our drone-enhanced MR system with 2D video and 360-degree video MR systems during inspection tasks revealed that the telepresence drone significantly improved collaboration efficiency and enhanced users' social presence and spatial presence. However, the remote users perceived a higher workload. Our findings demonstrate the benefits of independent spatial navigation through drones in remote collaboration and offer insights for future drone-enhanced MR remote collaboration systems.

The recent years have witnessed an increase in the use of augmented and virtual reality systems, changing the way we interact with our environments. Such systems are commonly associated with advertising, entertainment, medicine, training and education. However, with the increasing acceptance and availability of mobile and wearable devices (e.g. head-mounted displays (HMD)), the use of these technologies is moving towards professional and industrial environments, where they would be able to support employees in their daily tasks, increasing customer satisfaction and reducing business costs. This paper presents an innovative Mixed Reality (MR) system to assist field workforce in remote locations. As part of the overall implementation, the MR system uses fuzzy logic mechanisms to improve accuracy in user tracking and object monitoring, allowing the correct representation of users and objects in the Graphical User Interfaces (GUIs), and improving the experience for users.

➤ Extended reality is a term that encompasses different modalities, including virtual reality, augmented reality, and mixed reality.➤ Although fully immersive virtual reality has benefits for developing procedural memory and technical skills, augmented and mixed reality are more appropriate modalities for preoperative planning and intraoperative utilization.➤ Current investigations on the role of extended reality in preoperative planning and intraoperative utilization are still in the early stages, but preliminarily show that extended reality technologies can help surgeons to be more accurate and efficient.

There still lacks a collaborative framework that integrates existing and new virtual reality systems supporting remote collaboration for data visualization. The collaborative framework currently available might limit the collaborators for collaboration because it is not developed to support interaction with diverse 3D applications while using heterogeneous virtual reality (VR) systems remotely in a synchronized way. We will present our collaborative framework that supports remote collaboration across heterogeneous VR systems to interact with multiple 3D applications. This system is independent of any VR systems and can be used to interact collaboratively in real-time on a shared task. Additionally, we conducted a pilot study to gain perspective on the usability of the system for collaborative work across heterogeneous VR systems. We will present the results on the usability of the system, and the results obtained from the users are motivating and encouraging. Our system has the potential to increase task effectiveness and workflow because it enables shared and real-time interaction while remotely collaborating with diverse 3D applications across heterogeneous VR systems.

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patient's body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

Extended reality (XR), here jointly referring to virtual, augmented, and mixed (VR, AR, MR) reality, is becoming more common in everyday working life. This paper presents a systematic literature review of academic publications on XR indicating changes in practical organization of work. We analyse both application areas of XR and theoretical and methodological approaches of XR research. The review process followed the PRISMA statement. Design, remote collaboration, and training were the main application areas of XR. XR enabled overcoming of obstacles set by time and space, safety, and resources by mediating experience of space. Research on XR applications in actual working life settings is yet relatively rare and covers primarily three areas: collaboration, evaluation of knowledge transfer, and work practices. Virtual reality was the most common form of applied XR, although the hardware used varied case by case. We identified four research areas regarding XR: collaboration, work practices, and evaluation of knowledge transfer, which somewhat followed the application areas. We did not find XR-specific methodologies in the reviewed articles, only few recent studies used novel ways of collecting research material, such as recording the movement in virtual reality. For now, XR still holds significant potential rather than clearly confirmed general advantages in working life.

This paper describes a task-oriented approach to cope with the difference between the three dimensional (3D) space in a virtual reality (VR) system and that perceived by the user from its stereoscopic images with binocular disparity. This difference occurs for the reason that human visual perception is affected not only by binocular disparity but also by various kinds of visual cues. Although it is possible to set some of those cues consistent with each other when they are presented by a stereoscopic display, it is difficult to do the same for other cues especially vergence and focus. As the result, the users of VR systems often fail to manipulate virtual objects due to this difference. In this article, we propose to adjust the virtual space in a VR system through object manipulation by the user so that the user does not fail in each manipulation. We verified by experiments that this adjustment is effective to improve the rate of successful manipulation.

Many solutions tailored for intuitive visualization or teleoperation of virtual, augmented and mixed (VAM) reality systems are not robust to robot failures, such as the inability to detect and recognize objects in the environment or planning unsafe trajectories. In this paper, we present a novel virtual reality (VR) framework where users can (i) recognize when the robot has failed to detect a real-world object, (ii) correct the error in VR, (iii) modify proposed object trajectories and, (iv) implement behaviors on a real-world robot. Finally, we propose a user study aimed at testing the efficacy of our framework. Project materials can be found in the OSF repository.

Tangible physical systems are more intuitive than Intangible virtual Systems. Mixed reality systems are considered as an alternative to virtual systems, bringing advantages of tangible systems into an interaction. However, past research has mainly focussed on technical aspects of incorporating pervasive-ness and immersive-ness in the virtual systems. This paper reports on an empirical study of intuitive Interaction in a Mixed Reality game system for children and the design aspects that could facilitate intuitive Interaction in such systems. A related samples Friedman’s test showed that the Mixed Reality game system demonstrated more intuitive interactions than non‐intuitive Interactions. A linear regression analysis further established that the variation in intuitive Interaction in the Mixed Reality system could be statistically significantly explained primarily by physical affordances offered by the Mixed Reality system and to a lesser extent by the perceived affordances in the system. Design guidelines to develop intuitive Mixed Reality systems are discussed. These guidelines should allow designers to exploit the wonders of advances in technology and at the same time allow users to directly interact with the physical real world. This will allow users to access maximal physical affordances, which are primary contributors to intuitive interaction in Tangible and Mixed Reality systems. Keywords: Intuitive Interaction; Mixed Reality Systems; Tangibles; Child Computer Interaction

The article considers the possibility of using virtual and mixed reality in studying the geometry of the crystal lattice in the framework of the course of solid state physics. The application of mixed and virtual reality technologies in the study of various branches of physics is receiving more and more attention. Virtual reality headsets were used in practical and laboratory classes, where students had to create crystal structures and carry out modeling of various processes. When studying complex spatial objects of crystallography, virtual reality technology allowed students to familiarize themselves with the object of study in more detail, understand the structure of crystals and, accordingly, complete the task in a timely manner. At the same time, the prolonged use of virtual reality headsets led to rapid fatigue of students.

Over the past few years, virtual and mixed reality systems have evolved significantly yielding high immersive experiences. Most of the metaphors used for interaction with the virtual environment do not provide the same meaningful feedback, to which the users are used to in the real world. This paper proposes a cyber-glove to improve the immersive sensation and the degree of embodiment in virtual and mixed reality interaction tasks. In particular, we are proposing a cyber-glove system that tracks wrist movements, hand orientation and finger movements. It provides a decoupled position of the wrist and hand, which can contribute to a better embodiment in interaction and manipulation tasks. Additionally, the detection of the curvature of the fingers aims to improve the proprioceptive perception of the grasping/releasing gestures more consistent to visual feedback. The cyber-glove system is being developed for VR applications related to real estate promotion, where users have to go through divisions of the house and interact with objects and furniture. This work aims to assess if glove-based systems can contribute to a higher sense of immersion, embodiment and usability when compared to standard VR hand controller devices (typically button-based). Twenty-two participants tested the cyber-glove system against the HTC Vive controller in a 3D manipulation task, specifically the opening of a virtual door. Metric results showed that 83% of the users performed faster door pushes, and described shorter paths with their hands wearing the cyber-glove. Subjective results showed that all participants rated the cyber-glove based interactions as equally or more natural, and 90% of users experienced an equal or a significant increase in the sense of embodiment.