Mixed Reality or Simply Mobile? A Case Study on Enabling Less Skilled Workers to Perform Routine Maintenance Tasks

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

The priority of modern dentistry is to keep patients’ teeth for as long as possible. Tooth extraction is a procedure performed as a last resort when conservative methods and endodontic surgery procedures have not brought the expected results. As a consequence, the number of patients in dental offices, who require first and repeated endodontic treatment, is increasing. Thus, the development of new technologies in the medical industry, including microscopy, computer tomography (CT), as well as diode and neodymium-YAG-erbium lasers, enables dentists to increase the percentage of successful treatments. Moreover, mixed reality (MR) is a very new technology, in which the 3D view can help plan or simulate various types of tasks before they will be carried out in real life. In dentistry, 3D holography can be applied to display CT data to plan endodontic treatment. The most important element in effective root canal treatment is the precise imaging of the root canal. The CT scans allow dentists to view the anatomy of the patient’s tooth with much higher precision and understanding than using 2D radiography (RTG-radiographic photo) pictures. Recently, the development of new 3D technologies allows dentists to obtain even more data from existing CT scans. In this paper, the CT scan data were applied to generate patient teeth in 3D and simulate the view of the root canal’s anatomy in MR devices, i.e., Microsoft HoloLens 2. Using DICOM RAW data from the CT exam, we generated a 3D model of the jaw with a tooth. In the next step, the crown of the tooth was removed in a similar way to how a dentist would do this using a dental handpiece. Furthermore, all root canals were cleaned of everything inside. This way we achieved empty tunnels, namely root canals. Finally, we added appropriate lighting, similar to the type of lighting that dentists use. The proposed approach enables to display of the root canals in the same way as during the endodontic procedure using a microscope. It allows for the visualization of the root canal and changing its direction, in which dimensional accuracy is crucial. It turns out that mixed reality can be considered a complementary method to the traditional approach, which reduces the amount of time for the root canal treatment procedure by up to 72.25%, depending on the complexity of the case, and increases its effectiveness. Thus, the mixed reality-based system can be considered an effective tool for planning dental treatment.

Collaborative learning has been shown to be beneficial for children's learning performance, increasing the curiosity and intensity of the ability of cooperation. Mixed-Reality with collaborative learning is the trending research topic in the Human-Computer Interaction (HCI) area. Additionally, with the rise of attention to global warming which brings in more extreme weather and climate conditions, the earth science education would be one of the crucial topics for the next generation. Moreover, there are few augmented reality and mixed reality applications on earth science subject. In this paper, we propose a Mixed Reality Tornado Simulator which offers an earth science education in a collaborative setting. Students and the instructor can cooperate on learning the knowledge of the formation and its damage cause on human-built structures, farming, and vegetation by using our mixed reality application with the Microsoft HoloLens. Also, for evaluating the learning performance in this mixed reality setting, we propose to study the cognitive load while the student is learning the abstract knowledge in Earth Science. We will separate the student into a control group and experimental groups and use different teaching instruments to test the difference of cognitive load.

This paper shows our work in progress towards an easy to use development environment for Mixed Reality (MR) Applications. We argue that development of MR applications is a collaboration between interaction designers who know about user requirements, and expert developers who know the intricacies of MR development. This collaboration should be supported by tools that aid both roles and ease their communication. We also argue that real MR development should allow easy migration from one hardware setup to another, since hardware greatly varies in these type of applications, and it is important to fit a solution to the particular user's requirements and context.We show the foundational concepts in our work and current Integrated Development Environment (IDE) implementation. This work is based on InTml, a domain specific language for MR applications, and Eclipse, an open source, general purpose IDE.

In proposing a modern approach to process industry operational integrity management, a question that often surfaces is "What is the most complex activity within operational integrity management that affects performance, productivity and efficiency?". There are variety of answers to this question, one answer being "Human limitations in conceiving data to make informed decisions". The objective of this presentation is to outlines the key benefits of mixed reality platforms in presenting data to managers, operators and maintenance personnel working in an Oil & Gas facility. Process industry in the past did not generate as much data as it does today. With the amount of data generated every day, it is necessary to innovate so as to analyze data and visually represent them thereby providing a "total immersive knowledge". In light of the recent developments in mixed reality platforms, it is now possible to use this technology to visually present data focused around the needs of humans. Using mixed reality, one can merge the view of real equipment of a plant with a virtual world of data, simulation and instructions, such that both the physical and digital world co-exist. Few of the applications of this technology in the Oil & Gas industry are 1) providing safe operations, maintenance and repair instructions real time 2) monitor, record and analyze data real time during inspections 3) visualize data outside the control room 4) effectively train personnel during visits 5) analyze trends and predict outcomes 6) share data with managers, vendors for efficient decisions. The presentation outlines the key benefits of mixed reality platforms such as Microsoft Hololens in presenting data to managers, operators and maintenance personnel working in an Oil & Gas facility. The presentation will also discuss findings of a sponsored pilot project where mixed reality was implemented. At the end, the presentation will also raise the current challenges and possible solutions in successfully implementing such technologies.

This study aims to explore the application of mixed reality in experimental safety teaching, designing and implementing a mixed reality-based experimental teaching method to enhance the effectiveness of experiments and user experience. Through field visits to design laboratories and expert interviews, this research analyzes the existing problems in design experimental teaching and real user needs. It proposes a multi-level, multi-dimensional mixed reality innovation design experiment platform construction method from teaching to dialogue, from human-computer interaction to perception, from creation to manufacturing, and from communication to thinking. Finally, this study uses the intelligent cockpit interface experience perception mixed reality experimental platform as an example to confirm that mixed reality technology can provide an immersive learning experience. This research provides theoretical support and practical reference for the application of mixed reality in experimental safety teaching, and offers prospects for future teaching design and research.

With Generation Z becoming the primary group of nursing learners and the increasing shortage of nursing education resources, augmented reality and mixed reality based on head-mounted displays are being used more and more in nursing education. However, the current application landscape and the proper usage of these concepts remain unclear. Therefore, this study aims to conduct a scoping review to explore the current applications of head-mounted display-based augmented reality and mixed reality in nursing education and to clarify the definitions and usage of the concepts of augmented reality and mixed reality, ultimately providing directions for future applications and research. Based on the five-stage framework and PRISMA-ScR guidelines, a comprehensive collection and summarization of evidence regarding the application of head-mounted display-based augmented reality and mixed reality in nursing education were conducted. The databases retrieved include CNKI, Wanfang Database, VIP, CBM, PubMed, Cochrane Library, Embase, CINAHL, Web of Science, Scopus, IEEE Xplore, ACM Digital Library, and Ei. The languages of the included literature are Chinese and English. The retrieval was up to October 2024. A total of 44 studies were included in this review, covering three types of head-mounted displays: immersive head-mounted displays, smart glasses, and smartphone-based head-mounted displays. The main application areas were skills training and knowledge acquisition. Most of the studies were feasibility studies, though they also included some efficacy studies and research on personal experiences. In addition, these studies often employed vague or inconsistent definitions of augmented reality and mixed reality. Despite various explorations in the application of head-mounted display-based augmented reality and mixed reality in nursing education, there is still room for improvement in the relevant theory and utilization of this technology. In the future, interventions should use the three dimensions (observation of reality, real - virtual interaction, and fidelity of virtuality) described in detail, rather than simply employing the concepts of augmented reality or mixed reality. Efforts should be concentrated on developing and implementing head-mounted displays combined with other technologies that boast enhanced performance and cost - effectiveness, and further validating their effectiveness.

Augmented Reality (AR) and Mixed Reality (MR) are new emerging technologies that are becoming increasingly popular. Because these technologies are so new, there is a lack of standards and consistency in application and hardware design. This can make it difficult to learn and frustrating for users. One way to standardized design and enhance the usability of a product is through the use of heuristic evaluations. General heuristics, such as Nielsen’s 10 or Schneiderman’s 8 usability heuristics have been used to evaluate these technologies. These heuristics are a useful starting point because they bring attention to many crucial aspects of the usability of a product. However, additional aspects that could alter the users’ experience may not be assessed due to the uniqueness of AR and MR. There are very few validated AR and MR heuristics in the literature for practitioners to use. The purpose of this study was to create and validate a heuristic checklist that can be used to assess and inform design changes that influence the user experience of an AR or MR application and/or device. We followed an established and comprehensive 8-stage methodology developed by Quiñones, Rusu, & Rusu to create and validate our AR and MR usability heuristic checklist [4]. This included a search and summary of the current literature, formally defining heuristics based on this literature search, and validating through heuristic evaluations, expert reviews, and user testing. Our final revised heuristic checklist included 11 heuristics and 94 checklist items that encompasses usability aspects for AR and MR technologies. KeywordsHuman factorsUser experienceAugmented RealityMixed RealityUsabilityDesign guidelines

Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.

A Mixed Reality (MR) application using an optical see-through headset was developed to assess short-term spatial memory. A study with 29 participants was conducted. Data from this study were compared to two previous studies using mobile Augmented Reality (AR) and Virtual Reality (VR) with headsets. When comparing the three technologies (MR, AR, VR) for the performance variables, there were no statistically significant differences for either the total number of correctly placed objects or the total number of attempts. However, the MR application required more time than the AR and VR applications in the evaluation phase and more time than the VR application in the learning phase. Our arguments for the longer time are the novelty of the MR application for the participants and the characteristics of the applications. The key results from the MR study include the following: (1) the objects used in the MR application were correctly positioned on a map, which implies that the memory acquired with the MR application is effectively transferred to the user's mental map; (2) for the performance variables, there were no significant differences in the results by gender; (3) and the usability rating decreased with computer experience. The results show that the MR application is effective for spatial memory assessment and was well rated by the participants. The three technologies, along with suitable hardware, are effective for spatial memory assessment. However, MR using optical see-through headsets offers advantages over mobile AR and VR using headsets, discussed in this publication.

Evaluation of cognitive load for a mixed reality anatomy application

In Augmented, Virtual and Mixed Reality, users interact with content in 3D space. Currently, surprisingly little is known about where these user interfaces should be placed in 3D space for optimal comfort and productivity. In this presentation, we will discuss factors that affect content placement and put forward a set of best practices for interaction zones in 3D. The output of the project is a set of Human Interface Guidelines (HIG) for Interaction Zones in AR. The Guidelines can be thought of as an AR version of the original Human Interface Guidelines put forward by Macintosh in 1995. The purpose of the Guidelines is to help application designers and developers in AR to design user-centric content in AR. Given the limited research that exists on the range of factors affecting interaction zones in AR, this document will contribute towards growing shared knowledge on this topic. This project combined knowledge of human physiological capabilities and limitations, Anthropometric ranges of the American target population, and current ergonomics guidelines and principles, as well as functional capabilities of Magic Leap 2 mixed reality device in the establishment of the HIG. Specific focus areas of the Human Interface Guidelines (HIG) for Interaction Zones in AR include: Ergonomics and Visual Comfort, Direct vs Indirect Interactions (gesture, gaze, controller), Field of View, and UI Behavior. In Ergonomics and Visual Comfort, we provide recommendations on content placement based on natural human line of sight, ergonomically comfortable head movement, and natural range of eye rotation. In Direct vs Indirect Interactions, we describe that where content is presented relative to the user depends on the way in which the user is expected to interact with the content. For example, content placement for direct (i.e. touch) interactions may be dictated by arm length anthropometry, as well as the field of view of the gesture-perceiving cameras. In Field of View, we review the relationship between field of view and viewing distance, and provide some techniques that designers can use to mitigate the noticeability of cut-off content due to limited FOV. In UI Behavior, we put forward best practices for how the UI moves with the user to prevent discomfort. The methodology to create the Human Interface Guidelines for Interaction Zones in AR is a literature review of existing research and guidelines on ergonomics as well as primary research in AR. The Guidelines will provide functional instructions for Developers and Designers of Augmented, Virtual and Mixed Reality applications, informing the design of content that is comfortable, convenient and optimally placed for the user.

The paper presents a framework for interactive control of multi robots using mixed reality interfaces. The proposed approach can operate with both single and multi-robot systems, including interaction with industrial manipulators, mobile robots, and unmanned aerial vehicles (UAV). The human–robot interface allows both interactive and physical interaction with the robot. Particular attention is paid to a visualisation of the robots' intent during control. The presented framework has Robotic Operation System (ROS) as a core that can be connected to any robot and mixed reality interface. This allows us to make framework scalable and connect several different mixed reality devices and multiple heterogeneous robots. In the implementation part we connected our framework with Microsoft HoloLens, collaborative robot KUKA IIWA, mobile platform Plato, UAV designed on the basis of dji f450. The proposed framework was tested experimentally with real experimental setups mentioned above. The results showed the capabilities of the mixed reality system for interactive control of different robots' type.

PurposeThe purpose of this study is to explore the effectiveness of mixed reality (MR) technology in enhancing library tours for first-year students at a university academic library. It specifically aims to investigate whether MR tours can reduce library anxiety, improve knowledge acquisition and increase motivation when compared to traditional library tours.Design/methodology/approachThis study uses a quasi-experimental research design, comparing two groups of first-year students (n = 96). One group (n = 48) experiences a MR library tour using the HoloLens 2 device, while the other group (n = 48) participates in a traditional library tour led by library staff. The participants’ library anxiety, knowledge acquisition and motivation are measured using relevant questionnaires before and after the tours.FindingsThe findings indicate that while both MR and traditional tours are effective in imparting knowledge, there is no statistically significant difference in overall knowledge acquisition. However, the MR tour significantly enhances students’ perceived competence, interest and effort, providing higher engagement and motivation. Traditional tours, on the other hand, are more effective in reducing library anxiety, particularly in relation to interactions with librarians.Originality/valueThis study highlights the potential of MR technology to enhance library tours by balancing immersive experiences with ease of use. MR bridges the gap between virtual reality’s immersion and the accessibility, lower cost of content development and simplicity of application usage. It aligns with trends in academic libraries by offering high engagement without common health issues like motion sickness. However, while MR applications can be developed at a relatively lower cost, the high cost of MR equipment remains a limitation for institutions. Despite this, MR presents a promising solution for improving student engagement and learning, with the potential to become more accessible as hardware costs decrease.

Although object detection methods have significant potential to extend the boundaries of mixed reality (MR), they are often difficult to execute on current MR devices that have limited computing resources. In this paper, we propose an edge-assisted object detection method that uses the eye-tracking function of MR devices to determine whether an object detection task should be offloaded to an edge server for processing and thus reduce the MR device processing load. We implement and evaluate the proposed method using a Microsoft HoloLens 2 to confirm its effectiveness.