Facilitating HRI by Mixed Reality Techniques

Mobile robots start to appear in everyday life, e.g., in shopping malls or nursing homes. Often, they are operated by staff with no prior experience in robotics. False expectations regarding the capabilities of robots, however, may lead to disappointments and reservations when it comes to accepting robots in ones personal space. We make use of state-of-the-art Mixed Reality (MR) technology by integrating a Microsoft HoloLens into the robot»s operating space to facilitate acceptance and interaction. The MR device is used to increase situation awareness by (a) externalizing the robot»s behavior-state and sensor data and (b) projecting planned behavior. In addition to that, MR technology is (c) used as satellite sensing and display device for human-robot-communication.

In the mysterious realm full of emerging technologies, there is a fantastic role played by Mixed Reality techniques. The blending of real and virtual worlds that allows individualized user control opportunities of the environment around user remarks the unique feature of Mixed Reality. The current work shows the effort of many researchers who have tried to explore the field of Mixed Reality in their own planned ways for the creation of Mixed Reality smart classroom. The objective is to introduce, how Mixed Reality device can act as real-time analytical tool for the teacher in a classroom to enhance the learning process of the students. The first part introduces the advent of Mixed Reality with some defined theories that are widely accepted and a glimpse of approaches done by many researchers. Later, an overview of an attempt to create a Mixed Reality classroom, two different hypotheses incorporating the same Mixed Reality tool called Lumilo were experimented with the assistance of AIED and ITS systems. Followed by the results of the hypotheses, aiming one of them to be employed in the classroom environment where both teachers and students were most benefitted. This type of environment resulted in better modes of education when compared to the existing approaches of Mixed Reality classroom.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Recently MR (Mixed Reality) techniques are used in many fields, and one of them is work support. Work support using MR techniques can display work instructions directly in the work space and help user to work effectively especially in assembling tasks. MR work support for assembling tasks often uses HMD (Head Mounted Display) to construct MR environment. However, there are some problems for the use of HMD, such as a burden to the head, a narrow view and a motion picture sickness. One of the techniques to solve such problems is pepper's ghost which is an optical illusion using a glass. In this paper, we propose the naked eye MR work supporting system for assembling tasks using pepper's ghost. This system enables a beginner to assemble some blocks into one object by the naked eye with a few burdens.

Mixed-reality (MR) technology has the potential to enhance care delivery, but there remains a paucity of evidence for its efficacy and feasibility. To assess the efficacy and feasibility of MR technology to enhance emergency care delivery in a simulated environment. This pilot randomized crossover trial was conducted from September to November 2021 at a single center in a high-fidelity simulated environment with participants block randomized to standard care (SC) or MR-supported care (MR-SC) groups. Participants were 22 resident-grade physicians working in acute medical and surgical specialties prospectively recruited from a single UK Academic Health Sciences Centre. Data were analyzed from September to December 2022. Participants resuscitated a simulated patient who was acutely unwell, including undertaking invasive procedures. Participants completed 2 scenarios and were randomly assigned to SC or MR-SC for the first scenario prior to crossover. The HoloLens 2 MR device provided interactive holographic content and bidirectional audiovisual communication with senior physicians in the MR-SC group. The primary outcome was error rate assessed via the Imperial College Error Capture (ICECAP) multidimensional error-capture tool. Secondary outcomes included teamwork (Observational Teamwork Assessment for Surgery [OTAS]; range, 0-6 and Teamwork Skills Assessment for Ward Care [T-SAW-C]; range, 1-5), scenario completion, stress and cognitive load (NASA Task Load Index [NASA-TLX; range 0-100]), and MR device user acceptability. A total of 22 physicians (15 males [68.2%]; median [range] age, 28 [25-34] years) were recruited. MR technology significantly reduced the mean (SD) number of errors per scenario compared with SC (5.16 [3.34] vs 8.30 [3.09] errors; P = .003), with substantial reductions in procedural (0.79 [0.75] vs 1.52 [1.20] errors; P = .02), technical (1.95 [1.40] vs 3.65 [2.03] errors; P = .01), and safety (0.37 [0.96] vs 0.96 [0.85] errors; P = .04) domains. MR resulted in significantly greater scenario completion rates vs SC (22 scenarios [100%] vs 14 scenarios [63.6%]; P = .003). It also led to significant improvements in the overall quality of teamwork and interactions vs SC as measured by mean (SD) OTAS (25.41 [6.30] vs 16.33 [5.49]; P < .001) and T-SAW-C (27.35 [6.89] vs 18.37 [6.09]; P < .001) scores. As reported via mean (range) NASA-TLX score, there were significant reductions for MR-SC vs SC in participant temporal demands (38 [20-50] vs 46 [30-70]; P = .03) and significant improvements in self-reported task performance (50 [30-60] vs 39 [10-70]; P = .01). Overall, 19 participants (86.4%) reported that they were more confident in making clinical decisions and undertaking clinical procedures with MR support. This study found that the use of MR technology reduced error, improved teamwork, and enhanced practitioner confidence when used to support the delivery of simulated emergency medical care. ClinicalTrials.gov Identifier: NCT05870137.

Underground spaces are rapidly expanding, with their functions increasingly extending to commercial, entertainment, and residential areas. However, the inherent characteristics of closure and complexity in these environments significantly impact user experience and safety. An experimental system was developed to explore human perception in underground spaces. The system consisted of three main components: a mixed reality (MR) device, a set of wearable biosensors, and a data platform. Typical elements of the underground environment were built by digital tools and overlaid onto real-world physical space using the MR device. Scenarios, such as fire hazards or earthquakes, were simulated based on physical mechanism and introduced into the MR environment. During these simulations, human responses were monitored via wearable biosensors. All data were transmitted to and stored on the data platform, where analysis revealed correlations between human responses and environmental factors, providing insights for the optimization of underground spaces. A fire drill scenario was studied as the application of this system. Smoke is generated by mixed-reality device according to the data derived from Fire Dynamics Simulator(FDS), after which the participant was asked to find the exit under low visibility. Meanwhile, the response of participants, including electroencephalogram(EEG), electrodermal activity(EDA) and photoplethysmographic(PPG), was collected for further analysis. It was indicated that the system could effectively build a vivid experimental scenario with less modeling work and higher movement freedom than virtual reality(VR) and provided a more immersive experience than augmented reality(AR). The human physiological signal can be recorded and utilized as a supplement for subjective response in the future.

The paper presents a framework for interactive control of multi robots using mixed reality interfaces. The proposed approach can operate with both single and multi-robot systems, including interaction with industrial manipulators, mobile robots, and unmanned aerial vehicles (UAV). The human–robot interface allows both interactive and physical interaction with the robot. Particular attention is paid to a visualisation of the robots' intent during control. The presented framework has Robotic Operation System (ROS) as a core that can be connected to any robot and mixed reality interface. This allows us to make framework scalable and connect several different mixed reality devices and multiple heterogeneous robots. In the implementation part we connected our framework with Microsoft HoloLens, collaborative robot KUKA IIWA, mobile platform Plato, UAV designed on the basis of dji f450. The proposed framework was tested experimentally with real experimental setups mentioned above. The results showed the capabilities of the mixed reality system for interactive control of different robots' type.

This paper presents a novel method to implement an interactive mobile robot. The proposed mobile robot is implemented under the technique of mixed reality. The mixed reality technique is realized by both the Virtools tool and mobile robot. There are three basic interactive behaviors of human beings have been implemented on this proposed mobile robot. There are target navigation behavior, face tracking behavior, and obstacle avoidance behavior; the three behaviors are designed by the fuzzy rules method, respectively. These behaviors lead to the proposed path planning strategy possibly. On the other hand, a novel path planning strategy for a mobile robot is given in this paper. The path planning strategy is according to the experience that the mobile robot has been experienced in target navigation. Experiments show that the proposed behaviors are well designed. With the basic ability of the behaviors, the proposed robot can automatically learn from the previous navigation experience and can optimize the path planning.

Augmented Reality (AR) and Mixed Reality (MR) are new emerging technologies that are becoming increasingly popular. Because these technologies are so new, there is a lack of standards and consistency in application and hardware design. This can make it difficult to learn and frustrating for users. One way to standardized design and enhance the usability of a product is through the use of heuristic evaluations. General heuristics, such as Nielsen’s 10 or Schneiderman’s 8 usability heuristics have been used to evaluate these technologies. These heuristics are a useful starting point because they bring attention to many crucial aspects of the usability of a product. However, additional aspects that could alter the users’ experience may not be assessed due to the uniqueness of AR and MR. There are very few validated AR and MR heuristics in the literature for practitioners to use. The purpose of this study was to create and validate a heuristic checklist that can be used to assess and inform design changes that influence the user experience of an AR or MR application and/or device. We followed an established and comprehensive 8-stage methodology developed by Quiñones, Rusu, & Rusu to create and validate our AR and MR usability heuristic checklist [4]. This included a search and summary of the current literature, formally defining heuristics based on this literature search, and validating through heuristic evaluations, expert reviews, and user testing. Our final revised heuristic checklist included 11 heuristics and 94 checklist items that encompasses usability aspects for AR and MR technologies. KeywordsHuman factorsUser experienceAugmented RealityMixed RealityUsabilityDesign guidelines

Digital hydraulics is a discrete technology that integrates advanced dynamic system controls, digital electronics, and machine learning to enhance fluid power systems’ performance, overall efficiency, and controllability. A mechanically actuated inline three-piston variable displacement digital pump was previously proposed and designed. The inline three-piston pump incorporates complex mechanical and hydraulic subsystems and highly coupled mechanisms. The complexity of the utilized subsystems poses challenges when assessing the viability of the conceptual design. Therefore, this work focuses on designing, developing, and implementing a collaborative virtual platform involving a digitized module showcasing the internal mechanical structure of the digital pump utilizing mixed reality (MR) technology. MR technology is acknowledged as the forthcoming evolution of the human–machine interface in the real–virtual environment utilizing computers and wearables. This technology permits running simulations that examine the complexity of highly coupled systems, like the digital pump, where understanding the physical phenomenon is far too intricate. The developed MR platform permits multiple users to collaborate in a synchronized immersive MR environment to study and analyze the applicability of the pump’s design and the adequacy of the operated mechanisms. The collaborative MR platform was designed and developed on the Unity game engine, employing Microsoft Azure and Photon Unity Networking to set up the synchronized MR environment. The platform involves a fully interactive virtual module on the digital pump design, developed in multiple stages using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity and deployed in the synchronized MR environment through a HoloLens 2 MR headset. A research study involving 71 participants was carried out at Purdue University. The study’s objective was to explore the impact of the collaborative MR environment on understanding the complexity and operation of the digital pump. It also sought to assess the effectiveness of MR in facilitating collaboration among fluid power stakeholders in a synchronized digital reality setting to study, diagnose, and control their complex systems. Surveys were designed and completed by all 71 participants after experiencing the MR platform. The results indicate that approximately 75% of the participants expressed positive attitudes toward their overall MR platform experience, with particular appreciation for its immersive nature and the synchronized collaborative environment it provided. More than 70% of the participants agreed that the pump’s collaborative MR platform was essential for studying and understanding the complexity and intricacy of the digital pump’s mechanical structure. Overall, the results demonstrate that the MR platform effectively facilitates the visualization of the complex pump’s internal structure, inspection of the assembly of each of the involved subsystems, and testing the applicability of the complicated mechanisms.

Traditional Mixed Reality(MR) and Augmented Reality(AR) devices support a wide gamut of sensors, but the limited computational resource onboard such devices make advanced tasks difficult. With the introduction of Edge computing and more advanced Edge hardware, we are no longer bound to just the onboard processor or the Cloud as our only source of computational power. In our work, we introduce the use of Edge with MR devices to provide a cooperative perception capability to the MR device. We base our approach on the portability and low latency of the Edge. Through our prototype system, we demonstrate the potential of devices and evaluate the performance and feasibility through real world trials. Our evaluation proves that the system is capable of supporting cooperative perception tasks.

Mixed reality (MR) aims at combining virtual reality with the user’s surrounding real environment in a way that they form one, coherent reality. Some sectors have started to use mixed reality such as factory, business, entertainment, education, medical, research, manufacture, and many more. This makes mixed reality systems evolved towards an overly complex system that requires high computational power, especially on medical and gaming. Another problem is current mixed reality devices still need more improvements in some aspects to fill the needs of gaming such as performance, latency, graphic quality, and gaming experience. A possible way to solve this problem is to utilize a cloud computing platform. Mixed reality devices today still use its own resources to do various process such as compute, graphical rendering, and another processes. Moving the gaming process such as graphics rendering towards the cloud services might reduce the processing load on the mixed reality devices such as rendering and computing. With moving those processes, also might reduce the usage of the device storage. Thus, in this paper, we explore the feasibility and potential of such an approach according to various related researches. The results of this paper are we see any feasibility and potential for implementation.

Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.

Empowering industrial robotic training with kinesthetic learning and digital twins in human-centric industrial systems

Although object detection methods have significant potential to extend the boundaries of mixed reality (MR), they are often difficult to execute on current MR devices that have limited computing resources. In this paper, we propose an edge-assisted object detection method that uses the eye-tracking function of MR devices to determine whether an object detection task should be offloaded to an edge server for processing and thus reduce the MR device processing load. We implement and evaluate the proposed method using a Microsoft HoloLens 2 to confirm its effectiveness.