Designing Interactive Robotic Games based on Mixed Reality Technology

Digital hydraulics is a discrete technology that integrates advanced dynamic system controls, digital electronics, and machine learning to enhance fluid power systems’ performance, overall efficiency, and controllability. A mechanically actuated inline three-piston variable displacement digital pump was previously proposed and designed. The inline three-piston pump incorporates complex mechanical and hydraulic subsystems and highly coupled mechanisms. The complexity of the utilized subsystems poses challenges when assessing the viability of the conceptual design. Therefore, this work focuses on designing, developing, and implementing a collaborative virtual platform involving a digitized module showcasing the internal mechanical structure of the digital pump utilizing mixed reality (MR) technology. MR technology is acknowledged as the forthcoming evolution of the human–machine interface in the real–virtual environment utilizing computers and wearables. This technology permits running simulations that examine the complexity of highly coupled systems, like the digital pump, where understanding the physical phenomenon is far too intricate. The developed MR platform permits multiple users to collaborate in a synchronized immersive MR environment to study and analyze the applicability of the pump’s design and the adequacy of the operated mechanisms. The collaborative MR platform was designed and developed on the Unity game engine, employing Microsoft Azure and Photon Unity Networking to set up the synchronized MR environment. The platform involves a fully interactive virtual module on the digital pump design, developed in multiple stages using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity and deployed in the synchronized MR environment through a HoloLens 2 MR headset. A research study involving 71 participants was carried out at Purdue University. The study’s objective was to explore the impact of the collaborative MR environment on understanding the complexity and operation of the digital pump. It also sought to assess the effectiveness of MR in facilitating collaboration among fluid power stakeholders in a synchronized digital reality setting to study, diagnose, and control their complex systems. Surveys were designed and completed by all 71 participants after experiencing the MR platform. The results indicate that approximately 75% of the participants expressed positive attitudes toward their overall MR platform experience, with particular appreciation for its immersive nature and the synchronized collaborative environment it provided. More than 70% of the participants agreed that the pump’s collaborative MR platform was essential for studying and understanding the complexity and intricacy of the digital pump’s mechanical structure. Overall, the results demonstrate that the MR platform effectively facilitates the visualization of the complex pump’s internal structure, inspection of the assembly of each of the involved subsystems, and testing the applicability of the complicated mechanisms.

BackgroundThis was a prospective comparative study of mixed reality (MR) technology assisted lumbar pedicle screws placement and traditional lumbar pedicle screws placement.Material/MethodsFifty cases of lumbar pedicle screws placement were randomly divided into 2 groups: 25 cases with MR technology in group A, and 25 cases without MR technology in group B. All patients had their scores on the Oswestry disability index (ODI) of low back pain and the visual analog scale (VAS) of the affected lower limb recorded at pre-operation. Blood loss, operative duration, success rate of first penetration by tap, and number of times C-arm fluoroscopy was performed were recorded at intraoperation. The postoperative drainage was recorded. The ODI of low back pain and VAS of the affected lower limb were recorded at 1, 3, and 6 months after operation.ResultsGroup A had less bleeding, shorter operation time, higher success rate of first penetration by tap, and fewer times using C-arm fluoroscopy at intraoperation (P<0.05). There was significant difference in ODI scores and VAS scores at 1 mouth after operation (P<0.05). The postoperative drainage of group A was less than group B (P<0.05). The implantation accuracy of group A was higher than group B (P<0.05). The postoperative recovery rate of low back pain of group A was faster than group B (P<0.05).ConclusionsThe safety of spinal surgery and implantation accuracy of pedicle screw fixation system could be increased by MR technology.

The steady increase in the aging population worldwide is expected to cause a shortage of doctors and therapists for older people. This demographic shift requires more efficient and automated systems for rehabilitation and physical ability evaluations. Rehabilitation using mixed reality (MR) technology has attracted much attention in recent years. MR displays virtual objects on a head-mounted see-through display that overlies the user's field of vision and allows users to manipulate them as if they exist in reality. However, tasks in previous studies applying MR to rehabilitation have been limited to tasks in which the virtual objects are static and do not interact dynamically with the surrounding environment. Therefore, in this study, we developed an application to evaluate cognitive and motor functions with the aim of realizing a rehabilitation system that is dynamic and has interaction with the surrounding environment using MR technology. The developed application enabled effective evaluation of the user's spatial cognitive ability, task skillfulness, motor function, and decision-making ability. The results indicate the usefulness and feasibility of MR technology to quantify motor function and spatial cognition both for static and dynamic tasks in rehabilitation.

ObjectiveTo provide an overview of the current state of mixed reality (MR) technology’s application in medical education and to review a research scope of the area.MethodsAccording to the Joanna Briggs Institute’s scope-based review procedures, searches were conducted in the following databases: PubMed, Embase, Web of Science Core Collection, Cochrane Library, CNKI, WanFang Database, and VIP Database. The included literatures were compiled and examined, and the search period was limited to the period from the database’s creation to April 20, 2025.ResultsIn all, 4118 records were found. After eligibility assessment, 28 studies in English or Chinese that examine the use of MR in education for medical students were included. Data extraction focused on theoretical/skills test scores, students’ satisfaction, testing and evaluation of the system operation as well as students’ learning interest of MR. The majority of studies indicated MR technology effectively enhances students’ theoretical knowledge and skill proficiency. Eighteen studies consistently reported a high students’ satisfaction level with MR-assisted instruction; MR systems demonstrated favorable usability due to its high consistency. Ten studies showed students in the MR-assisted learning groups exhibit heightened learning interest.ConclusionMR Technology has demonstrated excellent performance in enhancing students’ mastery of theories/skills, improving students’ satisfaction, and increasing students’ interest. However, it has not yet been widely adopted due to the high costs associated with teacher training and the purchase and maintenance of equipment. The future should prioritize continuously developing the immersive experience and engagement of MR technology, integrating ecological instantaneous evaluation methodologies and artificial intelligence technology, and combining economic benefit analysis and feasibility testing to perform higher-quality original research.

In this paper, we report findings from a systematic mapping study, conducted to review the existing literature on collaborative educational environments incorporating mixed reality technologies. There is increasing interest in mixed reality technologies in education, especially with the introduction of new over head mounted displays (OHMDs), such as HoloLens, Oculus Rift, and HTC Vive. with the consideration of areas, such as education, dynamic technology, and complex environments, a research area is identified. We carried out an extensive review of the literature from 2007 to 2017 and conducted an analysis of the works on mixed reality technologies and its subcategories applied to collaborative education environments. Results highlighted the lack of research across the mixed reality spectrum, especially in the augmented virtuality subcategory, as well as technical limitations such as response time in the development of mixed reality technologies for collaborative environments. Furthermore, the difficulty of teaching professionals to replicate mixed reality experiments in real environments, due to the technical skills required, was identified. The main contribution of this paper is the discussion of the current works with visualization of the present state of the area, which is aimed to encourage educators to develop mixed reality artefacts and conduct further research to support collaborative educational environments.

The integration of mixed reality (MR) technology into physical systems represents a significant advancement in bridging the gap between educational environments and industrial applications. This study explores the development of a robotic arm prototype controlled through MR technology, aimed at enhancing educational experiences in the field of robotics. By utilizing augmented reality (AR), virtual reality (VR), and real-time physical interactions, students gain a comprehensive understanding of robotic systems. The project combines open-source platforms with MR to provide a hands-on, immersive learning environment. The primary objective is to foster high-impact learning and practical training, thus reducing the gap between educational settings and real-world industrial applications.

Mobile robots start to appear in everyday life, e.g., in shopping malls or nursing homes. Often, they are operated by staff with no prior experience in robotics. False expectations regarding the capabilities of robots, however, may lead to disappointments and reservations when it comes to accepting robots in ones personal space. We make use of state-of-the-art Mixed Reality (MR) technology by integrating a Microsoft HoloLens into the robot»s operating space to facilitate acceptance and interaction. The MR device is used to increase situation awareness by (a) externalizing the robot»s behavior-state and sensor data and (b) projecting planned behavior. In addition to that, MR technology is (c) used as satellite sensing and display device for human-robot-communication.

Mixed-reality (MR) technology has the potential to enhance care delivery, but there remains a paucity of evidence for its efficacy and feasibility. To assess the efficacy and feasibility of MR technology to enhance emergency care delivery in a simulated environment. This pilot randomized crossover trial was conducted from September to November 2021 at a single center in a high-fidelity simulated environment with participants block randomized to standard care (SC) or MR-supported care (MR-SC) groups. Participants were 22 resident-grade physicians working in acute medical and surgical specialties prospectively recruited from a single UK Academic Health Sciences Centre. Data were analyzed from September to December 2022. Participants resuscitated a simulated patient who was acutely unwell, including undertaking invasive procedures. Participants completed 2 scenarios and were randomly assigned to SC or MR-SC for the first scenario prior to crossover. The HoloLens 2 MR device provided interactive holographic content and bidirectional audiovisual communication with senior physicians in the MR-SC group. The primary outcome was error rate assessed via the Imperial College Error Capture (ICECAP) multidimensional error-capture tool. Secondary outcomes included teamwork (Observational Teamwork Assessment for Surgery [OTAS]; range, 0-6 and Teamwork Skills Assessment for Ward Care [T-SAW-C]; range, 1-5), scenario completion, stress and cognitive load (NASA Task Load Index [NASA-TLX; range 0-100]), and MR device user acceptability. A total of 22 physicians (15 males [68.2%]; median [range] age, 28 [25-34] years) were recruited. MR technology significantly reduced the mean (SD) number of errors per scenario compared with SC (5.16 [3.34] vs 8.30 [3.09] errors; P = .003), with substantial reductions in procedural (0.79 [0.75] vs 1.52 [1.20] errors; P = .02), technical (1.95 [1.40] vs 3.65 [2.03] errors; P = .01), and safety (0.37 [0.96] vs 0.96 [0.85] errors; P = .04) domains. MR resulted in significantly greater scenario completion rates vs SC (22 scenarios [100%] vs 14 scenarios [63.6%]; P = .003). It also led to significant improvements in the overall quality of teamwork and interactions vs SC as measured by mean (SD) OTAS (25.41 [6.30] vs 16.33 [5.49]; P < .001) and T-SAW-C (27.35 [6.89] vs 18.37 [6.09]; P < .001) scores. As reported via mean (range) NASA-TLX score, there were significant reductions for MR-SC vs SC in participant temporal demands (38 [20-50] vs 46 [30-70]; P = .03) and significant improvements in self-reported task performance (50 [30-60] vs 39 [10-70]; P = .01). Overall, 19 participants (86.4%) reported that they were more confident in making clinical decisions and undertaking clinical procedures with MR support. This study found that the use of MR technology reduced error, improved teamwork, and enhanced practitioner confidence when used to support the delivery of simulated emergency medical care. ClinicalTrials.gov Identifier: NCT05870137.

Within the emerging research area represented by robotic gaming and, specifically, in application domains in which the recent literature suggests to combine commercial off-the-shelf (COTS) robots and projected mixed reality (MR) technology in order to develop engaging games, one of the crucial issues to consider in the design process is how to make the player perceive the robot as having a key role, i.e., to valorize its presence from the user experience point of view. By moving from this consideration, this paper reports efforts that are being carried out with the aim to investigate the impact of diverse game design choices in the above perspective, while at the same time extracting preliminary insights that can be exploited to orient further research in the field of MR-based robotic gaming and related scenarios.

PD30-08 EVALUATION OF MIXED REALITY (MR) TECHNOLOGIES FOR REMOTE GUIDANCE DURING ULTRASOUND (US)-GUIDED PERCUTANEOUS RENAL ACCESS SIMULATION: A PROSPECTIVE, RANDOMIZED COMPARATIVE TRIAL

PurposeThe purpose of this study is to explore the effectiveness of mixed reality (MR) technology in enhancing library tours for first-year students at a university academic library. It specifically aims to investigate whether MR tours can reduce library anxiety, improve knowledge acquisition and increase motivation when compared to traditional library tours.Design/methodology/approachThis study uses a quasi-experimental research design, comparing two groups of first-year students (n = 96). One group (n = 48) experiences a MR library tour using the HoloLens 2 device, while the other group (n = 48) participates in a traditional library tour led by library staff. The participants’ library anxiety, knowledge acquisition and motivation are measured using relevant questionnaires before and after the tours.FindingsThe findings indicate that while both MR and traditional tours are effective in imparting knowledge, there is no statistically significant difference in overall knowledge acquisition. However, the MR tour significantly enhances students’ perceived competence, interest and effort, providing higher engagement and motivation. Traditional tours, on the other hand, are more effective in reducing library anxiety, particularly in relation to interactions with librarians.Originality/valueThis study highlights the potential of MR technology to enhance library tours by balancing immersive experiences with ease of use. MR bridges the gap between virtual reality’s immersion and the accessibility, lower cost of content development and simplicity of application usage. It aligns with trends in academic libraries by offering high engagement without common health issues like motion sickness. However, while MR applications can be developed at a relatively lower cost, the high cost of MR equipment remains a limitation for institutions. Despite this, MR presents a promising solution for improving student engagement and learning, with the potential to become more accessible as hardware costs decrease.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Artificial intelligence and mixed reality are realities now, not just ideas from science fiction films. AI analyzes data and can provide interpretation, just like the human brain. The efficiency and accuracy of this are causing the globe to adopt it presently. As opposed to mixed reality (MR), which combines the actual and virtual worlds to create new locations and scenarios, where things are both digital and tangible and interact with each other in real time. With a focus on Indian education, this paper provides a thorough overview of artificial intelligence (AI) and mixed reality technologies and their prospective educational applications. Together, artificial intelligence (AI) and mixed reality can revolutionize education by enhancing learning outcomes and bringing fresh resources to the classroom that have the power to change traditional methods of instruction. Like any new technology, there are hazards involved in using it, though. Experts are even in danger because of its immense power, which has the potential to fundamentally alter several industries and cause a large number of job losses. which has the potential to fundamentally alter several industries and cause a large number of job losses. Keywords -- Mixed Reality, Artificial Intelligence, Indian Education

Mixed reality technology has provided more possibilities for users to interact with both the virtual and real worlds. By integrating virtual elements with real scenes, mixed reality offers a novel immersive experience. Mixed reality also introduces complex perceptual and cognitive challenges when processing conflicting information from the virtual and real worlds. This study investigates the impact of color mismatches on response time in a mixed reality scenario. We recruited 15 participants to identify the font color of words while ignoring their meaning. The experiment included three levels of perceptual-conceptual matching. We performed a global comparison across all conditions, a within-group comparison, and a between-group comparison. Results show that in the mixed reality scenario, the processing of visual information relies on low-level physical properties (e.g., brightness and hue) and is modulated by high-level cognitive factors (e.g., conflict processing and expectation). These findings indicate that visual mismatches in mixed reality can significantly affect response time, particularly when perceptual salience is low and different from cognitive expectations. The results have implications for the design of mixed reality interfaces in complex environments where users must process information from the virtual and real worlds, such as head-up displays, warehouse management systems, and industrial instructional overlays.

Preliminary application of mxed reality in neurosurgery: Development and evaluation of a new intraoperative procedure