Augmented Visual Instruction for Surgical Practice and Training

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Computer visualization can effectively deliver instructions to a user whose task requires understanding of a real world scene. Consider the example of surgical telementoring, where a general surgeon performs an emergency surgery under the guidance of a remote mentor. The mentor guidance includes annotations of the operating field, which conventionally are displayed to the surgeon on a nearby monitor. However, this conventional visualization of mentor guidance requires the surgeon to look back and forth between the monitor and the operating field, which can lead to cognitive load, delays, or even medical errors. Another example is 3D acquisition of a real-world scene, where an operator must acquire multiple images of the scene from specific viewpoints to ensure appropriate scene coverage and thus achieve quality 3D reconstruction. The conventional approach is for the operator to plan the acquisition locations using conventional visualization tools, and then to try to execute the plan from memory, or with the help of a static map. Such approaches lead to incomplete coverage during acquisition, resulting in an inaccurate reconstruction of the 3D scene which can only be addressed at the high and sometimes prohibitive cost of repeating acquisition.Augmented reality (AR) promises to overcome the limitations of conventional out-of-context visualization of real world scenes by delivering visual guidance directly into the user's field of view, guidance that remains in-context throughout the completion of the task. In this thesis, we propose and validate several AR visual interfaces that provide effective visual guidance for task completion in the context of surgical telementoring and 3D scene acquisition.A first AR interface provides a mentee surgeon with visual guidance from a remote mentor using a simulated transparent display. A computer tablet suspended above the patient captures the operating field with its on-board video camera, the live video is sent to the mentor who annotates it, and the annotations are sent back to the mentee where they are displayed on the tablet, integrating the mentor-created annotations directly into the mentee's view of the operating field. We show through user studies that surgical task performance improves when using the AR surgical telementoring interface compared to when using the conventional visualization of the annotated operating field on a nearby monitor. A second AR surgical telementoring interface provides the mentee surgeon with visual guidance through an AR head-mounted display (AR HMD). We validate this approach in user studies with medical professionals in the context of practice cricothyrotomy and lower-limb fasciotomy procedures, and show improved performance over conventional surgical guidance. A comparison between our simulated transparent display and our AR HMD surgical telementoring interfaces reveals that the HMD has the advantages of reduced workspace encumbrance and of correct depth perception of annotations, whereas the transparent display has the advantage of reduced surgeon head and neck encumbrance and of annotation visualization quality. A third AR interface provides operator guidance for effective image-based modeling and rendering of real-world scenes. During the modeling phase, the AR interface builds and dynamically updates a map of the scene that is displayed to the user through an AR HMD, which leads to the efficient acquisition of a five-degree-of-freedom image-based model of large, complex indoor environments. During rendering, the interface guides the user towards the highest-density parts of the image-based model which result in the highest output image quality. We show through a study that first-time users of our interface can acquire a quality image-based model of a 13m $\times$ 10m indoor environment in 7 minutes.A fourth AR interface provides operator guidance for effective capture of a 3D scene in the context of photogrammetric reconstruction. The interface relies on an AR HMD with a tracked hand-held camera rig to construct a sufficient set of six-degrees-of-freedom camera acquisition poses and then to steer the user to align the camera with the prescribed poses quickly and accurately. We show through a study that first-time users of our interface are significantly more likely to achieve complete 3D reconstructions compared to conventional freehand acquisition. We then investigated the design space of AR HMD interfaces for mid-air pose alignment with an added ergonomics concern, which resulted in five candidate interfaces that sample this design space. A user study identified the aspects of the AR interface design that influence the ergonomics during extended use, informing AR HMD interface design for the important task of mid-air pose alignment.

Presently, the quick innovative development has prompted technology assessment that is in accordance with the presence of industrial revolution 4.0. Notwithstanding, the field of education is also not on left behind in the current 4.0 industry development which directed to education 4.0. There are 9 pillars of revolution 4.0 and essentially, augmented reality and internet of things (IoT) are the important pillars that has been used for this research. Likewise, revolution 4.0 is an innovation that moves from the digital age to the physical cyber era. With the launching of Malaysia Education Development Plan 2015-2025 for higher education steps have been taken to push the Technical and Vocational Education and Training (TVET) which includes special intervention in teaching and learning in this field. Similarly, augmented reality being the interactive method of technology that has been hypnotize peoples especially generation Z. In this regard, 21st Century learning methods are subjected to the technology based in order to attract the student. Consequently, the researchers have made a study on augmented reality technology approach towards Mechanical Workshop practice 1 especially in welding workshop, machine workshop and fitting workshop among the Mechanical engineering students in Politeknik Tuanku Syed Sirajuddin, Arau, Malaysia in order to produce an innovation that can be impacted and interactive to the students. As a result, the innovation of Mechanical Workshop Practice with Augmented Reality approach has been created. Product development of mechanical practice has been done using ADDIE model approach. Hence, this innovation shows that augmented technology was suitable to adapt in 21th century learning method that can be impacted to the students. This innovative product has wide opportunity to be marketed to all Polytechnics in Malaysia and all Malaysian institutes of higher education which have Mechanical Engineering department.

Recupera algunos de los elementos menos estudiados y reconocidos en el ensayo latinoamericano: su profunda vocación de reflexión, diálogo y mediación intelectual; su plasticidad y apertura a la búsqueda del sentido; su vínculo con la sociabilidad, la conversación y la generación de un espacio simbólico de encuentro; su capa cidad de establecer una nueva retroalimentación entre lectura y escritura.

Inherent to the field of computer-aided surgery (CAS) is the necessity to handle sophisticated equipment in the operating room; an undesired side-effect of this development is the fact that the surgeon’s attention is drawn from the operating field since surgical progress is partially monitored on the computer’s screen. The overlay of computer-generated graphics over a real-world scene, also referred to as augmented reality (AR), provides a possibility to solve this problem. The considerable technical problems associated with this approach such as viewing of the scenery within a common focal range on the head-mounted display (HMD) or latency in display on the HMD have, however, kept AR from widespread usage in CAS. In this paper, the concept of the Varioscope AR, a lightweight head-mounted operating microscope used as a HMD, is introduced. The registration of the patient to the preoperative image data as well as preoperative planning take place on VISIT, a surgical navigation system developed at our hospital. Tracking of the HMD and stereoscopic visualisation take place on a separate POSIX.4 compliant realtime operating system running on PC hardware. While being in a very early stage of laboratory testing, we were able to overcome the technical problems described above; our work resulted in an AR visualisation system with an update rate of 6 Hz and a latency below 130 ms. First tests with 2D/3D registration have shown a match between 3D world coordinates and 2D HMD display coordinates in the range of 1.7 pixels. It integrates seamlessly into a surgical navigation system and provides a common focus for both virtual and real world objects. On the basis of our current results, we conclude that the Varioscope AR with the realtime visualisation unit is a major step towards the introduction of AR into clinical routine.

Situation-dependent remote AR collaborations: Image-based collaboration using a 3D perspective map and live video-based collaboration with a synchronized VR mode

This chapter introduces design and principles of head mounted displays (HMDs), as well as their state-of-the-art examples, for augmented reality (AR). Section 2 introduces a brief history of head mounted displays, human vision system, and application examples of see-through HMDs. Section 3 describes designs and principles of HMDs, such as typical configurations of optics, typical display elements, and major categories of HMDs. Section 4 gives typical characteristics of HMDs, such as resolution, field of view, and distortion. Section 5 describes human perceptual and health issues such as depth perception and safety. Finally, Section 6 gives conclusions with future challenges and prospects. For researchers, learners and HMD developers, this chapter is a good start point to learn basics, state of the art technologies, and future research directions of HMDs. For system developers and end-users, this chapter will give a good insight to HMDs to choose a suitable HMD for their purposes.

In this article, we present a client/server augmented reality (AR) system for viewing complex assembly models on mobile phones. Complex model information is located on a PC which takes care of all of the heavy AR tracking and rendering computation. A camera phone is used as a client to show this information, augmented on still images as animated view. The mobile phone interface also supports correct masking of the assembly pieces as they come together, as well as a graphics only viewing mode intended for better understanding of the assembly process. In addition to describing the mobile augmented assembly system we present results from two pilot user studies evaluating elements of the user interface. INTRODUCTION In order to remain competitive, today’s industry is employing a growing number of product variations with shorter and shorter life cycles, requiring efficient ways to handle the production planning and processing. Assembly line workers face more and more complex tasks and an increasing memory load is posed by the new production needs. Experienced workers may be able to handle several product variants at the same time, but new workers require training and guidance to handle the work tasks. Related activities such as maintenance and repair also require a lot of know-how of complex products. Guidance in assembly tasks is commonly provided by using printed blueprints or other documents. However, these are mainly two dimensional images or text while the real assembly task is three dimensional, making the relationship between parts hard to understand. Computer based technologies have been developed such as electronic guides, multimedia training material and interactive documents. However, these typically require the user to turn away from the assembly task to study a computer screen separate from the workspace. In our research we explore how augmented reality (AR) technology can be used to provide more intuitive guides to assist with assembly, repair and training tasks. With AR technology, synthetic objects can be merged with the user’s view of the real world so that the user can perform the assembly task without needing to look at a separate screen. The virtual objects appear in the correct position so that AR enables the creation of realistic looking 3D animated assembly “manuals”. Other researchers have developed prototype AR systems for supporting assembly tasks. These are typically based on head mounted displays (HMDs) connected to desktop or wearable computers. However, wearable computers are relatively expensive, they have a short battery life, and taking them into a harsh industrial environment is not always possible. Head mounted displays can also be bulky, they involve safety issues, and many workers are reluctant to wear them. To overcome these disadvantages, we are interested in developing a reliable mobile phone based AR assembly system. Today’s mobile phones have fast processors, significant memory, 3D graphics support, and high resolution displays. They contain various accessories that can be used for developing mobile AR applications such as good quality cameras, connectivity by Bluetooth, WLAN and GPRS, and various SAM.700.003.1155

Medical simulations bridge theory and practice in immersive settings. During this unique form of education, learners experience challenging medical scenarios without risking patient harm. The effectiveness of simulations stem from their ability to induce autonomic stress responses, which potentiate memory formation. Educators have adopted virtual reality (VR) and augmented reality (AR) to supplement traditional, mannequin-based simulations. In this Pro-Con commentary article, we discuss the competitive advantages and limitations of VR and AR simulation for medical training. VR uses computer-generated images (CGI) within head-mounted displays (HMDs) to enhance learning. Gamification of medical procedures in VR improves acquisition of complex skills, positioning VR simulation as a pivotal training tool. Participant embodiment while in VR develops non-technical skills such as improved communication and enhanced empathy. The placement of participants within CGI worlds that would otherwise be difficult or impossible to recreate situate VR as the ideal next step in simulation evolution. VR is superior to AR given its lower hardware costs, expansive software libraries, and complete immersion in CGI compared to limited holographic fields of view within AR. By displaying holograms within the participant's natural world, AR is primed to replace screen and VR technologies as the premier learning adjunct. The integration of real-world instruments with AR holograms provides innumerable methods for teaching procedural skills. In addition, AR enhances pre- and intra-operative planning by overlaying holographic, radiographic patient images over their own bodies. Because AR has the unique feature of maintaining direct eye contact between participants, scenarios that improve communication skills are easily created with holograms in the real world, eliminating the need for an entirely CGI VR environment. Immersive simulation efforts should skip the costs of creating VR scenarios and redirect efforts to developing AR simulation software. Although both VR and AR have limitations related to cost and ergonomics, immersive technologies are relatively more affordable and versatile than traditional mannequins and physical simulation centers. As adjuncts to enhance traditional simulation curricula, we advocate for continued research to demonstrate the validity and utility of these immersive technologies.

Introduction Novel medical technologies are necessary to support virtual health in low bandwidth environments. When tele-mentoring is not possible, advanced clinical decision support tools can deliver essential medical knowledge to assist novice clinicians. These tools can fill knowledge gaps in specialty fields, like burn. We developed the Augmented Reality Burn Assist Manager (ARBAM), a novel, comprehensive augmented reality (AR)-based burn management platform to address key tasks in burn care. Methods A randomized 2x2 cross-over design was used to investigate if ARBAM could enhance a user’s ability to care for a simulated burn patient as compared to standard clinical practice guidelines (CPGs). We developed the software for use on a mixed reality, head-mounted device. Participants were clinicians with little to no burn care experience, and were asked to complete 4 key tasks, twice each, using either paper copies of the Joint Trauma System’s CPGs and supplemental worksheets first (Paper) or our AR technology. Tasks included calculating burn size (TBSA), performing medication dosage calculations (MC), managing a burn fluid resuscitation (FR), and performing a simulated escharotomy (EC). All tasks were evaluated for completion time and accuracy. Cross-over analysis of treatment effects used a two-sided, two-sample t-test or Wilcoxon Rank-Sum test as appropriate. Results Eleven participants were randomized: 5 started with Paper and 6 started with AR. Time to perform FR was statistically different (p< 0.05) with AR taking less time than Paper (4.8 vs. 7.0 mins, respectively). FR clinical performance was also better in the AR group with a mean accuracy score of 99% vs. 52% for Paper (p< 0.001). EC completion times were not clinically or statistically different. However, EC accuracy was 93% for AR compared to 55% for Paper (p< 0.001). Although not statistically different, MC performance accuracy was considerably better for AR (91%) than for Paper (66%). No differences between groups were observed for TBSA or MC completion times nor TBSA accuracy. Conclusions Compared to traditional tools, AR may enhance a clinician's ability to complete key tasks in the initial management of a burn patient. This technology may be helpful during disasters when tele-mentoring is not available. Limitations of this study include a small sample size, limited user training on the AR device, and potential carry-over effects. Therefore, generalizations cannot be made about wide-scale usage. Nonetheless, this study showed that AR may be helpful in managing lifesaving burn tasks. Applicability of Research to Practice ARBAM may enhance decision making by novice clinicians when caring for burn patients. Our data demonstrate the potential for AR-based support tools to improve burn care delivery and reduce error rates for novice clinicians, which could lead to reductions in morbidity and mortality. Future research is needed to investigate how AR technology can be used for other lifesaving interventions.

Team communication is crucial in multi-domain operations (MDOs) that require teammates to collaborate on complex tasks synchronously in dynamic unknown environments. In order to enable effective communication in human-robot teams, the human teammate must have an intuitive interface that supports and satisfies the time-sensitive nature of the task for communicating information to and from their robot teammate. Augmented Reality (AR) technologies can provide just such an interface by providing a medium for both active and passive robot communication. In this paper we propose a new Virtual Reality (VR) based framework for authoring AR visualizations, and demonstrate the use of this framework to produce AR visualizations that help facilitate high task performance in synchronized, time-dominant human-robot teaming. In this paper we propose a new framework that uses a virtual reality (VR) simulation environment for developing AR strategies as well as present a AR solution for maximizing task performance in synchronized, time-dominant human-robot teaming. The framework utilizes a Unity-based VR simulator that is run from the first person point of view of the human teammate and overlays AR features to virtually imitate the use of an AR headset in human-robot teaming scenarios. Then, we introduce novel AR visualizations that support strategic communication within teams by collecting information from each teammate and presenting it to the other in order to influence their decision making. Our proposed design framework and AR solution has the potential to impact any domain in which humans conduct synchronized multi-domain operations alongside autonomous robots in austere environments, including search and rescue, environmental monitoring, and homeland defense.

As the use of virtual and augmented reality applications becomes more common, the need to fully understand how observers perceive spatial relationships grows more critical. One of the key requirements in engineering a practical virtual or augmented reality system is accurately conveying depth and layout. This requirement has frequently been assessed by measuring judgments of egocentric depth. These assessments have shown that observers in virtual reality (VR) perceive virtual space as compressed relative to the real-world, resulting in systematic underestimations of egocentric depth. Previous work has indicated that similar effects may be present in augmented reality (AR) as well.This paper reports an experiment that directly measured egocentric depth perception in both VR and AR conditions; it is believed to be the first experiment to directly compare these conditions in the same experimental framework. In addition to VR and AR, two control conditions were studied: viewing real-world objects, and viewing real-world objects through a head-mounted display. Finally, the presence and absence of motion parallax was crossed with all conditions. Like many previous studies, this one found that depth perception was underestimated in VR, although the magnitude of the effect was surprisingly low. The most interesting finding was that no underestimation was observed in AR.

Two main concepts of Head Mounted Displays (HMD) for augmented reality (AR) visualization exist, the optical and video-see through type. Several research groups have pursued both approaches for utilizing HMDs for computer aided surgery. While the hardware requirements for a video see through HMD to achieve acceptable time delay and frame rate seem to be enormous the clinical acceptance of such a device is doubtful from a practical point of view. Starting from previous work in displaying additional computer-generated graphics in operating microscopes, we have adapted a miniature head mounted operating microscope for AR by integrating two very small computer displays. To calibrate the projection parameters of this so called Varioscope AR we have used Tsai's Algorithm for camera calibration. Connection to a surgical navigation system was performed by defining an open interface to the control unit of the Varioscope AR. The control unit consists of a standard PC with a dual head graphics adapter to render and display the desired augmentation of the scene. We connected this control unit to a computer aided surgery (CAS) system by the TCP/IP interface. In this paper we present the control unit for the HMD and its software design. We tested two different optical tracking systems, the Flashpoint (Image Guided Technologies, Boulder, CO), which provided about 10 frames per second, and the Polaris (Northern Digital, Ontario, Canada) which provided at least 30 frames per second, both with a time delay of one frame.

This study investigated the effects of teaching history of computing with videos of historical figures. Augmented reality (AR) techniques were applied to assist student accessing the videos of historical figures while reading a printed textbook. Whenever a student was interested in a historical figure, one could use a tablet PC to scan the figure's picture, and the corresponding video about the person would then be played on the screen. We adapted thirteen videos of historical figures in computer network field and adopted a quasi-experimental method to evaluate the effectiveness of the AR-based learning approach. Two classes of high school students, with a total of 84 students, participated in the experiment. One class of the students used Tablet PCs to access the videos of historical figures, and the other class using traditional didactic instruction served as the control group. The data collected for analysis are students' achievement test scores and answers to a questionnaire, which consists of questions on attitudes toward learning, perspectives of nature of science, and perceptions on the AR activities. Our findings showed that the AR-based historical figure videos helped students comprehend learning contents and promoted their attitudes toward learning. Students appreciated the convenience of using AR tools to access the history videos. Future research should investigate other approaches to integrate AR with the videos of historical figure, and in general, to integrate AR with other media format of computing history.

Introduction: Augmented Reality (AR) systems are systems in which users view and interact with virtual objects overlaying the real world. AR systems are used across a variety of disciplines, i.e., games, medicine, and education to name a few. Optical See-Through (OST) AR displays allow users to perceive the real world directly by combining computer-generated imagery overlaying the real world. While perception of depth and visibility of objects is a widely studied field, we wanted to observe how color, luminance, and movement of an object interacted with each other as well as external luminance in OST AR devices. Little research has been done regarding the issues around the effect of virtual objects’ parameters on depth perception, external lighting, and the effect of an object’s mobility on this depth perception.Methods: We aim to perform an analysis of the effects of motion cues, color, and luminance on depth estimation of AR objects overlaying the real world with OST displays. We perform two experiments, differing in environmental lighting conditions (287 lux and 156 lux), and analyze the effects and differences on depth and speed perceptions.Results: We have found that while stationary objects follow previous research with regards to depth perception, motion and both object and environmental luminance play a factor in this perception.Discussion: These results will be significantly useful for developers to account for depth estimation issues that may arise in AR environments. Awareness of the different effects of speed and environmental illuminance on depth perception can be utilized when performing AR or MR applications where precision matters.