Stereoscopic augmented reality for single camera endoscopy: a virtual study

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented reality (AR) systems can be effectively used to enhance manufacturing and industrial processes. However, not all the existing prototypes of AR systems can be used in an industrial environment due to heavy constraints such as low robustness or cumbersome equipment. Our AR system relies purely on passive techniques to solve the real-time registration problem, and it can run on a portable computer. We combined a powerful virtual reality (VR) component-based simulation framework with computer vision techniques, turning it into an AR system. The resulting system allows us to produce complex rendering and animation of avatars, and blend them into the real world. The system tracks the 3D camera position by means of a natural features tracker, which, given a rough CAD model, can deal with complex 3D objects. The tracking method is robust and can handle large camera displacements and aspect changes. The target applications of our AR system are industrial maintenance, repair and training. The tracking robustness makes the AR system able to work in real environments, such as industrial facilities, and not only in the laboratory.

The major challenges of augmented reality (AR) systems and virtual reality (VR) systems are varifocal images for vergence-accommodation conflict (VAC) and vision corrections. In this paper, we design a liquid crystal (LC) lens set consisting of three LC lenses for varifocal images and vision corrections in AR and VR. Four operating modes of such a LC lens set present three electrically tunable lens powers: 0, -0.79 diopters, -2 diopters, and -3.06 diopters by means of manipulation of polarization of incident light using electrically tunable half-wave-plates. The response time is fast(< 50 ms). We also demonstrate AR and VR systems by adopting the LC lens set to exhibit functions of varifocal images and vision corrections which enable to solve VAC as well as vision problem in AR and VR.

Teaching, learning and training in assembly or disassembly as well as maintenance operation to new users, employees, engineers or technicians is important for various education sector and industries for them to acquire new skills. However to train those users to efficiently perform new skills is become challenging. It is important to reduce the user's training and working times to avoid cost issues in training, material, unavailable expertise and avoid actual part damage. Therefore, this study is conducted to design and develop a Virtual Reality (VR) and Augmented Reality (AR) system for learning and training platform. The valve assembly system was selected as a task for mechanical assembly system. This study was designed to find the influence of interactive learning method using AR system. The outcome from this study showed that learning and training in AR and VR system resulted in better to user skill performance. User learn and trained in a ARVR group gives positive feedback; very easy, practical and as a solution of the training process without using the actual one. While learning using interactive learning method such as AR resulted with better knowledge with less assembly error.

Virtual reality (VR) and augmented reality (AR) systems are increasingly available to neurosurgeons. These systems may provide opportunities for technical rehearsal and assessments of surgeon performance. The assessment of neurosurgeon skill in VR and AR environments and the validity of VR and AR feedback has not been systematically reviewed. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was conducted through MEDLINE and PubMed. Studies published in English between January 1990 and February 2021 describing the use of VR or AR to quantify surgical technical performance of neurosurgeons without the use of human raters were included. The types and categories of automated performance metrics (APMs) from each of these studies were recorded. Thirty-three VR studies were included in the review; no AR studies met inclusion criteria. VR APMs were categorized as either distance to target, force, kinematics, time, blood loss, or volume of resection. Distance and time were the most well-studied APM domains, although all domains were effective at differentiating surgeon experience levels. Distance was successfully used to track improvements with practice. Examining volume of resection demonstrated that attending surgeons removed less simulated tumor but preserved more normal tissue than trainees. More recently, APMs have been used in machine learning algorithms to predict level of training with a high degree of accuracy. Key limitations to enhanced-reality systems include limited AR usage for automated surgical assessment and lack of external and longitudinal validation of VR systems. VR has been used to assess surgeon performance across a wide spectrum of domains. The VR environment can be used to quantify surgeon performance, assess surgeon proficiency, and track training progression. AR systems have not yet been used to provide metrics for surgeon performance assessment despite potential for intraoperative integration. VR-based APMs may be especially useful for metrics that are difficult to assess intraoperatively, including blood loss and extent of resection.

Introduction: Augmented Reality (AR) systems are systems in which users view and interact with virtual objects overlaying the real world. AR systems are used across a variety of disciplines, i.e., games, medicine, and education to name a few. Optical See-Through (OST) AR displays allow users to perceive the real world directly by combining computer-generated imagery overlaying the real world. While perception of depth and visibility of objects is a widely studied field, we wanted to observe how color, luminance, and movement of an object interacted with each other as well as external luminance in OST AR devices. Little research has been done regarding the issues around the effect of virtual objects’ parameters on depth perception, external lighting, and the effect of an object’s mobility on this depth perception.Methods: We aim to perform an analysis of the effects of motion cues, color, and luminance on depth estimation of AR objects overlaying the real world with OST displays. We perform two experiments, differing in environmental lighting conditions (287 lux and 156 lux), and analyze the effects and differences on depth and speed perceptions.Results: We have found that while stationary objects follow previous research with regards to depth perception, motion and both object and environmental luminance play a factor in this perception.Discussion: These results will be significantly useful for developers to account for depth estimation issues that may arise in AR environments. Awareness of the different effects of speed and environmental illuminance on depth perception can be utilized when performing AR or MR applications where precision matters.

Abstract. The attempt to work out the geomatic workflow of transforming low-level aerial imagery obtained with unmanned aerial vehicles (UAV) into a digital terrain model (DTM) and implementing the 3D model into the augmented reality (AR) system constitutes the main problem discussed in this article. The authors suggest the following workflow demonstrated in Fig. 1.The series of pictures obtained by means of UAV equipped with a HD camera was the source of data to be worked out in the final stage of the geovisualization. The series was then processed and a few point clouds were isolated from it, being later used for generating test 3D models.The practical aim of the research conducted was to work out, on the basis of the UAV pictures, the 3D geovisualization in the AR system that would depict the heap of the natural aggregate of irregular shape. The subsequent aim was to verify the accuracy of the produced 3D model. The object of the study was a natural aggregate heap of irregular shape and denivelations up to 11 meters.Based on the obtained photos, three point clouds (varying in the level of detail) were generated for the 20 000-meter-square area. The several-centimeter differences observed between the control points in the field and the ones from the model might corroborate the usefulness of the described algorithm for creating large-scale DTMs for engineering purposes. The method of transformation of pictures into the point cloud that was subsequently transformed into 3D models was employed in the research, resulting in the scheme depicting the technological sequence of the creation of 3D geovisualization worked out in the AR system. The geovisualization can be viewed thanks to a specially worked out mobile application for smartphones.

In recent years, hardware and software advances in digital technology have prompted not only the commercial use of augmented reality (AR) systems, but its implementation in education. Marker-based AR systems allow users to input pre-designed markers, and generate new scenes blending together the real world and computer-generated 3D images. Nevertheless, few AR systems permit simultaneous interactions of multiple markers. This study aims to construct an interactive multimedia learning system. The story script showcases the growth of flowers by applying a series of markers, where interactions with 3D objects can be achieved through AR technology. Learners can use a camera to capture the information of multiple AR markers, and 3D objects and media can be generated on-screen. With variations in AR markers, and time changes of the Arduino Clock, one can learn the growth of plants during day and night through a mixed-reality virtual experience that brings fun to learning. Accordingly, this e-learning system is capable of boosting students' learning interests and improving their learning effectiveness.

Augmented Reality (AR) is a technology that can allow a surgeon to see subsurface structures. This works by overlaying information from another modality, such as MRI and fusing it in real time with the endoscopic images. AR has never been developed for a very mobile organ like the uterus and has never been performed for gynecology. Myomas are not always easy to localize in laparoscopic surgery when they do not significantly change the surface of the uterus, or are at multiple locations. To study the accuracy of myoma localization using a new AR system compared to MRI-only localization. Ten residents were asked to localize six myomas (on a uterine model into a laparoscopic box) when either using AR or in conditions that simulate a standard method (only the MRI was available). Myomas were randomly divided in two groups: thecontrol group (MRI only, AR not activated) and theAR group (AR activated). Software was used to automatically measure the distance between the point of contact on the uterine surface and the myoma. We compared these distances to the true shortest distance to obtain accuracy measures. The time taken to perform the task was measured, and an assessment of the complexity was performed. The mean accuracy in the control group was 16.80mm [0.1-52.2] versus 0.64mm [0.01-4.71] with AR. In the control group, the mean time to perform the task was 18.68 [6.4-47.1]s compared to 19.6 [3.9-77.5]s with AR. The mean score of difficulty (evaluated for each myoma) was 2.36 [1-4] versus 0.87 [0-4], respectively, for the control and the AR group. We developed an AR system for a very mobile organ. This is the first user study to quantitatively evaluate an AR system for improving a surgical task. In our model, AR improves localization accuracy.

Concepts for user interfaces are developed in order to enable humans exchanging information and having a better understanding of different processes. With the rise of technology, concepts of enriching our environments with computer-generated graphics appeared in the 1960s. The expansion of visual perception of the surrounding is the core of Augmented Reality (AR). Many AR applications have been appearing in different areas, opening the door for revolutionizing and changing interactivity models and the way people perform tasks. Even though AR systems’ structures are well defined, applications are not yet widespread. One of the main reasons for this is that AR system developers face difficulties with developing their applications and adapting them to different hardware. Each hardware set (i.e. AR device) requires different software development kits and different technical requirements. That leaves AR concepts hard to be scaled and implemented in real life scenarios. One more additional challenge facing the implementing of AR is that AR mostly relies on complex vision-based algorithms. Based on that, this paper explores a lightweight, modular, and cross-platform solution. It presents a novel framework based on redesigning the AR pipeline, introducing a prototype of a solution where AR can operate within the cloud, converting all AR devices into thin clients with high performance capability. This also contributes directly to the Industry 4.0 aim of preparing the way for digitalized and optimized processes in a connected and integrated network.

This paper describes the collaborative augmented reality (AR) system with which multiple users can handwrite 3D lines in the air simultaneously and manipulate the lines directly in the real world. In addition, we propose a new technique for embodied communication utilizing the 3D drawing exercise. Up to now, the various 3D user interfaces have been proposed. Although most of them aim to solve the specific problems in the virtual environments, the possibility of the 3D drawing expression has not been explored yet. Accordingly, we paid special attention to the interaction with the real objects in daily life, and considered to manipulate real objects and 3D lines without any distinctions by the same action. The developed AR system consists of a head-mounted display, a drawing tool, 6DOF sensors, and the 3D user interface, which enables to push, grasp and pitch a 3D lines directly by use of the drawing tool. Additionally users can pick up desired color from either a landscape or a virtual line through the direct interaction with this tool. For sharing 3D lines among multiple users at the same place, the distributed-type AR system has been developed that mutually sends and receives drawn data between systems. With the developed system, users can proceed to design jointly in the real space through arranging each 3D drawing by direct manipulation. Moreover, a new application to the entertainment has become possible to play sports like catch, fencing match, or the like

Augmented reality (AR) is a new human-computer interaction technology that combines virtual reality, computer vision, and computer networks. With the rapid advancement of the medical field towards intelligence and data visualisation, AR systems are becoming increasingly popular in the medical field because they can provide doctors with clear enough medical images and accurate image navigation in practical applications. However, it has been discovered that different display types of AR systems have different effects on doctors' perception of the image after virtual-real fusion during the actual medical application. If doctors cannot correctly perceive the image, they may be unable to correctly match the virtual information with the real world, which will have a significant impact on their ability to recognise complex structures. This paper uses Citespace, a literature analysis tool, to visualise and analyse the research hotspots when AR systems are used in the medical field. A visual analysis of the 1163 articles retrieved from the Web of Science Core Collection database reveals that display technology and visualisation technology are the key research directions of AR systems at the moment. This paper categorises AR systems based on their display principles, reviews current image perception optimisation schemes for various types of systems, and analyses and compares different display types of AR systems based on their practical applications in the field of smart medical care so that doctors can select the appropriate display types based on different application scenarios. Finally, the future development direction of AR display technology is anticipated in order for AR technology to be more effectively applied in the field of smart medical care. The advancement of display technology for AR systems is critical for their use in the medical field, and the advantages and disadvantages of various display types should be considered in different application scenarios to select the best AR system.

With computer vision-based technologies, current Augmented reality (AR) systems can effectively recognize multiple objects with different visual characteristics. However, only limited degrees of distinctions can be offered among different objects with similar natural features, and inherent information about these objects cannot be effectively extracted. In this paper, we propose TaggedAR, i.e., an RFID-based approach to assist the recognition of multiple tagged objects in AR systems, by deploying additional RFID antennas to the COTS depth camera. By sufficiently exploring the correlations between the depth of field and the received RF-signal, we propose a rotate scanning-based scheme to distinguish multiple tagged objects in the stationary situation, and propose a continuous scanning-based scheme to distinguish multiple tagged human subjects in the mobile situation. By pairing the tags with the objects according to the correlations between the depth of field and RF-signals, we can accurately identify and distinguish multiple tagged objects to realize the vision of “tell me what I see” from the AR system. We have implemented a prototype system to evaluate the actual performance with case studies in a real-world environment. The experiment results show that our solution achieves an average match ratio of 91 percent in distinguishing up to dozens of tagged objects with a high deployment density.

INTRODUCTION: Augmented Reality (AR) offers the benefit of improved hand-eye coordination by accurately superimposing 3D stereoscopic displays for surgical instruments onto anatomic landmarks. METHODS: 120 pedicle screws were implanted using an AR system (OnPoint AR, OnPoint Surgical, Bedford, MA) (thoracic 50, lumbosacral 70; pre-op CT 40, O-arm 80). Heary grading was applied by an independent radiologist assessing the degree of pedicle breaches. 3D measurements of positional error (PE) and angular error (AE) were performed by comparing post-operative CT scans of implanted screws with planned trajectories. Statistical analyses compared PE and AE for OnPoint AR with the available values for other systems using an unequal variance t-test method. RESULTS: 113 screws placed demonstrated Heary grade 1 accuracy. 6 screws had a larger diameter than the pedicle and were planned with in-out-in technique with Heary grade 2 accuracy. One screw demonstrated a lateral breach (grade 2) caused by a loose connection to the screwdriver. PE and AE (mean ± 1SD) for OnPoint AR were the following: for intra-operative spin 1.1mm±0.5mm and 1.3°±0.5°; for pre-operative CT 1.9mm±1.2mm and 2.2°±1.3°. Percent increment in PE and AE for Medtronic Stealth Station, Brainlab navigation, Stryker nav3i, Medtronic Mazor X robot, Globus Excelsius robot, Augmedics Xvision, and Novarad VisAR compared to OnPoint AR ranged between 40–80%, with differences being highly significant for all comparisons. CONCLUSIONS: PE and AE for placing pedicle screws are statistically significantly smaller for the novel AR system compared to existing navigation, robotic, and AR systems. A positional error of 1.1mm and angular error of 1.3° makes this technology particularly suited for small pedicles, challenging anatomy, and MIS techniques.

Purpose To explore research relating to the use of Augmented Reality (AR) technology for rehabilitation after stroke in order to better understand the current, and potential future application of this technology to enhance stroke rehabilitation. Methods Database searches and reference list screening were conducted to identify studies relating to the use of AR for stroke rehabilitation. These studies were then reviewed and summarised. Results Eighteen studies were identified where AR was used for upper or lower limb rehabilitation following stroke. The findings of these studies indicate the technology is in the early stages of development and application. No clear definition of AR was established, with some confusion between virtual and augmented reality identified. Most AR systems engaged users in rote exercises which lacked an occupational focus and contextual relevance. User experience was mostly positive, however the poor quality of the studies limits generalisability of these findings to the greater stroke survivor population. Conclusion AR systems are currently being used for stroke rehabilitation in a variety of ways however the technology is in its infancy and warrants further investigation. A consistent definition of AR must be developed and further research is required to determine the possibilities of using AR to promote practice of occupations in a more contextually relevant environment to enhance motor learning and generalisation to other tasks. This could include using AR to bring the home environment into the hospital setting to enhance practice of prioritised occupations before returning home. IMPLICATIONS FOR REHABILITATION There is a developing body of evidence evaluating the use of various forms of AR technology for stroke rehabilitation. User motivation and engagement in rehabilitation may improve with the use of AR. A clear and consistent definition for AR must be developed. Ongoing work could explore how AR systems support engagement in, and promote motor learning that links to, meaningful occupations.