Markerless Registration Based on Natural Features Tracking for Augmented Reality

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Registration is one of the most difficult problems in augmented reality (AR) systems. In this paper, a simple registration method using natural features based on the projective reconstruction technique is proposed. This method consists of two steps: embedding and rendering. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In rendering, the Kanade-Lucas-Tomasi (KLT) feature tracker is used to track the natural feature correspondences in the live video. The natural features that have been tracked are used to estimate the corresponding projective matrix in the image sequence. Next, the projective reconstruction technique is used to transfer the four specified points to compute the registration matrix for augmentation. This paper also proposes a robust method for estimating the projective matrix, where the natural features that have been tracked are normalized (translation and scaling) and used as the input data. The estimated projective matrix will be used as an initial estimate for a nonlinear optimization method that minimizes the actual residual errors based on the Levenberg-Marquardt (LM) minimization method, thus making the results more robust and stable. The proposed registration method has three major advantages: 1) It is simple, as no predefined fiducials or markers are used for registration for either indoor and outdoor AR applications. 2) It is robust, because it remains effective as long as at least six natural features are tracked during the entire augmentation, and the existence of the corresponding projective matrices in the live video is guaranteed. Meanwhile, the robust method to estimate the projective matrix can obtain stable results even when there are some outliers during the tracking process. 3) Virtual objects can still be superimposed on the specified areas, even if some parts of the areas are occluded during the entire process. Some indoor and outdoor experiments have been conducted to validate the performance of this proposed method.

The perceptual issues in Augmented Reality (AR) systems, is drawing more and more attention these days. Among the perceptual issues, the incorrect depth interpretations degrade user experience significantly. One way to improve the depth interpretations in AR systems is to improve the consistency of the real scene and the virtual objects by using the depth cue method based on blurring effect. In the research of [Okumura 2006], a blur rendering method based on measuring the defocusing on the AR marker from the captured real scene has been proposed. However, the blur effect could only be rendered on the marker position where the blur in the real scene is measured. This limits the practicality of the system since in many applications the virtual objects are out of the marker's range or even moving. In our work, a new method to estimate the degree of blur is proposed. This method makes it possible to calculate the degree of blur that varies spatially with the scene.

Target tracking and feature detection in 3D registration of augmented reality(AR) system are vulnerable to environment, time-consuming algorithm and low accuracy. An AR tracking registration method based on improved Kernerlized Correlation Filter(KCF) KCF tracking and ORB (Oriented FAST and Rotated BRIEF) feature detection is proposed. Firstly, KCF algorithm is employed to track the registered target position. By considering the strategy of all previous frames, the problem of real-time updating of the weight vector of the target model in KCF tracking is solved, which makes the tracking of the target position real-time and effective. ORB operator with high real-time performance is used to detect the target position. Then the feature points of the detected target position are matched, and the three-dimensional registration matrix is calculated according to the matched feature points, and the three-dimensional registration is carried out. The experimental results show that the improved method achieves better real-time performance, stability and robustness.

The major challenges of augmented reality (AR) systems and virtual reality (VR) systems are varifocal images for vergence-accommodation conflict (VAC) and vision corrections. In this paper, we design a liquid crystal (LC) lens set consisting of three LC lenses for varifocal images and vision corrections in AR and VR. Four operating modes of such a LC lens set present three electrically tunable lens powers: 0, -0.79 diopters, -2 diopters, and -3.06 diopters by means of manipulation of polarization of incident light using electrically tunable half-wave-plates. The response time is fast(< 50 ms). We also demonstrate AR and VR systems by adopting the LC lens set to exhibit functions of varifocal images and vision corrections which enable to solve VAC as well as vision problem in AR and VR.

Teaching, learning and training in assembly or disassembly as well as maintenance operation to new users, employees, engineers or technicians is important for various education sector and industries for them to acquire new skills. However to train those users to efficiently perform new skills is become challenging. It is important to reduce the user's training and working times to avoid cost issues in training, material, unavailable expertise and avoid actual part damage. Therefore, this study is conducted to design and develop a Virtual Reality (VR) and Augmented Reality (AR) system for learning and training platform. The valve assembly system was selected as a task for mechanical assembly system. This study was designed to find the influence of interactive learning method using AR system. The outcome from this study showed that learning and training in AR and VR system resulted in better to user skill performance. User learn and trained in a ARVR group gives positive feedback; very easy, practical and as a solution of the training process without using the actual one. While learning using interactive learning method such as AR resulted with better knowledge with less assembly error.

Augmented reality (AR) systems can be effectively used to enhance manufacturing and industrial processes. However, not all the existing prototypes of AR systems can be used in an industrial environment due to heavy constraints such as low robustness or cumbersome equipment. Our AR system relies purely on passive techniques to solve the real-time registration problem, and it can run on a portable computer. We combined a powerful virtual reality (VR) component-based simulation framework with computer vision techniques, turning it into an AR system. The resulting system allows us to produce complex rendering and animation of avatars, and blend them into the real world. The system tracks the 3D camera position by means of a natural features tracker, which, given a rough CAD model, can deal with complex 3D objects. The tracking method is robust and can handle large camera displacements and aspect changes. The target applications of our AR system are industrial maintenance, repair and training. The tracking robustness makes the AR system able to work in real environments, such as industrial facilities, and not only in the laboratory.

Marker registration is a fundamental process to estimate camera poses in marker-based Augmented Reality (AR) systems. We developed AR system that creates correspondence virtual objects on handwritten text markers. This paper presents a new method for registration that is robust for low-content text markers, variation of camera poses, and variation of handwritten styles. The proposed method uses Maximally Stable Extremal Regions (MSER) and polygon simplification for a feature point extraction. The experiment shows that we need to extract only five feature points per image which can provide the best registration results. An exhaustive search is used to find the best matching pattern of the feature points in two images. We also compared performance of the proposed method to some existing registration methods and found that the proposed method can provide better accuracy and time efficiency.

INTRODUCTION: Augmented Reality (AR) offers the benefit of improved hand-eye coordination by accurately superimposing 3D stereoscopic displays for surgical instruments onto anatomic landmarks. METHODS: 120 pedicle screws were implanted using an AR system (OnPoint AR, OnPoint Surgical, Bedford, MA) (thoracic 50, lumbosacral 70; pre-op CT 40, O-arm 80). Heary grading was applied by an independent radiologist assessing the degree of pedicle breaches. 3D measurements of positional error (PE) and angular error (AE) were performed by comparing post-operative CT scans of implanted screws with planned trajectories. Statistical analyses compared PE and AE for OnPoint AR with the available values for other systems using an unequal variance t-test method. RESULTS: 113 screws placed demonstrated Heary grade 1 accuracy. 6 screws had a larger diameter than the pedicle and were planned with in-out-in technique with Heary grade 2 accuracy. One screw demonstrated a lateral breach (grade 2) caused by a loose connection to the screwdriver. PE and AE (mean ± 1SD) for OnPoint AR were the following: for intra-operative spin 1.1mm±0.5mm and 1.3°±0.5°; for pre-operative CT 1.9mm±1.2mm and 2.2°±1.3°. Percent increment in PE and AE for Medtronic Stealth Station, Brainlab navigation, Stryker nav3i, Medtronic Mazor X robot, Globus Excelsius robot, Augmedics Xvision, and Novarad VisAR compared to OnPoint AR ranged between 40–80%, with differences being highly significant for all comparisons. CONCLUSIONS: PE and AE for placing pedicle screws are statistically significantly smaller for the novel AR system compared to existing navigation, robotic, and AR systems. A positional error of 1.1mm and angular error of 1.3° makes this technology particularly suited for small pedicles, challenging anatomy, and MIS techniques.

Augmented reality (AR) is a new human-computer interaction technology that combines virtual reality, computer vision, and computer networks. With the rapid advancement of the medical field towards intelligence and data visualisation, AR systems are becoming increasingly popular in the medical field because they can provide doctors with clear enough medical images and accurate image navigation in practical applications. However, it has been discovered that different display types of AR systems have different effects on doctors' perception of the image after virtual-real fusion during the actual medical application. If doctors cannot correctly perceive the image, they may be unable to correctly match the virtual information with the real world, which will have a significant impact on their ability to recognise complex structures. This paper uses Citespace, a literature analysis tool, to visualise and analyse the research hotspots when AR systems are used in the medical field. A visual analysis of the 1163 articles retrieved from the Web of Science Core Collection database reveals that display technology and visualisation technology are the key research directions of AR systems at the moment. This paper categorises AR systems based on their display principles, reviews current image perception optimisation schemes for various types of systems, and analyses and compares different display types of AR systems based on their practical applications in the field of smart medical care so that doctors can select the appropriate display types based on different application scenarios. Finally, the future development direction of AR display technology is anticipated in order for AR technology to be more effectively applied in the field of smart medical care. The advancement of display technology for AR systems is critical for their use in the medical field, and the advantages and disadvantages of various display types should be considered in different application scenarios to select the best AR system.

Architects conduct site visits prior to a design activity to understand existing conditions. If the architect’s position and orientation are known on site, and augmented reality system has access to a location based content database of the site, then augmented reality system can display the content in 3D directly upon the architect’s view. Generally, architects use augmented reality as a visualization tool for presentation. It is also possible to collect data of a site and represent it in situ for architectural design. This paper is a survey of location based data representation in augmented reality systems to use in early stages of architectural design related to site. Initially, it describes the field of augmented reality including the characteristics and requirements. Then it surveys the state of the art by reviewing featured applications of location based augmented reality technology. Developments to the recent conditions from the first implementations have been revealed with components. At the same time, this paper aims to find common links between these featured applications and architectural site survey. Thus, it discusses opportunities of augmented reality to provide the needs of an architect as a site visit. However, it suggests which augmented reality components are more suitable in recent conditions for use in architectural design related to site. As a survey paper, it focuses on how location based augmented reality can be used in architectural design instead of presenting a model or an application.

Virtual reality (VR) and augmented reality (AR) systems are increasingly available to neurosurgeons. These systems may provide opportunities for technical rehearsal and assessments of surgeon performance. The assessment of neurosurgeon skill in VR and AR environments and the validity of VR and AR feedback has not been systematically reviewed. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was conducted through MEDLINE and PubMed. Studies published in English between January 1990 and February 2021 describing the use of VR or AR to quantify surgical technical performance of neurosurgeons without the use of human raters were included. The types and categories of automated performance metrics (APMs) from each of these studies were recorded. Thirty-three VR studies were included in the review; no AR studies met inclusion criteria. VR APMs were categorized as either distance to target, force, kinematics, time, blood loss, or volume of resection. Distance and time were the most well-studied APM domains, although all domains were effective at differentiating surgeon experience levels. Distance was successfully used to track improvements with practice. Examining volume of resection demonstrated that attending surgeons removed less simulated tumor but preserved more normal tissue than trainees. More recently, APMs have been used in machine learning algorithms to predict level of training with a high degree of accuracy. Key limitations to enhanced-reality systems include limited AR usage for automated surgical assessment and lack of external and longitudinal validation of VR systems. VR has been used to assess surgeon performance across a wide spectrum of domains. The VR environment can be used to quantify surgeon performance, assess surgeon proficiency, and track training progression. AR systems have not yet been used to provide metrics for surgeon performance assessment despite potential for intraoperative integration. VR-based APMs may be especially useful for metrics that are difficult to assess intraoperatively, including blood loss and extent of resection.

Introduction: Augmented Reality (AR) systems are systems in which users view and interact with virtual objects overlaying the real world. AR systems are used across a variety of disciplines, i.e., games, medicine, and education to name a few. Optical See-Through (OST) AR displays allow users to perceive the real world directly by combining computer-generated imagery overlaying the real world. While perception of depth and visibility of objects is a widely studied field, we wanted to observe how color, luminance, and movement of an object interacted with each other as well as external luminance in OST AR devices. Little research has been done regarding the issues around the effect of virtual objects’ parameters on depth perception, external lighting, and the effect of an object’s mobility on this depth perception.Methods: We aim to perform an analysis of the effects of motion cues, color, and luminance on depth estimation of AR objects overlaying the real world with OST displays. We perform two experiments, differing in environmental lighting conditions (287 lux and 156 lux), and analyze the effects and differences on depth and speed perceptions.Results: We have found that while stationary objects follow previous research with regards to depth perception, motion and both object and environmental luminance play a factor in this perception.Discussion: These results will be significantly useful for developers to account for depth estimation issues that may arise in AR environments. Awareness of the different effects of speed and environmental illuminance on depth perception can be utilized when performing AR or MR applications where precision matters.

Endoscopic surgery causes less tissue injury compared to open surgical techniques, thus promoting more rapid recuperation and reduced post-operative pain. Endoscopy, however, allows the surgeon to visualise only the anatomical surface of the surgical site, with a relatively narrow field of view. Moreover, the 2D video captured by the conventional endoscope does not provide depth perception of the surgical scene. In this study, these limitations have been addressed with the development of an augmented reality (AR) system with stereoscopic visualisation. A phantom and its 3D CT model were used, respectively, to form the real and virtual parts of the AR. The virtual environment camera pose was tracked using algorithms for image feature detection, feature matching and Perspective-n-Point applied on the endoscopic image and the 3D virtual model-rendered image. The endoscope video frame- and the virtual model-rendered images were superimposed to form the AR composite view. The depth buffer (z-buffer) of the rendering window was further used to make a stereo pair of the AR image. The AR system produced a stereo composite view having well-aligned real and virtual components. The RMS error of the real and virtual image contours registration was 9.6 ± 6.7 mm. Correlation coefficients between the depth map from z-buffer and a depth camera was between 0.60 and 0.96 (p < 0.05). The AR system requires further improvement to be applicable at a higher frame rate of endoscope image acquisition. It also needs to include motion and deformation models when applied to animals or patients.

Augmented Reality (AR) is one of the latest developments in human-computer interaction technology. It aims to generate illusions from seamless fusion of virtual objects and real world. Typical AR system requires two basic parts: three-dimensional registration and real-virtual fusion. Occlusion handling is crucial for visual realism. To optimize visual realism, we generated a real-time systematic architecture to operate occlusion handling. The architecture is based on RGBD images, and it consists of three parts: real-time camera tracking system, 3D reconstruction system and AR fusion system. Specifically, we used a two-pass scheme strategy to execute the AR system. The first pass tracks camera poses timely at video rate, which allows the reconstruction results be updated and visualized correspondingly during the scanning. The second pass takes place simultaneously to handle occlusion between virtual objects and real scene according to camera pose. Finally, the render results of virtual objects and the color images are fused to generate AR contents. Our results indicate that this method is stable and precise for occlusion handling, and can effectively improve realism in AR system.