3D optical see-through head-mounted display based augmented reality system and its application

An optical see-through head-mounted display (OST-HMD) can potentially improve the safety and accuracy of ultrasonography (US)-guided fine-needle aspiration. We aimed to evaluate the usefulness of an OST-HMD in US-guided needle-puncture procedures. We conducted a prospective randomized controlled study in which we compared the accuracy and safety of the US-guided needle puncture procedure and the stress on the practitioner when using OST-HMD versus standard US display (SUD). Inexperienced medical students were enrolled and randomly divided into two groups. A breast phantom was used to evaluate the required time and accuracy of the US-guided needle puncture. Practitioner stress was quantified using a visual analog scale (VAS). When the procedure was performed for the first time, the time required to reach the target lesion at a shallow depth was significantly shorter in the OST-HMD group (39.8 ± 39.9 s) than in the SUD group (71.0 ± 81.0 s) (p = 0.01). Using the OST-HMD significantly reduced the unintentional puncture of a non-target lesion (p = 0.01). Furthermore, the stress felt by the practitioners when capturing the image of the target lesion (p < 0.001), inserting and advancing the needle more deeply (p < 0.001), and puncturing the target lesion (p < 0.001) was significantly reduced in the OST-HMD group compared with that in the SUD group. Use of OST-HMD may improve the accuracy and safety of US-guided needle puncture procedures and may reduce practitioner stress during the procedure.

In Augmented Reality (AR) with an Optical See-Through Head-Mounted Display (OST-HMD), the spatial calibration between a user's eye and the display screen is a crucial issue in realizing seamless AR experiences. A successful calibration hinges upon proper modeling of the display system which is conceptually broken down into an eye part and an HMD part. This paper breaks the HMD part down even further to investigate optical aberration issues. The display optics causes two different optical aberrations that degrade the calibration quality: the distortion of incoming light from the physical world, and that of light from the image source of the HMD. While methods exist for correcting either of the two distortions independently, there is, to our knowledge, no method which corrects for both simultaneously. This paper proposes a calibration method that corrects both of the two distortions simultaneously for an arbitrary eye position given an OST-HMD system. We expand a light-field (LF) correction approach [8] originally designed for the former distortion. Our method is camera-based and has an offline learning and an online correction step. We verify our method in exemplary calibrations of two different OST-HMDs: a professional and a consumer OST-HMD. The results show that our method significantly improves the calibration quality compared to a conventional method with the accuracy comparable to 20/50 visual acuity. The results also indicate that only by correcting both the distortions simultaneously can improve the quality.

We propose a varifocal occlusion technique for optical see-through head-mounted displays (OST-HMDs). Occlusion in OST-HMDs is a powerful visual cue that enables depth perception in augmented reality (AR). Without occlusion, virtual objects rendered by an OST-HMD appear semi-transparent and less realistic. A common occlusion technique is to use spatial light modulators (SLMs) to block incoming light rays at each pixel on the SLM selectively. However, most of the existing methods create an occlusion mask only at a single, fixed depth-typically at infinity. With recent advances in varifocal OST-HMDs, such traditional fixed-focus occlusion causes a mismatch in depth between the occlusion mask plane and the virtual object to be occluded, leading to an uncomfortable user experience with blurred occlusion masks. In this paper, we thus propose an OST-HMD system with varifocal occlusion capability: we physically slide a transmissive liquid crystal display (LCD) to optically shift the occlusion plane along the optical path so that the mask appears sharp and aligns to a virtual image at a given depth. Our solution has several benefits over existing varifocal occlusion methods: it is computationally less demanding and, more importantly, it is optically consistent, i.e., when a user loses focus on the corresponding virtual image, the mask again gets blurred consistently as the virtual image does. In the experiment, we build a proof-of-concept varifocal occlusion system implemented with a custom retinal projection display and demonstrate that the system can shift the occlusion plane to depths ranging from 25 cm to infinity.

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic () and maxillofacial surgeries (). For preoperative input data, computed tomography (CT) (), and surface rendered models () were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (); this was achieved by surface tracking of fiducials (), external tracking (), or manual placement (). Microsoft HoloLens devices ( in 2021, in 2022) were the most frequently used OST-HMDs; gestures and/or voice () served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

We propose ARAMIS, a solution to provide real-time “x-ray see-through vision” of a patient’s internal structure to the surgeon, via an optical see-through head-mounted display (OST-HMD), in minimally invasive laparoscopic surgery. ARAMIS takes input imaging from a binocular endoscope, reconstructs a dense point cloud with a GPU-accelerated semi-global matching algorithm on a per-frame basis, and then wirelessly streams the point cloud to an untethered OST-HMD (currently, Microsoft HoloLens) for visualization. The OST-HMD localizes the endoscope distal tip by fusing fiducial-based tracking and self-localization. The point cloud is rendered on the OST-HMD with a custom shader supporting our data-efficient point cloud representation. ARAMIS is able to visualize the reconstructed point cloud (184k points) at 41.27 Hz with an end-to-end latency of 178.3 ms. A user study with 25 subjects, including 2 experienced users, compared ARAMIS to conventional laparoscopy during a peg transfer task on a deformable phantom. Results showed no significant difference in task completion time, but users generally preferred ARAMIS and reported improved intuitiveness, hand-eye coordination and depth perception. Inexperienced users showed a stronger preference for ARAMIS and achieved higher task success rates with the system, whereas the two experienced users indicated a slight preference for ARAMIS and succeeded in all tasks with and without assistance.

This study explores the design, implementation, and evaluation of an Augmented Reality (AR) prototype that assists novice operators in performing procedural tasks in simulator environments. The prototype uses an optical see-through head-mounted display (OST HMD) in conjunction with a simulator display to supplement sequences of interactive visual and attention-guiding cues to the operator’s field of view. We used a 2x2 within-subject design to test two conditions: with/without AR-cues, each condition had a voice assistant and two procedural tasks (preflight and landing). An experiment examined twenty-six novice operators. The results demonstrated that augmented reality had benefits in terms of improved situation awareness and accuracy, however, it yielded longer task completion time by creating a speed-accuracy trade-off effect in favour of accuracy. No significant effect on mental workload is found. The results suggest that augmented reality systems have the potential to be used by a wider audience of operators.

Most state-of-the-art optical see-through head-mounted display (OST-HMD) lacks mutual occlusion capability between computer-rendered and real objects so that the virtual view through an OST-HMD appears "ghost-like", floating in the real world. In this paper, we demonstrated a light-weight, compact OST-HMD with mutual occlusion capability by exploring a highly innovative optical approach based on emerging freeform optical design and fabrication technologies. Our approach enabled us to achieve an occlusion-capable OST-HMD system with a very compelling form factor and high optical performance. The proposed display technology is designed for highly efficient liquid crystal on silicon (LCoS) type spatial light modulator (SLM) and bright Organic LED (OLED) microdisplay, which is capable of working in both indoor and outdoor environments. Our current design offered a 1280×1024 color resolution with a field of view (FOV) of 40 degrees and lightweight optics about 30 grams per eye.

Real-time visualization of imaging data constitutes a critical part of surgical workflow. Augmented reality (AR) is a promising tool to assist in conventional surgical navigation systems. We have been developing an AR framework for clinical imaging and guidance using an optical see-through head-mounted display (OST-HMD) and fluorescence lifetime imaging (FLIm) instrumentation. This framework supports in vivo scanning of FLIm data and the real-time visualization of diagnostic information overlaid on the interrogated tissue area. With the high discriminative power of FLIm, our FLIm-AR concept has the potential for indicating tumor margins and assisting with tumor excision surgery.

The human visual system always focuses at a distinct depth. Therefore, objects that lie at different depths appear blurred, a phenomenon known as Depth of Field (DoF); as the user's focus depth changes, different objects come in and out of focus. Augmented Reality (AR) is a technology that superimposes computer graphics (CG) images onto a user's view of the real world. A commonly used AR display device is an Optical See-Through Head-Mounted Display (OST-HMD), enabling users to observe the real-world directly, with CG added to it. A common problem in such systems is the mismatch between the DoF properties of the user's eyes and the virtual camera used to generate CG.In this demonstration, we present an improved version of the system presented in [11] as two implementations: The first as a high quality tabletop system, the second as a component which has been integrated into the Microsoft Hololens [18].

Utilization of Augmented Reality (AR) technology as a learning medium about perfect and imperfect metamorphosis aims as a tool in the alternative learning and teaching process between teachers and students by visualizing 3D objects in realtime. This multimedia application for introducing perfect and imperfect metamorphosis is based on Augmented Reality by referring to the material in the Science books for Elementary School grade IV (Four) curriculum 2013. The methodology used in developing this application is the Luther Sutopo development method. Where this method consists of 6 (Six) stages, namely concept, design, material collecting, assembly, testing, and distribution. The results or the output of the application that the author built is an android application using Augmented Reality technology to show the shape or form of the stages of animals that experience perfect metamorphosis and imperfect metamorphosis. The conclusion of this study is that the overall response of the end user to this application is quite good. Where in some questions related to this AR application has given good answers. This is evidenced by some of the tests that the authors have conducted on applications where the testers are writers and end users.

Optical see-through head-mounted displays (OHMDs) blend digital content with the physical world, presenting unique color management challenges. Previous literature suggests using green as the main color, but this severely limits creative freedom. To address this, we conducted an empirical study with 30 participants, evaluating 216 colors under various OHMD usage conditions. Based on the results, we propose color guidelines indicating each hue's clear and comfortable saturation and brightness ranges, along with clarity and comfort scores across hues for different devices and lighting conditions. Our color guidelines expand the usable color palette, offering designers a wider range of color options. These guidelines were used and iteratively refined through feedback in a workshop with 12 designers, integrating them into practical design workflows. The resulting comprehensive color guide provides a valuable resource for OHMD interface designers, enhancing both the aesthetic possibilities and functional effectiveness of augmented reality experiences.

In order to align the virtual and real content precisely through augmented reality devices, especially in optical see-through head-mounted displays (OST-HMD), it is necessary to calibrate the device before using it. However, most existing methods estimated the parameters via 3D-2D correspondences based on the 2D alignment, which is cumbersome, time-consuming, theoretically complex, and results in insufficient robustness. To alleviate this issue, in this paper, we propose an efficient and simple calibration method based on the principle of directly calculating the projection transformation between virtual space and the real world via 3D-3D alignment. The proposed method merely needs to record the motion trajectory of the cube-marker in the real and virtual world, and then calculate the transformation matrix between the virtual space and the real world by aligning the two trajectories in the observed view. There are two advantages associated with the proposed method. First, the operation is simple. Theoretically, the user only needs to perform four alignment operations for calibration without changing the rotation variation. Second, the trajectory can be easily distributed throughout the entire observation view, resulting in more robust calibration results. To validate the effectiveness of the proposed method, we conducted extensive experiments on our self-built optical see-through head-mounted display (OST-HMD) device. The experimental results show that the proposed method can achieve better calibration results than other calibration methods.

Optical see-through head-mounted displays (OST HMDs) are a major output medium for Augmented Reality, which have seen significant growth in popularity and usage among the general public due to the growing release of consumer-oriented models, such as the Microsoft Hololens. Unlike Virtual Reality headsets, OST HMDs inherently support the addition of computer-generated graphics directly into the light path between a user's eyes and their view of the physical world. As with most Augmented and Virtual Reality systems, the physical position of an OST HMD is typically determined by an external or embedded 6-Degree-of-Freedom tracking system. However, in order to properly render virtual objects, which are perceived as spatially aligned with the physical environment, it is also necessary to accurately measure the position of the user's eyes within the tracking system's coordinate frame. For over 20 years, researchers have proposed various calibration methods to determine this needed eye position. However, to date, there has not been a comprehensive overview of these procedures and their requirements. Hence, this paper surveys the field of calibration methods for OST HMDs. Specifically, it provides insights into the fundamentals of calibration techniques, and presents an overview of both manual and automatic approaches, as well as evaluation methods and metrics. Finally, it also identifies opportunities for future research.

The ray-shift phenomenon means the apparent distance shift in the display image plane between virtual and physical objects. It is caused by the difference in the refraction of virtual display and see-through optical paths derived from optical combiners that are necessary to provide a see-through capability in optical see-through head-mounted displays. In this work, through a human-subject experiment, we investigated the effects of ray-shift phenomenon induced by the optical combiner on depth perception for near-field distances (40 cm-100 cm). In our experiment, we considered three different configurations of optical combiner: horizontal-tilt and vertical-tilt configurations (using plate beamsplitters horizontally and vertically tilted by 45°, respectively), and non-tilt configuration (using rectangular solid waveguides). Participants' depth perception errors in these configurations were compared with those in an ordinary condition (i.e., the condition where physical objects are directly shown without the displays) and theoretically estimated ones. According to the experimental results, the measured percentage depth perception errors were similar to the theoretically estimated ones, where the amount of estimated percentage depth errors was greater than 0.3%. Furthermore, the participants showed significantly larger depth perception errors in the horizontal-tilt configuration than in an ordinary condition, while no large errors were found in the vertical-tilt configuration. In the non-tilt configuration, the results were dependent on the thickness of optical combiner and target distance.

Visual field defect (VFD) refers to a symptom in which a patient loses part of his/her field of view (FoV). Medical therapy can halt the progression of VFD, but complete recovery is impossible. In this paper, we propose a computational method for alleviating the restricted FoV with an optical see-through head-mounted display (OST-HMD), where an overview scene captured by the installed camera is overlaid on the persisting FoV. Since the overview window occludes with the real world scene, there is a trade-off between the augmented contextual information and the local unscreened information. We hypothesized that such a trade-off can be resolved by taking into consideration the size of the overview window and its displacement from the center of the unimpaired FoV. We therefore conducted an empirical evaluation through a Whac-A-Mole type of task with ten VFD-imitative subjects, where three sizes of an overview window with a fixed aspect ratio and seven positions in terms of elevation and azimuth were used combinatorially on an OST-HMD to find the best size and position of the overview window. It was statistically proven that for left-sided homonymous VFD-imitative subjects, the performance of the task was better when the medium-sized overview window was placed in the lower right position. The obtained result can legitimate default settings for the proposed VFD alleviation method.