Optical see-through head-mounted display with occlusion capability

Most state-of-the-art optical see-through head-mounted display (OST-HMD) lacks mutual occlusion capability between computer-rendered and real objects so that the virtual view through an OST-HMD appears "ghost-like", floating in the real world. In this paper, we demonstrated a light-weight, compact OST-HMD with mutual occlusion capability by exploring a highly innovative optical approach based on emerging freeform optical design and fabrication technologies. Our approach enabled us to achieve an occlusion-capable OST-HMD system with a very compelling form factor and high optical performance. The proposed display technology is designed for highly efficient liquid crystal on silicon (LCoS) type spatial light modulator (SLM) and bright Organic LED (OLED) microdisplay, which is capable of working in both indoor and outdoor environments. Our current design offered a 1280×1024 color resolution with a field of view (FOV) of 40 degrees and lightweight optics about 30 grams per eye.

We propose a varifocal occlusion technique for optical see-through head-mounted displays (OST-HMDs). Occlusion in OST-HMDs is a powerful visual cue that enables depth perception in augmented reality (AR). Without occlusion, virtual objects rendered by an OST-HMD appear semi-transparent and less realistic. A common occlusion technique is to use spatial light modulators (SLMs) to block incoming light rays at each pixel on the SLM selectively. However, most of the existing methods create an occlusion mask only at a single, fixed depth-typically at infinity. With recent advances in varifocal OST-HMDs, such traditional fixed-focus occlusion causes a mismatch in depth between the occlusion mask plane and the virtual object to be occluded, leading to an uncomfortable user experience with blurred occlusion masks. In this paper, we thus propose an OST-HMD system with varifocal occlusion capability: we physically slide a transmissive liquid crystal display (LCD) to optically shift the occlusion plane along the optical path so that the mask appears sharp and aligns to a virtual image at a given depth. Our solution has several benefits over existing varifocal occlusion methods: it is computationally less demanding and, more importantly, it is optically consistent, i.e., when a user loses focus on the corresponding virtual image, the mask again gets blurred consistently as the virtual image does. In the experiment, we build a proof-of-concept varifocal occlusion system implemented with a custom retinal projection display and demonstrate that the system can shift the occlusion plane to depths ranging from 25 cm to infinity.

An optical see-through head-mounted display (OST-HMD) can potentially improve the safety and accuracy of ultrasonography (US)-guided fine-needle aspiration. We aimed to evaluate the usefulness of an OST-HMD in US-guided needle-puncture procedures. We conducted a prospective randomized controlled study in which we compared the accuracy and safety of the US-guided needle puncture procedure and the stress on the practitioner when using OST-HMD versus standard US display (SUD). Inexperienced medical students were enrolled and randomly divided into two groups. A breast phantom was used to evaluate the required time and accuracy of the US-guided needle puncture. Practitioner stress was quantified using a visual analog scale (VAS). When the procedure was performed for the first time, the time required to reach the target lesion at a shallow depth was significantly shorter in the OST-HMD group (39.8 ± 39.9 s) than in the SUD group (71.0 ± 81.0 s) (p = 0.01). Using the OST-HMD significantly reduced the unintentional puncture of a non-target lesion (p = 0.01). Furthermore, the stress felt by the practitioners when capturing the image of the target lesion (p < 0.001), inserting and advancing the needle more deeply (p < 0.001), and puncturing the target lesion (p < 0.001) was significantly reduced in the OST-HMD group compared with that in the SUD group. Use of OST-HMD may improve the accuracy and safety of US-guided needle puncture procedures and may reduce practitioner stress during the procedure.

In order to align the virtual and real content precisely through augmented reality devices, especially in optical see-through head-mounted displays (OST-HMD), it is necessary to calibrate the device before using it. However, most existing methods estimated the parameters via 3D-2D correspondences based on the 2D alignment, which is cumbersome, time-consuming, theoretically complex, and results in insufficient robustness. To alleviate this issue, in this paper, we propose an efficient and simple calibration method based on the principle of directly calculating the projection transformation between virtual space and the real world via 3D-3D alignment. The proposed method merely needs to record the motion trajectory of the cube-marker in the real and virtual world, and then calculate the transformation matrix between the virtual space and the real world by aligning the two trajectories in the observed view. There are two advantages associated with the proposed method. First, the operation is simple. Theoretically, the user only needs to perform four alignment operations for calibration without changing the rotation variation. Second, the trajectory can be easily distributed throughout the entire observation view, resulting in more robust calibration results. To validate the effectiveness of the proposed method, we conducted extensive experiments on our self-built optical see-through head-mounted display (OST-HMD) device. The experimental results show that the proposed method can achieve better calibration results than other calibration methods.

Surface-stabilized antiferroelectric liquid crystals (AFLCs) and V-shape smectics share the spontaneous generation of intermediate transmission states, i.e. analogue grayscale. On the other hand, liquid crystal on silicon (LCOS) appears to be a promising alternative to thin-film transistors (TFTs) in the preparation of reflective microdisplays and spatial light modulators (SLMs). The feasibility of preparing LCOS SLMs using AFLC and V-shaped LCs has been analyzed. A comparative study of a number of AFLC and V-shape materials as candidates for LCOS SLMs has been carried out. Test cells below 1 µm have been prepared using an ITO-coated transparent glass plate and an aluminized silicon back plate mimicking the actual LCOS device. AFLC waveforms compatible to low-voltage operation of the silicon backplane have been demonstrated. V-shape waveforms have been designed and demonstrated as well. The results show that both LC families may be employed in this specific application. Dynamic behavior, with frame rates over 200 Hz, is especially remarkable.

In Augmented Reality (AR) with an Optical See-Through Head-Mounted Display (OST-HMD), the spatial calibration between a user's eye and the display screen is a crucial issue in realizing seamless AR experiences. A successful calibration hinges upon proper modeling of the display system which is conceptually broken down into an eye part and an HMD part. This paper breaks the HMD part down even further to investigate optical aberration issues. The display optics causes two different optical aberrations that degrade the calibration quality: the distortion of incoming light from the physical world, and that of light from the image source of the HMD. While methods exist for correcting either of the two distortions independently, there is, to our knowledge, no method which corrects for both simultaneously. This paper proposes a calibration method that corrects both of the two distortions simultaneously for an arbitrary eye position given an OST-HMD system. We expand a light-field (LF) correction approach [8] originally designed for the former distortion. Our method is camera-based and has an offline learning and an online correction step. We verify our method in exemplary calibrations of two different OST-HMDs: a professional and a consumer OST-HMD. The results show that our method significantly improves the calibration quality compared to a conventional method with the accuracy comparable to 20/50 visual acuity. The results also indicate that only by correcting both the distortions simultaneously can improve the quality.

We propose ARAMIS, a solution to provide real-time “x-ray see-through vision” of a patient’s internal structure to the surgeon, via an optical see-through head-mounted display (OST-HMD), in minimally invasive laparoscopic surgery. ARAMIS takes input imaging from a binocular endoscope, reconstructs a dense point cloud with a GPU-accelerated semi-global matching algorithm on a per-frame basis, and then wirelessly streams the point cloud to an untethered OST-HMD (currently, Microsoft HoloLens) for visualization. The OST-HMD localizes the endoscope distal tip by fusing fiducial-based tracking and self-localization. The point cloud is rendered on the OST-HMD with a custom shader supporting our data-efficient point cloud representation. ARAMIS is able to visualize the reconstructed point cloud (184k points) at 41.27 Hz with an end-to-end latency of 178.3 ms. A user study with 25 subjects, including 2 experienced users, compared ARAMIS to conventional laparoscopy during a peg transfer task on a deformable phantom. Results showed no significant difference in task completion time, but users generally preferred ARAMIS and reported improved intuitiveness, hand-eye coordination and depth perception. Inexperienced users showed a stronger preference for ARAMIS and achieved higher task success rates with the system, whereas the two experienced users indicated a slight preference for ARAMIS and succeeded in all tasks with and without assistance.

The combination of health and entertainment becomes possible due to the development of wearable augmented reality equipment and corresponding application software. In this paper, we implemented a fast calibration extended from SPAAM for an optical see-through head-mounted display (OSTHMD) which was made in our lab. During the calibration, the tracking and recognition techniques upon natural targets were used, and the spatial corresponding points had been set in dispersed and well-distributed positions. We evaluated the precision of this calibration, in which the view angle ranged from 0 degree to 70 degrees. Relying on the results above, we calculated the position of human eyes relative to the world coordinate system and rendered 3D objects in real time with arbitrary complexity on OSTHMD, which accurately matched the real world. Finally, we gave the degree of satisfaction about our device in the combination of entertainment and prevention of cervical vertebra diseases through user feedbacks.

Digital Electro-optics Platform is the main concept of Jasper Display Corp. (JDC) to develop various applications. These applications are based on our X-on-Silicon technologies, for example, X-on-Silicon technologies could be used on Liquid Crystal on Silicon (LCoS), Micro Light-Emitting Diode on Silicon (μLEDoS), Organic Light-Emitting Diode on Silicon (OLEDoS), and Cell on Silicon (CELLoS), etc. LCoS technology is applied to Micro Display, Spatial Light Modulator (SLM), Dynamic Optics, Wavelength Selective Switch (WSS), Holographic Display, Microscopy, Bio-tech, 3D Printing and Adaptive Optics, etc. In addition, μLEDoS technology is applied to Augmented Reality (AR), Head Up Display (HUD), Head-mounted Display (HMD), and Wearable Devices.

Visual field defect (VFD) refers to a symptom in which a patient loses part of his/her field of view (FoV). Medical therapy can halt the progression of VFD, but complete recovery is impossible. In this paper, we propose a computational method for alleviating the restricted FoV with an optical see-through head-mounted display (OST-HMD), where an overview scene captured by the installed camera is overlaid on the persisting FoV. Since the overview window occludes with the real world scene, there is a trade-off between the augmented contextual information and the local unscreened information. We hypothesized that such a trade-off can be resolved by taking into consideration the size of the overview window and its displacement from the center of the unimpaired FoV. We therefore conducted an empirical evaluation through a Whac-A-Mole type of task with ten VFD-imitative subjects, where three sizes of an overview window with a fixed aspect ratio and seven positions in terms of elevation and azimuth were used combinatorially on an OST-HMD to find the best size and position of the overview window. It was statistically proven that for left-sided homonymous VFD-imitative subjects, the performance of the task was better when the medium-sized overview window was placed in the lower right position. The obtained result can legitimate default settings for the proposed VFD alleviation method.

The ray-shift phenomenon means the apparent distance shift in the display image plane between virtual and physical objects. It is caused by the difference in the refraction of virtual display and see-through optical paths derived from optical combiners that are necessary to provide a see-through capability in optical see-through head-mounted displays. In this work, through a human-subject experiment, we investigated the effects of ray-shift phenomenon induced by the optical combiner on depth perception for near-field distances (40 cm-100 cm). In our experiment, we considered three different configurations of optical combiner: horizontal-tilt and vertical-tilt configurations (using plate beamsplitters horizontally and vertically tilted by 45°, respectively), and non-tilt configuration (using rectangular solid waveguides). Participants' depth perception errors in these configurations were compared with those in an ordinary condition (i.e., the condition where physical objects are directly shown without the displays) and theoretically estimated ones. According to the experimental results, the measured percentage depth perception errors were similar to the theoretically estimated ones, where the amount of estimated percentage depth errors was greater than 0.3%. Furthermore, the participants showed significantly larger depth perception errors in the horizontal-tilt configuration than in an ordinary condition, while no large errors were found in the vertical-tilt configuration. In the non-tilt configuration, the results were dependent on the thickness of optical combiner and target distance.

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic () and maxillofacial surgeries (). For preoperative input data, computed tomography (CT) (), and surface rendered models () were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (); this was achieved by surface tracking of fiducials (), external tracking (), or manual placement (). Microsoft HoloLens devices ( in 2021, in 2022) were the most frequently used OST-HMDs; gestures and/or voice () served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

Optical see-through head-mounted displays (OHMDs) blend digital content with the physical world, presenting unique color management challenges. Previous literature suggests using green as the main color, but this severely limits creative freedom. To address this, we conducted an empirical study with 30 participants, evaluating 216 colors under various OHMD usage conditions. Based on the results, we propose color guidelines indicating each hue's clear and comfortable saturation and brightness ranges, along with clarity and comfort scores across hues for different devices and lighting conditions. Our color guidelines expand the usable color palette, offering designers a wider range of color options. These guidelines were used and iteratively refined through feedback in a workshop with 12 designers, integrating them into practical design workflows. The resulting comprehensive color guide provides a valuable resource for OHMD interface designers, enhancing both the aesthetic possibilities and functional effectiveness of augmented reality experiences.

Light-on-dark color schemes, so-called “Dark Mode,” are becoming more and more popular over a wide range of display technologies and application fields. Many people who have to look at computer screens for hours at a time, such as computer programmers and computer graphics artists, indicate a preference for switching colors on a computer screen from dark text on a light background to light text on a dark background due to perceived advantages related to visual comfort and acuity, specifically when working in low-light environments. In this paper, we investigate the effects of dark mode color schemes in the field of optical see-through head-mounted displays (OST-HMDs), where the characteristic “additive” light model implies that bright graphics are visible but dark graphics are transparent. We describe a human-subject study in which we evaluated a normal and inverted color mode in front of different physical backgrounds and among different lighting conditions. Our results show that dark mode graphics on OST-HMDs have significant benefits for visual acuity, fatigue, and usability, while user preferences depend largely on the lighting in the physical environment. We discuss the implications of these effects on user interfaces and applications.

Blowing in the wind: Increasing social presence with a virtual human via environmental airflow interaction in mixed reality