Predictive Tracking Control of a Camera - Head Mounted Display System subject to Communication Constraints

Ray tracing-based rendering creates by far more realistic images than the traditional rasterization-based rendering. However, it is still burdensome when implemented for a Head-Mounted Display (HMD) system that demands a wide field of view and a high display refresh rate. Furthermore, for presenting high-quality images on the HMD screen, a sufficient number of ray sampling should be carried out per pixel to alleviate visually annoying spatial and temporal aliases. In this paper, we extend the recent selective foveated ray tracing technique by Kim et al. [<xref ref-type="bibr" rid="B1">1</xref>], and propose an improved real-time rendering technique that realizes the rendering effect of the classic Whitted-style ray tracing on the HMD system. In particular, by combining the ray tracing hardware-based acceleration technique and time-constrained rendering scheme, we show that fast HMD ray tracing is possible that is well suited to human visual systems.

Immersive virtual reality (VR) emerges as a promising research and clinical tool. However, several studies suggest that VR induced adverse symptoms and effects (VRISE) may undermine the health and safety standards, and the reliability of the scientific results. In the current literature review, the technical reasons for the adverse symptomatology are investigated to provide suggestions and technological knowledge for the implementation of VR head-mounted display (HMD) systems in cognitive neuroscience. The technological systematic literature indicated features pertinent to display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware that should be considered by the researchers. Subsequently, a meta-analysis of 44 neuroscientific or neuropsychological studies involving VR HMD systems was performed. The meta-analysis of the VR studies demonstrated that new generation HMDs induced significantly less VRISE and marginally fewer dropouts. Importantly, the commercial versions of the new generation HMDs with ergonomic interactions had zero incidents of adverse symptomatology and dropouts. HMDs equivalent to or greater than the commercial versions of contemporary HMDs accompanied with ergonomic interactions are suitable for implementation in cognitive neuroscience. In conclusion, researchers' technological competency, along with meticulous methods and reports pertinent to software, hardware, and VRISE, are paramount to ensure the health and safety standards and the reliability of neuroscientific results.

The personal head-mounted display (HMD) has emerged as a novel image monitoring system. We present here the application of a high-definition organic electroluminescent binocular HMD in ureteral stent placement. Our HMD system displayed multiple forms of information such as integrated, sharp, high-contrast images using a four-split screen or a picture-in-picture technique both seamlessly and synchronously. When both the operator and the assistant wore an HMD, they could continuously and simultaneously monitor the cystoscopic and fluoroscopic images in an ergonomically natural position. Furthermore, each participant was able to modulate the displayed images depending on the procedure. In all five cases, both the operator and the assistant successfully used this system with no unfavorable event. No participants experienced any HMD wear-related adverse effects. We therefore believe this HMD system might be potentially beneficial during ureteral stent placement procedures. Furthermore, it is compact, easily introduced and affordable.

Many applications have used a Head-Mounted Display (HMD), such as in virtual and mixed realities, and tele-presence. The advantage of HMD systems is the ease of feeling a 3D world in the display of animation or movies. However, the field of view (FOV) of commercial HMD systems is too narrow for feeling immersion. The horizontal FOV of many commercial HMDs is around 60 degrees, significantly narrower than that of humans. In this paper, we propose a super wide field of view catadioptrical head-mounted display consisting of an ellipsoidal and a hyperboloidal curved mirror. The horizontal FOV of the proposed HMD is 180 degrees and includes the peripheral view of humans. It increases reality and immersion of users. As well, the central region (60 degrees) of the FOV can measure 3D distances using stereoscopics.

Video games and sport are an essential part in the life of millions of people. With recent advances of immersive virtual reality devices such as the HTC Vive, Oculus Rift, or PlayStation VR, the use of virtual environments (VE) for exergames is becoming more and more popular. An exergame combines a physical activity with video game elements by tracking body movements or reactions of user, attempting to engage users in a more enjoyable system. In this paper, we present the results of a subjective experiment carried out with the aim to compare different kinds of virtual environments with each other. A rowing ergometer, connected either to a virtual reality system using a head-mounted display (HMD) or to a CAVE environment, was used as an exergame device. While for rowing experts, fitness and performance improvements are of major interest, we wanted to focus on the motivation and engagement of non-professionals. By means of a series of questionnaires and a follow-up interview, the Quality of Experience of participants using the system was assessed. Measurements include concepts such as flow, presence, video quality and well-being. Results show significant advantages of the HMD as well as of the CAVE compared to a system without a VE for the overall quality, system feedback, and flow. While the CAVE and HMD system mainly differed in their autotelic experience, the HMD was favored by the majority of participants due to a superior feeling of presence.

Research has shown that people are able to judge distances accurately in full-cue, real-world environments using visually directed actions. However, in virtual environments viewed with head-mounted display (HMD) systems, there is evidence that people act as though the virtual space is smaller than intended. This is a surprising result given how well people act in real environments. The behavior in the virtual setting may be linked to distortions in the available visual cues or to a person's ability to locomote without vision. Either could result from issues related to added mass, moments of inertia, and restricted field of view in HMDs. This article describes an experiment in which distance judgments based on normal real-world and HMD viewing are compared with judgments based on real-world viewing while wearing two specialized devices. One is a mock HMD, which replicated the mass, moments of inertia, and field of view of the HMD and the other an inertial headband designed to replicate the mass and moments of inertia of the HMD, but constructed to not restrict the field of view of the observer or otherwise feel like wearing a helmet. Distance judgments using the mock HMD showed a statistically significant underestimation relative to the no restriction condition but not of a magnitude sufficient to account for all the distance compression seen in the HMD. Indicated distances with the inertial headband were not significantly smaller than those made with no restrictions.

We present a low-cost solution for yaw drift in head-mounted display systems that performs better than current commercial solutions and provides a wide capture area for pose tracking. Our method applies an extended Kalman filter to combine marker tracking data from an overhead camera with onboard head-mounted display accelerometer readings. To achieve low latency, we accelerate marker tracking with color blob localisation and perform this computation on the camera server, which only transmits essential pose data over WiFi for an unencumbered virtual reality system.

Effectively managing visual search tasks across multiple spatial regions during daily activities such as driving, cycling, and navigating complex environments often overwhelms visual processing capacity, increasing the risk of errors and missed critical information. This study investigates an integrated approach that combines an Ambient Display system utilizing peripheral vision cues with traditional Head-Mounted Displays (HMDs) to enhance spatial search efficiency while minimizing cognitive burden. We systematically evaluated this integrated HMD-Ambient Display system against standalone HMD configurations through comprehensive user studies involving target search scenarios across multiple spatial regions. Our findings demonstrate that the combined approach significantly improves user performance by establishing a complementary visual system where peripheral stimuli effectively capture initial attention while central HMD cues provide precise directional guidance. The integrated system showed substantial improvements in reaction time for rear visual region searches and higher user preference ratings compared with HMD-only conditions. This integrated approach represents an innovative solution that efficiently utilizes dual visual channels, reducing cognitive load while enhancing search efficiency across distributed spatial areas. Our contributions provide valuable design guidelines for developing assistive technologies that improve performance in multi-region visual search tasks by strategically leveraging the complementary strengths of peripheral and central visual processing mechanisms.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

We developed a new respiratory-gated positron emission tomography (PET) imaging method (RGV-PET) that phase-based respiratory gated PET imaging (RG-PET) combine with head-mounted display (HMD)-guided "visual feedback." The purpose of this study was to investigate whether RGV-PET is effective at improving the quantitative measurement of tracer uptake in tumors using the phase-based respiratory gating method. Of the 41 enrolled patients with hepatobiliary or pancreatic cancer, 20 patients underwent RGV-PET and the remaining 21 patients underwent RG-PET. We measured the peak standardized uptake value (SUVpeak) of the primary lesion in each five bins obtained from both RG-PET and RGV-PET. The SUVpeak change rate calculated based on the ungated PET imaging. To evaluate the quantitative variation, the coefficient of variation of the SUVpeak change rate was compared between RG-PET and RGV-PET. In addition, we performed qualitative evaluation using visual score for the incidence of artifacts on four-dimensional-CT. The coefficient of variation of the average SUVpeak change rate in RGV-PET was 7.01 ± 4.43, which was significantly lower than the values in RG-PET (10.72 ± 5.74, p < 0.05). A significant improvement in the SUVpeak change rate of RGV-PET was obtained in bins 1 and 2 compared to RG-PET ( p < 0.05). The visual score of RGV-PET was significantly lower than RG-PET ( p < 0.05). RGV-PET was effective for respiratory stabilization and improved respiratory gating imaging quality. The RGV-PET is applicable to various PET/CT respiratory gating imaging and may improve the quantitativeness of PET images.

We present the VISAR keyboard: An augmented reality (AR) head-mounted display (HMD) system that supports text entry via a virtualised input surface. Users select keys on the virtual keyboard by imitating the process of single-hand typing on a physical touchscreen display. Our system uses a statistical decoder to infer users’ intended text and to provide error-tolerant predictions. There is also a high-precision fall-back mechanism to support users in indicating which keys should be unmodified by the auto-correction process. A unique advantage of leveraging the well-established touch input paradigm is that our system enables text entry with minimal visual clutter on the see-through display, thus preserving the user’s field-of-view. We iteratively designed and evaluated our system and show that the final iteration of the system supports a mean entry rate of 17.75wpm with a mean character error rate less than 1%. This performance represents a 19.6% improvement relative to the state-of-the-art baseline investigated: A gaze-then-gesture text entry technique derived from the system keyboard on the Microsoft HoloLens. Finally, we validate that the system is effective in supporting text entry in a fully mobile usage scenario likely to be encountered in industrial applications of AR HMDs.

A foveated display is a promising technique to realize displays offering both a large field of view (FOV) and high spatial resolution. Although several prior works have attempted to apply a foveation method to the design of a head-mounted display (HMD) system, the common method is based on a dual-resolution dynamic foveation scheme which is inevitably complex and has a high cost due to the requirements for multiple display sources, a 2D steering mechanism, and eye tracker. In this paper, a new perceptual-driven approach to the design of a statically foveated HMD is proposed with the goal of offering a wide FOV across which the degradation of the perceived image resolution is nearly imperceptible or minimal within regions of frequent eye movements. Compared to a dual-resolution discrete and dynamic foveation approach in the prior art, the static foveation approach will not only maintain resolution continuity but also eliminate the need for a scanning mechanism, multiple display sources, and an eyetracker, and therefore minimize hardware complexity. We present the general approach for creating a static foveation scheme, performance metrics for evaluating the perceived image quality, and the process of optimizing a foveation scheme to meet different requirements. Finally, we experimentally demonstrate and validate the proposed foveation scheme using a testbed system. Overall, we demonstrate a statically foveated scheme is capable of offering a display with a total 160° FOV, a constant resolution of 0.5 or 1 arcminutes per pixel within the ±10° region where frequent eye movements occur, an adequate resolution no less than 45% of peak resolution within the parafovea region of ±30°, and a data sampling efficiency as high as 90%.

It has always been a challenge to break the resolution/field-of-view (FOV) invariant to design a large FOV and high-resolution optical system, especially for a head-mounted display (HMD) system. In this study, a tiled HMD using two compact rotationally symmetrical eyepieces was designed and developed. Some issues on exit pupil and eye relief were analyzed in detail and taken into consideration during the design procedure. The overall optical system is compact with high performance. The system volume is smaller than 30 mm×35 mm×30 mm. Based on two 0.61 in. microdisplay devices, the overall tiled system demonstrates an FOV of 66°(H)×32°(V) with a 7.5 mm exit pupil diameter and a 15.7 mm eye relief.

An energy-efficient speech extraction (SE) processor is proposed for robust user speech recognition (SR) in head-mounted display (HMD) systems. User SE is essential for robust user SR in a noisy environment. For the low-latency SE, the FastSE algorithm is proposed to overcome the time-consuming constrained-independent-component-analysis-based user speech selection process, which results in < 2-ms SE latency. Moreover, a reinforced-FastSE scheme is proposed to achieve 97.2% accuracy with only 33-kB FastSE on-chip memory for the low-power HMD applications. Also, a reconfigurable matrix operation accelerator is implemented for the energy-efficient acceleration of the dominant matrix operation in SE. As a result, the proposed SE processor achieves 1.3× higher speed with 4.24× smaller memory compared to the state-of-the-art work, so SR in a noisy environment becomes possible for mobile HMD applications.

This paper describes the design of an optical see-through head-mounted display (HMD) system for Augmented Reality (AR). Our goals were to make virtual objects perfectly indistinguishable from real objects, wherever the user roams, and to find out to which extent imperfections are hindering applications in art and design. For AR, fast and accurate measuring of head motions is crucial. We made a head-pose tracker for the HMD that uses error-state Kalman filters to fuse data from an inertia tracker with data from a camera that tracks visual markers. This makes on-line head-pose based rendering of dynamic virtual content possible.We measured our system, and found that with an A4-sized marker viewed from > 20° at 5 m distance with an SXGA camera (FOV 108°), the RMS error in the tracker angle was < 0.5° when moving the head slowly. Our Kalman filters suppressed the pose error due to camera delay, which is proportional to the angular and linear velocities, and the dynamic misalignment was comparable to the static misalignment. Applications of artists and designers lead to observations on the profitable use of our AR system. Their exhibitions at world-class museums showed that AR is a powerful tool for disclosing cultural heritage.