Binocular eye tracking in virtual reality for inspection training

This paper presents a novel three-dimensional (3-D) eye movement analysis algorithm for binocular eye tracking within virtual reality (VR). The user's gaze direction, head position, and orientation are tracked in order to allow recording of the user's fixations within the environment. Although the linear signal analysis approach is itself not new, its application to eye movement analysis in three dimensions advances traditional two-dimensional approaches, since it takes into account the six degrees of freedom of head movements and is resolution independent. Results indicate that the 3-D eye movement analysis algorithm can successfully be used for analysis of visual process measures in VR. Process measures not only can corroborate performance measures, but also can lead to discoveries of the reasons for performance improvements. In particular, analysis of users' eye movements in VR can potentially lead to further insights into the underlying cognitive processes of VR subjects.

The intent of this paper is to provide an introduction into the bourgeoning field of eye tracking in Virtual Reality (VR). VR itself is an emerging technology on the consumer market, which will create many new opportunities in research. It offers a lab environment with high immersion and close alignment with reality. An experiment which is using VR takes place in a highly controlled environment and allows for a more in-depth amount of information to be gathered about the actions of a subject. Techniques for eye tracking were introduced more than a century ago and are now an established technique in psychological experiments, yet recent development makes it versatile and affordable. In combination, these two techniques allow unprecedented monitoring and control of human behavior in semi-realistic conditions. This paper will explore the methods and tools which can be applied in the implementation of experiments using eye tracking in VR following the example of one case study. Accompanying the technical descriptions, we present research that displays the effectiveness of the technology and show what kind of results can be obtained when using eye tracking in VR. It is meant to guide the reader through the process of bringing VR in combination with eye tracking into the lab and to inspire ideas for new experiments.

Virtual reality presents an efficient and engaging approach to ophthalmological diagnosis and treatment in children. This systematic review investigates the current state of eye tracking technology integrated into virtual reality for the spectrum of pediatric ophthalmology. The search encompassed the MEDLINE database via PubMed, without imposing any time restrictions. A total of 20 relevant studies that met our inclusion criteria were incorporated and categorized into the following domains: diagnosis, examination, treatment, and rehabilitation use of virtual reality devices within the realm of pediatric ophthalmology. Strabismus examinations and postoperative monitoring, inherited retinal degeneration examination and visual function testing, therapy of amblyopia, glaucoma visual field testing, cerebral visual impairment rehabilitation, and neuro-ophthalmic disease examination were included in the final analysis. Pediatric ophthalmology offers a promising landscape for the integration of eye tracking technology within virtual reality, with accelerated, quantifiable, and objective examination and diagnosis, and precise, real-time measurements that are crucial in children. Virtual reality is an engaging experience, easily applied in a pediatric setting and facilitating compliance during examination and adherence to therapy. Although our systematic review provides insights into the current state of research, it is anticipated that further exploration is required for the widespread utilization of eye tracking in virtual reality within pediatric ophthalmology. [J Pediatr Ophthalmol Strabismus. 2024;61(6):381-390.].

We aim to develop a brain-machine interface (BMI) system that estimates user's gaze or attention on an object to pick it up in the real world. In Experiment 1 and 2 we measured steady-state visual evoked potential (SSVEP) using luminance and/or contrast modulated flickers of photographic scenes presented on a head-mounted display (HMD). We applied multiclass SVM to estimate gaze locations for every 2s time-window data, and obtained significantly good classifications of gaze locations with the leave-one-session-out cross validation. In Experiment 3 we measured SSVEP using luminance and contrast modulated flickers of real scenes that were online captured by a USB camera and presented on the HMD. We put AR markers on real objects and made their locations flickering on HMD. We obtained the best performance of gaze classification with highest luminance and contrast modulation (73–91% accuracy at chance level 33%), and significantly good classification with low (25% of the highest) luminance and contrast modulation (42–50% accuracy). These results suggest that the luminance-modulated flickers of real scenes through USB camera can be applied to BMI by using augmented reality technology.

HoloTabletop is a low-cost holographic-like tabletop interactive system. This system analyzes user's head position and gaze location in a real time setting and computes the corresponding anamorphic illusion image. The anamorphic illusion image is displayed on a 2D horizontally-located monitor, yet offers stereo vision to the user. The user is able to view and interact with the 3D virtual objects without wearing any special glasses or devices. The experimental results and user studies verify that the proposed HoloTabletop system offers excellent stereo vision while no visual fatigue will be caused to human eyes. This system is a great solution for many interactive applications such as 3D board games and stereo map browsing.

Neural correlates to seen gaze-direction and head orientation in the macaque monkey amygdala

Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

The effects of electrical vestibular stimulation upon movement and perception suggest two evoked sensations: head roll and inter-aural linear acceleration. The head roll vector causes walking subjects to turn in a direction dependent on head pitch, requiring generation of torque around a vertical axis. Here the effect of vestibular stimulation upon vertical torque (T(z)) was investigated during quiet stance. With the head tilted forward, square-wave stimuli applied to the mastoid processes evoked a polarity-specific T(z) response accompanied by trunk yaw. Stochastic vestibular stimulation (SVS) was used to investigate the effect of head pitch with greater precision; the SVS–T(z) cross-correlation displayed a modulation pattern consistent with the head roll vector and this was also reflected by changes in coherence at 2–3 Hz. However, a separate response at 7–8 Hz was unaffected by head pitch. Head translation (rather than rotation) had no effect upon this high frequency response either, suggesting it is not caused by a sense of body rotation induced by an inter-aural acceleration vector offset from the body. Instead, high coherence between medio-lateral shear force and T(z) at the same frequency range suggests it is caused by mechanical coupling to evoked medio-lateral sway. Consistent with this explanation, the 7–8 Hz response was attenuated by 90 deg head roll or yaw, both of which uncouple the inter-aural axis from the medio-lateral sway axis. These results demonstrate two vertical torque responses to electrical vestibular stimulation in standing subjects. The high frequency response can be attributed to mechanical coupling to evoked medio-lateral sway. The low frequency response is consistent with a reaction to a sensation of head roll, and provides a novel method for investigating proprioceptive-vestibular interactions during stance.

In this paper, we propose a system that presents correct gaze directions of each user in a group-to-group video conference. It is known that gaze direction plays a significant role in face-to-face communication. However, in ordinary video conference systems, there is a problem that gaze directions cannot be correctly transmitted to a user in the other side since the camera is placed on the top or bottom of the display. We propose a method to generate images of each person from an appropriate viewpoint position using Kinect and screen-embedded cameras and to present the images to the user. By estimating each user's gaze direction and rendering the image of the person who the user is looking at from the users' viewpoint position, it is possible to present an image suitable for each user even in a group-to-group situation. We show demonstration of a group-to-group video conference using the proposed system and show that the system can present images suitable for each user.

We present a study on continuously predicting the direction to a pointing target in virtual environments using motion and eye-tracker data throughout the pointing process. We first collect time series data for user motion and eye-tracker in a cursorless, single-target pointing task. Results from analyzing fixation points from different sensors and observing velocity profiles over the course of pointing provide insights into optimally configuring features for predicting the target angles. Following this analysis, we train a recurrent neural network that feeds on sliding window inputs for continuously operating target direction prediction from start to finish. The input window contains historical data from past to current frames, capturing temporal changes in the feature data. By feeding on this input, our model can predict the direction of the target at any given time during pointing. Our findings demonstrate that incorporating eye-tracker data into the prediction model boosts the maximum achievable accuracy by 2.5 times when compared to baselines without eye-tracker data inputs. The results suggest that using features from both the eye-tracker and joint motion contributes to higher prediction performance, as well as faster stabilization of output values at the starting phase of pointing.

We propose a telepresence system with a real human face-shaped screen. This system tracks the remote user's face and extracts the head motion and the face image. The face-shaped screen moves along three degree-of-freedom (DOF) by reflecting the user's head gestures. As the face-shaped screen is molded based on the 3D-shape scan data of the user, the projected image is accurate even when it is seen from different angles. We expect this system can accurately convey the user's nonverbal communication, in particular the user's gaze direction in 3D space that is not correctly transmitted by using a 2D screen (which is known as "the Mona Lisa effect"). To evaluate how this system can contribute to the communication, we conducted three experiments. The first one examines the blind angle of a face-shaped screen and a flat screen, and compares the ease with which users can distinguish facial expressions. The second one evaluates how the direction in which the remote user's face points can be correctly transmitted. The third experiment evaluates how the gaze direction can be correctly transmitted. We found that the recognizable angles of the face-shaped screen were larger, and that the recognition of the head directions was better than on a flat 2D screen. More importantly, we found that the face-shaped screen accurately conveyed the gaze direction, resolving the problem of the Mona Lisa effect.

Enhancing user experience and performance, including task load in immersive environments, requires accurate prediction of user gaze point, reaction time, and mental and physical load uptake. Current gaze prediction approaches focus primarily on motion-based information, lacking physiological data, which leads to poor prediction accuracy in highly dynamic virtual reality (VR) environments. Traditional cognitive load measurements rely on post-task analysis without proper multimodal data integration and fail to capture the real-time dynamics of user states during interaction. Likewise, reaction time or attention load are often assessed only after the interaction, without using real-time immersive sensor data, which limits adaptive responsiveness. To tackle these limitations, we leveraged a comprehensive multimodal dataset - VRWalking, which recorded timestamped eye-tracking metrics, physiological signals (heart rate and galvanic skin response), and behavioral performance data during real-time engagement in a VR environment. We developed a unified multitask model based on the MultiPatchFormer architecture, which processes multimodal VR signals through dual patch projection branches for gaze and classification inputs. The model employs multiscale patch embeddings, cross-attention between gaze and classification pathways, channel attention, and transformer encoders to jointly predict continuous user gaze and classify reaction time, cognitive load (mental load and physical load). Our methodology achieved excellent predictive performance: 95.64% for reaction time, 98.01% for mental load, and 97.45% for physical load, with a MAPE (Mean Absolute Percentage Error) of 15.24% for gaze prediction. We applied Shapley Additive explanations (SHAP) analysis to interpret the model’s behavior across all features, including eye-tracking, head-tracking, and physiological signals. The analysis revealed which features most influenced the predictions of user gaze, reaction time, mental load, and physical load. Our methods, while based only on the VRWalking dataset, demonstrated strong performance across all tasks, suggesting promising potential for real-world VR applications such as interactive training systems that respond to user attention lapses, educational platforms that adapt to cognitive load, and performance assessments that consider physiological indicators.

Changes in the position of the head are known to introduce errors when pointing to the remembered location of a visual target. Pointing to a visual target requires the integration of visual information and proprioceptive information about the position and movement of the body. Errors introduced when pointing to a visual target may be introduced either through misperception of sensory information or through errors in the transformation of information between visual and proprioceptive systems. It is proposed that proximal body segments such as the head, the eyes, or the trunk may act as a frame of reference during this transformation of sensory information, and therefore changes in the position of these body segments may introduce error. Previous studies that have investigated changes in head position during pointing tasks hypothesised that errors are primarily introduced in the perception of target location. One of the aims of this thesis was to assess the potential for changes in head position to introduce error in the proprioceptive component of the planning and performance of upper limb movement. A second aim was to disentangle the mechanisms behind the effect of head rotation on pointing to a visual target. A third aim was to investigate the accuracy of upper limb proprioception and pointing, following changes in head position, in people with chronic neck pain following a whiplash injury. The first series of studies (I to IV) investigated the accuracy of upper limb movement towards a proprioceptive goal. The effect of real and illusory changes in head position on elbow joint position error (JPE) was assessed in healthy subjects. In addition, the effect of changes in head position on elbow JPE was assessed in people with whiplash associated disorders (WAD). In these studies, error in the elbow proprioceptive task was present, but the pattern of error was dependent on the range of head movement and the modality used to introduce illusory changes in head position. Elbow proprioception was affected by smaller ranges of head rotation in people with WAD than in healthy people. These studies demonstrated that changes in the head position influence movement to a proprioceptive target and suggest that head position is used in sensorimotor transformations for upper limb movement even when the task does not involve visual cues. The second series of studies (V to VII) aimed to explore the mechanisms behind the observation that rotation of the head induces a contralateral bias in pointing. The studies measured the accuracy of pointing towards the remembered location of three visual targets after rotation of the head and/or the eyes. Experimental conditions included full and mid range head rotation and tasks that aimed to dissociate between the relative contributions of head position, eye-in-head position, and gaze deviation to the bias in pointing. The accuracy of pointing with head rotation was also assessed in people with WAD, who displayed the same pattern of error as healthy subjects. As in previous studies, head rotation induced contralateral deviations in pointing, which was dependent on the location of the target. These studies demonstrated that errors in pointing were present with mid range head rotation, and that the errors could not simply be explained by changes in gaze direction. The studies of this thesis demonstrate that changes in head position are important in the planning and performance of goal oriented movement. Knowledge of head position is not only important for the perception of target location but also for the accurate perception of upper limb position, particularly in people with WAD. In healthy subjects, errors in upper limb movement following changes in head position can be introduced within a functional range of head motion and cannot be completely explained by the changes in gaze that are associated with head rotation.

SummaryHumans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.

In this paper, we suggest a new system to estimate the direction of a user's eye gaze. Our system consisted of five IR LEDs and a CCD camera. The IR LEDs, which are attached to the corners of a computer monitor, make glints on the surface of the cornea of the eye. If the user sees the monitor, the center of a pupil is always in a polygon that is made by the glints. Consequently, the direction of the user's eye gaze can be computed without computing the geometrical relation between the eye, the camera and the monitor in 3D space. Our method is comparatively simple and fast. We introduce the method and show some experimental results.