Detection and Correspondence Matching of Corneal Reflections for Eye Tracking Using Deep Learning

Eye tracking systems that estimate the point-of-gaze are essential in extended reality (XR) systems as they enable new interaction paradigms and technological improvements. It is important for these systems to maintain accuracy when the headset moves relative to the head (known as device slippage) due to head movements or user adjustment. One of the most accurate eye tracking techniques, which is also insensitive to shifts of the system relative to the head, uses two or more infrared (IR) light emitting diodes to illuminate the eye and an IR camera to capture images of the eye. An essential step in estimating the point-of-gaze in these systems is the precise determination of the location of two or more corneal reflections (virtual images of the IR-LEDs that illuminate the eye) in images of the eye. Eye trackers tend to have multiple light sources to ensure at least one pair of reflections for each gaze position. The use of multiple light sources introduces a difficult problem: the need to match the corneal reflections with the corresponding light source over the range of expected eye movements. Corneal reflection detection and matching often fail in XR systems due to the proximity of camera and steep illumination angles of light sources with respect to the eye. The failures are caused by corneal reflections having varying shape and intensity levels or disappearance due to rotation of the eye, or the presence of spurious reflections. We have developed a fully convolutional neural network, based on the UNET architecture, that solves the detection and matching problem in the presence of spurious and missing reflections. Eye images of 25 people were collected in a virtual reality headset using a binocular eye tracking module consisting of five infrared light sources per eye. A set of 4,000 eye images were manually labelled for each of the corneal reflections, and data augmentation was used to generate a dataset of 40,000 images. The network is able to correctly identify and match 91% of corneal reflections present in the test set. This is comparable to a state-of-the-art deep learning system, but our approach requires 33 times less memory and executes 10 times faster. The proposed algorithm, when used in an eye tracker in a VR system, achieved an average mean absolute gaze error of 1°. This is a significant improvement over the state-of-the-art learning-based XR eye tracking systems that have reported gaze errors of 2-3°.

In human-computer interaction (HCI) applications, the performance degradation of gaze trackers in real-world environments is a critical issue. Typically, gaze trackers utilize the pupil center and corneal reflection (CR) obtained from an infrared (IR) light source to estimate the point of regard (POR). However, false CRs are often generated due to extraneous light sources such as sunlight or lamps. In this study, we propose a method of improving the robustness of gaze tracking under unconstrained illumination conditions. First, the proposed method generates a coded CR pattern by utilizing time-multiplexed IR light sources. Next, the CR candidates are detected in eye images, and their coordinates are compensated based on the head and eye movements of the user. Finally, true CRs are selected from the motion-compensated CR candidates by utilizing a novel cost function. Experimental results indicate that the gaze-tracking performance of the proposed method under various light conditions is considerably better than those of the conventional methods.

A set of methods in terms of both image processing and gaze estimation for accurate eye tracking is proposed. The eye-tracker used in this study relies on the dark-pupil method with up to 12 corneal reflections and offers an unprecedented high resolution imaging of the pupil and the cornea. The potential benefits of a higher number of glints and their optimum arrangement are analyzed considering distinct light sources configurations with 12, 8, 6, 4, and 2 corneal reflections. Moreover, a normalization factor of the pupil-glint vector is proposed for each configuration. There is a tendency for increasing accuracy with the number of glints, especially vertically (0.47deg for 12 glints configuration versus 0.65deg for 2 glints configuration). Besides the number of corneal reflections, their arrangement seems to have a stronger effect. A configuration that minimizes the interference of the eyelids with the corneal reflections is desired. Finally, the normalization of the pupil-glint vectors improves the vertical eye tracking accuracy up to 43.2%. In addition, the normalization also limits the need for a higher number of light sources to achieve better spatial accuracy.

In our pupil-corneal reflection-based gaze detection system, when users move the head quickly, the image difference method for detecting the pupils does not tend to function accurately. In addition, it becomes more difficult to detect the corneal reflection of the near-infrared light source as well as the pupil under face intense illumination condition because the pupils constrict and the disturbance light source, e.g. the sun, is misdetected as the corneal reflection. Moreover, when the users wear glasses, the reflections of glasses tend to be misdetected as the pupil and corneal reflection (feature points). In the present paper, we introduce a high-speed camera (2,000 fps) and propose a new detection method additionally acquiring a non-lighting image and then differentiating the image from the bright and dark images to detect the feature points even under the face intense illumination condition. In addition, a new light source for removing the glass reflections and enhancing the brightness difference between the bright and dark pupils is developed. The experimental results show that the robustness for the detection of the feature points is improved both under the face intense illumination condition and under the glasses-wearing condition.

Estimating the gaze direction with a digital video-based pupil and corneal reflection (P-CR) eye tracker is challenging partly since a video camera is limited in terms of spatial and temporal resolution, and because the captured eye images contain noise. Through computer simulation, we evaluated the localization accuracy of pupil-, and CR centers in the eye image for small eye rotations (≪ 1 deg). Results highlight how inaccuracies in center localization are related to 1) how many pixels the pupil and CR span in the eye camera image, 2) the method to compute the center of the pupil and CRs, and 3) the level of image noise. Our results provide a possible explanation to why the amplitude of small saccades may not be accurately estimated by many currently used video-based eye trackers. We conclude that eye movements with arbitrarily small amplitudes can be accurately estimated using the P-CR eye-tracking principle given that the level of image noise is low and the pupil and CR span enough pixels in the eye camera, or if localization of the CR is based on the intensity values in the eye image instead of a binary representation.

This study proposes a 3D shape measurement device based on an image scanner. The shape reconstruction method using the image scanner is one of the photometric stereo method which uses multiple parallel light sources, but our proposed method is applied to linear light sources. This method is constructed by several processes. First, we estimate an object shape using four images taken by two groups of two light sources located in same direction. And then, the color and specular components are estimated by using several images taken by light sources located in a circular path. In experiments, we investigate optimal number and location of light sources to estimate object shape, color and specular components.

A system for stabilising and monitoring eye movements for linac-based stereotactic radiotherapy associated with the mobile eye, the Eye Tracker, was developed. Whilst the Eye Tracker design is based on a previously reported system, the purpose of this study was to confirm that the modified version can be used with clinically acceptable treatment margins. We report the estimates of the margin required to account for inter- and intra-fraction eye motion based on data from 12 consecutive patients treated with the Eye Tracker system in place. Patients were immobilised in a head and neck mask and were required to fixate on a light source. A camera system monitored eye movements relative to CT simulation baseline measurements. The Exactrac system (Brainlab, Feldkirchen, Germany) combined with the Varian TrueBeamSTx (Varian Medical Systems, Palo Alto, CA) confirmed pre- and intra-treatment setup of the head position. Displacement/rotation of the image of the pupil/iris was determined in the lateral and superior-inferior directions using a video display. A standard margin equation was applied to estimate the margin required to account for inter- and intra-fraction eye movement. The average displacement in both directions was 0.1-0.2mm (0.36mm SD). All patients maintained a position within 1mm of the intended position during treatment. Based on a Bayesian estimation of the systematic and treatment errors, accounting for displacements in two-planes and a standard deviation of the penumbral width of 1.3mm, the estimated margins to achieve coverage of the GTV with the 95% isodose in 90% of patients was found to be less than 1mm. Small random and systematic uncertainties due to inter- and intra-fraction movement of the eye were achieved with the Eye Tracker. Whilst the estimated margins are small (<1mm) they need to be considered in addition to contouring and treatment delivery uncertainties.

This study proposes a simple, affordable and available technology for controlling the computer cursor with only your eyes. The proposed design method was designed and developed for people with significant upper limb disabilities that don’t use a keyboard or a mouse, and has been framed as a standard input method re-defining the notion of input by an eye-re-URL sing interface. Based upon a standard web camera and real-time image processing, the eye tracking system uses the direction of an individual’s facial gaze and blinking detect and translate that into the movement needed for the cursor action. The proposed eye tracking system is lighter and less accessible because it is not dependent upon sophisticated input-device hardware, thus able to obtain reliable cursor movement whenever the lit conditions in the environment are suitable to allow the eye-tracking input system to function smoothly. Based upon early trials, there has been a reported ease-of-use and high responsivity to potential use of a system using assistive technology capabilities. Keywords— Eye tracking ,Cursor control, Webcam interface Blink detection, Physically disabled

This paper discusses a method to estimate 3D shape, color and specular components using an image scanner. This shape reconstruction method is based on the photometric stereo method, but we consider multiple linear light sources in the shape reconstruction algorithm. In our previous methods, color and specular estimation needs many images. Therefore, we propose a new method to obtain the optimal light source positions to reduce the number of images. And, we also show the effectiveness of proposed method using synthetic scanned images.

Explicit (often verbal) reports are typically used to investigate memory (e.g. "Tell me what you remember about the person you saw at the bank yesterday."), however such reports can often be unreliable or sensitive to response bias 1, and may be unobtainable in some participant populations. Furthermore, explicit reports only reveal when information has reached consciousness and cannot comment on when memories were accessed during processing, regardless of whether the information is subsequently accessed in a conscious manner. Eye movement monitoring (eye tracking) provides a tool by which memory can be probed without asking participants to comment on the contents of their memories, and access of such memories can be revealed on-line 2,3. Video-based eye trackers (either head-mounted or remote) use a system of cameras and infrared markers to examine the pupil and corneal reflection in each eye as the participant views a display monitor. For head-mounted eye trackers, infrared markers are also used to determine head position to allow for head movement and more precise localization of eye position. Here, we demonstrate the use of a head-mounted eye tracking system to investigate memory performance in neurologically-intact and neurologically-impaired adults. Eye movement monitoring procedures begin with the placement of the eye tracker on the participant, and setup of the head and eye cameras. Calibration and validation procedures are conducted to ensure accuracy of eye position recording. Real-time recordings of X,Y-coordinate positions on the display monitor are then converted and used to describe periods of time in which the eye is static (i.e. fixations) versus in motion (i.e., saccades). Fixations and saccades are time-locked with respect to the onset/offset of a visual display or another external event (e.g. button press). Experimental manipulations are constructed to examine how and when patterns of fixations and saccades are altered through different types of prior experience. The influence of memory is revealed in the extent to which scanning patterns to new images differ from scanning patterns to images that have been previously studied 2, 4-5. Memory can also be interrogated for its specificity; for instance, eye movement patterns that differ between an identical and an altered version of a previously studied image reveal the storage of the altered detail in memory 2-3, 6-8. These indices of memory can be compared across participant populations, thereby providing a powerful tool by which to examine the organization of memory in healthy individuals, and the specific changes that occur to memory with neurological insult or decline 2-3, 8-10.

Deep Learning (DL) is continuously adopted in many industrial applications at a rapidly increasing pace. This includes safety- and security-critical applications where errors in the DL system can lead to massive or even fatal losses. With the rise of DL adoption, trustworthy AI initiatives have been introduced that cover quality assurance principles such as robustness, fairness, and security of a DL system. Robustness aims to address the DL system's ability to predict new unseen inputs that are relevant to the DL application. Fairness aims to enable equal performance for all demographics, such as gender, age or ethnicity. Security aims to address new DL threats and defenses. Combining all three principles enables a more trustworthy DL system, which gives benefits to developers and end users. Developers gain confidence in deployment and end users gain trust in adopting a DL system. So far, the principles of robustness, fairness and security have been outlined in trustworthy initiatives from an expectations point of view under ISO or regulatory acts under the European Commission\cite{ai_act}. However, these initiatives give little attention to methods and quantitative assessment strategies on how those principles can be enabled and validated. Without assessment strategies, forming benchmarks to determine the quality of a DL system becomes a challenging task. This motivates research questions such as: ``\textit{At what point can a DL system be considered sufficient for deployment?}” and ``\textit{Is the data used for testing relevant and complete for the target application?}” In this thesis, we aim to address such questions and present foundational improvements by introducing out-of-distribution (OOD) awareness to robustness, fairness and security. Finally, the research methodologies are integrated into a national standard of Singapore which represents a high impact outcome of this work. First, we conduct a large-scale OOD empirical study to analyse which OOD technique is most suited for real-world DL system testing. Then, we propose an OOD testing criteria to enhance error discovery and propose distribution aware robustness enhancement by showing that filtering errors far from the trained distribution increases robustness by 21.5\%. We then enhance the field of OOD by introducing a fine-grained OOD technique assessing the ground truth of data augmentation and increasing accuracy by 31.9\% by guiding data augmentation selection with distribution awareness. We further apply the same augmentation technique to subsets of the training data to balance accuracy among demographics to enhance fairness. Finally, we introduce OOD defense approaches against two novel DL threats which fully evade traditional defenses. Ultimately, the thesis concludes with increasing quality assurance research impact by presenting the methodology that led to launching a national standard. The standard integrates our security research and evaluates it among four real-world use cases. This represents an important step in creating and validating trustworthy DL systems enabling increased trust in the intelligent solution and thereby further accelerating adoption in a safeguarded manner.

This paper describes a flexible method for synchronizing electroencephalography (EEG) and eye tracking (ET) recordings to the presentation of visual stimuli. The method consists of embedding a synchronization signal in the visual stimuli, and recording this signal with both the EEG and ET equipment. The signal is recorded by the EEG device as an additional data channel, and with the camera used in the ET equipment by modulating the global illumination of the scene in time with the synchronizing signal. The prototype system where this method was implemented resulted in a single sample of jitter in both the EEG and ET system, while the ET system achieved a spatial resolution of 1.26 degrees. The system will be used in future work with augmented memory applications.

This paper describes a low-cost, robust, and accurate remote eye-tracking system that uses an industrial prototype smartphone with integrated infrared illumination and camera. Numerous studies have demonstrated the beneficial use of eye-tracking in domains such as neurological and neuropsychiatric testing, advertising evaluation, pilot training, and automotive safety. Remote eye-tracking on a smartphone could enable the significant growth in the deployment of applications in these domains. Our system uses a 3D gaze-estimation model that enables accurate point-of-gaze (PoG) estimation with free head and device motion. To accurately determine the input eye features (pupil center and corneal reflections), the system uses Convolutional Neural Networks (CNNs) together with a novel center-of-mass output layer. The use of CNNs improves the system’s robustness to the significant variability in the appearance of eye-images found in handheld eye trackers. The system was tested with 8 subjects with the device free to move in their hands and produced a gaze bias of 0.72°. Our hybrid approach that uses artificial illumination, a 3D gaze-estimation model, and a CNN feature extractor achieved an accuracy that is significantly (400%) better than current eye-tracking systems on smartphones that use natural illumination and machine-learning techniques to estimate the PoG.

Eye tracking has become an essential human–machine interaction modality in virtual reality (VR) and augmented reality (AR) applications requiring high throughput (e.g., more than 240 frames per second), small form factor, and enhanced visual privacy. Existing eye tracking systems have adopted bulky, lens-based cameras, and thus suffer from both a large form factor and high communication cost between the camera and back-end processor. This work presents a camera, algorithm, and accelerator co-designed lensless eye tracking system dubbed EyeCoD, which, to the best of our knowledge, is the first to provide a general, front-end eye tracking solution for AR/VR while satisfying the requirements for both high throughput and smaller form factor. Specifically, EyeCoD integrates system-, algorithm-, and accelerator-level techniques to boost system efficiency without sacrificing eye tracking accuracy. We believe that our EyeCoD system will pave the way for next-generation eye tracking solutions in VR/AR and shed light on future innovations for intelligent imaging systems.

To achieve the full potential of an area-of-interest (A0I) display requires that a high resolution area be accurately aligned with the direction of gaze. Two methods of eye position measurement with the Fiber Optic Helmet Mounted Display (FOHMD) have been developed and are described. This paper describes requirements necessary for successful eye tracking in aircraft simulators and introduces two approaches to monitoring eye position. In order to measure eye position over a wide field of view with sufficient accuracy, the oculometer must be able to measure various types of eye movements and also provide sufficient information to distinguish between eye movements and associated artifacts such as eye blinks and any anomalies introduced by spurious reflections or movement of the oculometer optics relative to the eye. In addition, the device must take into account variations in pupil size caused by changes in scene brightness and distinguish between pupil image displacements caused by actual eye movements or helmet slip. Under development are two oculometers that monitor both the center of the pupillary image and the corneal reflection and which possess both high temporal and spatial resolution.