Estimating heading during real and simulated eye movements

The patterns of optic flow seen during self-motion can be used to determine the direction of one's own heading. Tracking eye movements which typically occur during everyday life alter this task since they add further retinal image motion and (predictably) distort the retinal flow pattern. Humans employ both visual and nonvisual (extraretinal) information to solve a heading task in such case. Likewise, it has been shown that neurons in the monkey medial superior temporal area (area MST) use both signals during the processing of self-motion information. In this article we report that neurons in the macaque ventral intraparietal area (area VIP) use visual information derived from the distorted flow patterns to encode heading during (simulated) eye movements. We recorded responses of VIP neurons to simple radial flow fields and to distorted flow fields that simulated self-motion plus eye movements. In 59% of the cases, cell responses compensated for the distortion and kept the same heading selectivity irrespective of different simulated eye movements. In addition, response modulations during real compared with simulated eye movements were smaller, being consistent with reafferent signaling involved in the processing of the visual consequences of eye movements in area VIP. We conclude that the motion selectivities found in area VIP, like those in area MST, provide a way to successfully analyze and use flow fields during self-motion and simultaneous tracking movements.

Terrestrial navigation naturally involves translations within the horizontal plane and eye rotations about a vertical (yaw) axis to track and fixate targets of interest. Neurons in the macaque ventral intraparietal (VIP) area are known to represent heading (the direction of self-translation) from optic flow in a manner that is tolerant to rotational visual cues generated during pursuit eye movements. Previous studies have also reported that eye rotations modulate the response gain of heading tuning curves in VIP neurons. We tested the hypothesis that VIP neurons simultaneously represent both heading and horizontal (yaw) eye rotation velocity by measuring heading tuning curves for a range of rotational velocities of either real or simulated eye movements. Three findings support the hypothesis of a joint representation. First, we show that rotation velocity selectivity based on gain modulations of visual heading tuning is similar to that measured during pure rotations. Second, gain modulations of heading tuning are similar for self-generated eye rotations and visually simulated rotations, indicating that the representation of rotation velocity in VIP is multimodal, driven by both visual and extraretinal signals. Third, we show that roughly one-half of VIP neurons jointly represent heading and rotation velocity in a multiplicatively separable manner. These results provide the first evidence, to our knowledge, for a joint representation of translation direction and rotation velocity in parietal cortex and show that rotation velocity can be represented based on visual cues, even in the absence of efference copy signals.

When tracking or reaching a visual target with the hand, retinal information is key to guide the hand. Although the contribution of extraretinal signals associated with saccades is established for reaching movements, less is known about extraretinal contribution during manual tracking in which smooth pursuit dominates. To fill this gap, we conducted a study in which eye and hand tracking performance were compared across multiple conditions in which eye and hand movements performed similar/dissimilar tasks. In the SINGLE condition, participants (n = 26) were asked to track a unique target with both eyes and hand. In the DOUBLE condition, their eyes and hand had to track separate targets that followed the same trajectory but displayed on separate hemifields. In the FIX condition, participants tracked one target with the hand while fixating eyes away from that target. We reasoned that if extraretinal information is key for manual tracking, it should be less altered in SINGLE and DOUBLE than FIX. Indeed, results showed that not only manual tracking was better in DOUBLE than FIX, but was nearly equivalent to SINGLE. Surprisingly, foveating the target provided only modest benefits to manual tracking. Finally, analyses confirmed that smooth pursuit contributed more than saccades to overall eye movements, making it the primary contributor to extraretinal information. Taken together, these results show that foveating the target is not mandatory for accurate manual tracking as long as eyes and hand perform congruent tracking/motion, and speak for a key role of extraretinal information provided by smooth pursuit.NEW & NOTEWORTHY The contribution of extraretinal information associated with saccades is well established for reaching movements. Here, by using a continuous manual tracking task in which smooth pursuit dominates, we examined the contribution of extraretinal information. Critically, our findings show that extraretinal information is more essential than foveating the target for hand tracking performance. More generally, these results highlight the key role of smooth pursuit in supplying extraretinal signals for guiding the hand during manual tracking.

Modeling and simulation of human eye movement have a wide range of applications in many domains. Various attempts have been made to model and simulate eye movements in a physically accurate manner. All the existing models show limitations and problems in simulating secondary and tertiary eye movements. Recent investigation of pulley models (passive and active hypotheses) in representing human eye motion has recognized mathematical complexity in modeling eye behavior. Sophisticated techniques of modeling are required to investigate eye movements. This paper presents a procedure for eye movement simulation through geometrical modeling (an OpenSim script with its recent MATLAB binding) for binocular vision. First order neural dynamics with Millard's muscle model are used to actuate six Extra Ocular Muscles (EOMs). Pulse-step inputs are used to generate the muscle forces around the eye globe. The implemented model is successful in simulating horizontal and vertical movements of the human eye with respect to the prescribed activation. The developed technique is evaluated using responses from lumped parameter models and EOG recordings.

As we move forward in the environment, we experience a radial expansion of the retinal image, wherein the center corresponds to the instantaneous direction of self-motion. Humans can precisely perceive their heading direction even when the retinal motion is distorted by gaze shifts due to eye/body rotations. Previous studies have suggested that both retinal and extra-retinal strategies can compensate for the retinal image distortion. However, the relative contributions of each strategy remain unclear. To address this issue, we devised a two-alternative-headings discrimination task, in which participants had either real or simulated pursuit eye movements. The two conditions had the same retinal input but either with or without extra-retinal eye movement signals. Thus, the behavioral difference between conditions served as a metric of extra-retinal contribution. We systematically and independently manipulated pursuit speed, heading speed, and the reliability of retinal signals. We found that the levels of extra-retinal contributions increased with increasing pursuit speed (stronger extra-retinal signal), and with decreasing heading speed (weaker retinal signal). In addition, extra-retinal contributions also increased as we corrupted retinal signals with noise. Our results revealed that the relative magnitude of retinal and extra-retinal contributions was not fixed but rather flexibly adjusted to each specific task condition. This task-dependent, flexible integration appears to take the form of a reliability-based weighting scheme that maximizes heading performance.

To solve the problem of user visual attention distribution and spatial misalignment of interface elements in current user interface layout design due to reliance on subjective experience, this paper proposes an optimization method for fashion brand e-commerce user interface design that uses eye tracking simulation and integrates brand characteristics. This paper builds a fashion-specific visual behavior database through multimodal eye movement data collection, and uses the spatiotemporal attention mechanism of the Transformer-XL (Transformer with Extra Long Context) model to predict the gaze hotspots and scanning paths of users in dynamic tasks; then, this paper designs a multi-agent reinforcement learning optimizer, encodes aesthetic rules such as brand logo size and main color ratio as hard constraints, and generates an interface space distribution plan through element competition-collaboration game; finally, this paper develops a real-time interactive prototype based on the Unity engine to achieve dynamic layout closed-loop optimization driven by the collaborative efforts of brand constraints and eye movement simulation heat maps. Experimental results show that this method reduces the task completion time by 32% in the optimization of the light luxury clothing homepage, the overlap between the simulated and real eye movement hot spots reaches 88%, and the brand consistency score increases by 46% (from 3.2 to 4.7), significantly reducing the user's cognitive load and improving the user experience. The conclusion confirms that by integrating eye movement behavior quantification with brand characteristics, it is possible to break through the traditional design's reliance on static aesthetics, provide a “cognitive adaptation-brand expression” dual-goal collaborative intelligent design paradigm for fashion interfaces, and promote the simultaneous improvement of user conversion rate and brand value.

Cognitive dysfunction is the most common and important nonmotor symptom in Parkinson’s disease (PD) and can occur at any stage. However, there is still a lack of effective biomarkers to evaluate the decline in cognitive function and predict the progression of the disease, especially in the early stage. At present, the cognitive scale is widely used to evaluate the cognitive function of patients with PD, but its sensitivity and accuracy are relatively limited, especially in the early identification of mild cognitive impairment. Eye movement tracking is an advanced neurophysiological measurement method that serves as a powerful means to study the relationship between behavior and neural mechanisms. In recent years, eye movement tracking has been found to provide a nonverbal and less cognitive method to measure the disease progress of patients with cognitive impairment. Moreover, there is a good correlation between eye movement tracking and the traditional cognitive assessment scale, indicating that eye movement tracking can be used to evaluate and monitor the cognitive status, disease severity, and disease progression of patients with PD. Compared to the traditional cognitive scale, the eye movement detected by the instrument has better objectivity and repeatability. Existing studies have found that executive dysfunction is one of the most important manifestations of cognitive dysfunction in patients with PD and is related to an increase in the error rate of the saccade, an increase in the disinhibition of the delayed saccade task, and a prolongation of the saccade reaction time. This suggests that eye movement measurement plays an important role in the early diagnosis, progression, and differential diagnosis of PD and may even help to predict the disease progression of patients with PD and cognitive impairment. In this article, we review the correlation between cognitive impairment and eye movement disorder in patients with PD.

Relationship between eye acceleration and retinal image velocity during foveal smooth pursuit in man and monkey.

Eyes are the window of the mind, and eye movement contains a lot of effective information. With the breakthrough of artificial intelligence and human-computer interaction, eye movement tracking technology has gradually moved to the foreground in recent years. This paper takes medical health as a typical scenario, carries out innovative research around the potential application of eye movement tracking technology in patient communication and human-computer interaction, proposes and implements an intention estimation system based on eye movement tracking. Based on the physiology, medicine and ergonomics of eye movement, the system makes original innovations, extracts a set of eye movement parameters with high recognition accuracy and high expression efficiency, summarizes eye movement instructions that can meet the application of silent environment, and finally forms a series of embryonic applications related to interactive instructions.

Validating and improving the correction of ocular artifacts in electro-encephalography

Eye and hand movement tracking has been proven to be a successful tool and is widely used to figure out characteristics of human cognition in language or visual processing (Just & Carpenter, 1976 Cognitive Psychology8441–480). Eye movement has proven to be a successful measure to figure out characteristics of human language and visual processing (Rayner, 1998 Psychological Bulletin124(3) 372–422). Recently, mouse tracking was used for social-cognition-like categorization of sex-atypical faces and studying spoken-language processes (Magnuson, 2005 PNAS102(28) 9995–9996; Spivey et al., 2005 PNAS102 10393–10398). Here, we present a framework that uses both eye gaze and hand movement simultaneously for analyzing the relation of them during memory retrieval. We tracked eye and mouse movements when the subject was watching a drama and playing a multimodal memory game (MMG), a cognitive task designed to investigate the recall memory mechanisms in watching video dramas (Zhang, 2009 AAAI 2009 Spring Symposium: Agents that Learn from Human Teachers 144–149). Experimental results show that eye tracking and mouse tracking provide complementary information about underlying cognitive processes. Also, we found some interesting patterns in eye-hand movement during multimodal memory recall.

Color fusion and rivalry are two key information integration mechanisms in binocular vision, representing the visual system's processing patterns for consistent and conflicting inputs, respectively. This study hypothesizes that there are quantifiable differences in eye movement indicators under states of binocular color fusion and rivalry, which can be verified through multi-paradigm eye movement experiments. The experiment recruited eighteen subjects with normal vision (nine males and nine females), employing the Gaze Stability paradigm, Straight Curve Eye Hopping paradigm, and Smoothed Eye Movement Tracking paradigm for eye movement tracking. Each paradigm included a binocular color rivalry experimental group (R-G) and two binocular color fusion control groups (R-R, G-G). Data analysis indicates significant differences in indicators such as Average Saccade Amplitude, Median Saccade Amplitude, and SD of Saccade Amplitude between binocular color fusion and rivalry states. For instance, through Z-Score normalization and cross-paradigm merged analysis, specific ranges of these indicators were identified to distinguish between the two states. When the Average Saccade Amplitude falls within the range of -0.905--0.693, it indicates a state of binocular color rivalry; when the range is 0.608-1.294, it reflects a state of binocular color fusion. Subsequently, ROC curve analysis confirmed the effectiveness of the experimental paradigms in analyzing the mechanisms of binocular color fusion and rivalry, with AUC values of 0.990, 0.741, and 0.967, respectively. These results reveal the potential of eye movement behaviors as biomarkers for the dynamic processing of visual conflicts. This finding provides empirical support for understanding the neural computational models of binocular vision and lays a methodological foundation for developing visual impairment assessment tools based on eye movement features.

Heading detection using motion templates and eye velocity gain fields.

The neural basis of smooth pursuit eye movements in the rhesus monkey brain

Predictive eye movements do not discriminate between dyslexic and control children