Multimodal Emotion Recognition Using a Modified Dense Co-Attention Symmetric Network

In this paper, we call the point on an object where human looks a subjective viewpoint and we call the point where the object and the straight line going through the center of pupil and the center of the fovea crosses an objective viewpoint. We realize eye movements in reading on the computer when an objective viewpoint is a subjective viewpoint and when an objective viewpoint is shifted to the right of a subjective viewpoint and we investigate the characteristics of eye movements. First, we realize a computer-simulation of human eye movements in reading. Secondly, the superiority of the right visual field appears in the foveated vision realized by shifting an objective viewpoint to the right of a subjective viewpoint. And we confirm that the gap between the subjective viewpoint and the objective viewpoint is important by comparing the characteristics of human eye movements to the characteristics of eye movements when an objective viewpoint is shifted to the right of a subjective viewpoint. Furthermore, we perform eye movements on the computer while shifting an objective viewpoint to the right of a subjective viewpoint and measure the speed at which an English sentence is read. We conclude that reading speed when the foveated vision has the superiority of the right visual field is faster than that when the foveated vision doesn't have it.

Vision is one of the most important human senses, accounting for most of the external information pedestrians receive while crossing the street. However, distracted mobile phone usage during street crossing consumes pedestrians’ cognitive resources and diverts their visual attention. As a result, pedestrians may be unable to fully concentrate on observing the traffic environment and effectively planning their crossing path and behavior. This study evaluated the effect of pedestrian behavioral activities at street crossings on eye-movement (EM) characteristics. The crossing tasks were natural behavior, voice call, text messaging, and listening to music. The tasks were further categorized as simple or complex. A total of 29 participants were recruited in Nanjing: 18 males (62.1%) and 11 females (37.9%) with an average age of 23.59 years (SD = 2.44). The Friedman test was used to analyze differences in saccade frequency, fixation time, browsing number, and browsing time across different scenarios. Text messaging had the most significant impact on pedestrians’ EM characteristics, followed by voice call; music listening had a relatively weaker effect. Secondary task difficulty influenced the percentage of browsing, viewing, and to some extent gaze time. On the other hand, music rhythm and style only partially influenced the percentage of gaze and gaze time. Mobile phones substantially affected pedestrians’ EM characteristics and attention allocation for the same level of secondary task difficulty. These findings contribute to a better understanding of pedestrians’ visual characteristics under distracted mobile phone usage conditions and provide valuable insights for developing appropriate measures to enhance pedestrian safety.

BackgroundEye movement is a vital indicator of neurodegenerative diseases, brain health, and behavior. However, existing knowledge is limited to patient populations or cross-sectional samples. Little is known about eye movement in association with longitudinal cognitive and mobility decline in aging.ObjectiveInvestigate relationships between eye movement features with cognitive impairment, including Alzheimer's disease (AD), and longitudinal decline in cognition and mobility.MethodsIn 543 Baltimore Longitudinal Study of Aging participants (mean age = 71 years), we examined associations of eye movements with cognitive impairment of any severity, vascular conditions, and falls using logistic regression, and up to 18-year longitudinal changes in cognition and mobility using linear mixed-effects models. Four eye movement features (saccade, smooth pursuit, vergence, optokinetic nystagmus) were derived from a portable eye-tracking perimeter (Neurolign Dx100) using machine learning Least Absolute Shrinkage Selection Operator regression.ResultsHigher saccade, smooth pursuit, and vergence were bivariately or marginally associated with lower odds of cognitive impairment, including AD, and vascular diseases. In age- and sex-adjusted models, higher saccade was associated with slower declines in cognition (attention: Trail Making Test-Part A), mobility, and balance. Higher smooth pursuit was associated with slower decline in mobility and balance. Higher vergence was associated with slower cognitive decline (executive function: Trail Making Test-Part B; visuoperceptual speed: Digit Symbol Substitution Test). Higher optokinetic nystagmus was associated with lower odds of falls and slower decline in balance.ConclusionsSelect eye movement features may be indicators of age-related cognitive and mobility decline. Future studies are warranted to investigate underlying neuroimaging markers and brain pathology.

The onset of fatigue is dangerous in areas of activity that require high concentration of human attention, such as air traffic controllers, nuclear power plant operators, etc. It should be noted that these types of activities are characterized by the fact that most of the time the employee sits at the workplace and his gaze is directed at the monitor. The paper presents a method for dynamic detection of PC operator fatigue based on eye movement characteristics. The method for dynamic detection of fatigue implements a training scenario for a fatigue detection model and a fatigue detection scenario. Within the training scenario, eye movement characteristics are calculated and correlations with fatigue test results are searched for. Within the fatigue detection scenario, eye movement characteristics that most strongly correlate with fatigue are selected. These characteristics can also be divided by the types of physical events on which they are based. We can distinguish such characteristics as speed, time, quantity, size, percentage, frequency and ratio characteristics. To find correlations between eye movement characteristics and fatigue, a dataset of eye movements and the results of tests and questionnaires such as the go/no-go task, the Landolt ring test, and the VAS-F questionnaire were analyzed. The dataset consists of gaze coordinate recordings from 15 participants acting as PC operators. To determine the degree of fatigue, the participant completed the VAS-F questionnaire. The Landolt ring correction test is a test used to measure attention concentration. The labeled dataset is used to train a machine learning model that detects fatigue. The experimental results showed that using the characteristics selected in the study yielded the most promising results. This approach allowed us to achieve the highest F-measure and the best average accuracy, indicating the overall reliability of the model.

ObjectiveDepression is a complex affective disorder characterized by high prevalence and severe impact, commonly presenting with cognitive impairment. The objective diagnosis of depression lacks precise standards. This study investigates eye movement characteristics during emotional face recognition task (EFRT) in depressive patients to provide empirical support for objective diagnosis.MethodsWe recruited 43 patients with depression (Depressive patients, DP) from a psychiatric hospital and 44 healthy participants (Healthy Control, HC) online. All participants completed an EFRT comprising 120 trials. Each trial presented a gray screen for 800 ms followed by a stimulus image for judgment. Emotions were categorized as positive, neutral, or negative. Eye movement trajectories were recorded throughout the task. Latency of First Fixation (LFF), Latency of First Fixation for Eye AOI, and Latency of First Fixation for Mouth AOI were used as representative indicators of early attention, Proportion of Eye AOI, and Proportion of Mouth AOI as measures of intermediate attention, Accuracy (ACC) and Reaction Time (RT) as behavioral indicators of late-stage attention. In this study, these metrics were employed to explore the differences between patients with depression and healthy individuals.ResultsCompared to healthy participants, individuals with depression exhibit longer first fixation latencies on the eyes and mouth during the early attention stage of emotional face recognition, indicating an avoidance tendency toward key facial recognition cues. In the mid-to-late attention stages, depressive individuals show an increased fixation ratio on the eyes and a decreased fixation ratio on the mouth, along with lower accuracy and longer response times. These findings suggest that, relative to healthy individuals, individuals with depression have deficits in facial recognition.ConclusionThis study identified distinct attention patterns and cognitive deficits in emotional face recognition among individuals with depression compared to healthy individuals, providing an attention-based approach for exploring potential clinical diagnostic markers for depression.

To explore different operators’ eye movement characteristics, which can reflect their degree of attention and hazard identification ability in university laboratories, experiments were carried out with 60 junior students. The laboratories were divided into five areas by function and hazard type. The subjects were divided into two groups based on their safety experience. Using typical university laboratory images as stimulus materials, a Tobii X2 eye tracker was used to measure and record the eye movement parameters of the subjects in five laboratory areas. The results showed that the subjects exhibited greater interest in the equipment area (attracting the focus of 100.00% of the subjects) and less interest in the wireway area (attracting the focus of 46.67% of the subjects); no significant difference was observed between subjects who were or were not specialized in safety. Most subjects could identify the hazard sources in the five areas but paid little attention to electrical sockets. Thus, countermeasures including reasonably matching the color and layout in laboratories and popularizing laboratory safety education are proposed, which will provide a basis for strengthening safety management in university laboratories.

Oculomotor behavior and perceptual strategies in complex tasks

According to in-depth research on the perception ability of dangerous omens of excellent drivers, references can be provided for the development of brain-like intelligence and its transplantation, as well as applications in the field of autonomous driving, which will improve the active safety and intelligence level of vehicles. Previous studies have shown that there is indeed a dangerous omen before an accident occurs. However, current studies are still unclear about the bio-psychophysiological characteristics exhibited by drivers with high levels of sensory agility when they anticipate potential warning signs, and there is no method for screening such drivers who can perceive dangerous omens proposed by any research. To address the above issues, this paper conducts in-depth research. Firstly, through designing dangerous scenarios and conducting hazard perception tests, we collect physiological, psychological, and physical data, such as drivers' bioelectrical signals (electroencephalogram and electrocardiogram) and eye movements. Secondly, through playing back experimental videos, actively questioning drivers, and analyzing local changes in their electroencephalogram data, the driver's ability to identify a dangerous omen and the moment of perception are determined. Thirdly, based on techniques such as the Kolmogorov-Smirnov test and the Mann-Whitney U test, the differences in bioelectrical and eye movement characteristics between drivers who can perceive a dangerous omen and others can be further revealed. Finally, the driver's bioelectrical and eye movement characteristics are used as latent variables, and their corresponding data are utilized as observation indicators. We construct a structural equation model for screening drivers capable of perceiving a dangerous omen and conduct calibration and validation. This study provides inspirational ideas for empowering vehicles to identify potential hazards, advancing end-to-end and other higher-level autonomous driving technologies, and further enhancing road traffic safety.

In this paper, a user independent human computer interface system using eye movement and blink feature detection is introduced. A hands free interface between computer and human can potentially replace the traditional human computer interface devices like mouse, keyboard etc. This technology is intended to give functionality to peoples with severe motor disabilities to control a computer by just moving their eyes. This paper describes a method of controlling the mouse cursor on a computer screen using the electrical potentials developed by eye movements known as Electrooculography. Electrooculography (EOG) signal is used to detect eye movement and blink features. The EOG signal is recorded from electrodes placed at appropriate positions around the eyes. The captured EOG signal is then analyzed to detect and classify eye movement features of interest. The detected features were then used to generate control signals to control a mouse cursor. The cursor control application is implemented offline. The detection accuracy for blinks, horizontal and vertical saccades are 100%, 97% and 93% respectively.

Besides allowing us to perceive our surroundings, eye movements are also a window into our mind and a rich source of information on who we are, how we feel, and what we do. Here we show that eye movements during an everyday task predict aspects of our personality. We tracked eye movements of 42 participants while they ran an errand on a university campus and subsequently assessed their personality traits using well-established questionnaires. Using a state-of-the-art machine learning method and a rich set of features encoding different eye movement characteristics, we were able to reliably predict four of the Big Five personality traits (neuroticism, extraversion, agreeableness, conscientiousness) as well as perceptual curiosity only from eye movements. Further analysis revealed new relations between previously neglected eye movement characteristics and personality. Our findings demonstrate a considerable influence of personality on everyday eye movement control, thereby complementing earlier studies in laboratory settings. Improving automatic recognition and interpretation of human social signals is an important endeavor, enabling innovative design of human–computer systems capable of sensing spontaneous natural user behavior to facilitate efficient interaction and personalization.

Eye movement is a new kind of feature for biometrical recognition, it has many advantages compared with other features such as fingerprint, face, and iris. It is not only a sort of static characteristics, but also a combination of brain activity and muscle behavior, which makes it effective to prevent spoofing attack. In addition, eye movements can be incorporated with faces, iris and other features recorded from the face region into multimode systems. In this paper, we do an exploring study on eye movement identification based on the eye movement datasets provided by Komogortsev et al. in 2011 with different classification methods. The time of saccade and fixation are extracted from the eye movement data as the eye movement features. Furthermore, the performance analysis was conducted on different classification methods such as the BP, RBF, ELMAN and SVM in order to provide a reference to the future research in this field.

Color fusion and rivalry are two key information integration mechanisms in binocular vision, representing the visual system's processing patterns for consistent and conflicting inputs, respectively. This study hypothesizes that there are quantifiable differences in eye movement indicators under states of binocular color fusion and rivalry, which can be verified through multi-paradigm eye movement experiments. The experiment recruited eighteen subjects with normal vision (nine males and nine females), employing the Gaze Stability paradigm, Straight Curve Eye Hopping paradigm, and Smoothed Eye Movement Tracking paradigm for eye movement tracking. Each paradigm included a binocular color rivalry experimental group (R-G) and two binocular color fusion control groups (R-R, G-G). Data analysis indicates significant differences in indicators such as Average Saccade Amplitude, Median Saccade Amplitude, and SD of Saccade Amplitude between binocular color fusion and rivalry states. For instance, through Z-Score normalization and cross-paradigm merged analysis, specific ranges of these indicators were identified to distinguish between the two states. When the Average Saccade Amplitude falls within the range of -0.905--0.693, it indicates a state of binocular color rivalry; when the range is 0.608-1.294, it reflects a state of binocular color fusion. Subsequently, ROC curve analysis confirmed the effectiveness of the experimental paradigms in analyzing the mechanisms of binocular color fusion and rivalry, with AUC values of 0.990, 0.741, and 0.967, respectively. These results reveal the potential of eye movement behaviors as biomarkers for the dynamic processing of visual conflicts. This finding provides empirical support for understanding the neural computational models of binocular vision and lays a methodological foundation for developing visual impairment assessment tools based on eye movement features.

The characteristics of children’s eye movements during reading change as they graduallybecome better readers. However, few eye tracking studies have investigated children’sreading and reading development and little is known about the relationship between reading-related eye movement measures and reading assessment outcomes. We recorded andanalyzed three basic eye movement measures in an ecologically valid eye-tracking set-up.The participants were Swedish children (n = 2876) who were recorded in their normalschool environment. The relationship between eye movements and reading assessmentoutcomes was analyzed in using linear mixed effects models. We found similar age-relatedchanges in eye movement characteristics as established in previous studies, and that eyemovements seem to correlate with reading outcome measures. Additionally, our resultsshow that eye movements predict the results on several tests from a word reading assessment.Hence eye tracking may potentially be a useful tool in assessing reading development.

Reading is a complex cognitive process. Through eye tracking techniques the cognitive features can be observed in depth, and therefore a better grasp of the instant cognitive processing of language can be achieved. Compared to traditional methods, the greatest advantage of eye tracking is that is that the reader proceeds with reading coherently and fluently without interrupt and interference, which secures the results in higher ecological validity. This paper is mainly about studies on eye movement features in ESL reading, and particularly illustrates eye movement patterns, the relationship between eye movement and perceptual processing, and eye movement control, aiming at providing some basic evidences for explaining the relationship between eye movement behavior and cognitive processing thus helping to solve some theoretical problems in reading. DOI: http://dx.doi.org/10.11591/telkomnika.v13i2.6936

The present paper aims at describing the concept of how the interaction with computer devices may benefit from joint input of users emotional state and his/her eye movements characteristics. We claim that emotions regulate visual attention, changing its formal characteristics and thus in turn influence the user's behaviour. We hypothesised that during interaction with an interface, positive mood will enhance exploratory eye movements. Participants of the study completed positive or neutral mood induction procedure followed by free viewing of classical paintings on computer screen. During the whole experimental procedure, the users' eye movements as well as their brain activity were recorded. Results confirm that positive mood changes the dynamics of visual attention. When completing a computer task, people's eye movements indicate shift from ambient toward more focal attention mode. However, positive mood slows down that process, fostering visual exploration of the presented stimuli. Obtained results are discussed in the context of self-adaptive user interfaces idea suggesting that the information about the users' emotional state together with visual attention characteristics and gaze information could be used as an input channel in order to create better user experience.