Brain Region-Based Vigilance Assessment Using Electroencephalography and Eye Tracking Data Fusion

Frontiers Events is a rapidly growing calendar management system dedicated to the scheduling of academic events. This includes announcements and invitations, participant listings and search functionality, abstract handling and publication, related events and post-event exchanges. Whether an organizer or participant, make your event a Frontiers Event!

To develop and evaluate the concept of a non-invasive machine learning (ML) approach for detecting hypoglycaemia based exclusively on combined driving (CAN) and eye tracking (ET) data. We first developed and tested our ML approach in pronounced hypoglycaemia, and then we applied it to mild hypoglycaemia to evaluate its early warning potential. For this, we conducted two consecutive, interventional studies in individuals with type 1 diabetes. In study 1 (n=18), we collected CAN and ET data in a driving simulator during euglycaemia and pronounced hypoglycaemia (blood glucose [BG] 2.0-2.5 mmol L-1 ). In study 2 (n=9), we collected CAN and ET data in the same simulator but in euglycaemia and mild hypoglycaemia (BG 3.0-3.5 mmol L-1 ). Here, we show that our ML approach detects pronounced and mild hypoglycaemia with high accuracy (area under the receiver operating characteristics curve 0.88 ± 0.10 and 0.83 ± 0.11, respectively). Our findings suggest that an ML approach based on CAN and ET data, exclusively, enables detection of hypoglycaemia while driving. This provides a promising concept for alternative and non-invasive detection of hypoglycaemia.

Early Autism Spectrum Disorder (ASD) detection is important for early intervention.This study investigates the potential of eye-tracking (ET) data combined with machine learning (ML) models to classify ASD and Typically Developed (TD) children.Using a publicly available dataset, five ML models were evaluated: Support Vector Machine (SVM), Random Forest, Convolutional Neural Network (CNN), Artificial Neural Network (ANN), and Random Forest improved with Convolutional Filters (ConvRF).The models were trained and tested using a set of evaluation metrics, including accuracy, precision, recall, F1-score, and ROC Area Under the Curve (AUC).Among these, the ConvRF model attained superior performance, achieving a recall of 90% and an AUC of 88%, indicating its robustness in identifying ASD children.These results highlight the model's effectiveness in ensuring high sensitivity, which is critical for early ASD detection.This study shows the promise of combining ML and eye-tracking technology as accessible non-invasive tools for enhancing early ASD detection, resulting in timely and personalized interventions.Limitations and recommendations for future research are also included.

A classification algorithm of an SSVEP brain-Computer interface based on CCA fusion wavelet coefficients

In this paper, we propose a novel vigilance enhancement method based on audio stimulation of pure tone at 250 Hz. We induced two different levels of vigilance state; vigilance decrement (VD) and vigilance enhancement (VE). The VD state was induced by performing a modified version of the Stroop Color-Word Task (SCWT) for approximately 45 minutes. Likewise, the VE state was induced by incorporating audio stimulation of 250 Hz into the SCWT for 45 minutes. We assessed the levels of vigilance on 20 healthy subjects by utilizing Electroencephalogram (EEG) signals and machine learning. The EEG signals were analyzed using four types of entropies; Approximate Entropy (AE), Sample Entropy (SE), Fuzzy Entropy (FE), and Differential Entropy (DE). We then quantified vigilance levels using statistical analysis and support vector machines (SVM) classifier. We found that the proposed VE method has significantly reduced the reaction time (RT) by 44% and improved the accuracy of target detection by 25%, (p <; 0.001) compared to VD state. Besides, we found that 30 min of audio stimulation has reduced the RT by 32% from the beginning to the end of VE phase of the experiment. The entropy measures show that the temporal profile of the EEG signals has significantly increased with VE. The classification results showed that SVM technique with DE features across all frequency bands can detect VE levels with accuracy varying between (92.10± 02.24)% to (98.32± 01.14)%, sensitivity of (92.50± 02.33)% to (98.66± 01.00)%, and specificity of (91.70± 02.32)% to (97.99± 01.05)%. Results also showed that the classification performance using DE has outperformed the other entropy measures by an average of +8.07%. Our results demonstrate the effectiveness of the proposed 250 Hz audio stimulation method in improving vigilance level and suggest using it for future cognitive enhancement studies.

In the future, cognitive activity will be tracked in the same way how physical activity is tracked today. Eye-tracking technology is a promising off-body technology that provides access to relevant data for cognitive activity tracking. For building cognitive state models, continuous and longitudinal collection of eye-tracking and self-reported cognitive state label data is critical. In a field study with 11 students, we use experience sampling and our data collection system esmLoop to collect both cognitive state labels and eye-tracking data. We report descriptive results of the field study and develop supervised machine learning models for the detection of two eye-based cognitive states: cognitive load and flow. In addition, we articulate the lessons learned encountered during data collection and cognitive state model development to address the challenges of building generalizable and robust user models in the future. With this study, we contribute knowledge to bring eye-based cognitive state detection closer to real-world applications.

Stress is a feeling of emotional or physical tension, this makes a serious influence among undergraduates. There is a lack of stress prediction techniques that can detect what sort of stress level undergraduates are having from time to time. This research explored the prediction of undergraduate stress levels using eye-tracking. In this study, 306 PSS-10 data sets and 30600 eye-tracking data (Images) were collected from undergraduates at the University of Ruhuna using a questionnaire and a third-party eye-tracker application. PSS-10 (Perceived Stress Scale −10) and CNN (Convolutional Neural Network) was used to predict undergraduate stress levels. Stress levels are determined by the PSS-10 analysis, and divided into three classes: High, Moderate, and Low. Eye tracking data and stress classes are correlated in data pre-processing phase. The eye tracking images takes as input for well define eye tracking classification model; the model predict the stress level of the given undergraduate eye tracking data. According to the results, it was concluded that 19.7% of the undergraduates were in very high level of stress, 71.8% were in moderate level of stress and 8.5% were in low level of stress. However, it is certain that most of the undergraduates suffer from moderate levels of stress. The research will help predict undergraduate stress levels more accurately, and aid undergraduates managing their stress levels in academic life.

Color vision deficiency (CVD), a common visual impairment, affects individuals' ability to differentiate between various colors due to malfunctioning or absent color photoreceptors in the retina. Currently available diagnostic tests require a behavioral response, rendering them unsuitable for individuals with limited physical and communication abilities, such as those with locked-in syndrome. This study introduces a novel, non-invasive method that employs brain signals, specifically Steady-State Visually Evoked Potentials (SSVEPs), along with Ishihara plates to diagnose CVD. This method aims to provide an alternative diagnostic tool that addresses the limitations of current tests. Electroencephalography (EEG) recordings were obtained from 16 subjects, including 5 with CVD (specifically Deuteranomaly), using channels O1, O2, Pz, and Cz. The subjects were exposed to visual stimuli at frequencies of 15 Hz and 18 Hz to assess the proposed method. The subjects focused on specific visual stimuli in response to questions related to the Ishihara plates. Their responses were analyzed to determine the presence of CVD. Feature extraction was performed using Power Spectral Density (PSD), Canonical Correlation Analysis (CCA), and a combined PSD + CCA, followed by classification to categorize subjects into two classes: normal vision and CVD. The results indicate that the proposed method effectively diagnoses CVD in individuals with limited communication abilities. The classification accuracy of SSVEP exceeded 75% across the three classifiers: Decision Tree (DT), K-Nearest Neighbors (KNN), and Support Vector Machine (SVM). The SVM classifier demonstrated higher accuracy compared to the other classifiers, exceeding 90%. These observations suggest that the SVM classifier, utilizing the combined feature set of PSD + CCA, may be the most effective in this classification task. These findings demonstrate that the proposed method is an accurate and reliable diagnostic tool for CVD, particularly for individuals unable to communicate.

How do we come to the decision that we like a face? This thesis investigates this important aspect of social processing and communication by examining preference decisions for faces and the role that visual behavior plays in the process. I present a series of studies designed to investigate face preference formation and gaze patterns using eye-tracking and self-reported preference ratings. I tested healthy control subjects and two clinical populations known to have deficits in social processing: people with autism and patients with amygdala lesions. In studies one and two, I explore whether known social cognition deficits in people with autism and amygdala lesions also impair subjective decision-making regarding the attractiveness of faces. In study three, I investigate the flexibility of rule-based visual strategies used by these populations during face perception. Additionally, I present a custom algorithm developed to process raw eyetracking data, which was used to analyze all eyetracking data in this thesis. People with autism and patients with amygdala lesions are known to have general deficits in social processing, including difficulty orienting toward and evaluating faces. Nevertheless, I find that their behavior is markedly similar in many areas where we would expect them to have abnormalities or deficiencies. Their preference decisions when judging facial attractiveness were highly correlated with those made by controls, and both groups showed the same biases for familiar faces over novel faces. In addition, people with autism exhibit the same visual sampling behavior linking preference and attentional orienting, but reach their decisions faster than controls and also appear insensitive to the difficulty of the choice. Finally, gaze to the eye region appears normal in the absence of an explicit decision-making task, but only when analyzed in a similar manner as previous studies. However, when face sub-regions were analyzed in greater detail, people with autism demonstrate abnormalities in face gaze patterns, failing to emphasize the most information-rich regions of the face. Furthermore, people with autism demonstrate impairments in their ability to update those gaze patterns to accommodate different viewing restrictions. Taken together, these findings support the idea that the normal formation of face preferences can be preserved in the presence of general social processing impairments. Patterns in the eyetracking and behavioral data indicate that this is made possible, in part, by compensatory atypical processing and visual strategies.

The goal of this study was to use eye-tracking and log-file data to investigate the impact of prior knowledge on college students’ (N = 194, with a subset of n = 30 for eye tracking and sequence mining analyses) fixations on (i.e., looking at) self-regulated learning-related areas of interest (i.e., specific locations on the interface) and on the sequences of engaging in cognitive and metacognitive self-regulated learning processes during learning with MetaTutor, an Intelligent Tutoring System that teaches students about the human circulatory system. Results revealed that there were no significant differences in fixations on single areas of interest by the prior knowledge group students were assigned to; however there were significant differences in fixations on pairs of areas of interest, as evidenced by eye-tracking data. Furthermore, there were significant differences in sequential patterns of engaging in cognitive and metacognitive self-regulated learning processes by students’ prior knowledge group, as evidenced from log-file data. Specifically, students with high prior knowledge engaged in processes containing cognitive strategies and metacognitive strategies whereas students with low prior knowledge did not. These results have implications for designing adaptive intelligent tutoring systems that provide individualized scaffolding and feedback based on individual differences, such as levels of prior knowledge.

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

<title>Abstract</title> Marketing plays a crucial role in enhancing consumer awareness and affinity towards a product. Consumer research examines the motivations, trends, and patterns underlying purchasing behavior, thereby enabling institutions to refine their offerings and marketing strategies to improve their alignment with consumer decision-making and increase purchase probability. Consumer neuroscience aims to study the neurological factors influencing consumer preference, which cannot be determined using traditional methods like questionnaires and self-assessment. Machine learning techniques have proven effective in this domain, yet deep learning architectures that effectively fuse electroencephalography (EEG) signals with eye-tracking (ET) data remain largely unexplored. In this study, we propose a novel hybrid "Wide &amp; Deep" framework for multimodal classification of consumerpreference using EEG and eye-tracking data from the Neuma dataset. The proposed architecture employs an InceptionTime network with early fusion to process concatenated multimodal input (25 channels), extracting multi-scale temporal featuresthrough parallel convolutions with varying kernel sizes (10, 20, 40). The learned 128-dimensional neural embeddings are combined with 44 time-domain statistical features (temporal mean and standard deviation across all channels) and passed to a CatBoost gradient boosting classifier, forming a neural-symbolic ensemble. A modality dropout strategy (30% probability) is introduced during training to prevent over-reliance on eye-tracking signals and force robust EEG representation learning. The framework employs a grokking-oriented training regime with extended epochs (100), high weight decay (0.02), and strict checkpointing based on validation loss. The model is evaluated using 5-fold stratified cross-validation to assess generalizability. The proposed framework achieves a mean accuracy of 77.51% (±0.49%) and a weighted F1-score of 74.60% (±0.21%) in classifying consumer preferences, representing a significant improvement over prior methods on this dataset. This study demonstrates the effectiveness of multi-scale convolutional architectures combined with gradient boosting ensemble methodsfor multimodal consumer neuroscience applications.

Skill acquisition in the manufacturing industry is a crucial aspect of optimising performance, efficiency, and safety in complex work environments. Human factors play a significant role in skill acquisition, encompassing factors such as cognitive processes, perception, decision-making, and physical interactions within the work environment. Eye tracking data has emerged as a valuable tool for studying skill acquisition (Toker et al., 2014) in the manufacturing industry, offering insights into workers' visual attention, cognitive strategies, and task performance. Currently, in manufacturing settings, skill acquisition involves the mastery of various tasks, from operating machinery and assembly line processes to quality control and troubleshooting. Effective skill acquisition is essential for ensuring consistent product quality, minimising errors, and maximizing productivity. Human factors, including attentional processes, perception, and decision-making, influence how workers acquire and apply new skills in these dynamic and often high-pressure environments (Mark et al., 2020).The aim of the current research is to understand the duration required to acquire skills through procedural learning and the transition to routine development occurs when leaned behaviour becomes habitual and routine. Furthermore, the research also aims to understand how mental and physical fatigue impact their performance with manual quality control tasks. The study aims to showcase preliminary results regarding human factors and performance variations. Participants completed an inspection task that involved an industrial component (monitor) for their serial number, visual and tactile quality under a control condition: control (no manipulation) measuring physical demand and stress levels during each monitor inspection. Physiological measures were captured using a Empatica E4 wristband (capturing electrodermal activity (EDA), heart rate, skin temperature) and eye tracking was performed with Tobii Glasses 3, as well as subjective measures of performance via NASA TLX. The results from the physiological data show that the initial 10 minutes of the task showed a positive significant correlation between EDA and NASA Performance score (Spearman rho = .675, p =.016), the second set of 10 minutes positively correlated EDA and NASA TLX temporal demands (Spearman rho = .757, p =.004), while final 10 minutes showed a positive correlation between EDA and NASA TLX physical demand (Spearman rho = .639, p =.025). Such results indicate that skill acquisition over time goes through several stages – individual's anxiety of their performance, then concerns for timely performance, and finally experiencing physical impact – as well as that EDA is good indicator of changing workload demands.The current study will present the findings of the stress and physical discomfort levels, physiological data and preliminary eye tracking data whereby results found that for the initial inspection of the first two components participants move between the instructions for the inspection and the actual inspection, however, analysis of the third component and further components revealed that attention is mainly focused on the component inspection with little fixations towards the instructions. There was also an increase in the development of skill acquisition whereby by participants displayed a decrease in error count (from 1 to 0 over the inspection of 10 monitors), instruction analysis and increase in time of task completions as the average time reduced from 6 minutes 07 seconds for the first component to 1 minute 51 seconds for the last component. (The results from the eye tracking data reveal participants’ gaze patterns during the experiment highlighting the time spent on task comprehension and completion suggesting the skill was acquired during the experimental task. In summary, the current study demonstrates the development of skills through a variety of measures showing temporal progression, mental demand and the physicality of manual handling tasks is a complex process influenced by various human factors, including attention, fatigue, and tactile knowledge. By leveraging physiological data, eye tracking and stress levels the research provides scope into how human skill acquisition is progressive through various factors that can reduce overtime to upkeep and maintain the skills of workers perform in the manufacturing industry.

Locating the center of the pupils is the most important foundation and the core component of gaze tracking. The accuracy of gaze tracking largely depends on the quality of images, but additional constraints and large amount of calculation make gaze tracking impractical on high-resolution images. Although some eye-gaze trackers can get accurate result, improving the accuracy of pupil feature on low-resolution images and accurately recognizing closed eye images are still common tasks in the field of gaze estimation. Our aim is to get the accurate localization of pupil center on low-resolution image. To this aim, we proposed a simple but effective method which can accurately locate pupil center in real time. The method first gets initial eye center based on improved scale-invariant feature transform (SIFT) descriptor and support vector machine (SVM) classifier, and then gets final position of the pupil center through a size variable correction rectangular block. In this paper, comparing with the reported state-of-the-art methods,the experimental results demonstrate that our system can achieve a more accurate result on low-resolution images. On top of that, our approach shows robustness on closed eye images while some other methods would not recognize the closed eye images.

Distinguishing whether eye tracking data reflects reading or skimming already proved to be of high analytical value. But with a potentially more widespread usage of eye tracking systems at home, in the office or on the road the amount of environmental and experimental control tends to decrease. This in turn leads to an increase in eye tracking noise and inaccuracies which are difficult to address with current reading detection algorithms. In this paper we propose a method for constructing and training a classifier that is able to robustly distinguish reading from skimming patterns. It operates in real time, considering a window of saccades and computing features such as the average forward speed and angularity. The algorithm inherently deals with distorted eye tracking data and provides a robust, linear classification into the two classes read and skimmed. It facilitates reaction times of 750ms on average, is adjustable in its horizontal sensitivity and provides confidence values for its classification results; it is also straightforward to implement. Trained on a set of six users and evaluated on an independent test set of six different users it achieved a 86% classification accuracy and it outperformed two other methods.