Novel Framework for Multimodal Classification of Consumer Preference using EEG signals and eye-tracking data

To develop and evaluate the concept of a non-invasive machine learning (ML) approach for detecting hypoglycaemia based exclusively on combined driving (CAN) and eye tracking (ET) data. We first developed and tested our ML approach in pronounced hypoglycaemia, and then we applied it to mild hypoglycaemia to evaluate its early warning potential. For this, we conducted two consecutive, interventional studies in individuals with type 1 diabetes. In study 1 (n=18), we collected CAN and ET data in a driving simulator during euglycaemia and pronounced hypoglycaemia (blood glucose [BG] 2.0-2.5 mmol L-1 ). In study 2 (n=9), we collected CAN and ET data in the same simulator but in euglycaemia and mild hypoglycaemia (BG 3.0-3.5 mmol L-1 ). Here, we show that our ML approach detects pronounced and mild hypoglycaemia with high accuracy (area under the receiver operating characteristics curve 0.88 ± 0.10 and 0.83 ± 0.11, respectively). Our findings suggest that an ML approach based on CAN and ET data, exclusively, enables detection of hypoglycaemia while driving. This provides a promising concept for alternative and non-invasive detection of hypoglycaemia.

In the future, cognitive activity will be tracked in the same way how physical activity is tracked today. Eye-tracking technology is a promising off-body technology that provides access to relevant data for cognitive activity tracking. For building cognitive state models, continuous and longitudinal collection of eye-tracking and self-reported cognitive state label data is critical. In a field study with 11 students, we use experience sampling and our data collection system esmLoop to collect both cognitive state labels and eye-tracking data. We report descriptive results of the field study and develop supervised machine learning models for the detection of two eye-based cognitive states: cognitive load and flow. In addition, we articulate the lessons learned encountered during data collection and cognitive state model development to address the challenges of building generalizable and robust user models in the future. With this study, we contribute knowledge to bring eye-based cognitive state detection closer to real-world applications.

Frontiers Events is a rapidly growing calendar management system dedicated to the scheduling of academic events. This includes announcements and invitations, participant listings and search functionality, abstract handling and publication, related events and post-event exchanges. Whether an organizer or participant, make your event a Frontiers Event!

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

The application of neuroscience and neuroengineering methodologies to the study of product marketing is a new research area at the intersection of neuromarketing, neuroeconomics and consumer neuroscience. This emerging multidisciplinary field aims to better understand consumers and their interaction with products and services beyond the traditional self-report surveys and articulated responses from focus groups. Consumers make decisions every day, especially when deciding what food products to purchase or consume, yet little is known about how advertising and preparation affects those decisions. Knowing the main factors that influence consumers' food and drink choices provides important information for better understanding factors that affect consumers' preferences and purchase decisions. Currently, there is a disparity that current market research is not able to address between consumer attitudes and actual purchasing behavior. Therefore, by better understanding the factors influencing consumer preferences, it could lead to better success of products in the market. With traditional self-reported measures, consumers may not be able to fully articulate their preferences. Therefore, use of objective measures could complement self-reported measures to provide a richer more in-depth analysis of consumer decision making and preferences. In this thesis, we take a neuroergonomics approach and dovetail traditional self-reported approaches with objective neural measures to better understand consumer tendencies and preferences and how they affect their consumption behavior. This thesis will provide several novel contributions to the knowledge base that can further consumer neuroscience research. First, we utilized a multimodal approach to better understand food consumption and preferences. By utilizing this method, it is possible to add information to understanding the user experience by combining self-reported and behavioral measures with neural physiological data. This will be able to provide a more comprehensive analysis. Second, we investigated the effect of machine user interface ergonomics on consumer experience in a naturalistic office setting. In a neuroergonomic study, we investigated two competing coffee machine interfaces while users interacted with the machines. Users prepared and consumed hot drinks in a realistic office setting and were continuously monitored with wireless brain and body sensors. By allowing participants to move around untethered in a simulated office environment where they are able to interact with the machines and the product, we are able to get better responses that would be representative of real-world preferences. Next, we investigated the effect of different hot beverages on cognition and on consumer preference. In a controlled multi-day study, we explored coffee and tea products and how they influence consumer preference and cognitive ability. By investigating the effects of the ingredients in different coffee and tea products will help gain knowledge into the effects they have on cognitive ability and consumer preferences. Further, there has been a trend of increasing products on the market that provide a benefit to the consumer. By comparing these products to traditional hot beverages, we were able to better understand how these types of product further affect consumer preferences and consumption behavior. Lastly, we investigated the role of food advertisements on product likeability and how they affect consumption behavior. Advertising for food has been shown to have a significant influence on food consumption behaviors. By investigating the neural correlates while viewing food and non-food advertisements we can better understand the effect of the advertisements on consumers preferences and willingness to buy products. By combining both self-reported, behavioral and neurophysiological measures we were able to comprehensively and objectively understand different factors that affect consumer preferences for food products. The findings of this research can be used to enhance our understanding of food choices and purchase behavior, optimize methodology used for food consumption studies and provide guidance for further research in consumer neuroscience studies.

How do we come to the decision that we like a face? This thesis investigates this important aspect of social processing and communication by examining preference decisions for faces and the role that visual behavior plays in the process. I present a series of studies designed to investigate face preference formation and gaze patterns using eye-tracking and self-reported preference ratings. I tested healthy control subjects and two clinical populations known to have deficits in social processing: people with autism and patients with amygdala lesions. In studies one and two, I explore whether known social cognition deficits in people with autism and amygdala lesions also impair subjective decision-making regarding the attractiveness of faces. In study three, I investigate the flexibility of rule-based visual strategies used by these populations during face perception. Additionally, I present a custom algorithm developed to process raw eyetracking data, which was used to analyze all eyetracking data in this thesis. People with autism and patients with amygdala lesions are known to have general deficits in social processing, including difficulty orienting toward and evaluating faces. Nevertheless, I find that their behavior is markedly similar in many areas where we would expect them to have abnormalities or deficiencies. Their preference decisions when judging facial attractiveness were highly correlated with those made by controls, and both groups showed the same biases for familiar faces over novel faces. In addition, people with autism exhibit the same visual sampling behavior linking preference and attentional orienting, but reach their decisions faster than controls and also appear insensitive to the difficulty of the choice. Finally, gaze to the eye region appears normal in the absence of an explicit decision-making task, but only when analyzed in a similar manner as previous studies. However, when face sub-regions were analyzed in greater detail, people with autism demonstrate abnormalities in face gaze patterns, failing to emphasize the most information-rich regions of the face. Furthermore, people with autism demonstrate impairments in their ability to update those gaze patterns to accommodate different viewing restrictions. Taken together, these findings support the idea that the normal formation of face preferences can be preserved in the presence of general social processing impairments. Patterns in the eyetracking and behavioral data indicate that this is made possible, in part, by compensatory atypical processing and visual strategies.

The goal of this study was to use eye-tracking and log-file data to investigate the impact of prior knowledge on college students’ (N = 194, with a subset of n = 30 for eye tracking and sequence mining analyses) fixations on (i.e., looking at) self-regulated learning-related areas of interest (i.e., specific locations on the interface) and on the sequences of engaging in cognitive and metacognitive self-regulated learning processes during learning with MetaTutor, an Intelligent Tutoring System that teaches students about the human circulatory system. Results revealed that there were no significant differences in fixations on single areas of interest by the prior knowledge group students were assigned to; however there were significant differences in fixations on pairs of areas of interest, as evidenced by eye-tracking data. Furthermore, there were significant differences in sequential patterns of engaging in cognitive and metacognitive self-regulated learning processes by students’ prior knowledge group, as evidenced from log-file data. Specifically, students with high prior knowledge engaged in processes containing cognitive strategies and metacognitive strategies whereas students with low prior knowledge did not. These results have implications for designing adaptive intelligent tutoring systems that provide individualized scaffolding and feedback based on individual differences, such as levels of prior knowledge.

Stress is a feeling of emotional or physical tension, this makes a serious influence among undergraduates. There is a lack of stress prediction techniques that can detect what sort of stress level undergraduates are having from time to time. This research explored the prediction of undergraduate stress levels using eye-tracking. In this study, 306 PSS-10 data sets and 30600 eye-tracking data (Images) were collected from undergraduates at the University of Ruhuna using a questionnaire and a third-party eye-tracker application. PSS-10 (Perceived Stress Scale −10) and CNN (Convolutional Neural Network) was used to predict undergraduate stress levels. Stress levels are determined by the PSS-10 analysis, and divided into three classes: High, Moderate, and Low. Eye tracking data and stress classes are correlated in data pre-processing phase. The eye tracking images takes as input for well define eye tracking classification model; the model predict the stress level of the given undergraduate eye tracking data. According to the results, it was concluded that 19.7% of the undergraduates were in very high level of stress, 71.8% were in moderate level of stress and 8.5% were in low level of stress. However, it is certain that most of the undergraduates suffer from moderate levels of stress. The research will help predict undergraduate stress levels more accurately, and aid undergraduates managing their stress levels in academic life.

Early Autism Spectrum Disorder (ASD) detection is important for early intervention.This study investigates the potential of eye-tracking (ET) data combined with machine learning (ML) models to classify ASD and Typically Developed (TD) children.Using a publicly available dataset, five ML models were evaluated: Support Vector Machine (SVM), Random Forest, Convolutional Neural Network (CNN), Artificial Neural Network (ANN), and Random Forest improved with Convolutional Filters (ConvRF).The models were trained and tested using a set of evaluation metrics, including accuracy, precision, recall, F1-score, and ROC Area Under the Curve (AUC).Among these, the ConvRF model attained superior performance, achieving a recall of 90% and an AUC of 88%, indicating its robustness in identifying ASD children.These results highlight the model's effectiveness in ensuring high sensitivity, which is critical for early ASD detection.This study shows the promise of combining ML and eye-tracking technology as accessible non-invasive tools for enhancing early ASD detection, resulting in timely and personalized interventions.Limitations and recommendations for future research are also included.

Vigilance is the capacity to remain alert for an extended time while performing a task. Staying alert is obligatory in many jobs, particularly those that involve monitoring, such as surveillance tasks, security monitoring, and air traffic control. These monitoring tasks require a specific level of arousal to maintain an adequate level of cognitive efficiency. In this study, we investigate the possibility of assessing the vigilance levels using a fusion of electroencephalography (EEG) and eye tracking data. Vigilance levels were established by performing a modified version of the Stroop color word task (SCWT) for 30 minutes. Feature-level fusion based on the canonical correlation analysis (CCA) was employed to each brain region to improve the classification accuracy of vigilance level assessment. Results obtained using support vector machines (SVM) classifier show that fusion of EEG+eye tracking modalities has improved the classification accuracy compared to individual modality. The EEG+Eye tracking fusion on the right central brain region achieved the highest classification accuracy of 97.4 ± 1.3%, compared to the individual Beta EEG with 92.0±7.3% and Eye tracking with 76.8±8.4%, respectively. Likewise, EEG and Eye tracking fusion on the right frontal region showed classification accuracy of 96.9 ± 1.1% for both the Alpha and Beta bands. Meanwhile, when all brain regions were utilized, the highest classification accuracy of EEG+Eye tracking was 96.8 ± 0.6% using Delta band compared to the EEG alone with 88.18 ± 8.5% and eye tracking alone with 76.8 ± 8.4 %, respectively. The overall results showed that vigilance is a brain region specific and the fusion of EEG+ and Eye tracking data using CCA has significantly improved the classification accuracy of vigilance levels assessment.

Skill acquisition in the manufacturing industry is a crucial aspect of optimising performance, efficiency, and safety in complex work environments. Human factors play a significant role in skill acquisition, encompassing factors such as cognitive processes, perception, decision-making, and physical interactions within the work environment. Eye tracking data has emerged as a valuable tool for studying skill acquisition (Toker et al., 2014) in the manufacturing industry, offering insights into workers' visual attention, cognitive strategies, and task performance. Currently, in manufacturing settings, skill acquisition involves the mastery of various tasks, from operating machinery and assembly line processes to quality control and troubleshooting. Effective skill acquisition is essential for ensuring consistent product quality, minimising errors, and maximizing productivity. Human factors, including attentional processes, perception, and decision-making, influence how workers acquire and apply new skills in these dynamic and often high-pressure environments (Mark et al., 2020).The aim of the current research is to understand the duration required to acquire skills through procedural learning and the transition to routine development occurs when leaned behaviour becomes habitual and routine. Furthermore, the research also aims to understand how mental and physical fatigue impact their performance with manual quality control tasks. The study aims to showcase preliminary results regarding human factors and performance variations. Participants completed an inspection task that involved an industrial component (monitor) for their serial number, visual and tactile quality under a control condition: control (no manipulation) measuring physical demand and stress levels during each monitor inspection. Physiological measures were captured using a Empatica E4 wristband (capturing electrodermal activity (EDA), heart rate, skin temperature) and eye tracking was performed with Tobii Glasses 3, as well as subjective measures of performance via NASA TLX. The results from the physiological data show that the initial 10 minutes of the task showed a positive significant correlation between EDA and NASA Performance score (Spearman rho = .675, p =.016), the second set of 10 minutes positively correlated EDA and NASA TLX temporal demands (Spearman rho = .757, p =.004), while final 10 minutes showed a positive correlation between EDA and NASA TLX physical demand (Spearman rho = .639, p =.025). Such results indicate that skill acquisition over time goes through several stages – individual's anxiety of their performance, then concerns for timely performance, and finally experiencing physical impact – as well as that EDA is good indicator of changing workload demands.The current study will present the findings of the stress and physical discomfort levels, physiological data and preliminary eye tracking data whereby results found that for the initial inspection of the first two components participants move between the instructions for the inspection and the actual inspection, however, analysis of the third component and further components revealed that attention is mainly focused on the component inspection with little fixations towards the instructions. There was also an increase in the development of skill acquisition whereby by participants displayed a decrease in error count (from 1 to 0 over the inspection of 10 monitors), instruction analysis and increase in time of task completions as the average time reduced from 6 minutes 07 seconds for the first component to 1 minute 51 seconds for the last component. (The results from the eye tracking data reveal participants’ gaze patterns during the experiment highlighting the time spent on task comprehension and completion suggesting the skill was acquired during the experimental task. In summary, the current study demonstrates the development of skills through a variety of measures showing temporal progression, mental demand and the physicality of manual handling tasks is a complex process influenced by various human factors, including attention, fatigue, and tactile knowledge. By leveraging physiological data, eye tracking and stress levels the research provides scope into how human skill acquisition is progressive through various factors that can reduce overtime to upkeep and maintain the skills of workers perform in the manufacturing industry.

Eye trackers are sometimes used to study the miniature eye movements such as drift that occur while observers fixate a static location on a screen. Specifically, analysis of such eye-tracking data can be performed by examining the temporal spectrum composition of the recorded gaze position signal, allowing to assess its color. However, not only rotations of the eyeball but also filters in the eye tracker may affect the signal’s spectral color. Here, we therefore ask whether colored, as opposed to white, signal dynamics in eye-tracking recordings reflect fixational eye movements, or whether they are instead largely due to filters. We recorded gaze position data with five eye trackers from four pairs of human eyes performing fixation sequences, and also from artificial eyes. We examined the spectral color of the gaze position signals produced by the eye trackers, both with their filters switched on, and for unfiltered data. We found that while filtered data recorded from both human and artificial eyes were colored for all eye trackers, for most eye trackers the signal was white when examining both unfiltered human and unfiltered artificial eye data. These results suggest that color in the eye-movement recordings was due to filters for all eye trackers except the most precise eye tracker where it may partly reflect fixational eye movements. As such, researchers studying fixational eye movements should be careful to examine the properties of the filters in their eye tracker to ensure they are studying eyeball rotation and not filter properties.

The greatest challenge in autism spectrum disorder (ASD) detection and treatment arises from the elusive etiological origins and heterogeneity of ASD. ET-EEG correlative analytics refers to correlating simultaneously recorded eye tracking (ET) and electroencephalography (EEG) data or combining them to yield diagnostic biomarkers. This approach was recently applied in ASD research as EEG and video-based ET are suitable for children and do not interfere with each other. It allows researchers to associate neural correlates with gaze patterns in the same cognitive task. Besides, correlative analytics has shed light on the inconsistent findings derived from the unimodal approach, reveals the developmental trajectory of ASD, and provides biomarkers to assist ASD diagnosis or intervention. This review focuses on eight articles applying ET-EEG correlative analytics and synthesized the reported correlation between ET and EEG patterns. Studies concerning such patterns that acquired either ET or EEG data were also reviewed to provide comparisons. In recent years, correlative analytics has also been developed in other research fields, such as visual information processing and EEG artifact correction. Incorporating methodologies used in these research works into ASD studies would help identify cognitive alteration related to a specific visual pattern in naturalistic settings. The opportunities and limitations of this approach in ASD research are discussed at last.

Eye tracking technology is used to record the eye positions and the movements of the eye using the optical tracking of corneal reflections. Eye tracking data collected this way can be used in a wide variety of applications like gaming, marketing, cognitive ability and psychology. One of the applications of eye tracking data is to predict whether an individual has a learning disability like Dyslexia. Dyslexia is the most common neurological learning disability which manifests in the form of difficulty in reading and spelling. Although eye tracking data is recorded and available, there is a scarcity of studies done in analysing the data and understanding the hidden relationship and classifying it appropriately. This research intends to study different classification models like Logistic Regression, Gaussian NB, SVC, Decision Tree and so on. that are applied in the prediction of risk of Dyslexia. The paper also presents the result of the accuracy of different classification models in predicting the risk of Dyslexia.

Dyslexia is a neurological disorder which leads to learning disabilities mainly in reading.This learning disorder affects 5 to 10 % of peoples across worldwide.Usually, persons affected from dyslexia faces difficulties in spelling, reading and writing fluency.It affects any age peoples and it is not related to their intelligence.It is a hidden reading disability which is crucial to diagnosis.But an early and easy diagnosis of dyslexia will improve the abilities of affected peoples using specialized software or tools.The major objective of this model is to present a clean and accurate screening model for dyslexia classification using eye tracking movement data.This paper proposed a residual dense-assisted multi-attention transformer model for detecting and classifying dyslexia with data normalization.In this paper the risk of reading difficulties can be analyzed by the tracking movement data which is better than any screening methods.The proposed research attained high accuracy, precision, recall, F1-score, sensitivity and specificity of 99.48 %, 99.32 %, 99.27 %, 99.41 %, 99.53 % and 99.26 % respectively.A novel transformer based residual dense network with multi-attention module is developed to detect and classify the dyslexia disorder.This research is done with the eye tracking data with large number of attributes.This work attained high accuracy which implies that the proposed model outperformed well on others.