Investigating XR Pilot Training Through Gaze Behavior Analysis Using Sensor Technology

The manner in which drivers direct their gaze plays a pivotal role in the operation of a vehicle. As information acquisition and processing while driving form the basis for action decisions, their implementation and control in road traffic, the driver's gaze behavior largely determines his driving behavior and is therefore of central importance for traffic safety. With the help of modern measurement technology, eye movements can be precisely registered and gaze behavior can be recorded in real time while driving. The analysis of gaze behavior thus represents a new method for considering the perceptual-psychological factor in the interaction between driver, vehicle, and road. The gaze behavior of drivers is largely determined by the layout geometry of the road. It has been found that gaze patterns are significantly different when a vehicle is steered along straight sections and within horizontal curves. On highways, the fixation position and the standard lateral deviation from the main gaze axis exhibit notable The manner in which drivers direct their gaze plays a pivotal role in the operation of a vehicle. As information acquisition and processing while driving form the basis for action decisions, their implementation and control in road traffic, the driver's gaze behavior determines his driving behavior and is therefore of central importance for traffic safety. With the help of modern measurement technology, eye movements can be precisely registered and gaze behavior can be recorded in real time while driving. The analysis of gaze behavior thus represents a new method for considering the perceptual-psychological factor in the interaction between driver, vehicle, and road. The gaze behavior of drivers is largely determined by the layout geometry of the road. It has been found that gaze patterns are significantly different when a vehicle is steered along straight sections and within horizontal curves. On highways, the fixation position and the standard lateral deviation from the main gaze axis exhibit notable differences between straight sections and curves, as well as between left and right curves. The number of fixations in the area surrounding the road center, also referred to as the primary attention area or the vanishing point, is markedly higher on curves than on straight sections. This indicates an increased level of attention and concentration, as well as elevated stress levels for drivers. Additionally, there is a notable concentration of gaze on the vanishing point of the road on right curves compared to left curves.

Introduction Oculomotor behavioral metrics change according to time awake and circadian phase following a distinct pattern of impairment. Acute sleep deprivation (ASD) causes large decreases in pursuit initiation and steady-state gain, and a compensatory increase in saccadic rate without any systematic change in saccadic size. It also causes large deficits in visual processing of direction and speed, and impaired saccade dynamics. Such deterioration likely reflects changes in both higher cortical and brainstem function, explaining in-part how sleep loss and circadian misalignment affect cognition. It is unclear how oculomotor behavior might change according to chronic sleep restriction (CSR). Here, we measured the same eye-movement metrics during CSR. Methods Twelve healthy participants (6 females) kept a fixed wake-time sleep-wake schedule, at home for four weeks (weeks 1 and 3 = 9h time in bed (TIB); weeks 2 and 4 randomized to 5h or 9h TIB; actigraphy confirmed). Following weeks two and four, participants completed a 13-hour laboratory visit under dim light (<15 lux), where they maintained a semi-recumbent posture and were provided with hourly isocaloric snacks. A visual tracking task was performed hourly to assess pursuit and saccadic responses and visual motion processing. Performance metrics were computed using MATLAB, including pursuit gain (eye speed/target speed), the rate and amplitude of catch-up saccades, and the accuracy and precision of direction and speed processing. Results As expected, we found a small but significant (t(11)=-2.17, p<0.03) reduction in pursuit gain (mean+/-SEM: -0.028 +/-0.013 with a large (t(11)=2.96, p<0.01) increase in saccadic rate (0.37 +/-0.13 Hz). However, surprisingly, we found a significant (t(11)=-2.52, p<0.03) decrease in the amplitude of catch-up saccades (-0.15 +/-0.06 deg). The only systematic alteration to visual motion processing was a small reduction in horizontal-vertical asymmetry, which was previously observed with ASD. Conclusion A week of CSR to 5h is associated with only mild impairment in smooth pursuit eye movements with little impact on visual motion processing. However, CSR caused a maladaptive decrease in saccade amplitude that was not observed during ASD. Eye-movement metrics reveal differential neurological effects of CSR versus ASD. Support (if any) Force Health Protection Program, ONR (SAA2402925-1), NASA Human Research Program, and agreement NNX17AE07A

Nighttime affects pilot visual scan patterns and increase the risks of helicopter operations, contributing to many helicopter accidents and incidents. Several past studies have attempted to examine the effect of nighttime on helicopter pilot gaze behavior, but researchers had limited success due to the difficulty of collecting representative data under real flight conditions. The present study attempted to address this challenge by conducting a real flight study involving daytime and nighttime traffic pattern tasks and using a Tobii Glasses 3 eye-tracking device to collect helicopter pilot eye-tracking data. This study preliminarily explored the feasibility of data collection in real flight conditions in the context of eye-tracking research on civil helicopter pilots in China. Due to safety considerations, only one pilot was recruited to collect data in multiple tasks. Differences and correlations were examined for all gaze behavior metrics. The results suggested that pilot gaze behavior metrics and their correlations differed between daytime and nighttime flights in aspects critical to aviation safety. Pilot gaze behavior also varied with the flight phase. The findings from this study serve as a reference for optimizing helicopter pilot training systems, improving pilot performance during nighttime flights, and ensuring flight safety on helicopters.

Loss of control in-flight (LOC-I) has historically been the leading cause of commercial airline fatalities. In 2009, an investigation of LOC-I accidents revealed four primary causal factors, citing human factors errors as one of the factors. The investigation discovered that one reason for the human errors was due to deficiencies in pilot training from aircraft stalls and upsets. These deficiencies led the FAA and similar international organizations to implement upset prevention and recovery training (UPRT) guidelines for commercial air transport aircraft. The aviation industry uses flight simulators as part of a comprehensive program for pilot training and certification requirements. Although flight simulators can possess a high level of realism, deficiencies remain. Many pilots find simulators to be easier than their real-world counterparts. Even with a high-fidelity FFS, simulator training currently cannot accurately reproduce essential factors that are considered to be within the safety margins of the aircraft, such as roll and pitch rates and g-loading, all of which can lead to high levels of altitude loss, if not managed correctly. This thesis compared upset recovery maneuvers between a simulator and live flight. The study used in the research required the pilots to recover from an upset condition while also performing an additional task of serially adding two numbers. The secondary task provided a physiological baseline cognitive workload measurement that was used to compare the pilots’ increased workload levels during the recovery phase of the flight. The results indicated that the pilots generally performed better during the simulator flights and had less subjective workload and high levels of situation awareness. The physiological-based workload contrasted subjective measures. However, during a live flight upset recovery, pilots may not be able to transfer the skills acquired from a flight simulator and experience higher levels of workload and reduced levels of situation awareness, increasing their potential for an unsuccessful recovery. Suggestions are made that include more use of live aircraft aerobatic training for UPRT and the use of a physiological-based, unbiased workload assessment during pilot training. Additionally, it is suggested using an adaptive training approach to UPRT that incorporates the primary technical flight performance metrics with secondary task performance.

This paper introduces a quantitative methodology for assessing drone pilot performance, aiming to reduce drone-related incidents by understanding the human factors influencing performance. The challenge lies in balancing evaluations in operationally relevant environments with those in a standardized test environment for statistical relevance. The proposed methodology employs a novel virtual test environment that records not only basic flight metrics but also complex mission performance metrics, such as the video quality from a target. A group of Belgian Defence drone pilots were trained using this simulator system, yielding several practical results. These include a human-performance model linking human factors to pilot performance, an AI co-pilot providing real-time flight performance guidance, a tool for generating optimal flight trajectories, a mission planning tool for ideal pilot assignment, and a method for iterative training improvement based on quantitative input. The training results with real pilots demonstrate the methodology’s effectiveness in evaluating pilot performance for complex military missions, suggesting its potential as a valuable addition to new pilot training programs.

Identifying causal relationships is complicated. Researchers usually overlook causality behind relationships which can generate misleading associations. Herein, we carefully examine the parametric relationship and causality between wildfire smoke exposure and animal performance and behavior metrics over a period of 2 yr in Reno, Nevada. The animals in the 2020 smoke season were grain-finished (n = 12) and grass-finished (n = 12), whereas the animals during the 2021 season were fed under the same diet but finished with either a hormonal implant (n = 9), or without (n = 9). The dataset included daily records of feed intake (FI), body weight (BW), water intake (WI), average daily gain (ADG), and WI behavior (time spent drinking [TSD]; water intake events [WIE]; no-WIE [NWIE]). Variable tree length Bayesian additive regression trees (BART) were utilized to investigate the relationships between air quality index (AQI), particulate matter 2.5 μm (PM2.5) and 10 μm (PM10), NO2, SO2, Ozone, and CO levels in the air (sensors < 1.6 km from animals) with the animal data. Additionally, linear mixed models with a 7-d lag were used to evaluate parametric relationships among the same variables. All statistical analyses were performed on R Statistical Software (R Core Team 2023). Under the linear mixed model with a 7-d lag, significant positive and negative associations were found for all parameters examined (P < 0.05). Negative associations were found between FI, WI, ADG, BW, WIE, NWIE, TSD, and PM2.5 (P < 0.05) for at least one animal group. Positive linear associations between wildfire smoke parameters and the metrics evaluated were more variable and dependent on year, treatment, and smoke parameters. When examining the credible intervals and the variable importance in the BART, relationships were more difficult to identify. However, some associations were found for Ozone, AQI, NO2, CO, and PM10 (P < 0.05). Overall, our results carefully examine the relationship between smoke parameters and cattle performance and present interesting pathways previously unexplored that could guide early culling/finishing of animals to avoid economic losses associated with performance decrease in response to wildfire smoke exposure. Though interesting associations are found under linear mixed models, causality is difficult to establish, which highlights the need for controlled exposure experiments.

Nursing education relies on simulation to provide standardized, safe clinical experiences to prepare nurses for complex patient scenarios. Traditional simulation requires cost-prohibitive equipment and facilities, which are incompatible with complex scenarios and distance learning. Early studies indicate eXtended Reality technology is an effective modality to deliver educational simulation. However, the existing research is not yet conclusive. This study explored the use of a holographic video virtual standardized patient simulation to assess student knowledge, skill, and motivation to learn for a low-frequency, high-risk patient scenario surrounding anaphylaxis. Participating students (n=161) across three levels of nursing courses were stratified and randomly assigned to an anaphylaxis simulation research group delivered by three different modalities. Analysis of covariance showed differences in student knowledge, skill, and motivation to learn between modalities of a written case study, two-dimensional video, and eXtended Reality. These results indicated two-dimensional video provided the highest degree of knowledge and skill, while eXtended Reality simulation resulted in increased student motivation to learn. Stepwise regression indicated that conscious placement of the simulation across the learning continuum should be considered. These results have implications for affordable simulation to motivate learning and provide experiences with low-frequency, high-risk scenarios.

Exploring the Relationship Between Eye Metrics and Surgical Performance: A Study on Orthopedic Surgeons in Simulated Procedures.

BackgroundSurgical residents underutilize opportunities for traditional laparoscopic simulation training. Serious gaming may increase residents’ motivation to practice laparoscopic skills. However, little is known about the effectiveness of serious gaming for laparoscopic skills training.ObjectiveThe aim of this study was to establish construct validity for the laparoscopic serious game Underground.MethodsAll study participants completed 2 levels of Underground. Performance for 2 novel variables (time and error) was compared between novices (n=65, prior experience <10 laparoscopic procedures), intermediates (n=26, prior experience 10-100 laparoscopic procedures), and experts (n=20, prior experience >100 laparoscopic procedures) using analysis of covariance. We corrected for gender and video game experience.ResultsControlling for gender and video game experience, the effects of prior laparoscopic experience on the time variable differed significantly (F2,106=4.77, P=.01). Both experts and intermediates outperformed novices in terms of task completion speed; experts did not outperform intermediates. A similar trend was seen for the rate of gameplay errors. Both gender (F1,106=14.42, P<.001 in favor of men) and prior video game experience (F1,106=5.20, P=.03 in favor of experienced gamers) modulated the time variable.ConclusionsWe established construct validity for the laparoscopic serious game Underground. Serious gaming may aid laparoscopic skills development. Previous gaming experience and gender also influenced Underground performance. The in-game performance metrics were not suitable for statistical evaluation. To unlock the full potential of serious gaming for training, a more formal approach to performance metric development is needed.

Objective. The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain–machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behavior is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements. Approach. Eighteen participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection. Main results. A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved. Significance. The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g. in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user’s involvement during gaze-based interaction.

Eye tracking technologies have frequently been used in sport research to understand the interrelations between gaze behavior and performance, using a paradigm known as vision-for-action. This methodology has not been robustly applied within the field of interface design. The present work demonstrates the benefit of employing a vision-for-action paradigm for interface evaluation. This is demonstrated through the evaluation of a novel task-specific symbology set presented on a head-up-display (HUD), developed to support pilots conduct ground operations in low-visibility conditions. HUD gaze behavior was correlated with task performance to determine whether certain combinations of gaze behavior could produce effective predictive performance models. A human-in-the-loop experiment was conducted with 11 professional pilots who were required to taxi in a fixed-base flight simulator using the HUD symbology, while gaze data toward the different HUD symbology elements was collected. Performance was measured as centerline deviation error and taxiing speed. Results revealed that appropriately timed gaze behavior toward task-specific elements of the HUD were associated with superior performance. During turns, attention toward an undercarriage lateral position indicator was associated with reduced centerline deviation ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> < 0.05). The findings are interpreted alongside detailed posttrial user-feedback of the HUD symbology to illustrate how eye tracking methodologies can be incorporated into interface usability evaluations. The joint interpretation of these data demonstrates these novel procedures, the findings contribute to enhancing the wider domain of interface design evaluation.

Captions blocking visual information in live television news leads to dissatisfaction among Deaf and Hard of Hearing (DHH) viewers, who cannot see important information on the screen. Prior work has proposed generic guidelines for caption placement but not specifically for live television news, and important genre of television with dense placement of onscreen information regions, e.g., current news topic, scrolling news, etc. To understand DHH viewers’ gaze behavior while watching television news, both spatially and temporally, we conducted an eye-tracking study with 19 DHH participants. Participants’ gaze behavior varied over time as measured by their proportional fixation time on information regions on the screen. An analysis of gaze behavior coupled with open-ended feedback revealed four thematic categories of information regions. Our work motivates considering the time dimension when considering caption placement, to avoid blocking information regions, as their importance varies over time.

The integration of Artificial Intelligence (AI) with mobile learning platforms has revolutionized personalized education by dynamically adapting content to individual learner preferences. This paper presents the design and implementation of Smart Ed, an AI-driven mobile learning framework that leverages machine learning algorithms, natural language processing (NLP), and predictive analytics to customize educational experiences based on user behavior, learning patterns, and performance metrics. The proposed system consists of multiple components, including a personalized content recommendation engine, an adaptive assessment module, a content management system (CMS), cloud-based storage, and a real-time feedback mechanism. To enhance the learning process, the AI-powered recommendation engine continuously analyzes learner interactions, assessments, and engagement levels to provide customized study plans, interactive quizzes, and AI-assisted tutoring. The system integrates multi-factor authentication (MFA) and role-based access control (RBAC) to ensure secure access and data privacy. Furthermore, a real-time analytics dashboard enables educators to track student progress and optimize content delivery. Cloud-based storage and API integrations facilitate seamless accessibility across multiple devices. The framework is evaluated using key performance metrics such as content recommendation accuracy, student engagement levels, and learning efficiency. Results indicate that Smart Ed significantly improves student retention, comprehension, and overall learning outcomes compared to traditional e-learning models. By incorporating state-of-the-art AI techniques and adaptive learning methodologies, this project aims to bridge the gap between conventional education and digital learning, ensuring a personalized, scalable, and effective mobile learning experience.

Simulation is a key component of surgical training, enabling trainees to develop their skills in a safe environment. With simulators broadly grouped into physical models and virtual-reality (VR) simulators, it is important to evaluate the comparative effectiveness of the simulator types in terms of validity as well as cost. The review aims to compare the benefits and drawbacks of novel VR and physical simulators within the broader themes of endourology, laparoscopic and robotic operations, and other urological procedures. Key benefits of bench models include their comparatively lower cost, easy access and provision of haptic feedback, whereas VR simulators are generally self-sufficient, reusable and enable skills of haemostasis to be practised. The advent of perfused 3D printed simulators across a range of urological procedures may replace cadavers as the traditional gold-standard simulation modality. Although possessing differing strengths and downsides, VR and physical simulators when used together can have an additive effect due to skill transferability across the platforms. Further comparative studies are required to directly quantify the differences between physical models and VR simulators in terms of performance metrics and cost-effectiveness. There is lack of validated VR simulators for open and reconstructive procedures.

Microstrip circuits form the backbone of modern high-frequency communication systems, offering compact and efficient solutions for signal processing and transmission. However, the design of these circuits is challenging due to the intricate interplay of electromagnetic (EM) parameters, material properties, and circuit dimensions. Traditional EM simulation methods, while accurate, are computationally intensive and time-consuming, limiting their applicability for rapid prototyping and optimization. To address these challenges, this study integrates deep learning techniques with electromagnetic simulations to enhance microstrip circuit design efficiency. A Recurrent Neural Network (RNN)-based framework is proposed to predict the frequency-dependent behavior of microstrip circuits, leveraging temporal data from iterative EM simulations. The RNN model is trained on a diverse dataset of simulated circuit configurations, capturing the relationships between physical parameters, design constraints, and performance metrics. The proposed approach significantly reduces computational overhead by approximating the results of full-wave EM simulations while maintaining high accuracy. Validation against benchmark EM simulation tools shows that the RNN model achieves over 95% prediction accuracy with a 70% reduction in simulation time. Additionally, this framework enables real-time optimization of circuit designs, accelerating the iterative design process without compromising performance.