AR-Eye: A Custom, Low-Cost Eye Tracker for Mental Fatigue Detection with Pattern-Based Machine Learning on Augmented Reality Headsets

Augmented reality environments face significant challenges in achieving context and semantic awareness, which are relevant for enhancing user experiences. Although machine learning has been extensively applied to areas such as object and action recognition, there is a knowledge gap in the integration of these techniques to improve contextual understanding in AR environments. This study explores how machine learning enhances object and action recognition, improving context awareness in augmented reality environments. A systematic review of fifteen studies revealed advancements in deep learning, semantic web, knowledge graphs, and multimodal interfaces, which collectively improved the capabilities of augmented reality systems. The findings include innovations in aspects such as 3D hand pose estimation, semantic gaze analysis through eye tracking, and hybrid approaches that optimize real-time processing. These developments improve augmented and mixed reality environments by enabling more realistic, interactive, and responsive applications. However, persistent challenges such as latency, scalability, and environmental adaptation underscore the need for more research. These results suggest a guide for the development of highly immersive and efficient AR/MR systems, with the potential to improve user interaction and operational capabilities.

Three-dimensional sound's effectiveness in virtual reality (VR) environments has been widely studied. However, due to the big differences between VR and augmented reality (AR) systems in registration, calibration, perceptual difference of immersiveness, navigation, and localization, it is important to develop new approaches to seamlessly register virtual 3-D sound in AR environments and conduct studies on 3-D sound's effectiveness in AR context. In this paper, we design two experimental AR environments to study the effectiveness of 3-D sound both quantitatively and qualitatively. Two different tracking methods are applied to retrieve the 3-D position of virtual sound sources in each experiment. We examine the impacts of 3-D sound on improving depth perception and shortening task completion time. We also investigate its impacts on immersive and realistic perception, different spatial objects identification, and subjective feeling of human presence and collaboration. Our studies show that applying 3-D sound is an effective way to complement visual AR environments. It helps depth perception and task performance, and facilitates collaborations between users. Moreover, it enables a more realistic environment and more immersive feeling of being inside the AR environment by both visual and auditory means. In order to make full use of the intensity cues provided by 3-D sound, a process to scale the intensity difference of 3-D sound at different depths is designed to cater small AR environments. The user study results show that the scaled 3-D sound significantly increases the accuracy of depth judgments and shortens the searching task completion time. This method provides a necessary foundation for implementing 3-D sound in small AR environments. Our user study results also show that this process does not degrade the intuitiveness and realism of an augmented audio reality environment

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Augmented reality sickness questionnaire (ARSQ): A refined questionnaire for augmented reality environment

Augmented Reality (AR) has become hugely popular as a revolutionary aid for educational purposes, which allows learners to manipulate virtual content and to view it in a realistic, very interactive manner. Although it has the potential to positively impact engagement and motivation, AR environments can generate too much cognitive load as a result of quantity and complexity of the stimuli. This paper addresses this challenge by examining the use of state-of-the-art AI (Artificial Intelligence) models within AR learning systems for providing real-time, personalized teaching that successfully models and minimizes cognitive load. The ultimate aim is to design a framework that can adjust content, complexity and pacing to the learners’ cognitive state and learning styles. The proposed monitoring system uses DL and RL techniques to capture learner interaction behavior along with physiological signals (eye movement and reaction time) and performance characteristics. These datapoints are run through AI algos to estimate cognitive load and adjust AR content accordingly. For example, the system may reduce visual information, change the timing of feedback, or offer scaffolding support that is determined in real time. This adaptive learning process prevents the learners from getting too far ahead of themselves or being force-fed too much complexity, which in turn allows the system to be more easily related to and remembered. Personalization is performed by learner modeling, in which AI develops personalized cognitive models and learner-specific cognitive profiles by LSTM networks. These profiles drive the system’s decision making, guaranteeing that the content delivery is adapted to the user’s pace of learning, his/her preferences, and prior knowledge. By consistently modeling from user data, the AI system grows alongside the learner, providing more precise and beneficial customizations as it learns. Experimental investigations of the AI AR enhanced learning environment, with learners in various educational domains, i.e., natural and life sciences, mathematics and language learning, show considerable reduction in extraneous cognitive load, and enhanced learner satisfaction, task completion, and test performance. AI integration does not just improve usability it also means scalable learning solutions that cater to all - even in the case of limitations among users. This study emphasizes the need for intelligent systems that can lead to a new approach in digital education. By combining AR with live AI-driven personalization, the doors to new types of learner-centered pedagogical design can be laid open. The results add to the body of research in the emerging area of cognitive-aware learning technologies and have practical implications for developers and educators who seek to leverage AR to achieve the most effective, engaging, and cognitively balanced educational experiences.

The purpose of the research is to compare the impact of students’ learning performance between an augmented reality (AR) environment and in-class environment. To create an efficient AR environment, we used Microsoft HoloLens, which is the next generation of a see-through holographic computer. We developed the AR learning module: ergonomic guidelines for manual material handling (MMH). We hypothesize that AR changes the way students understand the concepts of MMH. Our analysis includes a careful evaluation of student experimental skills during the learning activities. This new AR environment could allow students to engage hands-on training of MMH and strengthen their understanding. Student test score was used as metrics for performance assessment. We found a significant improvement on student understanding of MMH lecture after they used the AR module. The findings of this study indicated the potential benefits of using AR environment interfaces in engineering education and training.

Recupera algunos de los elementos menos estudiados y reconocidos en el ensayo latinoamericano: su profunda vocación de reflexión, diálogo y mediación intelectual; su plasticidad y apertura a la búsqueda del sentido; su vínculo con la sociabilidad, la conversación y la generación de un espacio simbólico de encuentro; su capa cidad de establecer una nueva retroalimentación entre lectura y escritura.

Virtual reality is increasingly applied in training contexts across various domains, including education, industry, and medicine. As the cost of augmented reality (AR) technology declines, designers are gaining the capability to develop realistic experiences and interfaces in this medium. This introduces an unexplored variable in AR environments: the impact of simulated gravity. This thesis reports on a study conducted in AR, investigating three levels of simulated gravity. Participants, working at the Lynn and Arnold Irwin Advanced Perioperative Imaging Lab (APIL) at Toronto General Hospital, engaged in tasks using 3D puzzle pieces while their movements and interactions were tracked for data analysis. Post-task questionnaires assessed participants’ mental states. The study aims to provide insights into the effects of simulated gravity on a virtual assembly task as well as its potential use in other AR interface and user study design.<p></p>

Virtual characters are now widely used in games, computer-generated (CG) movies, virtual reality (VR), and communication media. The continued technological innovations in motion capture mean that a more natural representation of a three-dimensional character’s motion should be achievable. Many researchers have investigated how virtual characters interact with their surrounding environment through spatial relationships, which were introduced for adapting and preserving character motion. However, technical problems should be resolved to enable the control of characters in augmented reality (AR) environments that combine with the real world, and this can be achieved by adapting motion to environmental differences using original motion datasets. In this paper, we investigate a novel method for preserving automatic motion adaptation for a virtual character in AR environments. We used specific object (e.g., puddle) recognition and the spatial properties of the user’s surrounding space, e.g., object types and positions, and ran validation experiments to provide accurate motion to improve the AR experience. Our experimental study showed positive results in terms of smooth motion in AR configurations. We also found that the participants using AR felt a greater sense of co-presence with the character through adapted motion.

This study aims to explore the effect of using 3D Augmented Reality (AR) on cognitive load, collaboration tendency, and enjoyment in engineering education. A quasi-experimental design was conducted involving 111 students, divided into three groups: a control group (conventional), an experimental group with hints (AR + scaffolding), and an experimental group without hints (AR only). Data were collected through pretest-posttest scores and post-intervention questionnaires measuring Cognitive Load (CL), Collaboration Tendency (CT), and Enjoyment (EJ). Data analysis included normality tests (Shapiro–Wilk and Kolmogorov–Smirnov), Wilcoxon Signed-Rank Test, Kruskal–Wallis test, One-Way ANOVA, and Multiple Linear Regression. Results showed that all groups experienced learning improvement, with the highest gains in the AR with hints group. Effect size analysis using Cohen’s d indicated a large impact of AR with hints ( d = 1.10) and moderate impact of AR without hints ( d = 0.63) compared to the control group. Regression analysis revealed that EJ ( p = 0.012) and CT ( p = 0.034) significantly predicted learning performance, while CL had a negative effect ( p = 0.035). Two-Way ANOVA showed no significant gender effect on CL or CT, but found a significant interaction between gender and group on EJ ( p = 0.028), suggesting gender-based differences in emotional engagement with AR. Female students reported higher EJ without hints, while male students preferred guided AR learning. These findings highlight that 3D AR enhances learning outcomes not only cognitively but also socially and emotionally. The integration of scaffolding into AR environments increases effectiveness by reducing cognitive overload and fostering collaboration. Designing adaptive, gender-responsive AR learning environments can further optimize educational impact in technical disciplines.

Accurate implant orientation reduces wear and increases stability in arthroplasty but is a technically demanding skill. Augmented reality (AR) headsets overlay digital information on top of the real world. We have developed an enhanced AR headset capable of tracking bony anatomy in relation to an implant, but it has not yet been assessed for its suitability as a training tool for implant orientation. (1) In the setting of simulated THA performed by novices, does an AR headset improve the accuracy of acetabular component positioning compared with hands-on training by an expert surgeon? (2) What are trainees' perceptions of the AR headset in terms of realism of the task, acceptability of the technology, and its potential role for surgical training? Twenty-four study participants (medical students in their final year of school, who were applying to surgery residency programs, and who had no prior arthroplasty experience) participated in a randomized simulation trial using an AR headset and a simulated THA. Participants were randomized to two groups completing four once-weekly sessions of baseline assessment, training, and reassessment. One group trained using AR (with live holographic orientation feedback) and the other received one-on-one training from a hip arthroplasty surgeon. Demographics and baseline performance in orienting an acetabular implant to six patient-specific values on the phantom pelvis were collected before training and were comparable. The orientation error in degrees between the planned and achieved orientations was measured and was not different between groups with the numbers available (surgeon group mean error ± SD 16° ± 7° versus AR 14° ± 7°; p = 0.22). Participants trained by AR also completed a validated posttraining questionnaire evaluating their experiences. During the four training sessions, participants using AR-guidance had smaller mean (± SD) errors in orientation than those receiving guidance from the surgeon: 1° ± 1° versus AR 6° ± 4°, p < 0.001. In the fourth session's assessment, participants in both groups had improved (surgeon group mean improvement 6°, 95% CI, 4-8°; p < 0.001 versus AR group 9°, 95% CI 7-10°; p < 0.001). There was no difference between participants in the surgeon-trained and AR-trained group: mean difference 1.2°, 95% CI, -1.8 to 4.2°; p = 0.281. In posttraining evaluation, 11 of 12 participants would use the AR platform as a training tool for developing visuospatial skills and 10 of 12 for procedure-specific rehearsals. Most participants (11 of 12) stated that a combination of an expert trainer for learning and AR for unsupervised training would be preferred. A novel head-mounted AR platform tracked an implant in relation to bony anatomy to a clinically relevant level of accuracy during simulated THA. Learners were equally accurate, whether trained by AR or a surgeon. The platform enabled the use of real instruments and gave live feedback; AR was thus considered a feasible and valuable training tool as an adjunct to expert guidance in the operating room. Although there were no differences in accuracy between the groups trained using AR and those trained by an expert surgeon, we believe the tool may be useful in education because it demonstrates that some motor skills for arthroplasty may be learned in an unsupervised setting. Future studies will evaluate AR-training for arthroplasty skills other than cup orientation and its transfer validity to real surgery. Level I, therapeutic study.

This study is based on the science and technology acceptance model, the results that augmented reality system import Army Command and Staff learning methods, the impact of officer's situational awareness (SA) and intention to use (ITU). To develop an ARB-MDMP system for creating and presenting 3D image-based augmented reality, it provides realistic troops patterns, terrain environment, and immediate battlefield Intelligence.

Background Stroke is a leading cause of disability, often accompanied by unilateral spatial neglect (USN), which severely impairs recovery. Traditional assessments like paper-pencil tests provide limited insights into behaviors and eye–hand coordination during real-world tasks. Advances in hand pose estimation and eye tracking in combination with augmented reality (AR) offer potential for data-driven assessments of naturalistic interactions. Objective This proof-of-concept study presents and evaluates a multimodal behavioral tracking system that captures gaze, body, and hand movements during interactions within an AR environment. Our primary goals are to (1) validate that this system can achieve robust and accurate interaction data capture in clinical settings, (2) show that the system can reliably detect known USN behavioral patterns, and (3) explore how comprehensive data can provide new understanding of eye–hand coordination deficits in USN. Methods We developed an AR-based assessment system using Microsoft HoloLens 2 and an external body-tracking camera to capture real-time gaze, hand, and body movements in an interactive environment. Multimodal data streams were temporally synchronized, fused, and filtered to enhance spatial accuracy and availability. Tracking performance was benchmarked against a traditional optical motion-capture system to validate reliability. In a study, 7 patients with right-brain lesions with mild to moderate USN and 8 healthy controls participated. Each performed a designed reaching task, stamping virtual sheets of paper that appeared randomly on a table. We analyzed participants’ search behavior patterns to assess attentional biases and examined gaze anchoring timing during targeted reaching motions to explore potential eye–hand coordination deficits. Results The fusion of hand-tracking data from the HoloLens 2 and external system reduced tracking loss from 25.7% to 2.4%, with an absolute trajectory error of 3.27 cm. The system demonstrated high usability and was well accepted by patients. Data from the control group confirmed the absence of intrinsic lateral biases in the system and task design. The USN group displayed typical search behavior through ipsilesional biases in gaze direction during visual exploration (median deviation 7.46 [1.61-9.48] deg, P &lt;.05) and longer times to find contralesional targets (median difference 1.08 [0.20-1.80] s, P =.02). Additionally, the eye–hand coordination analysis revealed lateral differences in gaze anchoring during targeted reaching motions in the USN group, with earlier fixation on contralesional targets (median difference 112 [71-146] ms, P =.02). Conclusions The proposed AR framework provides a novel, comprehensive data-driven method for capturing interaction behavior in a controlled, yet naturalistic environment. Our results demonstrated the system’s effectiveness in measuring hallmark USN symptoms, such as gaze and head orientation biases, and highlighted its potential to complement traditional assessments by offering deep insights into torso rotation and eye–hand coordination with a high resolution and accuracy. This data-driven approach shows promise for enhancing current USN assessment practices and gaining new insights into patients’ behaviors.

As commodity virtual reality and augmented reality hardware becomes more accessible, the opportunity to use these systems for learning and training will increase. This study provides an exploratory look at performance differences for a simple memory matching task across four different technologies that could easily be used for learning and training. We compare time and number of attempts to successfully complete a memory matching game across virtual reality, augmented reality, a large touchscreen table-top display and a real environment. The results indicate that participants took more time to complete the task in both the augmented reality and real conditions. Augmented reality and real environments were statistically different than the fastest two conditions, which occurred in the virtual reality and table-top touch display conditions.

An increasingly large amount of data from the physical world has been collected, digitized, and stored. To communicate such data to general public effectively, visual data-driven storytelling has been widely used. Yet, the vast majority of data communication occurs on desktop computers separated from the physical world the data originates in and refers to. Recent advances in Augmented Reality (AR) have shed new light on data-driven storytelling, offering exciting possibilities for telling engaging, in-situ, and immersive stories by embedding the data in the real-world context. However, although creating such kind of AR data stories is demanding and requires considerable knowledge and skills from different fields (e.g., data visualization, computer graphics, computer vision, and human-machine interaction), prior research has rarely investigated the effective way to create visual data-driven stories in AR environments. This thesis aims to fill this gap by exploring the approaches to facilitate the authoring of infographics, a popular format for data-driven storytelling, in AR. Given that AR devices are still evolving rapidly, this thesis focuses on the interaction between reality and virtuality, an essence of AR that is independent of specific devices. The first system, MARVisT, leverages physical properties (e.g., size, positions) from real-world objects to assist non-experts in creating 3D infographics in mobile AR. Given the limited interaction capabilities of mobile devices, in the second work, LassoNet is proposed to facilitate the selection of 3D objects in a 2D screen based on a deep neural network. Besides augmenting the physical world, the third research studies augmenting the semantic content of real-world infographics in AR and introduces PapARVis Designer to allow designers to create augmented static visualizations. Finally, the fourth work explores automating the creation of timeline infographics in AR by developing a deep-learning based method that automatically extracts extensible templates from real timeline infographics to generate virtual timelines with new data in AR. The core idea of this thesis is to allow visualization designers to go beyond the desktop platform to the engaging, immersive, and promising AR platform that is seen to be the next-generation human-machine interaction platform. The resulting systems and techniques blaze a trail toward futuristic visual storytelling.