Gaze and voice controlled drawing

Eye tracking is a process that identifies a specific point in both space and time that is being looked at by the observer. This information can also be used in real-time to control applications using the eyes. Recent innovations in the video game industry include alternative input modalities to provide an enhanced, more immersive user experience. In particular, eye gaze control has recently been explored as an input modality in video games. This book is an introduction for those interested in using eye tracking to control or analyze video games and virtual environments. Key concepts are illustrated through three case studies in which gaze control and voice recognition have been used in combination to control virtual characters and applications. The lessons learned in the case studies are presented and issues relating to incorporating eye tracking in interactive applications are discussed. The reader will be given an introduction to human visual attention, eye movements and eye tracking technologies. Previous work in the field of studying fixation behavior in games and using eye tracking for video game interaction will also be presented. The final chapter discusses ideas for how this field can be developed further to create richer interaction for characters and crowds in virtual environments. Alternative means of interaction in video games are especially important for disabled users for whom traditional techniques, such as mouse and keyboard, may be far from ideal. This book is also relevant for those wishing to use gaze control in applications other than games.

Physical workload during use of speech recognition and traditional computer input devices

Speech recognition can be a powerful tool for individuals with physical disabilities that hinder their ability to use traditional input devices. State-of-the-art speech recognition systems typically provide mechanisms for both data entry and cursor control, but the researchers continue to investigate methods of improving these interactions. Numerous researchers are investigating methods to improve the underlying technologies that make speech recognition possible and others focus on understanding the difficulties users experience using dictation-oriented applications, but few researchers have investigated the issues involved in speech-based cursor control. In this article, we describe a study that investigates the efficacy of two variations of a standard speech-based cursor control mechanism. One employs the standard mouse cursor while the second provides a predictive cursor designed to help users compensate for the delays often associated with speech recognition. As expected, larger targets and shorter distances resulted in shorter target selection times while larger targets also resulted in fewer errors. Although there were no differences between the standard and predictive cursors, a relationship between the delays associated with spoken input, the speed at which the cursor moves, and the minimum size for targets that can be reliably selected emerged that can guide the application of similar speech-based cursor control mechanisms as well as future research.

Fusional reserves differ with the method of measurement. The goal of this study was to compare the subjective and objective responses during the measurement of positive and negative fusional reserves using both step and ramp methods. A haploscopic system was used to measure fusional reserves. Eye movements were recorded using an EyeLink 1000 Plus eye tracker (SR Research). The stimulus disparity was changed to either mimic a prism bar (steps) or a Risley prism (ramp). Subjective responses were obtained by pressing a key on the keyboard, whereas objective break and recovery points were determined offline using a custom algorithm coded in Matlab. Thirty-three adults participated in this study. For the ramp method, the subjective and objective responses were similar for the negative (break and recovery points (t(32) = -0.82, p = 0.42) and (t(32) = 0.42, p = 0.67), respectively) and positive fusional reserves (break and recovery points (U = -1.34, p = 0.18) and t(19) = -0.25, p = 0.81), respectively). For the step method, no significant differences in positive fusional reserves were observed when measured subjectively and objectively for the break (t(32) = 1.27, p = 0.21) or the recovery point (U = -2.02, Bonferroni-adjusted p = 0.04). For the negative fusional reserve, differences were not significant for either the break or recovery points (U = -0.10, p = 0.92 and t(19) = 1.17, p = 0.26, respectively). Subjective and objective responses exhibited good agreement when measured with the ramp and step methods.

Validating and improving the correction of ocular artifacts in electro-encephalography

Any architectural optimization aims at satisfying the end user. However, modern architectures execute with little to no knowledge about the individual user. If architectures could determine whether their users are satisfied, they could provide higher efficiency; improved reliability, reduced power consumption, increased security, and a better user experience. A major reason for this limitation is their input devices. Specifically, the traditional input devices (e.g., the mouse and keyboard) provide limited information about the user. In this paper, we make a case for the addition of new biometric input devices for providing the computer information about the userpsilas physiological traits. We explore three biometric devices as potential sensors: an eye tracker, a galvanic skin response (GSR) sensor, and force sensors. We first present two user studies that explore the link between the sensor readings and user satisfaction when the performance of the processor is varied as a video game is being played. In the first study, we drastically drop the processor clock frequency at a set point in the game. In the second study, we set the clock frequency to randomly-selected levels during game play. Both studies show that there are significant changes in human physiological traits as performance decreases. More importantly, we show that physiological changes correlate strongly to the satisfaction levels reported by the users. Based upon these observations, we construct a Physiological Traits-based Power-management (PTP) system that can be applied to existing dynamic voltage and frequency scaling (DVFS) schemes. We apply PTP to a typical CPU-utilization-based adaptive DVFS policy and evaluate our scheme using a third user study. An aggressive version of our PTP scheme reduces the total system power consumption of a laptop by up to 33.3% for an application averaged across users (18.1% averaged across three applications), while a conservative version reduces the total system power consumption by up to 25.6% across users (11.4% averaged across three applications).

Eye and hand movement tracking has been proven to be a successful tool and is widely used to figure out characteristics of human cognition in language or visual processing (Just & Carpenter, 1976 Cognitive Psychology8441–480). Eye movement has proven to be a successful measure to figure out characteristics of human language and visual processing (Rayner, 1998 Psychological Bulletin124(3) 372–422). Recently, mouse tracking was used for social-cognition-like categorization of sex-atypical faces and studying spoken-language processes (Magnuson, 2005 PNAS102(28) 9995–9996; Spivey et al., 2005 PNAS102 10393–10398). Here, we present a framework that uses both eye gaze and hand movement simultaneously for analyzing the relation of them during memory retrieval. We tracked eye and mouse movements when the subject was watching a drama and playing a multimodal memory game (MMG), a cognitive task designed to investigate the recall memory mechanisms in watching video dramas (Zhang, 2009 AAAI 2009 Spring Symposium: Agents that Learn from Human Teachers 144–149). Experimental results show that eye tracking and mouse tracking provide complementary information about underlying cognitive processes. Also, we found some interesting patterns in eye-hand movement during multimodal memory recall.

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.

Purpose:The purpose of this work was to develop a quality assurance (QA) system for high dose rate (HDR) brachytherapy to verify the absolute position of an 192Ir source in real time and to measure dwell time and position of the source simultaneously with a movie recorded by a web camera.Methods:A web camera was fixed 15 cm above a source position check ruler to monitor and record 30 samples of the source position per second over a range of 8.0 cm, from 1425 mm to 1505 mm. Each frame had a matrix size of 480×640 in the movie. The source position was automatically quantified from the movie using in‐house software (built with LabVIEW) that applied a template‐matching technique. The source edge detected by the software on each frame was corrected to reduce position errors induced by incident light from an oblique direction. The dwell time was calculated by differential processing to displacement of the source. The performance of this QA system was illustrated by recording simple plans and comparing the measured dwell positions and time with the planned parameters.Results:This QA system allowed verification of the absolute position of the source in real time. The mean difference between automatic and manual detection of the source edge was 0.04 ± 0.04 mm. Absolute position error can be determined within an accuracy of 1.0 mm at dwell points of 1430, 1440, 1450, 1460, 1470, 1480, 1490, and 1500 mm, in three step sizes and dwell time errors, with an accuracy of 0.1% in more than 10.0 sec of planned time. The mean step size error was 0.1 ± 0.1 mm for a step size of 10.0 mm.Conclusion:This QA system provides quick verifications of the dwell position and time, with high accuracy, for HDR brachytherapy.This work was supported by the Japan Society for the Promotion of Science Core‐to‐Core program (No. 23003).

Expert and novice group differences in eye movements when assessing biodiversity of harvested forests

Improvement of Gaze Control After Balance and Eye Movement Training in Patients With Progressive Supranuclear Palsy: A Quasi-Randomized Controlled Trial

The present study utilized a new experimental set-up synchronizing eye movements and head motion for investigating referential change occurring in a reading task. We examined the effects of a change in narrative perspective during reading on eye movements and head motion. Forty-four participants read texts on a digital tablet, and both participants' eye movements and head movements were recorded using eye tracker and motion capture. The results showed longer eye fixation duration, longer reading time and decreasing head motions when perspective changed. Recent studies have supported the dynamic engagement hypothesis suggesting that there is a fluctuation in cognitive engagement reflected by postural movements. Our findings on head movements seem to validate this hypothesis of a bodily engagement in reading. The results provided by our study showed that the novel methodology of eye and head movement recordings we used was proven to be informative in studying the reader's embodiment during reading.

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

Eye tracking techniques enable high-efficient, natural, and effortless human-machine interaction by detecting users’ eye movements and decoding their attention and intentions. Here, a miniature, imperceptible, and biocompatible smart contact lens is proposed for in situ eye tracking and wireless eye-machine interaction. Employing the frequency encoding strategy, the chip-free and battery-free lens successes in detecting eye movement and closure. Using a time-sequential eye tracking algorithm, the lens has a great angular accuracy of <0.5°, which is even less than the vision range of central fovea. Multiple eye-machine interaction applications, such as eye-drawing, Gluttonous Snake game, web interaction, pan-tilt-zoom camera control, and robot vehicle control, are demonstrated on the eye movement model and in vivo rabbit. Furthermore, comprehensive biocompatibility tests are implemented, demonstrating low cytotoxicity and low eye irritation. Thus, the contact lens is expected to enrich approaches of eye tracking techniques and promote the development of human-machine interaction technology.

RETRACTED: Real time speech recognition algorithm on embedded system based on continuous Markov model