TalkingEyes: Pluralistic Speech-Driven 3D Eye Gaze Animation.

This paper presents a realtime and accurate method for 3D eye gaze tracking with a monocular RGB camera. Our key idea is to train a deep convolutional neural network(DCNN) that automatically extracts the iris and pupil pixels of each eye from input images. To achieve this goal, we combine the power of Unet\cite{ronneberger2015u-net:} and Squeezenet\cite{iandola2017squeezenet:} to train an efficient convolutional neural network for pixel classification. In addition, we track the 3D eye gaze state in the Maximum A Posteriori (MAP) framework, which sequentially searches for the most likely state of the 3D eye gaze at each frame. When eye blinking occurs, the eye gaze tracker can obtain an inaccurate result. We further extend the convolutional neural network for eye close detection in order to improve the robustness and accuracy of the eye gaze tracker. Our system runs in realtime on desktop PCs and smart phones. We have evaluated our system on live videos and Internet videos, and our results demonstrate that the system is robust and accurate for various genders, races, lighting conditions, poses, shapes and facial expressions. A comparison against Wang et al.[3] shows that our method advances the state of the art in 3D eye tracking using a single RGB camera.

Previous work in automatic affect analysis (AAA) has emphasized static expressions to the neglect of the dynamics of facial movement and considered head movement only a nuisance variable to control. We investigated whether the dynamics of head and facial movements apart from specific facial expressions communicate affect in infants, an under-studied population in AAA. Age-appropriate tasks were used to elicit positive and negative affect in 31 ethnically diverse infants. 3D head and facial movements were tracked from 2D video. Head angles in the horizontal (pitch), vertical (yaw), and lateral (roll) directions were used to measure head movement; and the 3D coordinates of 49 facial points to measure facial movements. Strong effects were found for both head and facial movements. Angular velocity and angular acceleration of head pitch, yaw, and roll were higher during negative relative to positive affect. Amplitude, velocity, and acceleration of facial movement were higher as well during negative relative to positive affect. A linear discriminant analysis using head and facial movement achieved a mean classification rate of positive and negative affect equal to 65% (Kappa = 0.30). Head and facial movement individually and in combination were also strongly related to observer ratings of affect intensity. Our results suggests that the dynamics of head and facial movements communicate affect at ages as young as 13 months. These interdisciplinary findings from behavioral science and computer vision deepen our understanding of communication of affect and provide a basis for studying individual differences in emotion in socio-emotional development.

Seeing lexical tone: Head and face motion in production and perception of Cantonese lexical tones

Conclusions. Data about the recovery course of facial function after intratemporal facial nerve reconstruction using interposition nerve graft would provide useful information for clinicians to understand the regenerative process of the facial nerve after this type of surgery. It would also enable them to obtain informed consent from the patients by preoperatively explaining the predicted outcome of the postoperative facial paralysis. Objective. The purpose of this study was to describe the recovery course of facial movement and electrophysiological findings after intratemporal facial nerve reconstruction using interposition graft. Patients and methods. Five patients who underwent reconstruction of the facial nerve using interposition nerve graft immediately after facial nerve excision during surgery for temporal bone lesions were included in this study. Each patient was evaluated for facial movement (Yanagihara score), blink reflex (BR), and electroneurography (ENoG) preoperatively and postoperatively. Results. Improvement in facial movement began 8–10 months postoperatively. The score then gradually increased, and reached a plateau level by 2 years following surgery. The final score in four of the five patients ranged from 20 to 24 points, while the facial score of one patient only reached 12 points even at 3 years after surgery. All patients demonstrated moderate to severe synkinesis. The reappearance of R1 in BR occurred 7–10 months postoperatively, almost simultaneously with the beginning of recovery of facial movement. The latency of R1 on the operated side became shortened with increasing postoperative time, although it remained considerably longer than that on the unoperated side, even after 2 postoperative years. The onset of recovery of ENoG value (10–12 months postoperatively) was always delayed compared with the actual facial movement recovery and never returned to the level in the unoperated side.

In this data article, this dataset included raw data of head and eye movement that collected by Polhemus (Polhemus Inc) and SmartEye (Smart Eye AB) equipment. Subjects who have driver license participated in this experiment. The experiment was conducted with a driving simulator that was controlled by CarSim (Mechanical simulation Co., Anna Arbor, MI) with the vehicle motion. This data set not only contained the eye and head movement but also had eye gaze, pupil diameter, saccades, and so on. It can be used for the parameter identification of the vestibulor-ocular reflex (VOR) model, simulation eye movement, as well as running other analysis related to eye movement.

Depression is one of the most common mental health disorders with strong adverse effects on personal and social functioning. The absence of any objective diagnostic aid for depression leads to a range of subjective biases in initial diagnosis and ongoing monitoring. Psychologists use various visual cues in their assessment to quantify depression such as facial expressions, eye contact and head movements. This paper studies the contribution of (upper) body expressions and gestures for automatic depression analysis. A framework based on space-time interest points and bag of words is proposed for the analysis of upper body and facial movements. Salient interest points are selected using clustering. The major contribution of this paper lies in the creation of a bag of body expressions and a bag of facial dynamics for assessing the contribution of different body parts for depression analysis. Head movement analysis is performed by selecting rigid facial fiducial points and a new histogram of head movements is proposed. The experiments are performed on real-world clinical data where video clips of patients and healthy controls are recorded during interactive interview sessions. The results show the effectiveness of the proposed system to evaluate the contribution of various body parts in depression analysis.

We investigated whether the dynamics of head and facial movements apart from specific facial expressions communicate affect in infants. Age-appropriate tasks were used to elicit positive and negative affect in 28 ethnically diverse 12-month-old infants. 3D head and facial movements were tracked from 2D video. Strong effects were found for both head and facial movements. For head movement, angular velocity and angular acceleration of pitch, yaw, and roll were higher during negative relative to positive affect. For facial movement, displacement, velocity, and acceleration also increased during negative relative to positive affect. Our results suggest that the dynamics of head and facial movements communicate affect at ages as young as 12 months. These findings deepen our understanding of emotion communication and provide a basis for studying individual differences in emotion in socio-emotional development.

This work outlines a quantitative analysis of the relation between speech acoustics and the face and head motions that occur simultaneously [A. V. Barbosa, Ph.D. thesis, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil, 2004]. 2-D motion data is obtained by means of a video camera. An algorithm has been developed for tracking markers on the speaker’s face from the acquired video sequence [A. V. Barbosa, E. Vatikiotis-Bateson, and A. Daffertshofer, in Proceedings of the 8th ICSLP Interspeech 2004, Korea, 2004]. The motion domain is represented by the 2-D marker trajectories, whereas line spectrum pairs (LSP) coefficients and the fundamental frequency F0 are used to represent the speech acoustics domain. Mathematical models are trained to estimate the acoustic parameters (LSPs + F0) from the motion parameters (2-D marker positions). The estimated acoustic parameters are then used to synthesize the acoustic speech signal. Cross-domain analysis for undecomposed (i.e., full head + face) and decomposed (i.e., separated head and face) normalized 2-D motions is performed. Syntheses from each method using intelligibility tests and qualitative comparison of the original and synthesized utterances are being evaluated.

Driver behavioral cues may present a rich source of information and feedback for future intelligent driver assistance systems (IDAS). Two of the most useful cues might be eye gaze and head motion. Eye gaze provides a more accurate proxy than head motion for determining driver attention, whereas the measurement of head motion head motion as a derivative of pose is less cumbersome and more reliable in harsh driving conditions. With the design of a simple and robust IDAS in mind, we are interested in determining the most important driver cues for distinguishing driver intent. We use a lane change intent prediction system [1] to determine the relative usefulness of each cue for determining intent. Various combinations of input data are presented to a discriminative classifier, which is trained to output a prediction of probable lane change maneuver at a particular point in the future. Quantitative results using real-world data are presented and show that head motion, when combined with lane position and vehicle dynamics, is a reliable cue for lane change intent prediction. The addition of eye gaze does not improve performance as much as simpler head pose-based cues.

Research has shown that learning programs with well designed animated virtual pedagogical agents engage and motivate students, produce greater reported satisfaction and enjoyment by students, and produce greater learning gains than programs without these agents. How to design an animated virtual pedagogical agent behaves much like a sensitive and effective human tutor is a very challenge work to do. In this paper, we developed a procedure for producing head and face movements during speech by a virtual pedagogical agent by combining different voice recordings, different facial expressions and different head movement patterns. We then developed three experimental conditions to evaluate how the facial expressions and head movements contribute to learning experiences and outcomes. Results showed that facial expressions and head movements have great impact on student’s impressions and engagement with the virtual pedagogical agent. Experimental results suggest that virtual pedagogical agent that produce natural head movements and appropriate facial expressions while narrating a story produce much more positive user experiences than virtual pedagogical agent that lack these behaviors.

In this paper, computer mouse control with head movements and eye blinks was proposed for people with impaired spinal cord injury. The head mouse control is based on finding and predicting eye states and direction of the head. This human computer interface (HCI) is an assistant system for people with physical disabilities who are suffering from motor neuron diseases or severe cerebral palsy. By moving the head to the right, left, up and down moves the mouse pointer and sends mouse button commands using eye blinks. Here, left eye-blink triggers left mouse button, right eye-blink triggers right mouse button and double-eyed sends “holds” command. In this system, eye blink and head movement used same Convolutional Neural Network (CNN) architecture with different number of classes (output). In head movement part, CNN has 5 outputs, (forward, up, down, left and right), in eye-blink part CNN has 2 output either opened or closed. This combined system allows people with down to neck paralyzed, to control computer using head movement and eye blinking. The test results reveal that this system is robust and accurate. This invention allows people with disabilities to use computer with head movements and eye blinks without using any extra hardware.

A technique for the detection of facial gestures from low resolution video sequences is presented. The technique builds upon the automatic 3D head tracker formulation of [M. La Cascia et al., 2000]. The tracker is based on the registration of a texture-mapped cylindrical model. Facial gesture analysis is performed in the texture map by assuming that the residual registration error can be modeled as a linear combination of facial motion templates. Two formulations are proposed and tested. In one formulation, the head and facial motion are estimated in a single, combined linear system. In the other formulation, head motion and then facial motion are estimated in a two-step process. The two-step approach significantly yields better accuracy in facial gesture analysis. The system is demonstrated in detecting two types of facial gestures: "mouth opening" and "eyebrows raising." On a dataset with lots of head motion, the two-step algorithm achieved a recognition accuracy of 70% for the "mouth opening" and an accuracy of 66% for "eyebrows raising" gestures. The algorithm can reliably track and classify facial gestures without any user intervention and runs in real-time.

In human-robot interaction, human mental states in dialogue have attracted attention to human-friendly robots that support educational use. Although estimating mental states using speech and visual information has been conducted, it is still challenging to estimate mental states more precisely in the educational scene. In this paper, we proposed a method to estimate human mental state based on participants’ eye gaze and head movement information. Estimated participants’ confidence levels in their answers to the miscellaneous knowledge question as a human mental state. The participants’ non-verbal information, such as eye gaze and head movements during dialog with a robot, were collected in our experiment using an eye-tracking device. Then we collect participants’ confidence levels and analyze the relationship between human mental state and non-verbal information. Furthermore, we also applied a machine learning technique to estimate participants’ confidence levels from extracted features of gaze and head movement information. As a result, the performance of a machine learning technique using gaze and head movements information achieved over 80 % accuracy in estimating confidence levels. Our research provides insight into developing a human-friendly robot considering human mental states in the dialogue.

Advanced human-computer interaction (HCI) techniques are required to enhance current computer interfaces. In this paper, we present an eye gaze tracking system based on a robust low-cost real-time pupil detector, and we describe some eye-aware applications that are being developed to enhance HCI. Images of pupils are segmented using an active lighting scheme that exploits very particular properties of eyes. Once the pupil is detected, its center is tracked, along with the corneal reflection (CR) generated by the light sources. Assuming little head motion, the eye gaze direction is computed based on the vector between the centers of the CR and the pupil, after a brief calibration procedure. Other information, such as pupil size and blink rate, can also be made available. The current prototype runs at frame rate, providing 30 samples of the gaze position per second to gaze-aware applications, such as advanced pointing and selection mechanisms.

Gaze is defined as the coordinated sum of eye and head movements and gaze position is the position of the eye in space, while eye position is the position of the eye in the orbit. In this chapter, the authors discuss how several causal factors (protocol conditions including desired gaze amplitude and pathologic situations including reduced inclination to move the head in hemianopia) influence the selection of a gaze mode despite statistical selection. The results of the methods employed in this study show that there is a strong connection between the gaze type as defined by relative latency of eye and head movement and the VOR gain and gaze amplitude. It is also evident that some motor brain mechanism organizes a relative latency between head and eye movement.