Eye movement strategies in face ethnicity categorization vs. face identification tasks

Our perception of an event is often modulated by our past experience and expectations [1Ramachandran V.S. Armel C. Foster C. Stoddard R. Object recognition can drive motion perception.Nature. 1998; 395: 852-853Crossref PubMed Scopus (35) Google Scholar, 2O'Regan J.K. Noe A. A sensorimotor account of vision and visual consciousness.Behav. Brain Sci. 2001; 24: 939-973Crossref PubMed Scopus (1863) Google Scholar]. Here we used a magic trick [3Triplett N. The psychology of conjuring deceptions.Am. J. Psychol. 1900; 6: 439-510Crossref Google Scholar] to demonstrate how magicians can distort our subjective perception and we investigate the mechanisms behind this deception. We found that when a magician performed an illusion in which a ball was seen to vanish in the air, 63% of observers perceived the ball leave his hand, move upwards, and disappear even though the ball did not leave the magician's hand. Moreover, observers' illusory perception of the ball was determined by cues that indicated the ball's location, namely the magician's head direction, rather than the percept itself. Furthermore, eye movement records revealed participants' strategic use of social cues prior to looking at the ball. Surprisingly, however, when the ball was not physically present, observers did not look at the area where they claimed to have seen the ball vanish, suggesting that the oculomotor system was not fooled by the illusion. These results show that although people's subjective percept is determined by expectations, the oculomotor system is largely driven by accurate bottom-up information, which is consistent with the suggestion that there are separate mechanisms for perception and visuomotor control [4Goodale M.A. Milner A.D. Separate visual pathways for perception and action.Trends Neurosci. 1992; 15: 20-25Abstract Full Text PDF PubMed Scopus (3924) Google Scholar]. A striking example of magicians' deception is the vanishing ball illusion, in which a ball apparently disappears in mid air. This illusion is created by pretending to throw a ball up in the air, when in fact it remains secretly palmed in the magician's hand [3Triplett N. The psychology of conjuring deceptions.Am. J. Psychol. 1900; 6: 439-510Crossref Google Scholar]. If done convincingly, the observer should perceive the ball as moving up in the air even though it is no longer physically present. Similarly, when observers view a moving object that suddenly disappears, the final position of the object is usually perceived as being further along the path of motion than its actual final position [5Freyd J.J. Finke R.A. Representational Momentum.J. Exp. Psychol. Learn. Mem. Cogn. 1984; 10: 126-132Crossref Scopus (464) Google Scholar], a phenomenon known as representational momentum [6Assad J.A. Maunsell J.H.R. Neuronal correlates of inferred motion in primate posterior parietal carter.Nature. 1995; 373: 518-521Crossref PubMed Scopus (181) Google Scholar]. Much of the magician's deception relies on social cuing, in terms of his head and gaze direction, and we therefore predicted that the effectiveness of the vanishing ball illusion would be influenced by the magician's social cuing. Two different versions of the trick were created that addressed the extent to which the magician's social cues are responsible for the illusion. In the social cues pro-illusion condition, on the final 'fake' throw, the magician's eyes and head followed an imaginary ball moving upwards (Figure 1 and Supplemental data available on-line with this issue). In the social cues anti-illusion condition, the magic trick was identical except that on the final 'fake' throw, the magician looked at the hand concealing the ball, rather than following the imaginary ball (see Figure 2F). We also measured people's eye movements while they watched the video of the trick, to gain an online measure of the information obtained by the visual system. Immediately after having seen the magic trick, participants were questioned as to whether they perceived the 'illusory ball' moving towards the top of the screen on the final throw.Figure 2Participants' fixation points for the social cues pro-illusion (top row) and the social cues anti-illusion (bottom row) conditions.Show full captionThe dots represent the fixation points for the participants who experienced the illusion and the 'X' marks the fixation points for participants who did not experience the illusion. (A,D) At the point when the ball reached its zenith on the first throw. (B,E) at the point when the ball reached its zenith on the second throw. (C,F) At the point when the ball disappeared. For each participant the height of the terminal eye position on the monitor was calculated (distance between the fixation point and the bottom of the monitor, monitor height = 25°); each panel shows the means (M) and standard deviations (SD) for participants who experienced the illusion and those who did not. On the final throw (C,F) there was no significant difference between the final eye position of the participants who perceived the illusion compared to those who did not perceive it (t(36) < 1), which shows that participants' eye movements were not correlated with the perception of the illusion. On the final throw there was no significant difference in the terminal eye positions between the anti- and the pro-illusion condition (t(36) < 1), illustrating that participants' eye movements were not driven by the social cues. 19 participants experienced the illusion. For these participants the terminal eye position on the final throw was significantly lower than on the first (t(18) = 3.23, p = 0.005) and the second throw (t(18) = 3.57, p = 0.002).View Large Image Figure ViewerDownload Hi-res image Download (PPT) The dots represent the fixation points for the participants who experienced the illusion and the 'X' marks the fixation points for participants who did not experience the illusion. (A,D) At the point when the ball reached its zenith on the first throw. (B,E) at the point when the ball reached its zenith on the second throw. (C,F) At the point when the ball disappeared. For each participant the height of the terminal eye position on the monitor was calculated (distance between the fixation point and the bottom of the monitor, monitor height = 25°); each panel shows the means (M) and standard deviations (SD) for participants who experienced the illusion and those who did not. On the final throw (C,F) there was no significant difference between the final eye position of the participants who perceived the illusion compared to those who did not perceive it (t(36) < 1), which shows that participants' eye movements were not correlated with the perception of the illusion. On the final throw there was no significant difference in the terminal eye positions between the anti- and the pro-illusion condition (t(36) < 1), illustrating that participants' eye movements were not driven by the social cues. 19 participants experienced the illusion. For these participants the terminal eye position on the final throw was significantly lower than on the first (t(18) = 3.23, p = 0.005) and the second throw (t(18) = 3.57, p = 0.002). Participants who perceived the illusory ball had a vivid recollection of seeing the ball leave the screen at the top, and typically claimed that the illusion was created by someone catching it beyond the top of the screen. 68% of the participants in the social cues pro-illusion condition experienced the illusion, significantly more than in the social cues anti-illusion condition (32%, χ2 = 5.16, p = 0.025), thus demonstrating that the illusion was mediated by the social cuing. Most of the participants claimed that they spent their entire time looking at the ball. However, these reports deviated strongly from where they were actually looking. Figure 1 shows a timeline indicating where participants were looking during the magic trick. We found a very high consistency in the eye movement patterns between the participants, suggesting that people employed similar eye movement strategies. On throws where the ball was visible, participants generally looked at the ball once it reached the apex. But rather than merely tracking the ball, most of the participants glanced at the magician's face before looking at the ball, indicating that the visual system uses information about where the magician is looking as a way of predicting the location of the ball. Our results show that an observer's percept was driven by the magician's social cuing, and that participants utilized the magician's social cues. However, there was a surprising difference in the susceptibility of these two systems to the illusion. On the final illusory throw, most participants claimed to have seen the ball at the top of the screen. The oculomotor system on the other hand was not fooled by the illusion. Figure 2 shows that participants only looked at the top of the screen when the ball was physically present. These results illustrate a remarkable dissociation between what participants claimed to have seen and the way in which their eyes behaved. Whilst their percept was predominantly based on expectations, their eyes were largely driven by the actual visual input. Our results offer a striking example of how the dissociation between vision for perception and for action, well documented in neurological patients [4Goodale M.A. Milner A.D. Separate visual pathways for perception and action.Trends Neurosci. 1992; 15: 20-25Abstract Full Text PDF PubMed Scopus (3924) Google Scholar, 7Goodale M.A. Haffenden A. Frames of reference for perception and action in the human visual system.Neurosci. Biobehav. Rev. 1998; 22: 161-172Crossref PubMed Scopus (110) Google Scholar, 8Milner A.D. Goodale M.A. The Visual Brain in Action. Oxford University Press, Oxford1995Google Scholar] and neurologically intact subjects [9Aglioti S. Desouza J.F.X. Goodale M.A. Size-contrast illusions deceive the eye but not the hand.Curr. Biol. 1995; 5: 679-685Abstract Full Text Full Text PDF PubMed Scopus (721) Google Scholar], also translates to more real world situations. We would like to thank the reviewers and Cristiana Cavina-Pratesi for their valuable comments on the manuscript, and Tim Smith for his help with a control experiment. Download .pdf (.1 MB) Help with pdf files Document S1. Supplemental Experimental Procedures Download .mpg (1.09 MB) Help with mpg files Movie 1 Vanishing ball trick pro-illusion condition Download .mpg (1.12 MB) Help with mpg files Movie 2 Vanishing ball trick anti-illusion condition

Humans visually process the world with varying spatial resolution and can program their eye movements optimally to maximize information acquisition for a variety of everyday tasks. Diseases such as macular degeneration can change visual sensory processing, introducing central vision loss (a scotoma). However, humans can learn to direct a new preferred retinal location to regions of interest for simple visual tasks. Whether such learned compensatory saccades are optimal and generalize to more complex tasks, which require integrating information across a large area of the visual field, is not well understood. Here, we explore the possible effects of central vision loss on the optimal saccades during a face identification task, using a gaze-contingent simulated scotoma. We show that a new foveated ideal observer with a central scotoma correctly predicts that the human optimal point of fixation to identify faces shifts from just below the eyes to one that is at the tip of the nose and another at the top of the forehead. However, even after 5000 trials, humans of both sexes surprisingly do not change their initial fixations to adapt to the new optimal fixation points to faces. In contrast, saccades do change for tasks such as object following and to a lesser extent during search. Our findings argue against a central brain motor-compensatory mechanism that generalizes across tasks. They instead suggest task specificity in the learning of oculomotor plans in response to changes in front-end sensory processing and the possibility of separate domain-specific representations of learned oculomotor plans in the brain.SIGNIFICANCE STATEMENT The mechanism by which humans adapt eye movements in response to central vision loss is still not well understood and carries importance for gaining a fundamental understanding of brain plasticity. We show that although humans adapt their eye movements for simpler tasks such as object following and search, these adaptations do not generalize to more complex tasks such as face identification. We provide the first computational model to predict where humans with central vision loss should direct their eye movements in face identification tasks, which could become a critical tool in making patient-specific recommendations. Based on these results, we suggest a novel theory for oculomotor learning: a distributed representation of learned eye-movement plans represented in domain-specific areas of the brain.

Knowing where people look in a face provides an objective insight onto the information entering the visual system and into the cognitive processes involved in face perception. Eye-tracking studies on face perception have mostly investigated observers’ viewing behavior when studying single faces. However, in day-to-day situations, humans also compare faces or match a person’s face to a photograph. During comparison, facial information remains visually accessible, freeing observers from time and encoding constraints [1]. Here, we recorded eye movements of human participants while they compared two faces presented simultaneously. We used (i) two different tasks (discrimination or categorization), and (ii) faces differing either in identity or in sex. In addition, we varied (iii) task difficulty, i.e. the similarity of the two faces in a pair. Eye movements to previously defined areas of interest (AOIs) on the faces were analyzed in terms of frequency, duration and the temporal pattern of fixations made. We found that the eyes were fixated most often in the discrimination tasks (37 of all fixations) but the nose in the categorization task (34.5), while the total number of fixations increased with task difficulty. Faces differing in sex were more difficult to discriminate than faces differing in identity (63 versus 76 correct responses), which was also reflected in more fixations to face pairs differing in sex (14.4 versus 11.8 fixations per trial). With increasing task difficulty, fixations to only some AOIs increased, in accordance with the literature (more to the eyes in the sex and more over all areas in the identity discrimination tasks; [2]). Unexpectedly, we found a striking effect of tasks on performance measures, as over 80 of participants could detect the more feminine of two faces (categorization task) even at the most similar level, but for the same face pairs their performance in a discrimination task was less than 30 correct. Another interesting finding is that observers mostly compared the inner halves of the two faces of a pair, instead of the corresponding features (e.g., the left eye of the left face with the left eye of the right face). This viewing behavior remained the same in a control experiment where participants’ head was not fixed. Quite surprisingly, female participants fixated significantly more often the eyes of the face stimuli than male participants, but only when the sex of the faces was a relevant feature in the task.

An application of semi-supervised learning is made to the problem of person identification in low quality webcam images. Using a set of images of ten people collected over a period of four months, the person identification task is posed as a graph-based semi-supervised learning problem, where only a few training images are labeled. The importance of domain knowledge in graph construction is discussed, and experiments are presented that clearly show the advantage of semi-supervised learning over standard supervised learning. The data used in the study is available to the research community to encourage further investigation of this problem.

In comparison to the basal ganglia, prefrontal cortex, and medial temporal lobes, the cerebellum has been absent from recent research on the neural substrates of categorization and identification, two prominent tasks in the learning and memory literature. To investigate the contribution of the cerebellum to these tasks, we tested patients with cerebellar pathology (seven with bilateral degeneration, six with unilateral lesions, and two with midline damage) on rule-based and information-integration categorization tasks and an identification task. In rule-based tasks, it is assumed that participants learn the categories through an explicit reasoning process. In information-integration tasks, optimal performance requires the integration of information from multiple stimulus dimensions, and participants are typically unaware of the decision strategy. The identification task, in contrast, required participants to learn arbitrary, color-word associations. The cerebellar patients performed similar to matched controls on all three tasks and performance did not vary with the extent of cerebellar pathology. Although the interpretation of these null results requires caution, these data contribute to the current debate on cerebellar contributions to cognition by providing boundary conditions on understanding the neural substrates of categorization and identification, and help define the functional domain of the cerebellum in learning and memory.

Introduction: Visual abnormalities and manual motor control have been studied extensively after stroke, but an understanding of oculomotor control post-stroke has not. Recent studies have revealed that in visually guided reaches arm movements are planned during eye movement execution, which may contribute to increased task complexity. In fact, in healthy controls during visually guided reaches, the onset of eye movement is delayed, its velocity reduced, and endpoint errors are larger relative to isolated eye movements. Our objective in this experiment was to examine the temporal properties of eye movement execution for stroke patients with no diagnosed visual impairment. The goal is to improve understanding of oculomotor control in stroke relative to normal function, and ultimately further understand its coordination with manual motor control during joint eye and hand movements. We hypothesized that stroke patients would show abnormal initiation or onset latency for saccades made in an eye movement task, as compared to healthy controls. Methods: We measured the kinematics of eye movements during point-to-point saccades; there was an initial static, fixation point and the stimulus was a flashed target on a computer monitor. We used a video-based eye tracker for objective recording of the eye at a sampling frequency of 2000 Hz (SR Research, Eyelink). 10 stroke subjects, over 4 months from injury and with no diagnosed visual impairment, and 10 healthy controls completed 432 saccades in a serial fashion. Results: Stroke patients had significantly faster onset latencies as compared to healthy controls during saccades (99.5ms vs. 245.2ms, p=0.00058). Conclusion: A better understanding of the variations in oculomotor control post-stroke, which may go unnoticed during clinical assessment, may improve understanding of how eye control synchronizes with arm or manual motor control. This knowledge could assist in tailoring rehabilitative strategies to amplify motor recovery. For next steps, we will perform objective eye and hand recordings during visually guided reaches post-stroke to better understand the harmonization or lack thereof after neurologic insult.

Eye–hand coupling during closed-loop drawing: Evidence of shared motor planning?

A series of eye-tracking and categorization experiments investigated the use of speaking-rate information in the segmentation of Dutch ambiguous-word sequences. Juncture phonemes with ambiguous durations (e.g., [s] in 'eens (s)peer,' "once (s)pear," [t] in 'nooit (t)rap,' "never staircase/quick") were perceived as longer and hence more often as word-initial when following a fast than a slow context sentence. Listeners used speaking-rate information as soon as it became available. Rate information from a context proximal to the juncture phoneme and from a more distal context was used during on-line word recognition, as reflected in listeners' eye movements. Stronger effects of distal context, however, were observed in the categorization task, which measures the off-line results of the word-recognition process. In categorization, the amount of rate context had the greatest influence on the use of rate information, but in eye tracking, the rate information's proximal location was the most important. These findings constrain accounts of how speaking rate modulates the interpretation of durational cues during word recognition by suggesting that rate estimates are used to evaluate upcoming phonetic information continuously during prelexical speech processing.

The level of professional competence is presumably related to task performance and eye movement strategies. We have studied eye movements in experts and novices in the process of solving chemical problems. The subjects in the study were chemists with two different skill levels: professionals with ten years of experience and university graduates. First, they read the text describing complex chemical processes and then analyzed their schematic representation and filled in the gaps found in the scheme, using separately presented answers. Eye movements were recorded using the SMI iViewX Hi-Speed 1250 tracker with head support. As a result, a characteristic oculomotor pattern associated with the analysis and completion of conceptual schemes was identified and described. Compared to reading a text, schemes were completed with longer “concentrated” fixations, and longer and faster saccades. The experts read texts and solved schematically presented problems faster and also had more correct answers. When reading texts, there was no difference in eye movements between the experts and the novices. Analysis of eye movements has shown that experts are characterized by shorter first fixations and longer fixations on significant areas of graphical tasks. Novices usually have longer first fixations and evenly distributed shorter fixations on different parts of the task. It was also found that experts make fewer transitions between areas of interests than novices. Several indices of eye movements showed that success was associated with faster overall orientation in the main pattern and more in-depth analysis of the response options presented.

Last year, we showed that most of the visual processing for categorising centrally presented images could be done in 150 ms. The human subjects were given a go — no go categorisation task in which they had to release a button when they could identify an animal. All the pictures were natural scenes taken from a vast CD-ROM data bank allowing access to several thousands of stimuli so that each image was only seen once. Targets (animals) included fish, birds, mammals, reptiles, insects in their natural environment; distractors included landscapes, trees, flowers, fruits. They were presented for a very short duration (20 ms) in order to avoid eye movement. In the present study we wanted to evaluate how the performance was affected with extrafoveal stimulations. We used the same task, but the visual stimuli could be presented at random, either centrally where subjects had to maintain fixation or laterally (left or right hemifield). Behavioural data and concomitant evoked potentials were recorded. The subject's performance was extremely good with a mean score of 90% correct and a mean reaction time of 500 ms. Comparison of performance between central and lateral stimulations showed that most subjects could process lateral and central stimuli with the same speed (reaction time distributions were not statistically different), and that, although extrafoveal presentations induced a drop in the percentage of correct responses (mean decrease 5%), it was not significant in all subjects. Moreover, when compared with last year's study the increase of the number of spatial locations where the visual stimulus could appear did not induce a significant behavioural cost, in terms of speed or accuracy of the response. We also looked at a possible hemispheric specialisation with no clear results. The results obtained here show that even with shared spatial attention and extrafoveal vision, the human visual system is remarkably efficient in an identification task.

Do image properties drive first ocular explorations of natural scene during free viewing? The response is positive, according to previous works defining image properties with statistical measures or modeled with a saliency map. But, because these works were mainly based on correlation analysis (but see Einhauser, 2002), no causal link could be established between image properties and ocular fixations. Therefore, across three experiments (180 subjects) we manipulated the saliency and semantic congruency of small regions within 72 natural scenes during a categorization task. In the 'saliency' experiment, the mean saliency of one region was reduced for half the scenes. In the 'semantic' experiment, an incongruent object (object with a very low probability of occurrence in the scene context) was introduced. This object was substituted for another but without any modification in the saliency. For both experiments the luminance distribution of the modified regions were held constant. The third experiment was a control condition (without any manipulation). Eye movements were recorded during the 3s of scene presentation. The number of fixations and the time spent inside the manipulated regions were calculated. In contrast with control condition, data analysis showed that, first, when the saliency of one region was reduced the region was less attractive (the number of fixations and the time spent inside the region were lower than in the control condition). Secondly, when an incongruent object was inserted, in contrast to the expectations of the literature (Henderson & Hollingworth, 1998), the region was less attractive. In conclusion, on one hand; we establish a causal link between saliency and ocular fixations. On the other hand; keeping most of image properties constant, we show that ocular fixations are flexibly driven by the congruency of an object and its context. Incongruent object were less attractive during the first moment of exploration, in contrary to longer exploration.

In this paper, we investigated the parameters related to eye movement patterns of individuals while viewing images that consist of natural and man-made scenes. These parameters are as follows: number of fixations and saccades, fixation duration, saccade amplitude and distribution of fixation locations. We explored the way in which individuals look at images of different semantic categories, and used this information for automatic image classification. We showed that the eye movements and the contents of eye fixation locations of observers differ for images of different semantic categories. These differences were used effectively in automatic image categorization. Another goal of this study was to find the answer of this question that "whether the image patches of fixation points have sufficient information for image categorization?" To achieve this goal, a number of patches with different sizes from two different image categories was extracted. These patches, which were selected at the location of eye fixation points, were used to form a feature vector based on K-means clustering algorithm. Then, different statistical classifiers were trained for categorization purpose. The results showed that it is possible to predict the image category by using the feature vectors derived from the image patches. We found significant differences in parameters of eye movement pattern between the two image categories (average across subjects). We could categorize images by using these parameters as features. The results also showed that it is possible to predict the image category by using image patches around the subjects' fixation points.

Disentangling neglect and hemianopia

In this chapter, we report an investigation the influence of the saliency of another person's direction of gaze on an observer's eye movements through real-world scenes. Participants' eye movements were recorded while they viewed a sequence of scene photographs that told a story. A subset of the scenes contained an actor. The actor's face was highly likely to be fixated, and when it was, the observer's next saccade was more likely to be toward the object that was the focus of the actor's gaze than in any other direction. Furthermore, when eye movement patterns did not show an immediate saccade to the focused object, observers were nonetheless more likely to fixate the focused object than a control object within close temporal proximity of fixation on the face. We conclude that during real-world scene perception, observers are sensitive to another's direction of gaze and use it to help guide their own eye movements.

Work on eye movements in reading received a considerable new impetus starting in about 1975, when several research groups began using computers to control eye movement experiments “on-line” (McConkie & Rayner, 1975; O’Regan, 1979; Reder, 1973). The idea of a “moving window” allowing only a limited amount of text to be visible around the instantaneous eye position was an appealing technique that allowed researchers to measure the size of the “perceptual span,” that is, the zone around the fixation point from which different kinds of information is gathered. My own feeling at that time, and I believe also the feeling of other workers, was that the most reasonable thing that the eye might be doing in reading would be to adjust its saccades so that from moment to moment, the edges of successive perceptual spans would just touch, or overlap by a fixed amount. In that way, at moments when the text is easy to predict, or the words easy to recognize, the eye would be able to make larger saccades than when the eye was moving into less predictable or more difficult territory. This view of reading could be called the “perceptual span control” hypothesis (Jacobs & O’Regan, 1987). The perceptual span control hypothesis is at the root of a very large body of research in which perceptual span is measured by making perturbations in the information available in parafoveal vision (for a review see Rayner & Pollatsek, 1987): in this research, changes in eye movements are assumed to indicate changes in perceptual span. However, it has not usually been realized that this assumption can only be true to the extent that perceptual span actually directly determines eye movements.