- Conference Article
- 10.1109/aihcir67580.2025.11404921
GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes
- Nov 28, 2025
- Qianxi Yu + 5 more +5
To precisely recognize a visual object embedded in a complex visual scene, a person often has to adjust his or her gaze to fixate on the object, i.e., placing the object into the center of the visual field. Therefore, correctly understanding gaze direction and, more importantly, which object is being fixated on, is necessary to understand the intention of a person. To test how well vision large language models (VLLMs) can understand gaze direction and which object is being fixated on in a complex visual scene, the current study presents a novel dataset, referred to as the Gaze-based Intention Inference in Complex Scenes (GIICS), that comprises over 2800 pictures taken in 6 indoor and 14 outdoor backgrounds. Each picture contains two visual objects embedded in the visual background, with a viewer fixating on one of them. To verify if VLLMs can directly understand gaze direction instead of inferring gaze direction based on body posture or head direction, we constructed the GIICS-BodyPosture subset of images featuring uncommon combinations of body and head directions. Moreover, we constructed the GIICS-ObjectAngle subset, where we manipulated the visual angles between competing objects (e.g., objects placed to the left or right of the viewer) to investigate how angular intervals affect gaze target detection. We paired each picture with a visual question answering (VQA) question (e.g., "what is the person looking atƒ (A) the yellow object; (B) the blue object; (C) neither.") and evaluated how well nine popular VLLMs, e.g., Llama and Qwen2.5-VL, could answer the questions in both Chinese and English. The median VQA accuracy across models was 37.1% (VQA in Chinese) and 40.6% (VQA in English) and the highest performance was 57.1% (VQA in Chinese) and 58.8% (VQA in English). These results reveal significant deficits of current VLLMs when inferring human intention based on gaze direction in complex visual scenes.
Read more