• Home
  • Search
  • GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes
  • https://doi.org/10.1109/aihcir67580.2025.11404921Copy DOI Icon

GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes

  • Nov 28, 2025
  • Qianxi Yu +5 more
Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

To precisely recognize a visual object embedded in a complex visual scene, a person often has to adjust his or her gaze to fixate on the object, i.e., placing the object into the center of the visual field. Therefore, correctly understanding gaze direction and, more importantly, which object is being fixated on, is necessary to understand the intention of a person. To test how well vision large language models (VLLMs) can understand gaze direction and which object is being fixated on in a complex visual scene, the current study presents a novel dataset, referred to as the Gaze-based Intention Inference in Complex Scenes (GIICS), that comprises over 2800 pictures taken in 6 indoor and 14 outdoor backgrounds. Each picture contains two visual objects embedded in the visual background, with a viewer fixating on one of them. To verify if VLLMs can directly understand gaze direction instead of inferring gaze direction based on body posture or head direction, we constructed the GIICS-BodyPosture subset of images featuring uncommon combinations of body and head directions. Moreover, we constructed the GIICS-ObjectAngle subset, where we manipulated the visual angles between competing objects (e.g., objects placed to the left or right of the viewer) to investigate how angular intervals affect gaze target detection. We paired each picture with a visual question answering (VQA) question (e.g., "what is the person looking atƒ (A) the yellow object; (B) the blue object; (C) neither.") and evaluated how well nine popular VLLMs, e.g., Llama and Qwen2.5-VL, could answer the questions in both Chinese and English. The median VQA accuracy across models was 37.1% (VQA in Chinese) and 40.6% (VQA in English) and the highest performance was 57.1% (VQA in Chinese) and 58.8% (VQA in English). These results reveal significant deficits of current VLLMs when inferring human intention based on gaze direction in complex visual scenes.

Similar Papers
  • Conference Article

Reasoning-Enhanced Object-Centric Learning for Videos

  • Jul 20, 2025
  • Jian Li +3
  • Book Chapter
  • Citations2

A Comparative Analysis of Transformer-Based Models for Document Visual Question Answering

  • Jan 01, 2023
  • Vijay Kumari +2
  • Research Article
  • Citations36

Answer Questions with Right Image Regions: A Visual Attention Regularization Approach

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Yibing Liu +6
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article
  • Citations170

Object continuity enhances selective auditory attention

  • Sep 02, 2008
  • Proceedings of the National Academy of Sciences
  • Virginia Best +3
  • Conference Article
  • Citations11

Non-linear modeling of eye gaze perception as a function of gaze and head direction

  • Sep 01, 2014
  • Mar Gonzalez-Franco +1
  • Book Chapter

Boosting Visual Question Answering Through Geometric Perception and Region Features

  • Sep 28, 2023
  • Frontiers in artificial intelligence and applications
  • Hong Yu +3
  • Conference Article
  • Citations2

Exploring the Potential of Multimodal Large Language Models for Question Answering on Artworks

  • Jun 12, 2025
  • Alessio Ferrato +4
  • Conference Article
  • Citations29

Examine before You Answer

  • Oct 15, 2018
  • Lianli Gao +4
  • Conference Article
  • Citations211

Visual Question Generation as Dual Task of Visual Question Answering

  • Jun 01, 2018
  • Yikang Li +6
  • Research Article
  • Citations8

Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question Answering.

  • Aug 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Zhou Yu +4
  • Conference Article
  • Citations12

Self-Paced Curriculum Learning for Visual Question Answering on Remote Sensing Data

  • Jul 11, 2021
  • Zhenghang Yuan +2
  • Research Article
  • Citations21

Exploring Duality in Visual Question-Driven Top-Down Saliency.

  • Jan 01, 2019
  • IEEE Transactions on Neural Networks and Learning Systems
  • Shengfeng He +3
  • Conference Article
  • Citations6

Visual World to an Audible Experience: Visual Assistance for the Blind And Visually Impaired

  • Dec 10, 2020
  • S Shiv Mohith +3
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.