• Home
  • Search
  • Connecting Language and Vision to Actions
  • Cite Icon1
  • https://doi.org/10.18653/v1/p18-5004Copy DOI Icon

Connecting Language and Vision to Actions

  • Jan 1, 2018
  • Peter Anderson +2 more
Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

A long-term goal of AI research is to build intelligent agents that can see the rich visual environment around us, communicate this understanding in natural language to humans and other agents, and act in a physical or embodied environment. To this end, recent advances at the intersection of language and vision have made incredible progress – from being able to generate natural language descriptions of images/videos, to answering questions about them, to even holding free-form conversations about visual content! However, while these agents can passively describe images or answer (a sequence of) questions about them, they cannot act in the world (what if I cannot answer a question from my current view, or I am asked to move or manipulate something?). Thus, the challenge now is to extend this progress in language and vision to embodied agents that take actions and actively interact with their visual environments. To reduce the entry barrier for new researchers, this tutorial will provide an overview of the growing number of multimodal tasks and datasets that combine textual and visual understanding. We will comprehensively review existing state-of-the-art approaches to selected tasks such as image captioning, visual question answering (VQA) and visual dialog, presenting the key architectural building blocks (such as co-attention) and novel algorithms (such as cooperative/adversarial games) used to train models for these tasks. We will then discuss some of the current and upcoming challenges of combining language, vision and actions, and introduce some recently-released interactive 3D simulation environments designed for this purpose.

Loading PDF

Similar Papers
  • Research Article
  • Citations1

Sequential image encoding for vision-to-language problems

  • Dec 02, 2019
  • Multimedia Tools and Applications
  • Jicheng Wang +4
  • Research Article
  • Citations17

Inner Knowledge-based Img2Doc Scheme for Visual Question Answering

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Qun Li +4
  • Conference Article
  • Citations110

Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering

  • Jun 01, 2018
  • Unnat Jain +2
  • Research Article
  • Citations147

Vision-Language Models in Remote Sensing: Current progress and future trends

  • Jun 01, 2024
  • IEEE Geoscience and Remote Sensing Magazine
  • Xiang Li +4
  • Research Article
  • Citations5

Scaling up biomedical vision-language models: Fine-tuning, instruction tuning, and multi-modal learning.

  • Nov 01, 2025
  • Journal of biomedical informatics
  • Cheng Peng +5
  • Research Article
  • Citations41

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

  • Mar 05, 2025
  • ACM Computing Surveys
  • Jiayi Kuang +9
  • Conference Article

GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes

  • Nov 28, 2025
  • Qianxi Yu +5
  • Research Article
  • Citations53

Visual question answering: Which investigated applications?

  • Nov 01, 2021
  • Pattern Recognition Letters
  • Silvio Barra +3
  • Conference Article
  • Citations8

Vιsual question answering models Evaluation

  • Jun 01, 2020
  • S Sarath +1
  • Book Chapter
  • Citations2

Applying Vector Symbolic Architecture and Semiotic Approach to Visual Dialog

  • Jan 01, 2021
  • Alexey K Kovalev +3
  • Conference Article
  • Citations59

Visual Question Answering as Reading Comprehension

  • Jun 01, 2019
  • Hui Li +3
  • PDF
  • Research Article
  • Citations5

Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study

  • Aug 05, 2024
  • JMIR Medical Informatics
  • Usman Naseem +2
  • Research Article

Integrating features and harnessing pre-trained visual-language models for enhancing VQA reading comprehension

  • May 05, 2025
  • Journal of Computer Science and Cybernetics
  • Xuan Linh Truong
  • Conference Article
  • Citations2

Beyond Words: Exploring Co-Attention with BERT in Visual Question Answering

  • May 24, 2024
  • Vibhashree B S +4
  • Conference Article
  • Citations6

Visual World to an Audible Experience: Visual Assistance for the Blind And Visually Impaired

  • Dec 10, 2020
  • S Shiv Mohith +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.