• Home
  • Search
  • Image Captioning and Visual Question Answering Based on Attributes and External Knowledge.
  • Open Access IconOpen Access
  • Cite Icon447
  • https://doi.org/10.1109/tpami.2017.2708709Copy DOI Icon

Image Captioning and Visual Question Answering Based on Attributes and External Knowledge.

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Much of the recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We further show that the same mechanism can be used to incorporate external knowledge, which is critically important for answering high level visual questions. Specifically, we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. It particularly allows questions to be asked where the image alone does not contain the information required to select the appropriate answer. Our final model achieves the best reported results for both image captioning and visual question answering on several of the major benchmark datasets.

Similar Papers
  • Conference Article
  • Citations556

What Value Do Explicit High Level Concepts Have in Vision to Language Problems?

  • Jun 01, 2016
  • Qi Wu +4
  • Conference Article
  • Citations14

Image Captioning using Convolutional Neural Networks and Recurrent Neural Network

  • Apr 02, 2021
  • Rachel Calvin +1
  • Research Article

Image Captioning using Convolutional Neural Networks and Long Short Term Memory Cells

  • May 30, 2022
  • International Journal of Recent Technology and Engineering (IJRTE)
  • Hitoishi Das
  • Research Article
  • Citations29

Multi-source Multi-level Attention Networks for Visual Question Answering

  • Apr 30, 2019
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Dongfei Yu +3
  • Research Article
  • Citations21

Dynamic Convolution-based Encoder-Decoder Framework for Image Captioning in Hindi

  • Mar 24, 2023
  • ACM Transactions on Asian and Low-Resource Language Information Processing
  • Santosh Kumar Mishra +3
  • Research Article
  • Citations76

Recurrent convolutional neural network for answer selection in community question answering

  • Apr 11, 2017
  • Neurocomputing
  • Xiaoqiang Zhou +3
  • Conference Article
  • Citations6

Visual World to an Audible Experience: Visual Assistance for the Blind And Visually Impaired

  • Dec 10, 2020
  • S Shiv Mohith +3
  • PDF
  • Research Article
  • Citations1

Visual Question Answering using Convolutional Neural Networks

  • Apr 11, 2021
  • Turkish Journal of Computer and Mathematics Education (TURCOMAT)
  • K P Moholkar, Et Al
  • Conference Article
  • Citations4

An Image Sentence Generation Based on Deep Neural Network Using RCNN-LSTM Model

  • Nov 26, 2021
  • S Sai Satyanarayana Reddy +2
  • Research Article
  • Citations1

Sequential image encoding for vision-to-language problems

  • Dec 02, 2019
  • Multimedia Tools and Applications
  • Jicheng Wang +4
  • Research Article
  • Citations67

Dual self-attention with co-attention networks for visual question answering

  • Apr 09, 2021
  • Pattern Recognition
  • Yun Liu +6
  • Research Article
  • Citations226

EMG‐Based Estimation of Limb Movement Using Deep Learning With Recurrent Convolutional Neural Networks

  • Oct 25, 2017
  • Artificial Organs
  • Peng Xia +2
  • Book Chapter
  • Citations16

Learning Convolutional Text Representations for Visual Question Answering

  • May 07, 2018
  • Zhengyang Wang +1
  • Research Article
  • Citations78

Sequence to Sequence ECG Cardiac Rhythm Classification Using Convolutional Recurrent Neural Networks.

  • Feb 01, 2022
  • IEEE Journal of Biomedical and Health Informatics
  • Teeranan Pokaprakarn +5
  • Research Article
  • Citations1

Visual Question Answering with External Knowledge

  • Jan 01, 2021
  • SSRN Electronic Journal
  • Santhosh Voruganti +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.