• Home
  • Search
  • Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features
  • Cite Icon3
  • https://doi.org/10.1111/coin.12624Copy DOI Icon

Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Abstract In Visual question answering (VQA), a natural language answer is generated for a given image and a question related to that image. There is a significant growth in the VQA task by applying an efficient attention mechanism. However, current VQA models use region features or object features that are not adequate to improve the accuracy of generated answers. To deal with this issue, we have used a Two‐way Co‐Attention Mechanism (TCAM), which is capable enough to fuse different visual features (region, object, and concept) from diverse perspectives. These diverse features lead to different sets of answers, and also, there is an inherent relationship between these visual features. We have developed a powerful attention mechanism that uses these two critical aspects by using both bottom‐up and top‐down TCAM to extract discriminative feature information. We have proposed a Collective Feature Integration Module (CFIM) to combine multimodal attention features and thus capture the valuable information from these visual features by employing a TCAM. Further, we have formulated a Vertical CFIM for fusing the features belonging to the same class and a Horizontal CFIM for combining the features belonging to different types, thus balancing the influence of top‐down and bottom‐up co‐attention. The experiments are conducted on two significant datasets, VQA 1.0 and VQA 2.0. On VQA 1.0, the overall accuracy of our proposed method is 71.23 on the test‐dev set and 71.94 on the test‐std set. On VQA 2.0, the overall accuracy of our proposed method is 75.89 on the test‐dev set and 76.32 on the test‐std set. The above overall accuracy clearly reflecting the superiority of our proposed TCAM based approach over the existing methods.

Similar Papers
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Conference Article
  • Citations48

Coarse-to-Fine Reasoning for Visual Question Answering

  • Jun 01, 2022
  • Binh X Nguyen +5
  • Research Article
  • Citations17

Accuracy vs. complexity: A trade-off in visual question answering models

  • Jun 12, 2021
  • Pattern Recognition
  • Moshiur Farazi +2
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Research Article
  • Citations472

The multi-modal fusion in visual question answering: a review of attention mechanisms.

  • May 30, 2023
  • PeerJ Computer Science
  • Siyu Lu +5
  • Conference Article
  • Citations2

Estimating Viewed Images with Natural Language Question Answering from fMRI Data

  • Mar 01, 2020
  • Saya Takada +3
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
  • Research Article
  • Citations2

Hierarchical Attention Networks for Multimodal Machine Learning

  • Mar 01, 2022
  • Journal of Physics Conference Series
  • Haotian Liang +1
  • Conference Article
  • Citations11

The Impact of Explanations on AI Competency Prediction in VQA

  • Sep 01, 2020
  • Kamran Alipour +5
  • Conference Article
  • Citations2558

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Jul 01, 2017
  • Yash Goyal +4
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Book Chapter
  • Citations3

Optimal Image Feature Ranking and Fusion for Visual Question Answering

  • Sep 09, 2020
  • Sruthy Manmadhan +1
  • Research Article
  • Citations12

Comprehensive-perception dynamic reasoning for visual question answering

  • Jul 01, 2022
  • Pattern Recognition
  • Kai Shuang +2
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.