• Home
  • Search
  • Contrast and Classify: Training Robust VQA Models
  • Open Access IconOpen Access
  • Cite Icon29
  • https://doi.org/10.1109/iccv48922.2021.00163Copy DOI Icon

Contrast and Classify: Training Robust VQA Models

  • Oct 1, 2021
  • Yash Kant +4 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Recent Visual Question Answering (VQA) models have shown impressive performance on the VQA benchmark but remain sensitive to small linguistic variations in input questions. Existing approaches address this by augmenting the dataset with question paraphrases from visual question generation models or adversarial perturbations. These approaches use the combined data to learn an answer classifier by minimizing the standard cross-entropy loss. To more effectively leverage augmented data, we build on the recent success in contrastive learning. We propose a novel training paradigm (ConClaT) that optimizes both cross-entropy and contrastive losses. The contrastive loss encourages representations to be robust to linguistic variations in questions while the cross-entropy loss preserves the discriminative power of representations for answer prediction.We find that optimizing both losses – either alternately or jointly – is key to effective training. On the VQA-Rephrasings [44] benchmark, which measures the VQA model’s answer consistency across human paraphrases of a question, ConClaT improves Consensus Score by 1.63% over an improved baseline. In addition, on the standard VQA 2.0 benchmark, we improve the VQA accuracy by 0.78% overall. We also show that ConClaT is agnostic to the type of data-augmentation strategy used.

Similar Papers
  • PDF
  • Conference Article
  • Citations2

Can Pre-training help VQA with Lexical Variations?

  • Jan 01, 2020
  • Shailza Jolly +1
  • PDF
  • Conference Article
  • Citations27

Improving Visual Question Answering by Referring to Generated Paragraph Captions

  • Jan 01, 2019
  • Hyounghun Kim +1
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Conference Article
  • Citations668

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

  • Jun 01, 2018
  • Aishwarya Agrawal +3
  • Research Article
  • Citations3

HCCL: Hierarchical Counterfactual Contrastive Learning for Robust Visual Question Answering

  • Oct 29, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Dongze Hao +3
  • Research Article
  • Citations42

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding.

  • Jan 01, 2021
  • IEEE Transactions on Neural Networks and Learning Systems
  • Qingxing Cao +4
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
  • Research Article
  • Citations29

Learning visual question answering on controlled semantic noisy labels

  • Jan 13, 2023
  • Pattern Recognition
  • Haonan Zhang +5
  • Research Article
  • Citations15

Regularizing Attention Networks for Anomaly Detection in Visual Question Answering

  • May 18, 2021
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Doyup Lee +2
  • Research Article

Visual Question Answering Model Using Transformer-Based Multi-Head Cross Attention Network with Optimized Gold Rush Octave Convolution Network

  • Oct 28, 2025
  • International Journal of Image and Graphics
  • J Jinu Sophia +2
  • PDF
  • Research Article
  • Citations2

An Entropy Clustering Approach for Assessing Visual Question Difficulty

  • Jan 01, 2020
  • IEEE Access
  • Kento Terao +4
  • Conference Article
  • Citations2

Estimating Viewed Images with Natural Language Question Answering from fMRI Data

  • Mar 01, 2020
  • Saya Takada +3
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • PDF
  • Research Article
  • Citations86

MedFuseNet:\xa0An\xa0attention-based multimodal deep learning model for visual question answering in the medical domain

  • Oct 06, 2021
  • Scientific Reports
  • Dhruv Sharma +2
  • Conference Article
  • Citations211

Visual Question Generation as Dual Task of Visual Question Answering

  • Jun 01, 2018
  • Yikang Li +6
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.