• Home
  • Search
  • Multimodal Continuous Visual Attention Mechanisms
  • Open Access IconOpen Access
  • Cite Icon10
  • https://doi.org/10.1109/iccvw54120.2021.00122Copy DOI Icon

Multimodal Continuous Visual Attention Mechanisms

  • Oct 1, 2021
  • Antonio Farinhas +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Visual attention mechanisms are a key component of neural network models for computer vision. By focusing on a discrete set of objects or image regions, these mechanisms identify the most relevant features and use them to build more powerful representations. Recently, continuous-domain alternatives to discrete attention models have been proposed, which exploit the continuity of images. These approaches model attention as simple unimodal densities (e.g. a Gaussian), making them less suitable to deal with images whose region of interest has a complex shape or is composed of multiple non-contiguous patches. In this paper, we introduce a new continuous attention mechanism that produces multimodal densities, in the form of mixtures of Gaussians. We use the EM algorithm to obtain a clustering of relevant regions in the image, and a description length penalty to select the number of components in the mixture. Our densities decompose as a linear combination of uni-modal attention mechanisms, enabling closed-form Jacobians for the backpropagation step. Experiments on visual question answering in the VQA-v2 dataset show competitive accuracies and a selection of regions that mimics human attention more closely in VQA-HAT, substantiating the impact of the structure induced by our visual prior. We present several examples that suggest how multimodal attention maps are naturally more interpretable than their unimodal counterparts, showing the ability of our model to automatically segregate objects from ground in complex scenes.

Similar Papers
  • Research Article
  • Citations82

Co-Attending Free-Form Regions and Detections With Multi-Modal Multiplicative Feature Embedding for Visual Question Answering

  • Apr 27, 2018
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Pan Lu +4
  • Research Article
  • Citations13

Local relation network with multilevel attention for visual question answering

  • Jan 20, 2020
  • Journal of Visual Communication and Image Representation
  • Bo Sun +3
  • Conference Article
  • Citations5

Sparse And Structured Visual Attention

  • Sep 19, 2021
  • Pedro Henrique Martins +3
  • Conference Article
  • Citations5696

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

  • Jun 01, 2018
  • Peter Anderson +6
  • Research Article
  • Citations16

Image Segmentation Based on Visual Attention Mechanism

  • Dec 01, 2009
  • Journal of Multimedia
  • Qiaorong Zhang +2
  • PDF
  • Conference Article
  • Citations23

Supervised Visual Attention for Multimodal Neural Machine Translation

  • Jan 01, 2020
  • Tetsuro Nishihara +4
  • PDF
  • Research Article
  • Citations2

Supervised Visual Attention for Multimodal Neural Machine Translation

  • Jan 01, 2021
  • Journal of Natural Language Processing
  • Tetsuro Nishihara +4
  • Research Article
  • Citations55

Visual question answering model based on graph neural network and contextual attention

  • Mar 29, 2021
  • Image and Vision Computing
  • Himanshu Sharma +1
  • Conference Article

GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes

  • Nov 28, 2025
  • Qianxi Yu +5
  • Conference Article
  • Citations15

Feature Enhancement in Attention for Visual Question Answering

  • Jul 01, 2018
  • Yuetan Lin +3
  • Research Article
  • Citations50

DRAU: Dual Recurrent Attention Units for Visual Question Answering

  • May 20, 2019
  • Computer Vision and Image Understanding
  • Ahmed Osman +1
  • PDF
  • Research Article
  • Citations21

Interpretable and Reliable Oral Cancer Classifier with Attention Mechanism and Expert Knowledge Embedding via Attention Map

  • Feb 23, 2023
  • Cancers
  • Bofan Song +22
  • Conference Article
  • Citations852

Dual Attention Networks for Multimodal Reasoning and Matching

  • Jul 01, 2017
  • Hyeonseob Nam +2
  • Conference Article
  • Citations409

Improved Fusion of Visual and Language Representations by Dense Symmetric Co-attention for Visual Question Answering

  • Jun 01, 2018
  • Duy-Kien Nguyen +1
  • Conference Article
  • Citations38

SegEQA: Video Segmentation Based Visual Attention for Embodied Question Answering

  • Oct 01, 2019
  • Haonan Luo +5
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.