• Home
  • Search
  • Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.
  • Open Access IconOpen Access
  • Cite Icon617
  • https://doi.org/10.1109/tnnls.2018.2817340Copy DOI Icon

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Visual question answering (VQA) is challenging, because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multimodal feature fusion that is able to capture the complex interactions between multimodal features; and 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a "coattention" mechanism is developed using a deep neural network (DNN) architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multimodal feature fusion, a generalized multimodal factorized high-order pooling approach (MFH) is developed to achieve more effective fusion of multimodal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the Kullback-Leibler divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A DNN architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA data sets and win the runner-up in VQA Challenge 2017.

Similar Papers
  • Conference Article
  • Citations59

Visual Question Answering as Reading Comprehension

  • Jun 01, 2019
  • Hui Li +3
  • Research Article
  • Citations71

Multimodal feature fusion by relational reasoning and attention for visual question answering

  • Aug 19, 2019
  • Information Fusion
  • Weifeng Zhang +4
  • Conference Article
  • Citations785

Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering

  • Oct 01, 2017
  • Zhou Yu +3
  • Research Article
  • Citations2

GLFNet: An RGB-T Crowd Counting Network Based on Global–Local Multimodal Feature Fusion

  • Jan 01, 2025
  • IEEE Transactions on Instrumentation and Measurement
  • Yingxiang Hu +3
  • Research Article
  • Citations1

An Enhanced Term Weighted Question Embedding for Visual Question Answering

  • May 05, 2022
  • Journal of Information & Knowledge Management
  • Sruthy Manmadhan +1
  • Book Chapter
  • Citations4

Multi-modal Feature Fusion Based on Variational Autoencoder for Visual Question Answering

  • Jan 01, 2019
  • Liqing Chen +4
  • Research Article
  • Citations27

MFUR-Net: Multimodal feature fusion and unimodal feature refinement for RGB-D salient object detection

  • May 31, 2024
  • Knowledge-Based Systems
  • Zhengqian Feng +5
  • Conference Article
  • Citations46

Malaysia traffic sign recognition with convolutional neural network

  • Jul 01, 2015
  • Mian Mian Lau +2
  • Research Article

An Adaptive Multimodal Fusion Network Based on Multilinear Gradients for Visual Question Answering

  • Dec 24, 2024
  • Electronics
  • Chengfang Zhao +3
  • Research Article
  • Citations5

Multimodal feature fusion and exploitation with dual learning and reinforcement learning for recipe generation

  • Jul 09, 2022
  • Applied Soft Computing
  • Mengyang Zhang +4
  • Conference Article

Real-Time Optimization of Multimodal Visual Feature Fusion for Environmental Perception in Autonomous Driving

  • Nov 21, 2025
  • Panbo Li
  • Research Article
  • Citations2

A multimodal transformer-based visual question answering method integrating local and global information.

  • Jul 02, 2025
  • PloS one
  • Cuiyang Huang +1
  • Dissertation

Design and implementation of end-to-end deep neural network architectures for autonomous driving

  • Jan 01, 2025
  • Leanne Rebecca Miller Hayhurst
  • Research Article
  • Citations248

Survey on Deep Neural Networks in Speech and Vision Systems

  • Jul 26, 2020
  • Neurocomputing
  • M Alam +4
  • Research Article

Causality guided co-attention network for visual question answering.

  • Dec 01, 2025
  • Neural networks : the official journal of the International Neural Network Society
  • Jiali Miao +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.