• Home
  • Search
  • An Adaptive Multimodal Fusion Network Based on Multilinear Gradients for Visual Question Answering
  • https://doi.org/10.3390/electronics14010009Copy DOI Icon

An Adaptive Multimodal Fusion Network Based on Multilinear Gradients for Visual Question Answering

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

As an interdisciplinary field of natural language processing and computer vision, Visual Question Answering (VQA) has emerged as a prominent research focus in artificial intelligence. The core of the VQA task is to combine natural language understanding and image analysis to infer answers by extracting meaningful features from textual and visual inputs. However, most current models struggle to fully capture the deep semantic relationships between images and text owing to their limited capacity to comprehend feature interactions, which constrains their performance. To address these challenges, this paper proposes an innovative Trilinear Multigranularity and Multimodal Adaptive Fusion algorithm (TriMMF) that is designed to improve the efficiency of multimodal feature extraction and fusion in VQA tasks. Specifically, the TriMMF consists of three key modules: (1) an Answer Generation Module, which generates candidate answers by extracting fused features and leveraging question features to focus on critical regions within the image; (2) a Fine-grained and Coarse-grained Interaction Module, which achieves multimodal interaction between question and image features at different granularities and incorporates implicit answer information to capture complex multimodal correlations; and (3) an Adaptive Weight Fusion Module, which selectively integrates coarse-grained and fine-grained interaction features based on task requirements, thereby enhancing the model’s robustness and generalization capability. Experimental results demonstrate that the proposed TriMMF significantly outperforms existing methods on the VQA v1.0 and VQA v2.0 datasets, achieving state-of-the-art performance in question–answer accuracy. These findings indicate that the TriMMF effectively captures the deep semantic associations between images and text. The proposed approach provides new insights into multimodal interaction and fusion research, combining domain adaptation techniques to address a broader range of cross-domain visual question answering tasks.

Similar Papers
  • Conference Article
  • Citations2558

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Jul 01, 2017
  • Yash Goyal +4
  • Book Chapter
  • Citations4

Multi-modal Feature Fusion Based on Variational Autoencoder for Visual Question Answering

  • Jan 01, 2019
  • Liqing Chen +4
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Research Article

VTFCGNet: a novel cross-modal reasoning network integrating Fourier self-attention and graph attention for visual text question answering

  • Feb 01, 2026
  • Neural Computing and Applications
  • Yujie Huo +3
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Research Article
  • Citations3

Overcoming language priors in visual question answering with cumulative learning strategy

  • Aug 22, 2024
  • Neurocomputing
  • Aihua Mao +3
  • Research Article
  • Citations38

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

  • Jan 01, 2023
  • IEEE Transactions on Multimedia
  • Aihua Mao +4
  • Research Article
  • Citations53

Visual Question Answering: A Tutorial

  • Nov 01, 2017
  • IEEE Signal Processing Magazine
  • Damien Teney +2
  • Research Article
  • Citations17

Accuracy vs. complexity: A trade-off in visual question answering models

  • Jun 12, 2021
  • Pattern Recognition
  • Moshiur Farazi +2
  • Research Article
  • Citations472

The multi-modal fusion in visual question answering: a review of attention mechanisms.

  • May 30, 2023
  • PeerJ Computer Science
  • Siyu Lu +5
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Research Article
  • Citations28

Cross-Modal Multistep Fusion Network With Co-Attention for Visual Question Answering

  • Jan 01, 2018
  • IEEE Access
  • Mingrui Lao +3
  • Conference Article
  • Citations48

Coarse-to-Fine Reasoning for Visual Question Answering

  • Jun 01, 2022
  • Binh X Nguyen +5
  • Conference Article

A Crowdsourcing Tool for Data Augmentation in Visual Question Answering Tasks

  • Oct 16, 2018
  • Ramon Silva +4
  • Research Article
  • Citations617

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

  • Apr 09, 2018
  • IEEE Transactions on Neural Networks and Learning Systems
  • Zhou Yu +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.