• Home
  • Search
  • Visual question answering model for fruit tree disease decision-making based on multimodal deep learning
  • Cite Icon20
  • https://doi.org/10.3389/fpls.2022.1064399Copy DOI Icon

Visual question answering model for fruit tree disease decision-making based on multimodal deep learning

Show More
  • Abstract
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Visual Question Answering (VQA) about diseases is an essential feature of intelligent management in smart agriculture. Currently, research on fruit tree diseases using deep learning mainly uses single-source data information, such as visible images or spectral data, yielding classification and identification results that cannot be directly used in practical agricultural decision-making. In this study, a VQA model for fruit tree diseases based on multimodal feature fusion was designed. Fusing images and Q&A knowledge of disease management, the model obtains the decision-making answer by querying questions about fruit tree disease images to find relevant disease image regions. The main contributions of this study were as follows: (1) a multimodal bilinear factorized pooling model using Tucker decomposition was proposed to fuse the image features with question features: (2) a deep modular co-attention architecture was explored to simultaneously learn the image and question attention to obtain richer graphical features and interactivity. The experiments showed that the proposed unified model combining the bilinear model and co-attentive learning in a new network architecture obtained 86.36% accuracy in decision-making under the condition of limited data (8,450 images and 4,560k Q&A pairs of data), outperforming existing multimodal methods. The data augmentation is adopted on the training set to avoid overfitting. Ten runs of 10-fold cross-validation are used to report the unbiased performance. The proposed multimodal fusion model achieved friendly interaction and fine-grained identification and decision-making performance. Thus, the model can be widely deployed in intelligent agriculture.

Loading PDF

Similar Papers
  • Book Chapter
  • Citations4

Multi-modal Feature Fusion Based on Variational Autoencoder for Visual Question Answering

  • Jan 01, 2019
  • Liqing Chen +4
  • Conference Article
  • Citations59

Visual Question Answering as Reading Comprehension

  • Jun 01, 2019
  • Hui Li +3
  • Research Article
  • Citations617

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

  • Apr 09, 2018
  • IEEE Transactions on Neural Networks and Learning Systems
  • Zhou Yu +4
  • Research Article
  • Citations5

Multimodal feature fusion and exploitation with dual learning and reinforcement learning for recipe generation

  • Jul 09, 2022
  • Applied Soft Computing
  • Mengyang Zhang +4
  • Research Article
  • Citations71

Multimodal feature fusion by relational reasoning and attention for visual question answering

  • Aug 19, 2019
  • Information Fusion
  • Weifeng Zhang +4
  • Research Article

An Adaptive Multimodal Fusion Network Based on Multilinear Gradients for Visual Question Answering

  • Dec 24, 2024
  • Electronics
  • Chengfang Zhao +3
  • Research Article
  • Citations1

An Enhanced Term Weighted Question Embedding for Visual Question Answering

  • May 05, 2022
  • Journal of Information & Knowledge Management
  • Sruthy Manmadhan +1
  • Research Article
  • Citations1

Multimodal deep learning for breast tumor classification: Integrating mammography and ultrasound for enhanced diagnostic accuracy

  • Jan 01, 2026
  • Journal of Applied Clinical Medical Physics
  • Yu Yan +5
  • Research Article
  • Citations59

Analysis of multimodal data fusion from an information theory perspective

  • Dec 09, 2022
  • Information Sciences
  • Yinglong Dai +4
  • Research Article
  • Citations27

MFUR-Net: Multimodal feature fusion and unimodal feature refinement for RGB-D salient object detection

  • May 31, 2024
  • Knowledge-Based Systems
  • Zhengqian Feng +5
  • Research Article
  • Citations3

Bilinear pooling in video-QA: empirical challenges and motivational drift from neurological parallels

  • Jun 03, 2022
  • PeerJ Computer Science
  • Thomas Winterbottom +3
  • Research Article
  • Citations245

BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection

  • Jul 17, 2019
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Hedi Ben-Younes +3
  • Research Article
  • Citations4

A short video sentiment analysis model based on multimodal feature fusion

  • Sep 10, 2024
  • Systems and Soft Computing
  • Hongyu Shi
  • Research Article

Causality guided co-attention network for visual question answering.

  • Dec 01, 2025
  • Neural networks : the official journal of the International Neural Network Society
  • Jiali Miao +4
  • Research Article
  • Citations70

DMRFNet: Deep Multimodal Reasoning and Fusion for Visual Question Answering and explanation generation

  • Feb 12, 2021
  • Information Fusion
  • Weifeng Zhang +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.