• Home
  • Search
  • Bridging the Gap: Leveraging Textual and Visual Contexts for PreciseMedical Visual Question Answering
  • https://doi.org/10.1145/3675888.3676107Copy DOI Icon

Bridging the Gap: Leveraging Textual and Visual Contexts for PreciseMedical Visual Question Answering

  • Aug 8, 2024
  • Pratiksh Kumar +3 more
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

The advent of Visual Question Answering (VQA) technology has brought significant advancements in the medical field, offering transformative potential in clinical diagnostics and patient care. This research explores the application of VQA within the medical domain, highlighting its critical role in interpreting complex visual data, such as radiological images, pathology slides, and other diagnostic visuals. Traditional diagnostic processes often rely heavily on human expertise, which can be time-consuming and prone to variability. VQA systems, powered by sophisticated machine learning models, provide consistent and accurate interpretations, thus enhancing diagnostic accuracy and efficiency. Visual Question Answering (VQA) in the medical field necessitates extracting information from both textual and visual inputs to provide accurate answers, a critical requirement for supporting medical decision-making. This research introduces a novel approach to address VQA challenges in the medical domain using Bi-Directional Layout with Positional Encoding (BLIP) models. Our methodology seamlessly integrates text and image processing within a unified framework, enabling precise interactions between textual queries and medical imaging data. We commence with textual inputs, encoded by BLIP processors, and medical images, encoded by BLIP image processors. A custom VQA dataset, specifically designed for the medical field, includes textual questions and their corresponding medical image features. We employ a BLIP-based Question Answering architecture, fine-tuned on our medical VQA dataset, and optimized using the AdamW optimizer with a learning rate of 0.00005, ensuring efficient convergence. Additionally, we introduce attention mechanisms using Coarse and Fine Attention blocks for enhanced feature fusion and accurate answer prediction. Our results are highly encouraging, demonstrating competitive metrics in extensive VQA task experiments on both training and validation datasets. Qualitative analysis of sample predictions indicates the model’s capability to provide accurate answers for diverse visual and textual medical inputs. This work holds significant promise for improving automated medical image analysis and supporting clinical decision-making.

Similar Papers
  • PDF
  • Research Article
  • Citations86

MedFuseNet:\xa0An\xa0attention-based multimodal deep learning model for visual question answering in the medical domain

  • Oct 06, 2021
  • Scientific Reports
  • Dhruv Sharma +2
  • Conference Article
  • Citations2

VQAR: Review on Information Retrieval Techniques based on Computer Vision and Natural Language Processing

  • Mar 01, 2019
  • Shivangi Modi +1
  • Research Article
  • Citations49

Medical knowledge-based network for Patient-oriented Visual Question Answering

  • Dec 21, 2022
  • Information Processing & Management
  • Jian Huang +7
  • Book Chapter
  • Citations2

A VQA System for Medical Image Classification Using Transfer Learning

  • Jan 01, 2021
  • C Dhanush +2
  • Research Article
  • Citations4

A comprehensive interpretation for medical VQA: Datasets, techniques, and challenges

  • Apr 03, 2023
  • Journal of Intelligent & Fuzzy Systems
  • Sheerin Sitara Noor Mohamed +1
  • Book Chapter
  • Citations49

Visual Question Answering Using Deep Learning: A Survey and Performance Analysis

  • Jan 01, 2021
  • Yash Srivastava +3
  • Book Chapter

Multi-view Visual Question Answering Dataset for Real Environment Applications

  • Jan 01, 2020
  • Yue Qiu +3
  • Research Article
  • Citations1

Counting in Visual Question Answering: Methods, Datasets, and Future Work

  • Oct 20, 2023
  • International Journal of Image and Graphics
  • Tesfayee Meshu Welde +1
  • Research Article

Incorporating Verb Semantic Information in Visual Question Answering Through Multitask Learning Paradigm

  • Jun 01, 2020
  • International Journal of Semantic Computing
  • Mehrdad Alizadeh +1
  • Research Article
  • Citations503

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Sep 11, 2018
  • International Journal of Computer Vision
  • Yash Goyal +5
  • Conference Article
  • Citations603

VizWiz Grand Challenge: Answering Visual Questions from Blind People

  • Jun 01, 2018
  • Danna Gurari +7
  • Research Article
  • Citations4

BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Minjun Kim +4
  • Dissertation
  • Citations1

A Convolutional Neural Network Based Approach For Visual Question Answering

  • Apr 01, 2018
  • Lavanya Abhinaya Koduri
  • PDF
  • Conference Article
  • Citations1

Unsupervised Keyword Extraction for Full-Sentence VQA

  • Jan 01, 2020
  • Kohei Uehara +1
  • Book Chapter
  • Citations2

A-VQA: Attention Based Visual Question Answering Technique for Handling Improper Count of Occluded Object

  • Jan 01, 2021
  • Shivangi Modi +1
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.