• Home
  • Search
  • Fine-tuning multimodal large language models for medical visual question answering
  • https://doi.org/10.62791/20485Copy DOI Icon

Fine-tuning multimodal large language models for medical visual question answering

  • Jan 1, 2025
  • Ankitaben Mungalpara
Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Multimodal large language models (MLLMs), which can reason across different modalities, are opening up new frontiers in AI development that will soon enable complex reasoning with both visual and textual modalities. One application of multimodal MLLMs is Visual Question Answering (VQA) in healthcare; for example, a clinician can ask questions about medical images in natural language to receive detailed explanations. As suitably adapted for the modern age: "The art of medicine consists in amusing the patient while nature cures the disease - but in the age of AI, assisting the doctor while learning from the data." - Inspired by Voltaire, adapted for modern AI healthcare This research examines the efficacy of fine-tuning multimodal (vision and language) foundational models for performing medical visual question answering. This study aims to evaluate how effectively these models can interpret and respond to medical queries based on visual inputs, ultimately enhancing diagnostic accuracy and patient care. By leveraging the strengths of both vision and language processing, the research seeks to advance the capabilities of AI in medical settings. The author studies three fine-tuned variants derived from the baseline LLaVA-Med architecture: a caption-only baseline trained to generate global descriptions of medical images; a model that uses instruction tuning to better answer diagnostic and region-specific questions, and a variant that incorporates Alpha-CLIP with LLaVA-Med to allow spatially targeted understanding by conditioning responses on user-defined regions of interest (ROIs). All variants are adapted utilizing Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning (PEFT) technique that facilitates minimal computational overhead. The results show that the foundational LLaVA-Med model, which was trained only on image-caption pairs, performs fairly well with BLEU (0.09) and ROUGE-2 (0.14), but it struggles to answer specific questions compared to the improved version that has the best ROUGE-1 (0.59) and ROUGE-L (0.54). The region-focused Alpha-CLIP combined with LLaVA-Med achieved the best performance on both the ROUGE-2 (0.40) and BLEU (0.28), while still exhibiting accurate and context-sensitive reasoning, which points to the importance of task-specific adaptation in medical VQA. The combined results suggest that concentrating attention on specific areas within images, along with instruction-based reasoning, is essential for medical AI systems that manage diverse types of data. This approach facilitates the comparison of methods and provides a detailed strategy for improving specialized diagnostic medical AI assistants.

Similar Papers
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Research Article
  • Citations63

AMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering

  • Aug 27, 2022
  • Knowledge-Based Systems
  • Haiwei Pan +5
  • Research Article
  • Citations1

Caption-augmented reasoning model with Hierarchical rank LoRA finetuing for medical visual question Answering.

  • Dec 01, 2025
  • Journal of biomedical informatics
  • Yong Li +3
  • Research Article
  • Citations8

Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Xiaoshuang Huang +6
  • Research Article

MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models.

  • Apr 01, 2025
  • ... International Conference on Learning Representations
  • Mohammad Shahab Sepehri +3
  • Research Article
  • Citations2

Unveiling the power of multimodal large language models for radio astronomical image understanding and question answering

  • Oct 10, 2025
  • Machine Learning: Science and Technology
  • Fuyong Zhao +7
  • Research Article
  • Citations8

Parallel multi-head attention and term-weighted question embedding for medical visual question answering.

  • Mar 11, 2023
  • Multimedia Tools and Applications
  • Sruthy Manmadhan +1
  • Research Article

DAMON: Difference-Aware Medical Visual Question Answering via Multimodal Large Language Model.

  • Aug 01, 2026
  • IEEE journal of biomedical and health informatics
  • Zefan Zhang +3
  • Research Article
  • Citations1

Advancements and Challenges of Multimodal Models in Medical Applications

  • Feb 27, 2025
  • Applied and Computational Engineering
  • Zimeng He
  • Conference Article

Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts

  • Jan 01, 2025
  • Michal Golovanevsky +5
  • Research Article
  • Citations3

Uncertainty-Aware Medical Diagnostic Phrase Identification and Grounding.

  • Dec 01, 2025
  • IEEE transactions on pattern analysis and machine intelligence
  • Ke Zou +11
  • Research Article
  • Citations4

Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion

  • Apr 24, 2025
  • Applied Sciences
  • Junkai Zhang +2
  • Conference Article
  • Citations24

Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

  • Jan 01, 2024
  • Ankit Pal +1
  • Research Article
  • Citations41

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

  • Mar 05, 2025
  • ACM Computing Surveys
  • Jiayi Kuang +9
  • Research Article

A Global Visual Information Intervention Model for Medical Visual Question Answering.

  • Jun 01, 2025
  • Computers in biology and medicine
  • Peixi Peng +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.