• Home
  • Search
  • MLeVLM: Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering
  • Cite Icon9
  • https://doi.org/10.18653/v1/2024.findings-acl.296Copy DOI Icon

MLeVLM: Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering

  • Jan 1, 2024
  • Dexuan Xu +10 more
Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

Medical visual question answering (MVQA) requires in-depth understanding of medical images and questions to provide reliable answers.We summarize multi-level progressive capabilities that models need to focus on in MVQA: recognition, details, diagnosis, knowledge, and reasoning.Existing MVQA models tend to ignore the above capabilities due to unspecific data and plain architecture.To address these issues, this paper proposes Multi-level Visual Language Model (MLeVLM 1 ) for MVQA.On the data side, we construct a high-quality multi-level instruction dataset MLe-VQA via GPT-4, which covers multi-level questions and answers as well as reasoning processes from visual clues to semantic cognition.On the architecture side, we propose a multi-level feature alignment module, including attentionbased token selector and context merger, which can efficiently align features at different levels from visual to semantic.To better evaluate the model's capabilities, we manually construct a multi-level MVQA evaluation benchmark named MLe-Bench.Extensive experiments demonstrate the effectiveness of our constructed multi-level instruction dataset and the multi-level feature alignment module.It also proves that MLeVLM outperforms existing medical multimodal large language models.

Similar Papers
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Dissertation

Fine-tuning multimodal large language models for medical visual question answering

  • Jan 01, 2025
  • Ankitaben Mungalpara
  • Research Article

DAMON: Difference-Aware Medical Visual Question Answering via Multimodal Large Language Model.

  • Aug 01, 2026
  • IEEE journal of biomedical and health informatics
  • Zefan Zhang +3
  • Book Chapter
  • Citations6

A Pre-trained Language Model for Medical Question Answering Based on Domain Adaption

  • Jan 01, 2022
  • Lang Liu +5
  • Research Article
  • Citations63

AMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering

  • Aug 27, 2022
  • Knowledge-Based Systems
  • Haiwei Pan +5
  • Conference Article
  • Citations8

Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

  • Jan 01, 2024
  • Cuong Ha +5
  • Research Article
  • Citations1

Caption-augmented reasoning model with Hierarchical rank LoRA finetuing for medical visual question Answering.

  • Dec 01, 2025
  • Journal of biomedical informatics
  • Yong Li +3
  • Research Article
  • Citations8

Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine

  • Apr 11, 2025
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Xiaoshuang Huang +6
  • Research Article
  • Citations17

Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework.

  • Dec 27, 2024
  • Journal of medical systems
  • Ying Chang +5
  • Research Article

Medical question answering: A comprehensive multimodal and LLM-driven survey.

  • Oct 01, 2026
  • Computer methods and programs in biomedicine
  • Eya Mhedhbi +4
  • Research Article

MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models.

  • Apr 01, 2025
  • ... International Conference on Learning Representations
  • Mohammad Shahab Sepehri +3
  • Research Article
  • Citations24

Generative Models in Medical Visual Question Answering: A Survey

  • Mar 10, 2025
  • Applied Sciences
  • Wenjie Dong +5
  • Research Article

Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework

  • May 07, 2025
  • Mathematics
  • Yusong Ke +4
  • Research Article

Optimizing multimodal models for medical visual question answering: A comparative study of LoRA and AdaLoRA on VQA-RAD and SLAKE-VQA.

  • Jan 01, 2026
  • Computers in biology and medicine
  • Zahra Rezaei +2
  • Research Article
  • Citations58

Leveraging long context in retrieval augmented language models for medical question answering

  • May 02, 2025
  • npj Digital Medicine
  • Gongbo Zhang +10
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.