• Home
  • Search
  • Medical Vision-Language Modeling With Semantic Interaction and Adaptive Refinement Prompting for Bias Mitigation.
  • https://doi.org/10.1109/jbhi.2025.3631270Copy DOI Icon

Medical Vision-Language Modeling With Semantic Interaction and Adaptive Refinement Prompting for Bias Mitigation.

Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities across various medical tasks, including report generation and visual question answering (VQA). However, pixel-level tasks such as image segmentation remain relatively underexplored, despite their critical importance for clinical decision-making, surgical planning, and model interpretability. Moreover, the scarcity of high-quality segmentation annotations in the medical domain often leads to biased data distributions, characterized by imbalances in disease types, anatomical coverage, and image quality. These biases are frequently overlooked during both model development and evaluation, limiting the robustness and real-world applicability of VLMs in healthcare scenarios. In this study, we propose a unified medical vision-language model applicable for a variety of clinical tasks, including report generation, VQA, and pixel-level image segmentation. Within the model, we propose a semantic interaction mechanism aimed at enhancing pixel-level vision and language representation learning. To mitigate the impact of biased data distributions, we explicitly develop an adaptive refinement prompting method involving the iterative re-prompting of hard samples. The proposed method is thoroughly validated through experiments on eight datasets and comparisons with nine state-of-the-art methods. The experimental results indicate that our model achieves superior performance in both medical VQA and segmentation tasks. These results highlight the potential of our approach in advancing the deployment of medical VLMs in real-world clinical applications. Code will be released at: https://github.com/SZUHvern/Unified-Medical-Vision-Language-Modeling.

Similar Papers
  • Research Article
  • Citations30

Improving Automatic VQA Evaluation Using Large Language Models

  • Mar 24, 2024
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Oscar Mañas +2
  • Book Chapter
  • Citations5

Neural Networks for Detecting Irrelevant Questions During Visual Question Answering

  • Jan 01, 2020
  • Mengdi Li +2
  • Research Article
  • Citations53

Visual Question Answering: A Tutorial

  • Nov 01, 2017
  • IEEE Signal Processing Magazine
  • Damien Teney +2
  • Conference Article
  • Citations48

Coarse-to-Fine Reasoning for Visual Question Answering

  • Jun 01, 2022
  • Binh X Nguyen +5
  • Book Chapter

Multi-view Visual Question Answering Dataset for Real Environment Applications

  • Jan 01, 2020
  • Yue Qiu +3
  • Research Article
  • Citations15

MobiVQA

  • Jul 04, 2022
  • Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
  • Qingqing Cao +3
  • Book Chapter
  • Citations3

Optimal Image Feature Ranking and Fusion for Visual Question Answering

  • Sep 09, 2020
  • Sruthy Manmadhan +1
  • Conference Article

Bridging the Gap: Leveraging Textual and Visual Contexts for PreciseMedical Visual Question Answering

  • Aug 08, 2024
  • Pratiksh Kumar +3
  • Research Article
  • Citations3

Enhancing visual question answering with a two‐way co‐attention mechanism and integrated multimodal features

  • Dec 21, 2023
  • Computational Intelligence
  • Mayank Agrawal +2
  • Conference Article
  • Citations2558

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

  • Jul 01, 2017
  • Yash Goyal +4
  • Research Article
  • Citations6

Explanation vs. attention: A two-player game to obtain attention for VQA and visual dialog

  • Jul 23, 2022
  • Pattern Recognition
  • Badri N Patro +2
  • Research Article
  • Citations29

Explanation vs Attention: A Two-Player Game to Obtain Attention for VQA

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Badri Patro +2
  • Research Article
  • Citations2

Hierarchical Attention Networks for Multimodal Machine Learning

  • Mar 01, 2022
  • Journal of Physics Conference Series
  • Haotian Liang +1
  • Research Article
  • Citations27

Toward Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline

  • Jun 01, 2023
  • IEEE Transactions on Circuits and Systems for Video Technology
  • Lichen Zhao +8
  • Research Article

NeSyVQA: Neurosymbolic Visual Question Answering with Knowledge-Enriched Scene Graphs

  • Jan 01, 2025
  • IEEE Access
  • M Jaleed Khan +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.