• Home
  • Search
  • Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study
  • Cite Icon5
  • https://doi.org/10.2196/56627Copy DOI Icon

Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study

Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

BackgroundMedical image analysis, particularly in the context of visual question answering (VQA) and image captioning, is crucial for accurate diagnosis and educational purposes.ObjectiveOur study aims to introduce BioMedBLIP models, fine-tuned for VQA tasks using specialized medical data sets such as Radiology Objects in Context and Medical Information Mart for Intensive Care-Chest X-ray, and evaluate their performance in comparison to the state of the art (SOTA) original Bootstrapping Language-Image Pretraining (BLIP) model.MethodsWe present 9 versions of BioMedBLIP across 3 downstream tasks in various data sets. The models are trained on a varying number of epochs. The findings indicate the strong overall performance of our models. We proposed BioMedBLIP for the VQA generation model, VQA classification model, and BioMedBLIP image caption model. We conducted pretraining in BLIP using medical data sets, producing an adapted BLIP model tailored for medical applications.ResultsIn VQA generation tasks, BioMedBLIP models outperformed the SOTA on the Semantically-Labeled Knowledge-Enhanced (SLAKE) data set, VQA in Radiology (VQA-RAD), and Image Cross-Language Evaluation Forum data sets. In VQA classification, our models consistently surpassed the SOTA on the SLAKE data set. Our models also showed competitive performance on the VQA-RAD and PathVQA data sets. Similarly, in image captioning tasks, our model beat the SOTA, suggesting the importance of pretraining with medical data sets. Overall, in 20 different data sets and task combinations, our BioMedBLIP excelled in 15 (75%) out of 20 tasks. BioMedBLIP represents a new SOTA in 15 (75%) out of 20 tasks, and our responses were rated higher in all 20 tasks (P<.005) in comparison to SOTA models.ConclusionsOur BioMedBLIP models show promising performance and suggest that incorporating medical knowledge through pretraining with domain-specific medical data sets helps models achieve higher performance. Our models thus demonstrate their potential to advance medical image analysis, impacting diagnosis, medical education, and research. However, data quality, task-specific variability, computational resources, and ethical considerations should be carefully addressed. In conclusion, our models represent a contribution toward the synergy of artificial intelligence and medicine. We have made BioMedBLIP freely available, which will help in further advancing research in multimodal medical tasks.

Loading PDF

Similar Papers
  • Research Article
  • Citations447

Image Captioning and Visual Question Answering Based on Attributes and External Knowledge.

  • May 26, 2017
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Qi Wu +4
  • Research Article
  • Citations112

Federated Learning for Vision-and-Language Grounding Problems

  • Apr 03, 2020
  • Proceedings of the AAAI Conference on Artificial Intelligence
  • Fenglin Liu +4
  • Research Article
  • Citations12

RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering

  • Jan 01, 2024
  • IEEE Transactions on Geoscience and Remote Sensing
  • Yuduo Wang +1
  • Research Article
  • Citations11

Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering

  • Nov 12, 2024
  • Journal of Biomedical Informatics
  • Gang Liu +4
  • Conference Article

Bridging the Gap: Leveraging Textual and Visual Contexts for PreciseMedical Visual Question Answering

  • Aug 08, 2024
  • Pratiksh Kumar +3
  • Research Article
  • Citations4

A comprehensive interpretation for medical VQA: Datasets, techniques, and challenges

  • Apr 03, 2023
  • Journal of Intelligent &amp; Fuzzy Systems
  • Sheerin Sitara Noor Mohamed +1
  • Video Transcripts

All You May Need for VQA are Image Captions

  • Jun 29, 2022
  • Underline Science Inc.
  • Idan Szpektor +2
  • Research Article
  • Citations17

Inner Knowledge-based Img2Doc Scheme for Visual Question Answering

  • Mar 04, 2022
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Qun Li +4
  • Conference Article
  • Citations388

In Defense of Grid Features for Visual Question Answering

  • Jun 01, 2020
  • Huaizu Jiang +4
  • Research Article
  • Citations53

Visual question answering: Which investigated applications?

  • Nov 01, 2021
  • Pattern Recognition Letters
  • Silvio Barra +3
  • Research Article

English Visual Question Answering: Building a Culturally Relevant Dataset from Image Captions

  • Mar 15, 2026
  • International Journal of Scientific Research in Engineering and Management
  • Dr Md Sirajul Huque +3
  • Research Article
  • Citations18

RingMoGPT: A Unified Remote Sensing Foundation Model for Vision, Language, and Grounded Tasks

  • Jan 01, 2025
  • IEEE Transactions on Geoscience and Remote Sensing
  • Peijin Wang +10
  • Research Article
  • Citations35

Interpretable medical image Visual Question Answering via multi-modal relationship graph learning

  • Jul 20, 2024
  • Medical Image Analysis
  • Xinyue Hu +7
  • Research Article

MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry.

  • Mar 15, 2026
  • Journal of dental research
  • M-X Li +8
  • Research Article

Qualitative content analysis in visual question answering-based datasets and algorithms

  • Jan 01, 2020
  • International Journal of High Performance Computing and Networking
  • Prateek Ralhan +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.