• Home
  • Search
  • Adapting transformer-based language models for heart disease detection and risk factors extraction
  • Cite Icon26
  • https://doi.org/10.1186/s40537-024-00903-yCopy DOI Icon

Adapting transformer-based language models for heart disease detection and risk factors extraction

Show More
  • Abstract
  • Highlights & Summary
  • PDF
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Efficiently treating cardiac patients before the onset of a heart attack relies on the precise prediction of heart disease. Identifying and detecting the risk factors for heart disease such as diabetes mellitus, Coronary Artery Disease (CAD), hyperlipidemia, hypertension, smoking, familial CAD history, obesity, and medications is critical for developing effective preventative and management measures. Although Electronic Health Records (EHRs) have emerged as valuable resources for identifying these risk factors, their unstructured format poses challenges for cardiologists in retrieving relevant information. This research proposed employing transfer learning techniques to automatically extract heart disease risk factors from EHRs. Leveraging transfer learning, a deep learning technique has demonstrated a significant performance in various clinical natural language processing (NLP) applications, particularly in heart disease risk prediction. This study explored the application of transformer-based language models, specifically utilizing pre-trained architectures like BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, BioClinicalBERT, XLNet, and BioBERT for heart disease detection and extraction of related risk factors from clinical notes, using the i2b2 dataset. These transformer models are pre-trained on an extensive corpus of medical literature and clinical records to gain a deep understanding of contextualized language representations. Adapted models are then fine-tuned using annotated datasets specific to heart disease, such as the i2b2 dataset, enabling them to learn patterns and relationships within the domain. These models have demonstrated superior performance in extracting semantic information from EHRs, automating high-performance heart disease risk factor identification, and performing downstream NLP tasks within the clinical domain. This study proposed fine-tuned five widely used transformer-based models, namely BERT, RoBERTa, BioClinicalBERT, XLNet, and BioBERT, using the 2014 i2b2 clinical NLP challenge dataset. The fine-tuned models surpass conventional approaches in predicting the presence of heart disease risk factors with impressive accuracy. The RoBERTa model has achieved the highest performance, with micro F1-scores of 94.27%, while the BERT, BioClinicalBERT, XLNet, and BioBERT models have provided competitive performances with micro F1-scores of 93.73%, 94.03%, 93.97%, and 93.99%, respectively. Finally, a simple ensemble of the five transformer-based models has been proposed, which outperformed the most existing methods in heart disease risk fan, achieving a micro F1-Score of 94.26%. This study demonstrated the efficacy of transfer learning using transformer-based models in enhancing risk prediction and facilitating early intervention for heart disease prevention.

Loading PDF

Similar Papers
  • Research Article
  • Citations20

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

  • May 05, 2024
  • Artificial Intelligence In Medicine
  • Shoya Wada +6
  • Research Article
  • Citations29

Critical assessment of transformer-based AI models for German clinical notes

  • Oct 04, 2022
  • JAMIA Open
  • Manuel Lentzen +11
  • Research Article
  • Citations1

Transformer-based models for ADR detection: cross-drug validation and benchmarking against large language models

  • Dec 18, 2025
  • Therapeutic Advances in Drug Safety
  • Minjung Kim +5
  • PDF
  • Research Article
  • Citations18

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

  • Mar 02, 2024
  • Информатика. Экономика. Управление - Informatics. Economics. Management
  • Rajesh Gupta
  • Research Article

Performance Analysis of Transformer Based Models for Automatic Short Answer Grading

  • May 30, 2025
  • Journal of Information Systems Engineering and Management
  • Rupal Chaudhari
  • Research Article

Performance Analysis of Transformer Based Models for Automatic Short Answer Grading

  • May 30, 2025
  • Journal of Information Systems Engineering and Management
  • Rupal Chaudhari
  • Research Article

Analisis Sentimen Ulasan iPhone di Amazon Menggunakan Model Deep Learning BERT Berbasis Transformer

  • Dec 22, 2024
  • Elkom: Jurnal Elektronika dan Komputer
  • Arif Fitra Setyawan +3
  • PDF
  • Research Article
  • Citations61

Measurement of Semantic Textual Similarity in Clinical Texts: Comparison of Transformer-Based Models.

  • Nov 23, 2020
  • JMIR Medical Informatics
  • Xi Yang +5
  • PDF
  • Research Article
  • Citations10

Comparative Study of Multiclass Text Classification in Research Proposals Using Pretrained Language Models

  • Apr 29, 2022
  • Applied Sciences
  • Eunchan Lee +2
  • PDF
  • Research Article
  • Citations18

A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study.

  • Oct 04, 2023
  • Journal of Medical Internet Research
  • Maarten Homburg +10
  • Conference Article
  • Citations111

How Does BERT Answer Questions?

  • Nov 03, 2019
  • Betty Van Aken +3
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Conference Article
  • Citations3

Engineering Document Summarization Using Sentence Representations Generated by Bidirectional Language Model

  • Aug 17, 2021
  • Yunjian Qiu +1
  • PDF
  • Research Article
  • Citations25

Identification of asthma control factor in clinical notes using a hybrid deep learning model

  • Nov 01, 2021
  • BMC Medical Informatics and Decision Making
  • Bhavani Singh Agnikula Kshatriya +6
  • Research Article
  • Citations2

CACER: Clinical concept Annotations for Cancer Events and Relations.

  • Sep 03, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Yujuan Velvin Fu +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.