• Home
  • Search
  • Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT
  • Cite Icon20
  • https://doi.org/10.1016/j.artmed.2024.102889Copy DOI Icon

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

Show More
  • Abstract
  • Highlights & Summary
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

BackgroundPretraining large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has improved significantly in both the general and medical domains. However, it is difficult to train specific BERT models to perform well in domains for which few databases of a high quality and large size are publicly available. ObjectiveWe hypothesized that this problem could be addressed by oversampling a domain-specific corpus and using it for pretraining with a larger corpus in a balanced manner. In the present study, we verified our hypothesis by developing pretraining models using our method and evaluating their performance. MethodsOur proposed method was based on the simultaneous pretraining of models with knowledge from distinct domains after oversampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pretrained with complete PubMed abstracts in a balanced manner. We then compared their performance with those of conventional models. ResultsOur English BERT pretrained using both general and small medical domain corpora performed sufficiently well for practical use on the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than the conventional methods for each biomedical corpus of the same corpus size in the general domain. Our Japanese medical BERT outperformed the other BERT models built using a conventional method for almost all the medical tasks. The model demonstrated the same trend as that of the first experiment in English. Further, our enhanced biomedical BERT model, which was not pretrained on clinical notes, achieved superior clinical and biomedical scores on the BLUE benchmark with an increase of 0.3 points in the clinical score and 0.5 points in the biomedical score. These scores were above those of the models trained without our proposed method. ConclusionsWell-balanced pretraining using oversampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.

Similar Papers
  • Conference Article
  • Citations3

Engineering Document Summarization Using Sentence Representations Generated by Bidirectional Language Model

  • Aug 17, 2021
  • Yunjian Qiu +1
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Research Article

Spam Detection Based on BERT Model: A Descriptive Study

  • Jan 01, 2023
  • Theory and practice of science and technology
  • Hailin Xu
  • Research Article
  • Citations4

Artificial Intelligence for CT and MRI Protocoling: A Meta-Analysis of Traditional Machine Learning, BERT, and Large Language Models.

  • Oct 29, 2025
  • AJR. American journal of roentgenology
  • Ethan Sacoransky +2
  • Research Article
  • Citations1

Natural language processing models for predicting treatment outcomes in internet-delivered cognitive behavioral therapy

  • Oct 10, 2025
  • Internet Interventions
  • Nils Hentati Isacsson +4
  • PDF
  • Research Article
  • Citations25

Identification of asthma control factor in clinical notes using a hybrid deep learning model

  • Nov 01, 2021
  • BMC Medical Informatics and Decision Making
  • Bhavani Singh Agnikula Kshatriya +6
  • Research Article

Cybershield Monitor Using Deep Learning and BERT Model

  • Feb 24, 2025
  • INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
  • A Nandhini
  • Research Article
  • Citations9

Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation.

  • Oct 30, 2024
  • JMIR medical informatics
  • Kyungmo Kim +11
  • Book Chapter
  • Citations35

5 - Utilizing BERT for biomedical and clinical text mining

  • Oct 23, 2020
  • Data Analytics in Biomedical Engineering and Healthcare
  • Runjie Zhu +2
  • Research Article
  • Citations30

Semantic Textual Similarity in Japanese Clinical Domain Texts Using BERT.

  • Jun 01, 2021
  • Methods of Information in Medicine
  • Faith Wavinya Mutinda +3
  • Research Article
  • Citations64

Contextual semantic embeddings based on fine-tuned AraBERT model for Arabic text multi-class categorization

  • Feb 18, 2021
  • Journal of King Saud University - Computer and Information Sciences
  • Fatima-Zahra El-Alami +2
  • Conference Article
  • Citations7

Intelligent BERT-BiLSTM-CRF Based Legal Case Entity Recognition Method

  • Jul 30, 2021
  • Mingdong Sun +2
  • Research Article
  • Citations9

An Analysis of BERT (NLP) for Assisted Subject Indexing for Project Gutenberg

  • Oct 21, 2022
  • Cataloging & Classification Quarterly
  • Charlene Chou +1
  • PDF
  • Research Article

Automatic Short Answer Grading for Enhancing Educational Assessment Using BERT and USE Models

  • Feb 22, 2026
  • Indian Journal Of Science And Technology
  • Chandralika Chakraborty +2
  • Research Article

T-­BERT:臺灣語言模型–以臺灣在地語言預訓練BERT模型

  • Jan 01, 2020
  • 鍾明諺
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.