• Home
  • Search
  • Does BERT need domain adaptation for clinical negation detection?
  • Cite Icon62
  • https://doi.org/10.1093/jamia/ocaa001Copy DOI Icon

Does BERT need domain adaptation for clinical negation detection?

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Classifying whether concepts in an unstructured clinical text are negated is an important unsolved task. New domain adaptation and transfer learning methods can potentially address this issue. We examine neural unsupervised domain adaptation methods, introducing a novel combination of domain adaptation with transformer-based transfer learning methods to improve negation detection. We also want to better understand the interaction between the widely used bidirectional encoder representations from transformers (BERT) system and domain adaptation methods. We use 4 clinical text datasets that are annotated with negation status. We evaluate a neural unsupervised domain adaptation algorithm and BERT, a transformer-based model that is pretrained on massive general text datasets. We develop an extension to BERT that uses domain adversarial training, a neural domain adaptation method that adds an objective to the negation task, that the classifier should not be able to distinguish between instances from 2 different domains. The domain adaptation methods we describe show positive results, but, on average, the best performance is obtained by plain BERT (without the extension). We provide evidence that the gains from BERT are likely not additive with the gains from domain adaptation. Our results suggest that, at least for the task of clinical negation detection, BERT subsumes domain adaptation, implying that BERT is already learning very general representations of negation phenomena such that fine-tuning even on a specific corpus does not lead to much overfitting. Despite being trained on nonclinical text, the large training sets of models like BERT lead to large gains in performance for the clinical negation detection task.

Similar Papers
  • PDF
  • Research Article
  • Citations18

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

  • Mar 02, 2024
  • Информатика. Экономика. Управление - Informatics. Economics. Management
  • Rajesh Gupta
  • Research Article
  • Citations20

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

  • May 05, 2024
  • Artificial Intelligence In Medicine
  • Shoya Wada +6
  • Research Article

METAHEURISTIC-DRIVEN HYPERPARAMETER OPTIMIZATION FOR BERT IN SENTIMENT ANALYSIS

  • Oct 30, 2025
  • Archives for Technical Sciences
  • Alaa A El-Demerdash +1
  • Research Article
  • Citations8

GRU-BERT for NILM: A Hybrid Deep Learning Architecture for Load Disaggregation

  • Sep 22, 2025
  • AI
  • Annysha Huzzat +5
  • PDF
  • Research Article
  • Citations26

Adapting transformer-based language models for heart disease detection and risk factors extraction

  • Apr 04, 2024
  • Journal of Big Data
  • Essam H Houssein +3
  • Research Article
  • Citations39

A contextual multi-task neural approach to medication and adverse events identification from clinical text

  • Dec 04, 2021
  • Journal of Biomedical Informatics
  • Sankaran Narayanan +5
  • PDF
  • Research Article
  • Citations28

Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT)

  • Jul 30, 2022
  • BMC Medical Informatics and Decision Making
  • Jia Li +10
  • Book Chapter
  • Citations24

A Literature Review on Bidirectional Encoder Representations from Transformers

  • Jan 01, 2022
  • S Shreyashree +3
  • Book Chapter
  • Citations8

Natural Language Processing with “More Than Words – BERT”

  • Jan 01, 2021
  • Saranlita Chotirat +1
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Research Article
  • Citations30

Semantic Textual Similarity in Japanese Clinical Domain Texts Using BERT.

  • Jun 01, 2021
  • Methods of Information in Medicine
  • Faith Wavinya Mutinda +3
  • PDF
  • Research Article
  • Citations34

MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain

  • Jun 28, 2021
  • Applied Sciences
  • Muzamil Hussain Syed +1
  • Research Article
  • Citations1

Analisis Perbandingan Model Bert Dan Xlnet Untuk Klasifikasi Tweet Bully Pada Twitter

  • Dec 10, 2024
  • Jurnal Teknologi Informasi dan Ilmu Komputer
  • Teuku Radillah +2
  • Conference Article
  • Citations10

Classification of Fire Related Tweets on Twitter Using Bidirectional Encoder Representations from Transformers (BERT)

  • Nov 28, 2021
  • Jairus Mingua +2
  • Conference Article
  • Citations111

How Does BERT Answer Questions?

  • Nov 03, 2019
  • Betty Van Aken +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.