• Home
  • Search
  • Protected Health Information Recognition by Fine-Tuning a Pre-training Transformer Model
  • Cite Icon17
  • https://doi.org/10.4258/hir.2022.28.1.16Copy DOI Icon

Protected Health Information Recognition by Fine-Tuning a Pre-training Transformer Model

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

ObjectivesDe-identifying protected health information (PHI) in medical documents is important, and a prerequisite to de-identification is the identification of PHI entity names in clinical documents. This study aimed to compare the performance of three pre-training models that have recently attracted significant attention and to determine which model is more suitable for PHI recognition.MethodsWe compared the PHI recognition performance of deep learning models using the i2b2 2014 dataset. We used the three pre-training models—namely, bidirectional encoder representations from transformers (BERT), robustly optimized BERT pre-training approach (RoBERTa), and XLNet (model built based on Transformer-XL)—to detect PHI. After the dataset was tokenized, it was processed using an inside-outside-beginning tagging scheme and WordPiece-tokenized to place it into these models. Further, the PHI recognition performance was investigated using BERT, RoBERTa, and XLNet.ResultsComparing the PHI recognition performance of the three models, it was confirmed that XLNet had a superior F1-score of 96.29%. In addition, when checking PHI entity performance evaluation, RoBERTa and XLNet showed a 30% improvement in performance compared to BERT.ConclusionsAmong the pre-training models used in this study, XLNet exhibited superior performance because word embedding was well constructed using the two-stream self-attention method. In addition, compared to BERT, RoBERTa and XLNet showed superior performance, indicating that they were more effective in grasping the context.

Similar Papers
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Conference Article
  • Citations10

Classification of Fire Related Tweets on Twitter Using Bidirectional Encoder Representations from Transformers (BERT)

  • Nov 28, 2021
  • Jairus Mingua +2
  • PDF
  • Research Article
  • Citations18

Chinese Named Entity Recognition Method in History and Culture Field Based on BERT

  • Sep 22, 2021
  • International Journal of Computational Intelligence Systems
  • Shuang Liu +3
  • Research Article
  • Citations13

Sentiment analysis of Malayalam tweets using bidirectional encoder representations from transformers: a study

  • Mar 01, 2023
  • Indonesian Journal of Electrical Engineering and Computer Science
  • Syam Mohan Elankath +1
  • Book Chapter
  • Citations24

A Literature Review on Bidirectional Encoder Representations from Transformers

  • Jan 01, 2022
  • S Shreyashree +3
  • Book Chapter
  • Citations17

Document Classification by Word Embeddings of BERT

  • Jan 01, 2020
  • Hirotaka Tanaka +4
  • PDF
  • Research Article
  • Citations34

MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain

  • Jun 28, 2021
  • Applied Sciences
  • Muzamil Hussain Syed +1
  • Research Article
  • Citations1

Building a Korean morphological analyzer using two Korean BERT models.

  • May 02, 2022
  • PeerJ Computer Science
  • Yong-Seok Choi +2
  • PDF
  • Research Article
  • Citations3

Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method

  • Apr 02, 2024
  • Applied Sciences
  • Fenfang Li +3
  • PDF
  • Research Article
  • Citations18

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

  • Mar 02, 2024
  • Информатика. Экономика. Управление - Informatics. Economics. Management
  • Rajesh Gupta
  • Conference Article
  • Citations5

Construction of document feature vectors using BERT

  • Dec 01, 2020
  • Hirotaka Tanaka +4
  • Research Article
  • Citations5

Layer-Wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-Trained Models: An Evolutionary Approach

  • Nov 25, 2024
  • ACM Transactions on Evolutionary Learning and Optimization
  • Chenyang Bu +6
  • PDF
  • Research Article
  • Citations33

An ERNIE-Based Joint Model for Chinese Named Entity Recognition

  • Aug 18, 2020
  • Applied Sciences
  • Yu Wang +4
  • Conference Article

Domain-Specific Japanese ELECTRA Model Using a Small Corpus

  • Jan 01, 2021
  • Youki Itoh +1
  • Research Article
  • Citations1

Improving Systematic Review Updates With Natural Language Processing Through Abstract Component Classification and Selection: Algorithm Development and Validation.

  • Aug 13, 2024
  • JMIR medical informatics
  • Tatsuki Hasegawa +7
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.