• Home
  • Search
  • A Literature Review on Bidirectional Encoder Representations from Transformers
  • Cite Icon24
  • https://doi.org/10.1007/978-981-16-6723-7_23Copy DOI Icon

A Literature Review on Bidirectional Encoder Representations from Transformers

  • Jan 1, 2022
  • S Shreyashree +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Abstract Transfer learning is a technique of training a model for a specific problem and using it as a base for training another related problem. It has been proved to be very effective and has two phases: the pre-training phase (generation of pre-trained models) and the adaptive phase (reuse of pre-trained models). Auto-encoding pre-trained learning model is one type of pre-trained model, which uses the transformer model’s encoder component to perform natural language understanding. This work discusses the bidirectional encoder representations from transformers (BERT) and its variants and relative performances. BERTs are transformer-based models developed for pre-training unlabeled texts, bidirectional, by considering the semantics of texts from both sides of the word being processed. The model implements the above function using two specific functions: masked language modeling (MLM) and next sequence prediction (NSP). The robustly optimized BERT (RoBERTa) variant of BERT with few modifications has significant improvements in removing NSP loss function due to its inefficiency. SpanBERT is another variant that modifies MLM tasks by masking contagious random spans and also uses the span-boundary objective (SBO) loss function. A lite BERT (ALBERT) is another variant with two-parameter reduction techniques: factorized embedding parameterization and cross-layer parameter sharing. It also uses inter-sentence coherence loss instead of NSP. The performance of the BERT’s variants is found to be better than BERT, with few modifications as per the available literature.KeywordsBidirectional encoder representations from transformers (BERT)Robustly optimized BERT (RoBERTa)A lite BERT (ALBERT)Span-boundary objective (SBO)Masked language modeling (MLM)Next sequence prediction (NSP)Sequence-to-sequence (Seq2Seq)

Similar Papers
  • PDF
  • Research Article
  • Citations18

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

  • Mar 02, 2024
  • Информатика. Экономика. Управление - Informatics. Economics. Management
  • Rajesh Gupta
  • Abstract
  • Citations5

Bert model fine-tuning for text classification in knee OA radiology reports

  • Apr 01, 2020
  • Osteoarthritis and Cartilage
  • L Chen +5
  • Conference Article
  • Citations10

Classification of Fire Related Tweets on Twitter Using Bidirectional Encoder Representations from Transformers (BERT)

  • Nov 28, 2021
  • Jairus Mingua +2
  • Research Article

METAHEURISTIC-DRIVEN HYPERPARAMETER OPTIMIZATION FOR BERT IN SENTIMENT ANALYSIS

  • Oct 30, 2025
  • Archives for Technical Sciences
  • Alaa A El-Demerdash +1
  • Research Article
  • Citations12

Positional embeddings and zero-shot learning using BERT for molecular-property prediction

  • Feb 05, 2025
  • Journal of Cheminformatics
  • Medard Edmund Mswahili +4
  • Research Article
  • Citations10

Fine-Tuning BERT for Automated News Classification

  • Jun 04, 2025
  • Engineering, Technology & Applied Science Research
  • Mohammed I Salih +4
  • PDF
  • Research Article
  • Citations28

Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT)

  • Jul 30, 2022
  • BMC Medical Informatics and Decision Making
  • Jia Li +10
  • Conference Article

Domain-Specific Japanese ELECTRA Model Using a Small Corpus

  • Jan 01, 2021
  • Youki Itoh +1
  • Research Article
  • Citations20

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

  • May 05, 2024
  • Artificial Intelligence In Medicine
  • Shoya Wada +6
  • Book Chapter
  • Citations8

Natural Language Processing with “More Than Words – BERT”

  • Jan 01, 2021
  • Saranlita Chotirat +1
  • Research Article
  • Citations1

Analisis Perbandingan Model Bert Dan Xlnet Untuk Klasifikasi Tweet Bully Pada Twitter

  • Dec 10, 2024
  • Jurnal Teknologi Informasi dan Ilmu Komputer
  • Teuku Radillah +2
  • Research Article
  • Citations62

Does BERT need domain adaptation for clinical negation detection?

  • Feb 11, 2020
  • Journal of the American Medical Informatics Association
  • Chen Lin +5
  • PDF
  • Research Article
  • Citations45

Korean clinical entity recognition from diagnosis text using BERT

  • Sep 01, 2020
  • BMC Medical Informatics and Decision Making
  • Young-Min Kim +1
  • Conference Article
  • Citations3

Geoscience Language Processing for Exploration

  • Dec 09, 2021
  • Huseyin Denli +4
  • Research Article
  • Citations17

Protected Health Information Recognition by Fine-Tuning a Pre-training Transformer Model

  • Jan 01, 2022
  • Healthcare Informatics Research
  • Seo Hyun Oh +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.