• Home
  • Search
  • A Systematic Literature Review of BERT-based Models for Natural Language Processing Tasks
  • Cite Icon2
  • https://doi.org/10.20895/infotel.v16i4.1206Copy DOI Icon

A Systematic Literature Review of BERT-based Models for Natural Language Processing Tasks

Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

Research area in natural language processing (NLP) domain has made major advances in recent years. The Bidirectional Encoder Representations from Transformers (BERT) and its derivative models have been at the vanguard, gaining notice for their exceptional performance across a variety of NLP applications. As a response to this context, hence, this study aims to conduct a systematic literature review on current research in BERT-based models in order to describe their characteristic variations on three frequently demanded natural language processing (NLP) tasks, i.e. text classification, question answering, and text summarization. This study employed a systematic literature review method as prescribed by Kitchenham. We collected 4,120 papers from publications indexed by Scopus and Google Scholar from which 42 complied to our defined review criteria and finally chosen for further analysis. Our review came up with three conclusions. First, in order to select appropriate models for particular NLP tasks, three primary concerns should be considered: i) the type of NLP problem to be resolved (i.e. NLP task to be served), ii) the specific domain to be handled (such as financial, medical, law/legal or others), and iii) the intended language to be applied (such as English or others). Second, learning rate, batch size, and the type of optimizer were the three most considered hyperparameters to be properly arranged in model training. Third, the most widely used metrics for text classification tasks were F1-score, accuracy, precision, and sensitivity (recall), while question answering, and text summarization tasks were mostly used the Exact Match and ROUGE respectively.

Similar Papers
  • Research Article
  • Citations170

BERT applications in natural language processing: a review

  • Mar 15, 2025
  • Artificial Intelligence Review
  • Nadia Mushtaq Gardazi +5
  • Conference Article
  • Citations12

Cross2Self-attentive Bidirectional Recurrent Neural Network with BERT for Biomedical Semantic Text Similarity

  • Dec 16, 2020
  • Zhengguang Li +5
  • PDF
  • Research Article
  • Citations173

Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)–Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study

  • Sep 12, 2019
  • JMIR Medical Informatics
  • Fei Li +5
  • PDF
  • Research Article
  • Citations18

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

  • Mar 02, 2024
  • Информатика. Экономика. Управление - Informatics. Economics. Management
  • Rajesh Gupta
  • Research Article
  • Citations5

Layer-Wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-Trained Models: An Evolutionary Approach

  • Nov 25, 2024
  • ACM Transactions on Evolutionary Learning and Optimization
  • Chenyang Bu +6
  • Conference Article
  • Citations4

Deep neural networks with pre-train model BERT for aspect-level sentiments classification

  • Jun 01, 2020
  • Yunxiang Zhang +1
  • PDF
  • Research Article
  • Citations16

Learned Text Representation for Amharic Information Retrieval and Natural Language Processing

  • Mar 20, 2023
  • Information
  • Tilahun Yeshambel +2
  • Research Article

The Effect of Various Text Representation Methods for Sentiment Analysis on Movie Review Data with Different Machine Learning Methods

  • Dec 31, 2024
  • Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım ve Teknoloji
  • Veysel Göç +1
  • Research Article
  • Citations3

Use of deep learning-based NLP models for full-text data elements extraction for systematic literature review tasks

  • Jun 03, 2025
  • Scientific Reports
  • Jingcheng Du +9
  • PDF
  • Research Article
  • Citations14

Improving Sentence Representations via Component Focusing

  • Feb 02, 2020
  • Applied Sciences
  • Xiaoya Yin +4
  • Research Article
  • Citations15

Bioformer: an efficient transformer language model for biomedical text mining

  • Feb 03, 2023
  • ArXiv
  • Li Fang +4
  • Research Article

Performance Analysis of Transformer Based Models for Automatic Short Answer Grading

  • May 30, 2025
  • Journal of Information Systems Engineering and Management
  • Rupal Chaudhari
  • Research Article

Performance Analysis of Transformer Based Models for Automatic Short Answer Grading

  • May 30, 2025
  • Journal of Information Systems Engineering and Management
  • Rupal Chaudhari
  • Research Article
  • Citations10

Fine-Tuning BERT for Automated News Classification

  • Jun 04, 2025
  • Engineering, Technology & Applied Science Research
  • Mohammed I Salih +4
  • Research Article
  • Citations628

Transformer models for text-based emotion detection: a review of BERT-based approaches

  • Feb 08, 2021
  • Artificial Intelligence Review
  • Francisca Adoma Acheampong +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.