• Home
  • Search
  • Clinical-ShiftEval: a framework for simulating and evaluating model adaptation in dynamic clinical NLP tasks
  • https://doi.org/10.1186/s12911-026-03538-6Copy DOI Icon

Clinical-ShiftEval: a framework for simulating and evaluating model adaptation in dynamic clinical NLP tasks

Show More
  • Abstract
  • Literature Map
  • Similar Papers
Abstract

BackgroundClinical natural language processing (NLP) models are widely used to extract information from electronic health records (EHR) and support healthcare decision-making. However, most existing models are evaluated under the assumption of static data distributions and fixed task definitions, overlooking the dynamic and evolving nature of real clinical environments. Distributional changes arising from updated guidelines, emerging diseases, or institutional restructuring can lead to substantial model degradation in practice.MethodsWe introduce Clinical-ShiftEval, a framework designed to simulate and evaluate model adaptation in dynamic clinical NLP tasks using real-world clinical text data. Our framework parameterizes two prevalent types of evolution observed in healthcare: (1) label-set incompatibility (LSI), where the set of prediction targets changes over time, and (2) task definition evolution (TDE), where the clinical meaning or labeling rule is revised. These changes are operationalized through controlled transitions between two periods in real datasets. As a case study, we applied Clinical-ShiftEval to the Chilean Waiting List Corpus, using referral specialty classification to model LSI and disease prioritization to model TDE, and systematically compared continual training, in-context learning with large language models, and a hybrid approach.ResultsWe validated that Clinical-ShiftEval reliably simulates realistic levels of clinical change and produces controlled, interpretable performance drops. This confirms its utility as a framework to generate reproducible scenarios for benchmarking model robustness. As a case study, we compared multiple model adaptation strategies under LSI and TDE. Conventional supervised models showed severe degradation under these shifts (up to a drop of 82% F1 in LSI and 43% in TDE). In contrast, in-context learning reduced the drop to 35% in LSI and 10% in TDE, while a hybrid method further improved robustness, limiting the drop to approximately 8% in both settings. Continual adaptation with incremental retraining gradually restored performance, but only surpassed in-context and hybrid approaches after incorporating at least 30% of data from the second period.ConclusionsClinical-ShiftEval enables robust benchmarking of model adaptation in realistic, dynamic settings using authentic EHR text. Both in-context learning and hybrid approaches substantially mitigate the impact of evolving clinical data, achieving strong performance even without access to new labeled data. For optimal recovery of baseline accuracy, continual training with a moderate amount of new data is most effective, eventually surpassing the other methods as more new data become available. These findings provide practical guidance for the deployment of resilient NLP models in dynamic healthcare settings.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12911-026-03538-6.

Similar Papers
  • Research Article
  • Citations6

NLP modeling recommendations for restricted data availability in clinical settings

  • Mar 07, 2025
  • BMC Medical Informatics and Decision Making
  • Fabián Villena +2
  • Research Article
  • Citations173

An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study.

  • Apr 08, 2024
  • JMIR Medical Informatics
  • Sonish Sivarajkumar +4
  • Research Article

Integrating NLP into Radiation Oncology: A Practical Guide to Transformer Architecture and Large Language Models

  • Aug 13, 2025
  • BJR|Artificial Intelligence
  • Reza Khanmohammadi +10
  • Research Article
  • Citations8

Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models.

  • Oct 31, 2024
  • European radiology
  • Marcel C Langenbach +7
  • Research Article
  • Citations7

Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify Arrhythmia Recurrence.

  • Dec 16, 2024
  • Circulation. Arrhythmia and electrophysiology
  • Ruibin Feng +16
  • PDF
  • Research Article
  • Citations12

Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation

  • Jul 06, 2025
  • Scientific Reports
  • Thomas Kallstenius +3
  • Research Article
  • Citations12

Improving the robustness and accuracy of biomedical language models through adversarial training

  • Jun 15, 2022
  • Journal of Biomedical Informatics
  • Milad Moradi +1
  • Research Article

0510 Large Language Models Streamline the Identification of an Insomnia Phenotype in Electronic Health Records

  • May 19, 2025
  • SLEEP
  • Guillermo Lopez-Garcia +7
  • Research Article
  • Citations17

Getting More Out of Large Databases and EHRs with Natural Language Processing and Artificial Intelligence: The Future Is Here.

  • Oct 19, 2022
  • Journal of Bone and Joint Surgery
  • Bardia Khosravi +2
  • Conference Article
  • Citations11

Applications of Natural Language Processing in Clinical Research and Practice

  • Jan 01, 2019
  • Yanshan Wang +3
  • Research Article
  • Citations38

“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks

  • Jul 22, 2022
  • Journal of Biomedical Informatics
  • Jinghui Liu +3
  • Research Article
  • Citations9

A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept extraction.

  • May 14, 2024
  • Journal of the American Medical Informatics Association : JAMIA
  • Sunyang Fu +13
  • Research Article
  • Citations2

Natural Language Processing and ICD-10 Coding for Detecting Bleeding Events in Discharge Summaries: Comparative Cross-Sectional Study

  • Aug 29, 2025
  • JMIR Medical Informatics
  • Frederic Gaspar +6
  • Research Article
  • Citations15

Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review

  • Jan 22, 2025
  • Frontiers in Medicine
  • Mahmud Omar +5
  • Abstract

Measuring Adoption of Patient Priorities-Aligned Care Using Natural Language Processing

  • Dec 16, 2020
  • Innovation in Aging
  • Javad Razjouyan +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.