• Home
  • Search
  • Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study.
  • https://doi.org/10.2196/86365Copy DOI Icon

Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study.

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Automated structuring of radiology reports is essential for data utilization and the development of medical artificial intelligence models. However, manual annotation by experts is labor-intensive, and processing real clinical data through commercial large language models (LLMs) presents significant privacy risks. These challenges are particularly pronounced for non-English languages like Japanese, where specialized medical corpora are scarce. While synthetic data generation offers a potential privacy-preserving alternative, its effectiveness in capturing complex clinical nuances-such as negation and contextual dependencies-to train robust classification models without any real-world training data has not been fully established. This study aimed to develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline, thereby eliminating reliance on real-world clinical data during the development phase. Furthermore, we sought to evaluate the generalizability of this approach by validating the model's performance on diverse, multi-institutional, real-world reports. Japanese radiology reports (n=3104) were generated using GPT-4.1 and automatically annotated at the sentence level into 4 categories (background, positive finding, negative finding, and continuation) using GPT-4.1-mini. The synthetic data were partitioned into training (n=2670), validation (n=334), and test (n=100) sets. We fine-tuned several models, including lightweight local LLMs (Qwen3 and Llama 3.2 series) using low-rank adaptation and Japanese text classification models (Bidirectional Encoder Representations from Transformers [BERT]-base Japanese v3, Japanese Medical Robustly Optimized BERT Pretraining Approach [JMedRoBERTa]-base, and ModernBERT-Ja-130M). External validation was performed using 280 real-world reports (3477 sentences) from 7 institutions in the Japan Medical Image Database, with ground-truth labels established by board-certified radiologists. Evaluation metrics included accuracy, macro-averaged F1 (macro F1) score, and positive predictive value for positive findings (PPV_1). All models achieved high performance on the synthetic test set (accuracy: 0.938-0.951; macro F1-score: 0.924-0.940). Overall performance declined on the external validation dataset (accuracy: 0.783-0.813; macro F1-score: 0.761-0.790), reflecting distributional differences between synthetic and real-world reports; however, PPV_1 remained stable and high across datasets (eg, 0.957 on the synthetic test set vs 0.952 on the external validation dataset for Qwen3 [4B]). Parsing errors occurred in LLM-based approaches (19-260 sentences, 0.55%-7.48% in the external dataset). This study demonstrates the feasibility of developing context-aware sentence classification models for Japanese radiology reports using a training pipeline based entirely on synthetic data. The stability of PPV_1 indicates that the models successfully captured the essential clinical terminology and linguistic patterns required to identify positive findings in real-world reports, despite the observed performance degradation during external validation. This approach substantially reduces manual annotation requirements and privacy risks, providing a scalable foundation for constructing structured radiology datasets to support the development of clinically relevant medical artificial intelligence models.

Similar Papers
  • Research Article
  • Citations4

Enhancing Pulmonary Disease Prediction Using Large Language Models With Feature Summarization and Hybrid Retrieval-Augmented Generation: Multicenter Methodological Study Based on Radiology Report

  • Jun 11, 2025
  • Journal of Medical Internet Research
  • Ronghao Li +8
  • Research Article

Detecting Stigmatizing Language in Clinical Notes with Large Language Models for Addiction Care

  • Aug 12, 2025
  • medRxiv
  • R Sethi +8
  • Research Article
  • Citations548

Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs.

  • Sep 25, 2018
  • Radiology
  • Ju Gang Nam +10
  • PDF
  • Research Article
  • Citations14

Privacy-Preserving Federated Model Predicting Bipolar Transition in Patients With Depression: Prediction Model Development Study.

  • Jul 20, 2023
  • Journal of Medical Internet Research
  • Dong Yun Lee +12
  • Research Article

Structural fatigue modeling of cumulative mechanical stress in the athletic elbow using routine radiographs.

  • Jun 01, 2026
  • Journal of orthopaedics
  • I Govindharaj +5
  • Research Article
  • Citations2

Large Language Models in Crisis Informatics for Zero and Few-Shot Classification

  • May 22, 2025
  • ACM Transactions on the Web
  • Cinthia Sánchez +2
  • PDF
  • Research Article
  • Citations55

Machine Learning-Based Prediction of Small Intracranial Aneurysm Rupture Status Using CTA-Derived Hemodynamics: A Multicenter Study.

  • Mar 04, 2021
  • American Journal of Neuroradiology
  • Z Shi +10
  • Research Article
  • Citations3

Improving drug identification in overdose death surveillance by using clinical natural language processing models.

  • Feb 08, 2026
  • Journal of forensic sciences
  • Arthur J Funnell +8
  • PDF
  • Research Article
  • Citations7

Evaluation of near-infrared hyperspectral imaging for the assessment of potato processing aptitude

  • Oct 17, 2022
  • Frontiers in Nutrition
  • Ainara López-Maestresalas +6
  • PDF
  • Research Article
  • Citations24

Classifying Ruptured Middle Cerebral Artery Aneurysms With a Machine Learning Based, Radiomics-Morphological Model: A Multicentral Study.

  • Aug 11, 2021
  • Frontiers in Neuroscience
  • Dongqin Zhu +12
  • Research Article

Enhanced Language Models for Predicting and Understanding HIV Care Disengagement: A Case Study in Tanzania

  • May 08, 2025
  • Research Square
  • Waverly Wei +16
  • Research Article
  • Citations7

Retinal Photograph-based Deep Learning System for Detection of Thyroid-Associated Ophthalmopathy.

  • Dec 11, 2023
  • The Journal of craniofacial surgery
  • Xue Jiang +4
  • Research Article
  • Citations8

External Validation of a Winning Artificial Intelligence Algorithm from the RSNA 2022 Cervical Spine Fracture Detection Challenge.

  • Feb 24, 2025
  • AJNR. American journal of neuroradiology
  • James P Harper +5
  • Research Article

Opportunistic Osteoporosis Screening from Routine Knee Radiographs Using a Multi-Stage CNN Framework with External Validation.

  • Jul 03, 2026
  • Journal of clinical medicine
  • Nitiphoom Sinnathakorn +3
  • Research Article

Deep Learning-Based Multiclass Classification of Mitral Valve Etiologies Using Limited B-Mode and Color Doppler Echocardiography: Internal and External Validation.

  • Jun 15, 2026
  • Journal of the American Society of Echocardiography : official publication of the American Society of Echocardiography
  • Dawun Jeong +11
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.