Multi-Task Learning for Radiology Report Generation with Structured Findings Consistency

Chest radiology report generation based on cross-modal multi-scale feature fusion

BackgroundIt is essential for radiologists to communicate actionable findings to the referring clinicians reliably. Natural language processing (NLP) has been shown to help identify free-text radiology reports including actionable findings. However, the application of recent deep learning techniques to radiology reports, which can improve the detection performance, has not been thoroughly examined. Moreover, free-text that clinicians input in the ordering form (order information) has seldom been used to identify actionable reports. This study aims to evaluate the benefits of two new approaches: (1) bidirectional encoder representations from transformers (BERT), a recent deep learning architecture in NLP, and (2) using order information in addition to radiology reports.MethodsWe performed a binary classification to distinguish actionable reports (i.e., radiology reports tagged as actionable in actual radiological practice) from non-actionable ones (those without an actionable tag). 90,923 Japanese radiology reports in our hospital were used, of which 788 (0.87%) were actionable. We evaluated four methods, statistical machine learning with logistic regression (LR) and with gradient boosting decision tree (GBDT), and deep learning with a bidirectional long short-term memory (LSTM) model and a publicly available Japanese BERT model. Each method was used with two different inputs, radiology reports alone and pairs of order information and radiology reports. Thus, eight experiments were conducted to examine the performance.ResultsWithout order information, BERT achieved the highest area under the precision-recall curve (AUPRC) of 0.5138, which showed a statistically significant improvement over LR, GBDT, and LSTM, and the highest area under the receiver operating characteristic curve (AUROC) of 0.9516. Simply coupling the order information with the radiology reports slightly increased the AUPRC of BERT but did not lead to a statistically significant improvement. This may be due to the complexity of clinical decisions made by radiologists.ConclusionsBERT was assumed to be useful to detect actionable reports. More sophisticated methods are required to use order information effectively.

Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports

e18093 Background: Pulmonary nodule incidental findings challenge providers to balance resource efficiency and high clinical quality. Incidental findings tend to be undertreated with studies reporting appropriate follow-up rates as low as 29%. Ensuring appropriate follow-up on all incidental findings is labor-intensive; requires the clinical reading and classification of radiology reports to identify high-risk lung nodules. We tested the feasibility of automating this process with natural language processing (NLP) and machine learning (ML). Methods: In cooperation with Sarah Cannon Research Institute (SCRI), we conducted a series of data science experiments utilizing NLP and ML computing techniques on 8,879 free-text, narrative CT (computerized tomography) radiology reports. Reports used were dated from Dec 8, 2015 - April 23, 2017, came from SCRI-affiliated Emergency Department, Inpatient, and Outpatient facilities and were a representative, random sample of the patient populations. Reports were divided into a development set for model training and validation, and a test set to evaluate model performance. Two models were developed - a “Nodule Model” was trained to detect the reported presence of a pulmonary nodule and a rules-based “Sizing Model” was developed to extract the size of the nodule in millimeters. Reports were bucketed into three prediction groups: > = 6 mm, < 6 mm, and no size indicated. Nodules were considered positives and placed in a queue for follow-up if the nodule was predicted > = 6 mm, or if the nodule had no size indicated and the radiology report contained the word “mass.” The Fleischner Society Guidelines and clinical review informed these definitions. Results: Precision and recall metrics were calculated for multiple model thresholds. A threshold was selected based on the validation set calculations and a success criterion of 90% queue precision was selected to minimize false positives. On the test dataset, the F1 measure of the entire pipeline (lung nodule classification model and size extraction model) was 72.9%, recall was 60.3%, and queue precision was 90.2%, exceeding success criteria. Conclusions: The experiments demonstrate the feasibility of NLP and ML technology to automate the detection and classification of pulmonary nodule incidental findings in radiology reports. This approach promises to improve healthcare quality by increasing the rate of appropriate lung nodule incidental finding follow-up and treatment without excessive labor or risking overutilization.

The increasing number of computed tomography (CT) scan examinations and the time-intensive nature of manual analysis necessitate efficient automated methods to assist radiologists in managing their increasing workload. While deep learning approaches primarily classify abnormalities from three-dimensional (3D) CT images, radiologists also incorporate clinical indications and patient demographics, such as age and sex, for diagnosis. This study aims to enhance multilabel abnormality classification and automated report generation by integrating imaging and non-imaging data. We propose a multimodal deep learning model that combines 3D chest CT scans, clinical information reports, patient age, and sex to improve diagnostic accuracy. Our method extracts visual features from 3D volumes using a visual encoder, textual features from clinical indications via a pretrained language model, and demographic features through a lightweight feedforward neural network. These extracted features are projected into a shared representation space, concatenated, and processed by a projection head to predict abnormalities. For the multilabel classification task, incorporating clinical indications and patient demographics into an existing visual encoder, called CT-Net, improves the F1 score to 51.58, representing a increase over CT-Net alone. For the automated report generation task, we extend two existing methods, CT2Rep and CT-AGRG, by integrating clinical indications and demographic data. This integration enhances Clinical Efficacy metrics, yielding an F1 score improvement of for the CT2Rep extension and for the CT-AGRG extension. Our findings suggest that incorporating patient demographics and clinical information into deep learning frameworks can significantly improve automated CT scan analysis. This approach has the potential to enhance radiological workflows and facilitate more comprehensive and accurate abnormality detection in clinical practice.

Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets. We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data. In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain. With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better. Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

The purpose of this study is to evaluate the performance of a natural language processing (NLP) system in classifying a database of free-text knee MRI reports at two separate academic radiology practices. An NLP system that uses terms and patterns in manually classified narrative knee MRI reports was constructed. The NLP system was trained and tested on expert-classified knee MRI reports from two major health care organizations. Radiology reports were modeled in the training set as vectors, and a support vector machine framework was used to train the classifier. A separate test set from each organization was used to evaluate the performance of the system. We evaluated the performance of the system both within and across organizations. Standard evaluation metrics, such as accuracy, precision, recall, and F1 score (i.e., the weighted average of the precision and recall), and their respective 95% CIs were used to measure the efficacy of our classification system. The accuracy for radiology reports that belonged to the model's clinically significant concept classes after training data from the same institution was good, yielding an F1 score greater than 90% (95% CI, 84.6-97.3%). Performance of the classifier on cross-institutional application without institution-specific training data yielded F1 scores of 77.6% (95% CI, 69.5-85.7%) and 90.2% (95% CI, 84.5-95.9%) at the two organizations studied. The results show excellent accuracy by the NLP machine learning classifier in classifying free-text knee MRI reports, supporting the institution-independent reproducibility of knee MRI report classification. Furthermore, the machine learning classifier performed well on free-text knee MRI reports from another institution. These data support the feasibility of multiinstitutional classification of radiologic imaging text reports with a single machine learning classifier without requiring institution-specific training data.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

As a task at the intersection of computer vision and natural language processing, image captioning offers significant application value in domains such as intelligent human–computer interaction, accessibility support and multimedia content retrieval. The primary objective is to generate natural language descriptions by interpreting visual features, traditionally relying on heterogeneous single‐stream grid features and region features. However, existing approaches face limitations: grid features struggle to balance global semantic perception with local detail analysis, and region features exhibit weakened spatial modelling efficacy due to sparse semantic correlations. Furthermore, fusing heterogeneous visual features often lacks effective control over complementarity and redundancy, leading to descriptions prone to semantic bias or detail omission. To address these challenges, we propose a novel Multi‐Gated Dual‐Stream Visual Feature Fusion (MGDSF) for Image Captioning. Our approach enhances the semantic accuracy and completeness of generated captions through dual‐stream feature extraction and a multi‐gated fusion (MGF) mechanism. First, we employ a Mamba‐like linear attention mechanism to construct a grid feature network with hierarchical positional awareness. This network achieves global modelling while maintaining local sensitivity by dynamically modulating information flow. Second, based on the Detection Transformer (DETR) framework, we design a region feature extractor to provide complementary local object visual information. Finally, we introduce a MGF module that balances the complementarity of dual‐stream visual features and suppresses cross‐modal information redundancy via multiple context‐aware gates, thereby achieving fine‐grained visual‐semantic alignment. Experiments on MS COCO demonstrate that MGDSF surpasses existing methods on multiple evaluation metrics, achieving METEOR, ROUGE‐L and CIDEr scores of 30.0%, 59.8% and 140.1%, respectively. These results validate the effectiveness of our proposed method and indicate its broad application potential.

Abstract. To Electronic health records (EHRs) are a treasure trove of patient information, including clinical notes. However, extracting meaningful insights from these unstructured narratives remains a challenge. This paper explores the potential of natural language processing (NLP) and deep learning to unlock the rich clinical narrative data and revolutionize healthcare. Clinical notes capture a wealth of information beyond structured data points. They document patient history, symptoms, treatment progress, and physician observations. Analyzing this narrative unlocks a deeper understanding of a patient's health journey, aiding in NLP can identify subtle patterns in language usage that might be indicative of specific diseases. This can assist clinicians in arriving at more accurate diagnoses, particularly for complex cases. Deep learning models can analyze narratives to identify patients at higher risk of developing certain conditions. This allows for proactive interventions and preventative measures. Extracting details about a patient's lifestyle, social context, and emotional well-being from narratives can facilitate the development of personalized treatment plans that cater to individual needs. NLP techniques like named entity recognition, sentiment analysis, and topic modeling enable the extraction of key clinical entities, emotions, and themes from narratives. Deep learning, with its ability to learn complex relationships in text data, empowers NLP tasks. Deep learning models can be trained on massive datasets of clinical notes, achieving superior performance in tasks like information extraction compared to traditional NLP methods. Deep learning models can handle the vast amount of text data generated in healthcare settings, enabling large- scale analysis of clinical narratives. By learning from diverse patient populations, deep learning models can improve their generalizability and adapt to variations in language usage within clinical documentation. Integrating NLP and deep learning into clinical workflows promises a future where the patient narrative is not just documented but actively analyzed to inform better healthcare decisions. This can lead to earlier diagnoses, more effective treatments, and ultimately, improved patient outcomes. By addressing the challenges and fostering responsible development, we can unlock the true potential of clinical narratives and empower a new era of data-driven healthcare.

In clinical diagnosis, radiological reports are essential to guide the patient’s treatment. However, writing radiology reports is a critical and time-consuming task for radiologists. Existing deep learning methods often ignore the interplay between medical findings, which may be a bottleneck limiting the quality of generated radiology reports. Our paper focuses on the automatic generation of medical reports from input chest X-ray images. In this work, we mine the associations between medical discoveries in the given texts and construct a knowledge graph based on the associations between medical discoveries. The patient’s chest X-ray image and clinical history file were used as input to extract the image–text hybrid features. Then, this feature is used as the input of the adjacency matrix of the knowledge graph, and the graph neural network is used to aggregate and transfer the information between each node to generate the situational representation of the disease with prior knowledge. These disease situational representations with prior knowledge are fed into the generator for self-supervised learning to generate radiology reports. We evaluate the performance of the proposed method using metrics from natural language generation and clinical efficacy on two public datasets. Our experiments show that our method outperforms state-of-the-art methods with the help of a knowledge graph constituted by prior knowledge of the patient.

HomeRadiology: Artificial IntelligenceVol. 1, No. 1 Next EditorialFree AccessArtificial Intelligence, Real RadiologyCharles E. Kahn, Jr Charles E. Kahn, Jr Charles E. Kahn, Jr Published Online:Jan 30 2019https://doi.org/10.1148/ryai.2019184001MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In IntroductionGlendower:I can call spirits from the vasty deep.Hotspur:Why, so can I, or so can any man; But will they come when you do call for them?– William Shakespeare, Henry IV, Part IWelcome to this inaugural issue of Radiology: Artificial Intelligence. Our journal’s mission is to publish high-quality scientific work that advances our understanding of artificial intelligence (AI) in radiology.AI has become a topic of great interest—especially the application of machine learning techniques to medical images—but AI itself is not new. The term artificial intelligence was proposed in 1956 to describe efforts to understand, simulate, and improve upon human qualities such as reasoning, learning, solving problems, understanding verbal and written language, processing visual information, and playing games like chess and poker.What is new is a resurgence of interest in AI, particularly in the use of machine learning to recognize patterns in images. And, curiously, it is game playing that has opened this new frontier—but not the games of chess, checkers, or Go. Rather, think Xbox and PlayStation. Video games require a rapidly changing three-dimensional scene to be transformed into two-dimensional images shown in real time. The need to compute images efficiently spurred the development of highly parallelized graphics processing units. These specialized processors, in turn, have powered software for increasingly complex and sophisticated “deep” artificial neural network models. Whereas neural networks developed 10 years ago typically had three or four layers, today’s deep networks comprise hundreds of layers (1).Deep learning models have engendered both great excitement and a great deal of hyperbole. After all, if AI systems can pick out pictures of cats on the web, then surely such systems are ready to replace radiologists, right? Well, perhaps not, at least not right now. There is much work to be done to build and validate systems that can detect and characterize the thousands of imaging findings and their associated diseases that can be seen across a panoply of radiology studies.And that brings us to the quote from Shakespeare. Anyone can claim to build an AI system, but that doesn’t mean that the system will do their bidding as imagined. Our journal is here to assure that the science and applications of AI in radiology are built on thoughtful, innovative, and well-validated research.What sorts of topics will this journal publish? We will bring you the same high caliber of research that is found in RSNA’s flagship scientific journal, Radiology, but focused here on AI, machine learning, and data science in radiology. In particular, we seek to publish first-rate work that provides rigorous evaluation of AI’s applications to clinical problems in radiology.We invite manuscripts that show the impact of AI to extract information, diagnose and manage disease in patients, streamline radiology workflow, or improve health care outcomes. We’re interested in image segmentation, image reconstruction, automated detection of abnormalities, diagnostic reasoning, natural language processing, clinical workflow analysis, radiomics, and radiogenomics. We also invite manuscripts that demonstrate novel applications of AI in radiology or highlight innovative AI methodologies. Developers of publicly available sets of radiologic images, image annotations, radiology reports, or algorithms can present their work as a Data Resources report.AI and radiology do not exist in isolation: they are part of broad endeavors to advance knowledge and improve health. As such, this journal will feature articles on the ethical, legal, social, and economic implications of AI in radiology.AI is and must be a human—and humane—activity (2). We must engage in this work with an eye to how these technologies will help us care for our patients more effectively and humanely. Our goal is not to replace, but rather to extend our human abilities to provide medical care—and to improve the lives of those we are privileged to serve.All RSNA members receive access to this online bimonthly journal. We invite all readers (RSNA members or not) to sign up for our Editor’s Blog, The Vasty Deep (https://pubs.rsna.org/page/ai/blog) and to follow us on Twitter (@Radiology_AI). These social media platforms will augment the journal and offer innovative online features.Again, welcome!

Fake news is a growing problem on social media and can have significant negative consequences for individuals and society owing to its accessibility, low cost, and quick distribution. It is difficult to automatically identify bogus news that defies the current content-based analysis techniques. One of the key reasons is that current NLP algorithms still lack common sense, which is frequently necessary for understanding how to read the news. Recent research has demonstrated that the propagation patterns of true and fake news differ on social media. With the potential for automatic fake news detection, this study investigates the use of Deep Learning (DL) models to detect fake news on social media. A Neural Network (NN) model was developed using Natural Language Toolkit (NLTK), TensorFlow, and Natural Language Processing (NLP) for textual analysis. The model was trained on a dataset of fake and real news articles, and was able to achieve high accuracy in identifying fake news. The results of this study suggest that DL models can be valuable tools for detecting fake news on social media.

A unified modeling framework for lane change intention recognition and vehicle status prediction

For Heterogeneous Multi-Processor System-on-Chips (HMPSoCs), a reasonable pipeline design can significantly improve the inference performance of Deep Learning (DL) models. The pipeline design optimization can be modeled as a search problem where an accurate prediction model can efficiently speed up the search process. However, the performance prediction of DL models for the pipeline inference strategy is challenging because of the inter-layer effect, inference details, and variety of model structures. In this paper, we propose TPPNet, a transformer-based model for predicting the inference performance of various DL models with the pipeline inference strategy. TPPNet represents the DL model as an execution sequence with operators and hardware details to extract the hidden factors between layers. Moreover, we apply the Multi-task Learning (MTL) method to accurately predict throughput and latency metrics by constructing a predictive model. To the best of our knowledge, this is the first study dedicated to pipeline inference performance prediction for the DL model on HMPSoCs. We evaluate TPPNet on six well-known DL models using RK3399. The experimental outcomes affirm the high accuracy of TPPNet and its capability to significantly reduce the time overhead associated with pipeline exploration.