Unsupervised Contrastive Refinement with Graph-Aware Multimodal Interaction for Radiology Report Generation.

Chest radiology report generation based on cross-modal multi-scale feature fusion

Intelligent medicine is eager to automatically generate radiology reports to ease the tedious work of radiologists. Previous researches mainly focused on the text generation with encoder-decoder structure, while CNN networks for visual features ignored the long-range dependencies correlated with textual information. Besides, few studies exploit cross-modal mappings to promote radiology report generation. To alleviate the above problems, we propose a novel end-to-end radiology report generation model dubbed Self-Supervised dual-Stream Network (S3-Net). Specifically, a Dual-Stream Visual Feature Extractor (DSVFE) composed of ResNet and SwinTransformer is proposed to capture more abundant and effective visual features, where the former focuses on local response and the latter explores long-range dependencies. Then, we introduced the Fusion Alignment Module (FAM) to fuse the dual-stream visual features and facilitate alignment between visual features and text features. Furthermore, the Self-Supervised Learning with Mask(SSLM) is introduced to further enhance the visual feature representation ability. Experimental results on two mainstream radiology reporting datasets (IU X-ray and MIMIC-CXR) show that our proposed approach outperforms previous models in terms of language generation metrics.

The potential of automated radiology report generation in alleviating the time-consuming tasks of radiologists is increasingly being recognized in medical practice. Existing report generation methods have evolved from using image-level features to the latest approach of utilizing anatomical regions, significantly enhancing interpretability. However, directly and simplistically using region features for report generation compromises the capability of relation reasoning and overlooks the common attributes potentially shared across regions. To address these limitations, we propose a novel region-based Attribute Prototype-guided Iterative Scene Graph generation framework (AP-ISG) for report generation, utilizing scene graph generation as an auxiliary task to further enhance interpretability and relational reasoning capability. The core components of AP-ISG are the Iterative Scene Graph Generation (ISGG) module and the Attribute Prototype-guided Learning (APL) module. Specifically, ISSG employs an autoregressive scheme for structural edge reasoning and a contextualization mechanism for relational reasoning. APL enhances intra-prototype matching and reduces inter-prototype semantic overlap in the visual space to fully model the potential attribute commonalities among regions. Extensive experiments on the MIMIC-CXR with Chest ImaGenome datasets demonstrate the superiority of AP-ISG across multiple metrics.

Inspired by the tremendous success of Large Language Models (LLMs), existing Radiology report generation methods attempt to leverage large models to achieve better performance. They usually adopt a Transformer to extract the visual features of a given X-ray image, and then, feed them into the LLM for text generation. How to extract more effective information for the LLMs to help them improve final results is an urgent problem that needs to be solved. Additionally, the use of visual Transformer models also brings high computational complexity. To address these issues, this paper proposes a novel context-guided efficient radiology report generation framework. Specifically, we introduce the Mamba as the vision backbone with linear complexity, and the performance obtained is comparable to that of the strong Transformer model. More importantly, we perform context retrieval from the training set for samples within each mini-batch during the training phase, utilizing both positively and negatively related samples to enhance feature representation and discriminative learning. Subsequently, we feed the vision tokens, context information, and prompt statements to invoke the LLM for generating high-quality medical reports. Extensive experiments on three X-ray report generation datasets (i.e., IU X-Ray, MIMIC-CXR, CheXpert Plus) fully validated the effectiveness of our proposed model. The source code is available at https://github.com/Event-AHU/Medical_ Image_Analysis.

In response to the worldwide COVID-19 pandemic, advanced automated technologies have emerged as valuable tools to aid healthcare professionals in managing an increased workload by improving radiology report generation and prognostic analysis. This study proposes a Multi-modality Regional Alignment Network (MRANet), an explainable model for radiology report generation and survival prediction that focuses on high-risk regions. By learning spatial correlation in the detector, MRANet visually grounds region-specific descriptions, providing robust anatomical regions with a completion strategy. The visual features of each region are embedded using a novel survival attention mechanism, offering spatially and risk-aware features for sentence encoding while maintaining global coherence across tasks. A cross-domain LLMs-Alignment is employed to enhance the image-to-text transfer process, resulting in sentences rich with clinical detail and improved explainability for radiologists. Multi-center experiments validate the overall performance and each module's composition within the model, encouraging further advancements in radiology report generation research emphasizing clinical interpretation and trustworthiness in AI models applied to medical studies.

Radiology report generation, which aims to relieve the heavy workload of radiologists and reduce the risks of misdiagnosis and overlooked diagnoses, is of great significance in current clinical medicine. Most existing methods mainly formulate radiology report generation as a problem similar to image captioning. Nevertheless, in the medical domain, these data-driven methods are plagued by two key issues: the insufficient utilization of expert knowledge and visual–textual biases. To solve these problems, this study presents a novel knowledge-augmented adaptive mechanism (KAM) for radiology report generation. In detail, our KAM first introduces two distinct types of medical knowledge: prior knowledge, which is input-independent and reflects the accumulated expertise of radiologists, and posterior knowledge, which is input-dependent and mimics the process of identifying abnormalities, thereby mitigating the issue of visual–textual bias. To optimize the utilization of both types of knowledge, this study develops a knowledge-augmented adaptive mechanism, which integrates the visual characteristics of radiological images with prior and posterior knowledge into the decoding process. Experimental evaluations on the publicly accessible IU X-ray and MIMIC-CXR datasets indicate that our approach is on par with the current common methods.

Recently, automatic radiology report generation, which targets to generate multiple sentences that can accurately describe medical observations for given X-ray images, has gained increasing attention. Existing methods commonly employ the attention mechanism for accurate word generation. However, such attention-based methods fail to leverage useful image-level global features, thereby limiting the model's reasoning ability. To tackle this challenge, we propose two-stage global enhancement layers to facilitate the Transformer to generate more reliable reports from a global perspective. Specifically, the 1st Global Enhancement Layer (1st GEL) is designed to capture the global visual context features by establishing the relationships between image-level global features and previously generated words. The 2nd Global Enhancement Layer (2nd GEL) is devised to capture the region-global level features by building the relationships between image-level global features and region-level information. The experiments demonstrate that by integrating the aforementioned two-stage global enhancement layers into the Transformer model, our proposal achieves state-of-the-art (SOTA) performance on various Natural Language Generation (NLG) evaluation metrics. Further Clinical Efficacy (CE) evaluations also validate that our proposal is able to predict more critical information.

The automatic generation of radiology reports from medical imaging data is a burgeoning field at the intersection of computer vision and natural language processing. While deep learning models, particularly those based on the encoder-decoder paradigm, have shown promise in generating fluent narrative text, they frequently suffer from factual inconsistencies, commonly referred to as hallucinations. These models often fail to capture the structured clinical findings accurately, leading to reports that are grammatically correct but clinically erroneous. This paper proposes a novel Multi-Task Learning (MTL) framework designed to enforce structured findings consistency. By simultaneously training the model to predict discrete clinical tags (classification) and generate narrative reports (generation), we bridge the semantic gap between visual features and linguistic representations. We introduce a Shared Semantic Alignment Module that explicitly maps visual regions to structured disease labels before decoding the text, ensuring that the generated narrative aligns with the detected pathologies. Extensive experiments on the MIMIC-CXR and IU X-Ray datasets demonstrate that our approach outperforms state-of-the-art baselines in both natural language metrics and clinical efficacy metrics. The results suggest that integrating structured supervision significantly mitigates hallucination and improves the diagnostic validity of automated radiology reporting.

Automatic report generation has the potential to alleviate the heavy workload of radiologists. However, existing methods often generate low-quality reports. To this end, we propose a large language model enhanced contrastive learning (LLM-ECL) approach to improve the quality and accuracy of breast ultrasound report generation. LLM-ECL leverages the pretrained MiniGPT-4 to extract visual features and employs an encoder-decoder architecture to generate reports. To address the issue that models trained with conventional cross-entropy loss tend to generate reports lacking specific abnormality descriptions, we introduce the contrastive loss. We perform contrastive learning between visual features and textual features generated by LLaMA2 to enhance image-text consistency. Experiments conducted on a breast ultrasound dataset comprising 655 patient cases demonstrate that LLM-ECL produces reports with high professionalism and accuracy.

Unsupervised domain adaptive video action recognition aims to recognize actions of a target domain using a model trained with only out-of-domain (source) annotations. The inherent complexity of videos makes this task challenging but also provides ground for leveraging multi-modal inputs (e.g., RGB, Flow, Audio). Most previous works utilize the multi-modal information by either aligning each modality individually or learning representation via cross-modal self-supervision. Different from previous works, we find that the cross-domain alignment can be more effectively done by using cross-modal interaction first. Cross-modal knowledge interaction allows other modalities to supplement missing transferable information because of the cross-modal complementarity. Also, the most transferable aspects of data can be highlighted using cross-modal consensus. In this work, we present a novel model that jointly considers these two characteristics for domain adaptive action recognition. We achieve this by implementing two modules, where the first module exchanges complementary transferable information across modalities through the semantic space, and the second module finds the most transferable spatial region based on the consensus of all modalities. Extensive experiments validate that our proposed method can significantly outperform state-of-the-art methods on multiple benchmark datasets, including the complex fine-grained dataset EPIC-Kitchens-100.

Dual Graph Convolutional Networks for Chest X-ray Report Generation

Pneumonia screening is one of the most crucial steps in the pneumonia diagnosing system, which can improve the work efficiency of the radiologists and prevent delayed treatments. In this paper, we propose a deep regression framework for automatic pneumonia screening, which jointly learns the multi-channel images and multi-modal information (i.e., clinical chief complaints, age, and gender) to simulate the clinical pneumonia screening process. We demonstrate the advantages of the framework in three ways. First, visual features from multi-channel images (Lung Window Images, High Attenuation Images, Low Attenuation Images) can provide more visual features than single image channel, and improve the ability of screening pneumonia with severe diseases. Second, the proposed framework treats chest CT scans as short video frames and analyzes them by using Recurrent Convolutional Neural Network, which can automatically extract multiple image features from multi-channel image slices. Third, chief complaints and demographic information can provide valuable prior knowledge enhancing the features from images and further promote performance. The proposed framework has been extensively validated in 900 clinical cases. Compared to the baseline, the proposed framework improves the accuracy by 2.3% and significantly improves the sensitivity by 3.1%. To the best of our knowledge, we are the first to screen pneumonia using multi-channel images, multi-modal demographic and clinical information based on the large scale clinical raw dataset.

Automatic report generation is a technology that automatically generates documents in the form of report by summarizing various materials according to a specific topic in time sequence or subject. Although the main content of the report is text, insertion of appropriate images can improve the completeness of the report. In this paper, we propose an image recommendation method for automatically selecting and inserting appropriate images corresponding to a specific part of a report. In our proposed method, reevaluation of the candidate images is performed based on the semantic similarity between query and the contents of the images. In order to transform semantic information of text query and image into one vector space, we extracted semantic information from image as a set of tags form using deep learning based object detection module. Also, we extracted tags from the given title of the image so that the proposed system can evaluate the candidate images even in the case that the given query includes specific keywords or proper nouns which were not learned by object detection and recognition module in advance. In this paper, we conducted experiments on eight queries related to recent events to verify the applicability of our proposed image recommendation system and evaluate the image selection accuracy.

Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years. Most of the existing methods either transfer the knowledge of image-text pretraining model to video-text retrieval task without fully exploring the multi-modal information of videos, or simply fuse multi-modal features in a brute force manner without explicit guidance. In this paper, we integrate multi-modal information in an explicit manner by tagging, and use the tags as the anchors for better video-text alignment. Various pretrained experts are utilized for extracting the information of multiple modalities, including object, person, motion, audio, etc. To take full advantage of these information, we propose the TABLE (TAgging Before aLignmEnt) network, which consists of a visual encoder, a tag encoder, a text encoder, and a tag-guiding cross-modal encoder for jointly encoding multi-frame visual features and multi-modal tags information. Furthermore, to strengthen the interaction between video and text, we build a joint cross-modal encoder with the triplet input of [vision, tag, text] and perform two additional supervised tasks, Video Text Matching (VTM) and Masked Language Modeling (MLM). Extensive experimental results demonstrate that the TABLE model is capable of achieving State-Of-The-Art (SOTA) performance on various video-text retrieval benchmarks, including MSR-VTT, MSVD, LSMDC and DiDeMo.

Purpose:Automatic surgical instruction generation is a crucial part for intra-operative surgical assistance. However, understanding and translating surgical activities into human-like sentences are particularly challenging due to the complexity of surgical environment and the modal gap between images and natural languages. To this end, we introduce SIG-Former, a transformer-backboned generation network to predict surgical instructions from monocular RGB images.Methods:Taking a surgical image as input, we first extract its visual attentive feature map with a fine-tuned ResNet-101 model, followed by transformer attention blocks to correspondingly model its visual representation, text embedding and visual–textual relational feature. To tackle the loss-metric inconsistency between training and inference in sequence generation, we additionally apply a self-critical reinforcement learning approach to directly optimize the CIDEr score after regular training.Results:We validate our proposed method on DAISI dataset, which contains 290 clinical procedures from diverse medical subjects. Extensive experiments demonstrate that our method outperforms the baselines and achieves promising performance on both quantitative and qualitative evaluations.Conclusion:Our experiments demonstrate that SIG-Former is capable of mapping dependencies between visual feature and textual information. Besides, surgical instruction generation is still at its preliminary stage. Future works include collecting large clinical dataset, annotating more reference instructions and preparing pre-trained models on medical images.