A Unified Visual Information Preservation Framework for Self-supervised Pre-Training in Medical Image Analysis.

Supervised deep learning methods have produced state-of-the-art results with large labeled datasets. However, accessing large labeled datasets is difficult in medical image analysis because of a shortage of medical experts, expensive annotations, and privacy constraints in the healthcare domain. Self-supervised learning is a branch of machine learning that exploits unlabeled data to encourage network weights toward a valid latent representation of the data during a so-called pretext task. The features learned by the model while solving pretext tasks are transferred to a downstream task where limited annotations are available. In this work, we propose PatchLoc, a novel pretext task whose objective is to find the location of a given patch from an image as a source of supervision. We validated the effectiveness of PatchLoc on a downstream segmentation task using three different medical datasets. PatchLoc yields substantial improvements compared to U-Net trained from scratch and other pretext task-based approaches in a low data regime.

Distributed contrastive learning for medical image segmentation.

Brain tumour segmentation is critical in medical image analysis, facilitating diagnosis and treatment planning in neurosurgery. Brain tumour segmentation with supervised learning shows robust results in medical imaging; however, it requires a sufficient amount of annotated data for effective learning. It is important to detect boundaries of tumour subregions accurately in fine-grained segmentation. We propose a novel approach that uses a unique dual-decoder architecture, focusing on edge identification and segmentation accuracy enhancement. Utilising a dual-decoder 3D-UNet model, we prioritise accuracy and fine-grained details in tumour segmentation and introduce an additional tumour edge detection task, aiming to move beyond traditional single-decoder approaches. Incorporating a 3D SimSiam network as the self-supervised pretraining technique, we aim to address the limitation of annotated data and enhance the segmentation accuracy. Our model surpasses many supervised variants of U-net architectures and self-supervised approaches, highlighting the importance of edge detection in tumour segmentation. The proposed approach enhances segmentation accuracy by showing an accuracy of 98.1% and provides critical boundary details for clinical decision-making. Visualisations of segmentation and edge masks further validate the effectiveness of the proposed method.

Whether tracking patient progress for clinical decision making, or investigating novel therapies, automated analysis of Computed Tomography (CT) imaging data is essential for the future of digital radiomics. In digital radiomics, as in all medical imaging, well annotated data is scarce, whereas unlabelled images are relatively plentiful. In other fields of image processing and medical imaging, the application of self-supervised learning (SSL) to large quantities of unlabelled data has resulted in great strides forward for fast, scalable, and interpretable image analysis. In this work we present a new approach applying SSL to CT imaging data which allows for: 1) Improved performance on image classification tasks, based on 2) dramatically reduced quantity of annotated CT imaging data, whilst also 3) enabling easy exploration and interpretation of the image regions. We applied a selection of self-supervised approaches (BYOL, DINO, SimCLR, & inpainting) to CT imaging data. Because the high dimensionality of CT data prevents us from using them directly to out-of the box SSL models, we adopt a 3D patching approach to reduce the dimensionality of the neural net input, and process each patch independently. We train our self-supervised models on public datasets (DeepLesion, NSCLC), and we specialize these models for tumor classification tasks that we evaluate on AstraZeneca sponsored clinical trials. The specialization is done in two different ways: 1) we use the pre-trained SSL model as an encoder that transforms the images of the clinical study into embeddings, on which we apply supervised classification models; 2) we use transfer learning to fine-tune supervised classification models that take the patches directly as inputs. We find that self-supervised pre-training significantly improves the accuracy on tumor classification tasks compared against a supervised learning baseline. Additionally, using the SSL embeddings we build an interactive map of CT imaging data enabling quick and intuitive inspection of the relevant regions. Our findings show that SSL constitutes an important tool for medical imaging analysis. SSL results in models that generalize better, and enable improved downstream interpretability and predictions. Furthermore, well trained SSL models can be re-applied to multiple indications because they are pre-trained on broad and diverse CT imaging data. Citation Format: Leon Fedden, Zhenning Zhang, Khan Baykaner, Qin Li, Lucas Bordeaux. DIME-CT: Self-supervised learning for medical image analysis using patch-based embeddings [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 1937.

The success of supervised deep learning heavily depends on large labeled datasets whose construction is often challenging in medical image analysis. Contrastive learning, a variant of self-supervised learning, is a potential solution to alleviate the strong demand for data annotation. In this work, we extend the contrastive learning framework to 3D volumetric medical imaging. Specifically, we propose (1) multiview contrasting strategy to maximize the mutual information between three views of 3D image to learn global representations and (2) long-short spatial contrasting strategy to learn local representations by matching a short spatial clip to a long spatial clip in the latent space. To combine these two strategies, we propose multiview long-short spatial contrastive learning (MLSSCL) framework, which can effectively learn generic 3D representations. Our extensive experiments on two brain Magnetic Resonance Imaging (MRI) datasets demonstrate that MLSSCL significantly outperforms learning from scratch and other self-supervised learning methods on both classification and segmentation tasks.

This article reports the results of the second iteration of the autoPET challenge on automated lesion segmentation in whole-body PET/CT, held in conjunction with the 26th International Conference on Medical Image Computing and Computer Assisted Intervention in 2023. In contrast to the first autoPET challenge, which served as a proof of concept, this study investigates whether machine learning-based segmentation models trained on data from a single source can maintain performance across clinically relevant variations in PET/CT data, reflecting the demands of real-world deployment. Methods: A comprehensive biomedical segmentation challenge on PET/CT domain generalization was designed and conducted. Participants were tasked to train machine learning models on annotated whole-body 18F-FDG data (n = 1,014). These models were then evaluated on a test set of 200 samples from 5 clinically relevant domains, including variations in institutions, pathologies, and populations and a different tracer. Performance was measured in terms of average dice similarity coefficient, average false-positive volume, and average false-negative volume. The best-performing teams were awarded in 3 categories. Furthermore, a detailed analysis was conducted after the challenge, examining results across domains and unique instances, along with a ranking analysis. Results: Generalization from a single-source domain remains a significant challenge. Seventeen international teams successfully participated in the challenge. The best-performing team reached an average dice similarity coefficient of 0.5038, a mean false-positive volume of 87.8388 mL, and a mean false-negative volume of 8.4154 mL on the test set. nnU-Net was the most commonly used framework, with most participants using a 3-dimensional U-Net. Despite competitive in-domain results, out-of-domain performance deteriorated substantially, particularly on pediatric and prostate-specific membrane antigen data. Detailed error analysis revealed frequent false-positives due to physiologic uptake and decreased sensitivity in detecting small or low-uptake lesions. A majority-vote ensemble offered minimal performance gains, whereas an oracle ensemble indicates hypothetical gains. Ranking analysis showed no single team consistently outperformed all others across ranking schemes. Conclusion: The second autoPET challenge provides a comprehensive evaluation of the current state of automated PET/CT tumor segmentation, highlighting both progress and persistent challenges of single-source domain generalization and the need for diverse public datasets to enhance algorithm robustness.

Chapter 4 - 4D medical image analysis: a systematic study on applications, challenges, and future research directions

Continual source-free active domain adaptation for nasopharyngeal carcinoma tumor segmentation across multiple hospitals.

Segmentation of a brain tumor is one of the most difficult tasks in the analysis of medical images. The purpose of brain tumor segmentation is to create an accurate outline of brain tumor areas. Gliomas are the most common type of brain tumors. Diagnosis of patients with this disease is based on the analysis of the results of magnetic resonance imaging and segmentation of the tumor boundaries manually. However, due to the time-consuming nature of the manual segmentation process and errors, there is a need for a fast and reliable automatic segmentation algorithm. In recent years, deep learning methods have shown promising effectiveness in solving various computer vision problems, such as image classification, object detection and semantic segmentation. A number of methods based on deep learning have been applied to segmentation of brain tumors, and promising results have been achieved. The article proposes a hybrid method for solving the problem of segmentation of brain tumors based on its MRI images based on the U-Net architecture, the encoder of which uses a model of a deep convolutional neural network pre-trained on a set of ImageNet images. Among such models were used VGG16, VGG19, MobileNetV2, Inception, ResNet50, EfficientNetb7, InceptionResnetV2, DenseNet201, DenseNet121. Based on the hybrid method, the TL-U-Net model was implemented, and numerical experiments were carried out to train it with different encoder models for segmentation of brain tumors based on its MRI images. Computer experiments on a set of MRI images of the brain showed the effectiveness of the proposed approach, the best encoder model turned out to be the neural network Densenet121, which provided indicators of segmentation accuracy MeanIoU=90.34%, MeanDice=94.33%, accuracy=94.17%. The obtained estimates of segmentation accuracy are comparable or exceed similar estimates obtained by other researchers.

Most of abdominal CT images include Gaussian noise, and CT scans form a blurry vision because of the internal fat tissue inside of abdomen. These two handicaps (noise and fat tissue) constitute an impediment in front of an accurate abdominal organ & tumour segmentation. Also segmentation techniques generally fall into error on segmentation of close grayscale regions. Therefore, denoising and enhancement parts are crucial for better segmentation results on CT images. In this paper, we form a tool including three efficient algorithms for the purpose of image enhancement before abdominal organ & tumour segmentation. At first, the denoising process is realized by Block Matching and 3D Filtering (BM3D) algorithm for elimination of Gaussian noise stated in arterial phase CT images. At second, Fast Linking Spiking Cortical Model (FL-SCM) is used for removing the internal fat tissue. At last, Otsu algorithm is processed to remove the redundant parts within the image. In experiments, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) index are used to evaluate the performance of proposed method, and a visual comparison is presented. According to results, it is seen that proposed tool obtains the best PSNR and SSIM values in comparison with two steps of pipeline (FL-SCM and BM3D & FL-SCM). Consequently, BM3D & FL-SCM & Otsu (BFO) ensures a clean abdomen particularly for segmentation of liver, spleen, pancreas, adrenal tumours, aorta, ribs, spinal cord and kidneys.

Three-dimensional (3D) medical images, such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), are essential for clinical applications. However, the need for diverse and comprehensive representations is particularly pronounced when considering the variability across different organs, diagnostic tasks, and imaging modalities. How to effectively interpret the intricate contextual information and extract meaningful insights from these images remains an open challenge to the community. While current self-supervised learning methods have shown potential, they often consider an image as a whole thereby overlooking the extensive, complex relationships among local regions from one or multiple images. In this work, we introduce a pioneering method for learning 3D medical image representations through an autoregressive pre-training framework. Our approach sequences various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence. By employing an autoregressive sequence modeling task, we predict the next visual token in the sequence, which allows our model to deeply understand and integrate the contextual information inherent in 3D medical images. Additionally, we implement a random startup strategy to avoid overestimating token relationships and to enhance the robustness of learning. The effectiveness of our approach is demonstrated by the superior performance over others on nine downstream tasks in public datasets.

In this paper, we present a new cross-architecture contrastive learning (CACL) framework for self-supervised video representation learning. CACL consists of a 3D CNN and a video transformer which are used in parallel to generate diverse positive pairs for contrastive learning. This allows the model to learn strong representations from such diverse yet meaningful pairs. Furthermore, we introduce a temporal self-supervised learning module able to predict an Edit distance explicitly between two video sequences in the temporal order. This enables the model to learn a rich temporal representation that compensates strongly to the video-level representation learned by the CACL. We evaluate our method on the tasks of video retrieval and action recognition on UCF101 and HMDB51 datasets, where our method achieves excellent performance, surpassing the state-of-the-art methods such as VideoMoCo [23] and MoCo+BE [34] by a large margin.

Self-supervised anomaly pattern detection for large scale industrial data

Background Respiratory diseases, including chronic obstructive pulmonary disease (COPD), asthma, and pneumonia, are leading causes of global morbidity and mortality. Early and accurate diagnosis remains challenging, especially in resource-limited settings. Automated respiratory sound classification, supported by machine learning and deep learning, has emerged as a non-invasive and cost-effective diagnostic alternative. This systematic review examines the role of multistage classification frameworks and ensemble learning techniques in improving the accuracy, robustness, and clinical applicability of respiratory sound analysis. Methods A comprehensive literature search was conducted across major scientific databases, focusing on studies employing multistage, machine learning, or ensemble-based approaches for respiratory sound classification. Eligible studies were screened based on predefined inclusion criteria related to pulmonary acoustics, feature extraction, signal processing, and classification models. Extracted data were synthesized to compare methodologies, feature engineering strategies, ensemble algorithms, model architectures, benchmark datasets (ICBHI 2017, PhysioNet), and performance metrics such as accuracy, precision, recall, F1-score, and AUC-ROC. Results Findings indicate that multistage classification pipelines significantly enhance model reliability by integrating sequential preprocessing, feature extraction, dimensionality reduction, and classifier fusion. Ensemble learning methods—including bagging, boosting, random forests, stacking, and deep hybrid models—consistently outperform single-stage classifiers in handling noisy, non-stationary respiratory sounds and imbalanced datasets. Deep learning models such as convolutional neural networks and transformer-based architectures show superior performance when combined with advanced feature representations. However, challenges persist related to dataset variability, limited annotations, lack of standardization, and insufficient real-time or wearable-device compatibility. Explainable AI approaches are increasingly incorporated to support clinical interpretability. Conclusion Multistage and ensemble learning techniques demonstrate strong potential to improve automated respiratory sound classification and support early diagnosis of respiratory diseases. Despite promising results, advancements are needed in dataset standardization, multimodal integration, self-supervised learning, and IoT-enabled real-time monitoring. Future research should prioritize clinically interpretable, scalable, and regulation-ready AI systems to enable widespread deployment in respiratory healthcare.

To alleviate the need for large-scale pixel-wise annotations, domain adaptation for semantic segmentation trains segmentation models on synthetic data (source) with computer-generated annotations, which can be then generalized to segment realistic images (target). Recently, self-supervised learning (SSL) with a combination of image-to-image translation shows great effectiveness in adaptive segmentation. The most common practice is to perform SSL along with image translation to well align a single domain (source or target). However, in this single-domain paradigm, unavoidable visual inconsistency raised by image translation may affect subsequent learning. In addition, pseudo labels generated by a single segmentation model aligned in either the source or target domain may be not accurate enough for SSL. In this paper, based on the observation that domain adaptation frameworks performed in the source and target domain are almost complementary, we propose a novel adaptive dual path learning (ADPL) framework to alleviate visual inconsistency and promote pseudo-labeling by introducing two interactive single-domain adaptation paths aligned in source and target domain respectively. To fully explore the potential of this dual-path design, novel technologies such as dual path image translation (DPIT), dual path adaptive segmentation (DPAS), dual path pseudo label generation (DPPLG) and Adaptive ClassMix are proposed. The inference of ADPL is extremely simple, only one segmentation model in the target domain is employed. Our ADPL outperforms the state-of-the-art methods by large margins on GTA5 →Cityscapes, SYNTHIA → Cityscapes and GTA5 →BDD100K scenarios. Code and models are available at https://github.com/royee182/DPL.