Contrastive image adaptation for acquisition shift reduction in medical imaging

Generalising deep models to new data from new centres (termed here domains) remains a challenge. This is largely attributed to shifts in data statistics (domain shifts) between source and unseen domains. Recently, gradient-based meta-learning approaches where the training data are split into meta-train and meta-test sets to simulate and handle the domain shifts during training have shown improved generalisation performance. However, the current fully supervised meta-learning approaches are not scalable for medical image segmentation, where large effort is required to create pixel-wise annotations. Meanwhile, in a low data regime, the simulated domain shifts may not approximate the true domain shifts well across source and unseen domains. To address this problem, we propose a novel semi-supervised meta-learning framework with disentanglement. We explicitly model the representations related to domain shifts. Disentangling the representations and combining them to reconstruct the input image allows unlabeled data to be used to better approximate the true domain shifts for meta-learning. Hence, the model can achieve better generalisation performance, especially when there is a limited amount of labeled data. Experiments show that the proposed method is robust on different segmentation tasks and achieves state-of-the-art generalisation performance on two public benchmarks.

Blood cell morphology assessment via light microscopy constitutes a cornerstone of haematological diagnostics, providing crucial insights into diverse pathological conditions. This complex task demands expert interpretation owing to subtle morphological variations, biological heterogeneity and technical imaging factors that obstruct automated approaches. Conventional machine learning methods using discriminative models struggle with domain shifts, intraclass variability and rare morphological variants, constraining their clinical utility. We introduce CytoDiffusion, a diffusion-based generative classifier that faithfully models the distribution of blood cell morphology, combining accurate classification with robust anomaly detection, resistance to distributional shifts, interpretability, data efficiency and uncertainty quantification that surpasses clinical experts. Our approach outperforms state-of-the-art discriminative models in anomaly detection (area under the curve, 0.990 versus 0.916), resistance to domain shifts (0.854 versus 0.738 accuracy) and performance in low-data regimes (0.962 versus 0.924 balanced accuracy). In particular, CytoDiffusion generates synthetic blood cell images that expert haematologists cannot distinguish from real ones (accuracy, 0.523; 95% confidence interval: [0.505, 0.542]), demonstrating good command of the underlying distribution. Furthermore, we enhance model explainability through directly interpretable counterfactual heat maps. Our comprehensive evaluation framework establishes a multidimensional benchmark for medical image analysis in haematology, ultimately enabling improved diagnostic accuracy in clinical settings.

The effect of loss function on conditional generative adversarial networks

Supervised deep learning methods have produced state-of-the-art results with large labeled datasets. However, accessing large labeled datasets is difficult in medical image analysis because of a shortage of medical experts, expensive annotations, and privacy constraints in the healthcare domain. Self-supervised learning is a branch of machine learning that exploits unlabeled data to encourage network weights toward a valid latent representation of the data during a so-called pretext task. The features learned by the model while solving pretext tasks are transferred to a downstream task where limited annotations are available. In this work, we propose PatchLoc, a novel pretext task whose objective is to find the location of a given patch from an image as a source of supervision. We validated the effectiveness of PatchLoc on a downstream segmentation task using three different medical datasets. PatchLoc yields substantial improvements compared to U-Net trained from scratch and other pretext task-based approaches in a low data regime.

Domain shift is a significant challenge in medical image segmentation, primarily due to variations in image acquisition protocols, modalities, etc. Domain shift often causes models trained on a source domain to perform poorly on unseen target domains. In this work, we introduce the Domain-Generalized Discrete Diffusion Model for Segmentation (DG-DDM-Seg), a diffusion-based generative model designed for single-source domain generalization in medical image segmentation. DG-DDM-Seg generates discrete conditional distributions of segmentation masks. To ensure domain independence, we employ two key strategies: 1) We extract robust features from conditional images to enhance the domain independence of diffusion model. 2) We use both conditional images and pseudo-labels as inputs to improve cross-domain segmentation performance. Along this idea, we propose a two-path reverse diffusion process during training, utilizing Robust Feature Extraction Subnet and Mask-Generation Transformer to learn a domain-generalized discrete conditional distribution based on robust image features and pseudo-labels. This learned distribution is then used to generate segmentation masks for unseen target domains. Experimental results demonstrate that DG-DDM-Seg achieves state-of-the-art performance in cross-domain medical image segmentation, with domain shifts in modality, sequence, and site. The code is available at https://github.com/HeranYang/DG-DDM-Seg.

Medical image analysis plays a pivotal role in modern healthcare, aiding clinicians in accurate diagnosis and treatment planning. However, the complexity and diversity of medical images pose significant challenges for traditional image processing methods. Existing methods often struggle to precisely delineate structures in medical images, leading to suboptimal diagnostic accuracy. The demand for automated and accurate segmentation tools in medical imaging has grown, highlighting the necessity for robust and efficient algorithms capable of handling diverse anatomical variations and pathologies. While CNNs have shown promise in image analysis, their application to medical images requires customization to accommodate unique challenges. The literature lacks comprehensive studies that bridge the gap between general-purpose CNNs and the specific demands of medical image segmentation, especially concerning the diverse and intricate structures present in medical imagery. This study addresses the need for advanced techniques by leveraging Convolutional Neural Networks (CNNs) for semantic segmentation in medical image analysis. Our approach involves the design and implementation of a specialized CNN architecture tailored to the nuances of medical image data. We employ state-of-the-art techniques for data preprocessing, model training, and validation. The model is trained on a diverse dataset encompassing various medical imaging modalities, ensuring its adaptability and generalizability. The proposed CNN-based semantic segmentation model demonstrates superior performance in accurately delineating anatomical structures compared to traditional methods. Evaluation metrics, including Dice coefficient and sensitivity, indicate the model efficacy in achieving precise segmentation. The results underscore the potential of CNNs in advancing medical image analysis for improved clinical outcomes.

Medical image segmentation tasks hitherto have achieved excellent progresses with large-scale datasets, which empowers us to train potent deep convolutional neural networks (DCNNs). However, labeling such large-scale datasets is laborious and error-prone, which leads the noisy (or incorrect) labels to be an ubiquitous problem in the real-world scenarios. In addition, data collected from different sites usually exhibit significant data distribution shift (or domain shift). As a result, noisy label and domain shift become two common problems in medical imaging application scenarios, especially in medical image segmentation, which degrade the performance of deep learning models significantly. In this paper, we identify a novel problem hidden in medical image segmentation, which is unsupervised domain adaptation on noisy labeled data, and propose a novel algorithm named "Self-Cleansing Unsupervised Domain Adaptation" (S-CDUA) to address such issue. S-CUDA sets up a realistic scenario to solve the above problems simultaneously where training data (i.e., source domain) not only shows domain shift w.r.t. unsupervised test data (i.e., target domain) but also contains noisy labels. The key idea of S-CUDA is to learn noise-excluding and domain invariant knowledge from noisy supervised data, which will be applied on the highly corrupted data for label cleansing and further data-recycling, as well as on the test data with domain shift for supervised propagation. To this end, we propose a novel framework leveraging noisy-label learning and domain adaptation techniques to cleanse the noisy labels and learn from trustable clean samples, thus enabling robust adaptation and prediction on the target domain. Specifically, we train two peer adversarial networks to identify high-confidence clean data and exchange them in companions to eliminate the error accumulation problem and narrow the domain gap simultaneously. In the meantime, the high-confidence noisy data are detected and cleansed in order to reuse the contaminated training data. Therefore, our proposed method can not only cleanse the noisy labels in the training set but also take full advantage of the existing noisy data to update the parameters of the network. For evaluation, we conduct experiments on two popular datasets (REFUGE and Drishti-GS) for optic disc (OD) and optic cup (OC) segmentation, and on another public multi-vendor dataset for spinal cord gray matter (SCGM) segmentation. Experimental results show that our proposed method can cleanse noisy labels efficiently and obtain a model with better generalization performance at the same time, which outperforms previous state-of-the-art methods by large margin. Our code can be found at https://github.com/zzdxjtu/S-cuda.

While deep learning models become more widespread, their ability to handle unseen data and generalize for any scenario is yet to be challenged. In medical imaging, there is a high heterogeneity of distributions among images based on the equipment that generates them and their parametrization. This heterogeneity triggers a common issue in machine learning called domain shift, which represents the difference between the training data distribution and the distribution of where a model is employed. A high domain shift often results in a poor generalization performance from the models. In this work, we evaluate the extent of which domain shift damages model performance on four of the largest datasets of chest radiographs. We show how training and testing with different datasets (e.g., training in ChestX-ray14 and testing in CheXpert) drastically affects model performance, posing a big question over the reliability of deep learning models trained on public datasets. We also show that models trained on CheXpert and MIMIC-CXR generalized better to other datasets.KeywordsDeep learningDomain shiftChest radiographs

This dissertation explores the critical role of loss functions in enhancing the predictive performance of deep machine learning models. Loss functions are an integral element of all the ongoing advances we witness daily in this domain. I design custom loss functions and their impacts on various machine learning tasks, particularly in computer vision. In the first stage of my research, I aim to improve the prediction performance of deep learning models by providing them with more precise feedback associated with task requirements. This led me to create the concept of assistive loss functions. My first proposed loss function, inspired by the Active Shape Model (ASM), is ASM Loss, designed to improve the detection of facial landmarks and head pose accuracy in facial images. Additionally, influenced by Knowledge Distillation (KD), I introduce KD-Loss, a loss designed specifically for regression tasks, focusing on detecting facial landmarks. Proceeding to the second phase of my research, I focus on regression problems. Building upon my prior work, I introduce Adaptive Coordinate-based Regression (ACR)-Loss, a context-aware loss function with face alignment implication. I provide a mechanism to dynamically measure how hard the localization of each landmark point is, and accordingly, ACR-Loss provides unique feedback to the model. Furthermore, I introduce Intensity-aware Categorical (IC)-Loss for detecting sagittal cervical spine landmark points in medical images. It effectively deploys characteristics from classification problems to enhance regression optimization. Both ACR-Loss and IC-Loss comprise contextual information, leading to a significant enhancement in localization precision. In the third phase, I turned my attention toward classification problems within the domain of deep machine learning. I introduce Adaptive Correlation-based (Ad-corre) Loss, designed to enhance the discriminative capabilities of deep learning models, with a specific focus on improving classification accuracy in automatic facial expression recognition. Furthermore, I introduce Informative (Info) Loss, a specialized loss function developed for medical applications, specifically aimed at distinguishing between Mild Cognitive Impairment and Normal Cognition groups based on transcriptions of interviews conducted as part of the I-CONECT study. Overall, I have proposed custom loss functions to improve the predictive performance in both regression and classification problems. Furthermore, I have introduced assistive loss functions, designed to enhance the performance of the main loss functions. The collective outcomes of this extensive exploration highlight the critical role of loss functions, adapting and optimizing them for various domains in deep machine learning. This research suggests and proposes a variety of methodologies for designing adaptive and context-aware loss functions within deep machine learning.

Deep learning techniques have achieved significant advancements in medical image segmentation in recent years. However, model generalization remains severely constrained by domain shift, particularly in cross-modal medical image segmentation tasks. Traditional segmentation models struggle to generalize effectively to unseen target domains due to differences in joint probability distributions across medical imaging modalities. Existing methods primarily focus on unsupervised domain adaptation (UDA) and domain generalization (DG) techniques. While UDA methods face practical limitations due to difficulties in obtaining target domain data, current DG approaches often overlook inherent anatomical priors in medical images as well as the heterogeneity and sparsity of lesion regions. To address these challenges, this paper proposes a cross-modal medical image segmentation framework that integrates the Vision Mamba model with dynamic domain generalization. The framework achieves cross-domain feature alignment and multi-scale feature fusion by leveraging bidirectional state-space sequence modeling, Bezier curve-style enhancement, and a dual-normalization strategy. Additionally, the VEBlock module is introduced, which effectively combines the dynamic sequence modeling capabilities of the Mamba model with non-local attention mechanisms to better capture cross-modal global dependencies. Experimental results on the BraTS 2018 and cross-modal cardiac datasets demonstrate significant improvements in cross-modal segmentation. For example, in T2 → T1 tasks, our framework achieves an average Dice score of 56.22%, outperforming baseline methods by 1.78% while reducing Hausdorff distance for tumor boundaries to 13.26mm. Furthermore, in the Cardiac CT → MRI tasks, the Hausdorff distance is optimized to 27.34mm, validating the proposed framework's strong generalization capability for complex anatomical structures.

TC-Net: A joint learning framework based on CNN and vision transformer for multi-lesion medical images segmentation

Automated, anatomically coherent retinal layer segmentation in optical coherence tomography (OCT) is one of the most important components of retinal disease management. However, current methods rely on large amounts of labeled data, which can be difficult and expensive to obtain. In addition, these systems tend often propose anatomically impossible results, which undermines their clinical reliability. This study introduces a semi-supervised approach to retinal layer segmentation that leverages large amounts of unlabeled data and anatomical prior knowledge related to the structure of the retina. During training, we use a novel topological engine that converts inferred retinal layer boundaries into pixel-wise structured segmentations. These compose a set of anatomically valid disentangled representations which, together with predicted style factors, are used to reconstruct the input image. At training time, the retinal layer boundaries and pixel-wise predictions are both guided by reference annotations, where available, but more importantly by innovatively exploiting anatomical priors that improve the performance, robustness and coherence of the method even if only a small amount of labeled data is available. Exhaustive experiments with respect to label efficiency, contribution of unsupervised data and robustness to different acquisition settings were conducted. The proposed method showed state of-the-art performance on all the studied public and internal datasets, specially in low annotated data regimes. Additionally, the model was able to make use of unlabeled data from a different domain with only a small performance drop in comparison to a fully-supervised setting. A novel, robust, label-efficient retinal layer segmentation method was proposed. The approach has shown state-of-the-art layer segmentation performance with a fraction of the training data available, while at the same time, its robustness against domain shift was also shown.

• Efficiently transfers zero-shot capabilities from CLIP to pre-trained vision encoders • Improves performance over previous SOTA work in domain • Training data diversity/coverage improves mapping quality and zero-shot performance • Increased data coverage achieved through the use of tailored loss functions • Training data is entirely unlabelled and unpaired image and text data Foundation models like CLIP demonstrate exceptional capabilities over a broad domain of knowledge, such as with zero-shot classification; however, they also require significant computational resources, narrowing their real-world utility. Recent studies have shown that mapping features from pre-trained vision encoders into CLIP’s latent space can transfer some of CLIP’s abilities to smaller vision encoders, offering a promising alternative. Yet, the performance of these vision encoders still falls short of CLIP’s native capabilities, particularly in low-data regimes. In this work, we argue that enhancing training data coverage/diversity significantly improves mapping efficacy. We achieve this using tailored loss functions rather than relying on data augmentation or increasing training samples. For instance, we exploit the inherent multimodal nature of CLIP’s latent space, by incorporating cycle-consistency loss as one of our loss functions. Moreover, the mapping is learned using entirely unlabelled and unpaired data, eliminating the need for manual labelling or data pairing in novel domains. From these findings, our resulting method (Zoom-shot) offers a viable path to flexible zero-shot models for resource-limited, data-scarce settings. We test Zoom-shot’s zero-shot performance across various pre-trained vision encoders on coarse- and fine-grained datasets and achieve superior performance compared to recent works. In our ablations, we find Zoom-shot allows for a trade-off between data and compute during training; allowing for a significant reduction in required training data. All code and models are available on GitHub.

Meta-learning methods aim to build learning algorithms capable of quickly adapting to new tasks in low-data regime. One of the most difficult benchmarks of such algorithms is a one-shot learning problem. In this setting many algorithms face uncertainties associated with limited amount of training samples, which may result in overfitting. This problem can be resolved by providing additional information to the model. One of the most efficient ways to do this is multi-task learning. In this paper we investigate the modification of a standard meta-learning pipeline. The proposed method simultaneously utilizes information from several meta-training tasks in a common loss function. The impact of these tasks in the loss function is controlled by a per task weight. Proper optimization of the weights can have big influence on training and the final quality of the model. We propose and investigate the use of methods from the family of Simultaneous Perturbation Stochastic Approximation (SPSA) for optimization of meta-train tasks weights. We also demonstrate superiority of stochastic approximation in comparison to gradient-based method. The proposed Multi-Task Modification can be applied to almost all meta-learning methods. We study applications of this modification on Model-Agnostic Meta-Learning and Prototypical Network algorithms on CIFAR-FS, FC100, miniImageNet and tieredImageNet one-shot learning benchmarks. During these experiments Multi-Task Modification has demonstrated improvement over original methods. SPSA-Tracking algorithm first adapted in this paper for multi-task weight optimization shows the largest accuracy boost that is competitive to the state-of-the-art meta-learning methods.

Classification task is one of the most common tasks in machine learning. This supervised learning problem consists in assigning each input to one of a finite number of discrete categories. Classification task appears naturally in numerous applications, such as medical image processing, speech recognition, maintenance systems, accident detection, autonomous driving etc.In the last decade methods of deep learning have proven to be extremely efficient in multiple machine learning problems, including classification. Whereas the neural network architecture might depend a lot on data type and restrictions posed by the nature of the problem (for example, real-time applications), the process of its training (i.e. finding model’s parameters) is almost always presented as loss function optimization problem.Cross-entropy is a loss function often used for multiclass classification problems, as it allows to achieve high accuracy results.Here we propose to use a generalized version of this loss based on Renyi divergence and entropy. We remark that in case of binary labels proposed generalization is reduced to cross-entropy, thus we work in the context of soft labels. Specifically, we consider a problem of image classification being solved by application of convolution neural networks with mixup regularizer. The latter expands the training set by taking convex combination of pairs of data samples and corresponding labels. Consequently, labels are no longer binary (corresponding to single class), but have a form of vector of probabilities. In such settings cross-entropy and proposed generalization with Renyi divergence and entropy are distinct, and their comparison makes sense.To measure effectiveness of the proposed loss function we consider image classification problem on benchmark CIFAR-10 dataset. This dataset consists of 60000 images belonging to 10 classes, where images are color and have the size of 32×32. Training set consists of 50000 images, and the test set contains 10000 images.For the convolution neural network, we follow [1] where the same classification task was studied with respect to different loss functions and consider the same neural network architecture in order to obtain comparable results.Experiments demonstrate superiority of the proposed method over cross-entropy for loss function parameter value α < 1. For parameter value α > 1 proposed method shows worse results than cross-entropy loss function. Finally, parameter value α = 1 corresponds to cross-entropy.