Transformers and contrastive semi-supervised learning for medical image segmentation

Semi-supervised learning is a combination of conventional supervised methods with weakly supervised learning. A recent development in neural networks allows to achieve high-quality results but the training requires a large amount of annotated examples. This hinders the applicability of deep learning to some problems, especially medical imaging. In this paper, we present a semi-supervised learning approach based on convolutional neural networks (CNN) for medical image segmentation. A network is trained on a combination of fully labeled samples that have segmentation masks available and weakly labeled samples that only have class labels. We performed experiments that compare the results of the semi-supervised model with the baseline supervised method. Experiment results show the superiority of suggested methods on a low amount of fully annotated samples for lung nodules CT images.

Due to the lack of quality annotation in medical imaging community, semi-supervised learning methods are highly valued in image semantic segmentation tasks. In this paper, an advanced consistency-aware pseudo-label-based self-ensembling approach is presented to fully utilize the power of Vision Transformer (ViT) and Convolutional Neural Network (CNN) in semi-supervised learning. Our proposed framework consists of a feature-learning module which is enhanced by ViT and CNN mutually, and a guidance module which is robust for consistency-aware purposes. The pseudo labels are inferred and utilized recurrently and separately by views of CNN and ViT in the feature-learning module to expand the data set and are beneficial to each other. Meanwhile, a perturbation scheme is designed for the feature-learning module, and averaging network weight is utilized to develop the guidance module. By doing so, the framework combines the feature-learning strength of CNN and ViT, strengthens the performance via dual-view co-training, and enables consistency-aware supervision in a semi-supervised manner. A topological exploration of all alternative supervision modes with CNN and ViT are detailed validated, demonstrating the most promising performance and specific setting of our method on semi-supervised medical image segmentation tasks. Experimental results show that the proposed method achieves state-of-the-art performance on a public benchmark data set with a variety of metrics. The code is publicly available (https://github.com/ziyangwang007/CV-SSL-MIS).

Abstract. Crowd counting, a critical component in the management and safety planning of large gatherings and public spaces, is essential for ensuring smooth event operations and preventing potential overcrowding issues. While the standard convolutional neural network (CNN) based model performs well in head counting tasks, it has certain drawbacks when applied to complex scenarios. With the rapid development of artificial intelligence, Transformer models that rely on self-attention mechanisms, as Swin Transformer, have demonstrated exceptional performance in visual tasks, such as image classification, and segmentation in recent times. This study examines the experimental findings of Swin Transformer's head counting tasks and contrasts them with the CNN-based model. Mean Absolute Error (MAE) and Mean Square Error (MSE) evaluation indicators show that the Transformer model outperforms the classic CNN model in terms of generalization abilities when dealing with complicated scenarios. Future research work will increase the diversity of data sets and focus on optimizing model structure and improving training efficiency.

Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation

Automatic analysis methods of electrocardiograms (ECGs) usually required large-scale annotated training data, but the annotation process is extremely time-consuming. While semi-supervised learning can leverage unlabeled data, its performance depends heavily on the quality of the initial labeled subset. Active learning has been used to identify the most informative samples for annotation, but conventional approaches face three critical limitations: (1) dependency on manual intervention for iterative query design, (2) prohibitive computational costs during sample selection, and (3) limited compatibility with semi-supervised learning frameworks. To address these limitations, we proposed an Unsupervised Active Feature-selective Semi-Supervised Learning (UAFSSL) framework for ECG analysis, including an unsupervised feature selection-based active learning module and a semi-supervised learning module. UAFSSL captures latent data distributions via unsupervised feature extraction, selects diverse and representative samples using pseudo-label clustering, and integrates seamlessly with semi-supervised learning to eliminate human intervention. We validated our algorithm on an ECG waveform segmentation task and an atrial fibrillation detection task. In the waveform segmentation task, our method improved the F1-score for P-wave delineation by 2.4% compared to random sampling, using only 5% of labeled samples. For the atrial fibrillation detection task, we evaluated our method on both the AFDB and a 24-hour dataset collected from 500 atrial fibrillation patients. Using only 200 labeled samples for model training, our method achieved AUC improvements of 2.5% and 2.2% over random sampling in five-fold cross validation. This is the first study to integrate unsupervised active learning with semi-supervised learning for automatic ECG analysis, offering a robust, automated solution to reduce annotation costs while enhancing clinical applicability.

Most recently transformer models became the state of the art in various medical image segmentation tasks and challenges, outperforming most of the conventional deep learning approaches. Picking up on that trend, this study aims at applying various transformer models to the highly challenging task of colorectal cancer (CRC) segmentation in CT imaging and assessing how they hold up to the current state-of-the-art convolutional neural network (CNN), the nnUnet. Furthermore, we wanted to investigate the impact of the network size on the resulting accuracies, since transformer models tend to be significantly larger than conventional network architectures. For this purpose, six different transformer models, with specific architectural advancements and network sizes were implemented alongside the aforementioned nnUnet and were applied to the CRC segmentation task of the medical segmentation decathlon. The best results were achieved with the Swin-UNETR, D-Former, and VT-Unet, each transformer models, with a Dice similarity coefficient (DSC) of 0.60, 0.59 and 0.59, respectively. Therefore, the current state-of-the-art CNN, the nnUnet could be outperformed by transformer architectures regarding this task. Furthermore, a comparison with the inter-observer variability (IOV) of approx. 0.64 DSC indicates almost expert-level accuracy. The comparatively low IOV emphasizes the complexity and challenge of CRC segmentation, as well as indicating limitations regarding the achievable segmentation accuracy. As a result of this study, transformer models underline their current upward trend in producing state-of-the-art results also for the challenging task of CRC segmentation. However, with ever smaller advances in total accuracies, as demonstrated in this study by the on par performances of multiple network variants, other advantages like efficiency, low computation demands, or ease of adaption to new tasks become more and more relevant.

Rapid advancements in artificial intelligence and generative artificial intelligence have enabled the creation of fake images and videos that appear highly realistic. According to a report published in 2022, approximately 71% of people rely on fake videos and become victims of blackmail. Moreover, these fake videos and images are used to tarnish the reputation of popular public figures. This has increased the demand for deepfake detection techniques. The accuracy of the techniques proposed in the literature so far varies with changes in fake content generation techniques. Additionally, these techniques are computationally intensive. The techniques discussed in the literature are based on convolutional neural networks, Linformer models, or transformer models for deepfake detection, each with its advantages and disadvantages. In this manuscript, a hybrid architecture combining transformer and Linformer models is proposed for deepfake detection. This architecture converts an image into patches and performs position encoding to retain spatial relationships between patches. Its encoder captures the contextual information from the input patches, and Gaussian Error Linear Unit resolves the vanishing gradient problem. The Linformer component reduces the size of the attention matrix. Thus, it reduces the execution time to half without compromising accuracy. Moreover, it utilizes the unique features of transformer and Linformer models to enhance the robustness and generalization of deepfake detection techniques. The low computational requirement and high accuracy of 98.9% increase the real-time applicability of the model, preventing blackmail and other losses to the public. The proposed hybrid model utilizes the strength of the transformer model in capturing complex patterns in data. It uses the self-attention potential of the Linformer model and reduces the computation time without compromising the accuracy. Moreover, the models were implemented on patch sizes of 6 and 11. It is evident from the obtained results that increasing the patch size improves the performance of the model. This allows the model to capture fine-grained features and learn more effectively from the same set of videos. The larger patch size also enables the model to better preserve spatial details, which contributes to improved feature extraction.

In medical image segmentation, U-Net and its variants are widely utilized. However, CNN-based models are limited in long-range dependency modeling, while Transformer-based models are constrained by quadratic computational complexity. Recently, state space models (SSMs) such as Mamba have emerged as a promising solution, as they excel at modeling long-range interactions while maintaining linear computational complexity. This paper proposes the CIHM, a hybrid Mamba model that provides contextual insights. Its key component is the context-insight Mamba-CNN layer (CiMC layer). The CiMC layer adopts a hybrid Mamba-CNN approach to process features across different scales and levels in parallel, thereby enhancing the semantic segmentation of medical images. Specifically, the SSM branch of this layer captures long-range dependencies with comprehensive contextual semantic understanding, while the CNN block achieves superior local feature extraction using T-shaped convolution that focuses on patch centers. Moreover, to further aggregate multi-scale contextual information of medical images, we design the multiscale refining detail bridge (MRDB) module. This module significantly improves feature representation through the dense multiplication and concatenation of features for fusion. Comprehensive experiments on the ISIC2017, ISIC2018, Synapse, and DSB18 datasets demonstrate that CIHM exhibits strong competitive performance and efficiency in medical image segmentation tasks. Notably, CIHM achieves better segmentation performance than the well-known U-Net and the modern hybrid network U-Mamba, while reducing the parameter count by 40 times and 345 times, respectively. This demonstrates the potential of our approach to advance model lightweighting.

Eye diseases can affect vision and well-being, so early, accurate diagnosis is crucial to prevent serious impairment. Deep learning models have shown promise for automating the diagnosis of eye diseases from images. However, current methods mostly use single-model architectures, including convolutional neural networks (CNNs), which might not adequately capture the long-range spatial correlations and local fine-grained features required for classification. To address these limitations, this study proposes a multi-stage framework for eye diseases (MST-EDS), including two stages: hybrid and stacking models in the categorization of eye illnesses across four classes: normal, diabetic_retinopathy, glaucoma, and cataract, utilizing a benchmark dataset from Kaggle. Hybrid models are developed based on Transformer models: Vision Transformer (ViT), Data-efficient Image Transformer (DeiT), and Swin Transformer are used to extract deep features from images, Principal Component Analysis (PCA) is used to reduce the complexity of extracted features, and Machine Learning (ML) models are used as classifiers to enhance performance. In the stacking model, the outputs of the best hybrid models are stacked, and they are used to train and evaluate meta-learners to improve classification performance. The experimental results show that the MST-EDS-RF model recorded the best performance compared to individual Transformer and hybrid models, with 97.163% accuracy.

Stroke is a major factor in causing disability and fatalities. Doctors use computerized tomography (CT) and magnetic resonance imaging (MRI) scans to assess the severity of a stroke. Automatic image segmentation can help doctors diagnose strokes more quickly and accurately, but it is challenging due to the variability of stroke lesions and the limited availability of labeled data. Deep learning is the cutting-edge technique of machine learning and artificial intelligence, which needs an extensive labeled dataset for effective training. Unfortunately, in the medical domain, the availability of labeled data is severely limited, posing a challenge for conventional deep- learning approaches. In this article, we introduce a system that utilizes deep learning in the form of fusing transformer-based and convolutional neural network (CNN)-based features and few-shot learning techniques to segment ischemic strokes in multimedia MRIs. To accomplish this, we employ two different methods. The first method involves parallel fusion, where we combine CNN-based and transformer-based features. The second method utilizes serial fusion, combining CNN-based and transformer models using few-shot learning. Through the integration of transformer and CNN models, we can extract both global and local features and enhance the system's performance. Moreover, we tackle the issue of limited labeled data by integrating few-shot learning techniques. Additionally, our system optimizes efficiency by selecting only the slices with lesions, disregarding unlesioned slices. The system under consideration is trained with the BraTS2020 dataset, evaluated on the ISLES 2015 dataset, and contrasted the performance with cutting-edge systems. The suggested system attains a dice coefficient score of 0.76, surpassing the scores of previous cutting-edge systems by a substantial margin.

Semi-supervised deep learning for medical image segmentation is an intriguing area of research as far as the requirement for an adequate amount of labeled data is concerned. In this context, we propose Cross Task Temporal Consistency, a novel Semi-Supervised Learning framework that combines a self-ensembled learning strategy with cross-consistency constraints derived from the implicit perturbations between the incongruous tasks of multi-headed architectures. More specifically, the Signed Distance Map output of a teacher model is transformed to an approximate segmentation map which acts as a pseudo target for the student model. Simultaneously, the teacher’s segmentation task output is utilized as the objective for the student’s Signed Distance Map derived segmentation output. Our proposed framework is intuitively simple and can be plugged into existing segmentation architectures with minimal computational overhead. Our work focuses on improving the segmentation performance in very low-labeled data proportions and has demonstrated marked superiority in performance and stability over existing SSL techniques, as evidenced through extensive evaluations on two standard datasets: ACDC and LA.KeywordsSemi supervised learningConvolutional neural networksMedical image segmentation

Recent advances in deep learning have made a progressive impact on BCI researches. In particular, convolutional neural networks (CNNs) with different architectural forms have been studied for spatio-temporal or spatio-spectral feature representation learning. However, there still remain many challenges and limitations due to the necessity of a large annotated training samples for robustness. In this paper, we propose a semi-supervised deep adversarial learning framework that effectively utilizes generated artificial samples along with labeled and unlabelled real samples in discovering class-discriminative features to boost robustness of a classifier, thus to enhance BCI performance. It is also noteworthy that the proposed framework allows to exploit unlabelled real samples to better uncover the underlying patterns inherent in a user’s EEG signals. In order to justify the validity of the proposed framework, we conducted exhaustive experiments with ‘Recurrent Spatio-Temporal Neural Network’ CNN architectures over the public BCI Competition IV-IIa dataset. From our experiments, we could observe statistically significant improvements on performance, compared to the competing methods with the conventional framework. We have also visualized learned convolutional weights in terms of activation pattern maps, separability of extracted features, and validity of generated artificial samples.

Handwritten mathematical expression recognition (HMER) poses a formidable challenge due to the intricate two-dimensional structures and diverse handwriting styles. This paper introduces a novel approach to improve HMER accuracy by employing an integrated, high-capacity architecture that combines Transformer and Convolutional Neural Network (CNN) models, along with a denoising diffusion probabilistic model (DDPM)-based data augmentation technique. We explore three combination strategies for an attention-based encoder-decoder (AED) HMER model: 1) The “Tandem” strategy, which harnesses CNN features within a Transformer encoder to capture global interdependencies; 2) The “Parallel” strategy, which integrates Transformer encoder outputs with CNN outputs to achieve comprehensive feature fusion; 3) The “Mixing” strategy, which introduces multi-head self-attention (MHSA) at the final stage of the CNN. We evaluate these methods using the CROHME benchmark dataset and conduct a detailed comparative analysis. All three approaches significantly enhance model performance. Notably, the “Tandem” approach achieves expression recognition rates (ExpRate) of 54.85% and 58.56% on the CROHME 2016 and 2019 test sets, respectively, while the “Parallel” method attains 55.63% and 57.39% on the same test sets. Furthermore, we introduce an innovative data augmentation approach that utilizes DDPM to generate synthetic training samples. The DDPM, conditioned on LaTeX-rendered images, bridges the gap between printed and handwritten expressions, enabling the creation of realistic, stylistically diverse handwriting samples. This augmentation boosts the ExpRates of all strategies on both CROHME 2016 and 2019 test sets, yielding improvements of 1.6-4.6% relative to the unaugmented dataset.

Medical image segmentation using deep learning typically requires a large quantity of well-annotated data. However, the acquisition of pixel-level annotations is arduous and expensive, often requiring the expertise of experienced medical professionals. Recent advancements in few-shot learning can help address label scarcity in medical image segmentation by leveraging a small amount of labeled data. Most research on prototypical learning for medical image segmentation involves prototypes averaged across the entire class, ignoring fine details. In this work, we propose a novel dual prototype network and introduce part prototypes to supplement local prototypes for extracting fine-grained features and enhancing model performance in one-shot medical image segmentation. Extensive experiments on the CHAOS dataset demonstrate the effectiveness of the proposed method on one-shot segmentation, outperforming other methods by a significant margin.Clinical relevance- The proposed method can address the problem of prototypical self-supervised learning in the one-shot medical image segmentation scenario, saving annotation costs in clinical practice.

Medical image segmentation is crucial for clinical diagnosis and treatment planning, but it requires models to capture fine local details and understand global contextual relationships. Although traditional convolutional neural networks (CNNs) perform well in extracting local features, their inherent local receptive fields limit their ability to model long-range dependencies. On the other hand, although Transformer models excel at capturing global information through self attention mechanisms, they may lose low-level spatial details that are crucial for accurate segmentation when processing images. To overcome these challenges, this study proposes a novel parallel branch architecture called Trans CNN Fusion Network (TCF Net).This architecture synergistically combines a CNN encoder and a Swin Transformer encoder to extract both local texture features and global semantic information in parallel. The core innovation we designed is a Multi Scale Attention Fusion (MSAF) module that can intelligently and adaptively merge feature streams from two parallel branches, significantly outperforming simple feature concatenation methods. In addition, to address the common issue of class imbalance in medical images, we employed a hybrid Dice Focal loss function for model optimization. The experimental results on the publicly available BraTS 2021 brain tumor segmentation dataset show that TCF Net outperforms existing baseline models in all evaluation metrics, with an average Dice coefficient improved by 3.5% compared to TransUNet, demonstrating its outstanding performance and effectiveness in segmenting complex tumor subregions. This work provides a powerful new paradigm for high-precision automated medical image analysis.