Self-supervised learning and transformer-based technologies in breast cancer imaging

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

POPAR: Patch Order Prediction and Appearance Recovery for self-supervised learning in chest radiography.

Our long-term goal was to ensure nurse clinical decision support works as intended before full deployment in clinical practice. As part of a broader effort, this pilot project explored factors influencing acceptance/nonacceptance of eight clinical decision support suggestions displayed in an electronic health record-based nursing plan of care software prototype. A diverse sample of 21 nurses participated in this high-fidelity clinical simulation experience and completed a questionnaire to assess reasons for accepting/not accepting the clinical decision support suggestions. Of 168 total suggestions displayed during the experiment (eight for each of the 21 nurses), 123 (73.2%) were accepted, and 45 (26.8%) were not accepted. The mode number of acceptances by nurses was seven of eight, with only two of 21 nurses accepting all. The main reason for clinical decision support acceptance was the nurse's belief that the suggestions were good for the patient (100%), with other features providing secondary reinforcement. Reasons for nonacceptance were less clear, with fewer than half of the subjects indicating low confidence in the evidence. This study provides preliminary evidence that high-quality simulation and targeted questionnaires about specific clinical decision support selections offer a cost-effective means for testing before full deployment in clinical practice.

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

The integration of artificial intelligence (AI) in cancer research has significantly advanced radiology, pathology, and multimodal approaches, offering unprecedented capabilities in image analysis, diagnosis, and treatment planning. AI techniques provide standardized assistance to clinicians, in which many diagnostic and predictive tasks are manually conducted, causing low reproducibility. These AI methods can additionally provide explainability to help clinicians make the best decisions for patient care. This review explores state-of-the-art AI methods, focusing on their application in image classification, image segmentation, multiple instance learning, generative models, and self-supervised learning. In radiology, AI enhances tumor detection, diagnosis, and treatment planning through advanced imaging modalities and real-time applications. In pathology, AI-driven image analysis improves cancer detection, biomarker discovery, and diagnostic consistency. Multimodal AI approaches can integrate data from radiology, pathology, and genomics to provide comprehensive diagnostic insights. Emerging trends, challenges, and future directions in AI-driven cancer research are discussed, emphasizing the transformative potential of these technologies in improving patient outcomes and advancing cancer care. This article is part of a special series: Driving Cancer Discoveries with Computational Research, Data Science, and Machine Learning/AI.

cancer remains a leading cause of morbidity and mortality worldwide, with nearly 20 million new cases and 9.7 million deaths reported in 2022. The increasing burden of cancer, driven by population growth and aging, necessitates innovative solutions to improve diagnosis, prognosis, and treatment outcomes. Artificial Intelligence has emerged as a transformative tool in oncology, offering significant potential in cancer detection, diagnosis, and personalized treatment strategies. This review explores the real-world applications of Artificial Intelligence in oncology, focusing on lung cancer and breast cancer, two of the most prevalent and deadly cancers globally. Artificial Intelligence-driven technologies, particularly in imaging, pathology, and genomics, have demonstrated remarkable success in enhancing early detection, diagnostic accuracy, and treatment planning. In lung cancer, Artificial Intelligence-powered imaging tools, such as deep learning models, have shown high sensitivity and specificity in detecting small pulmonary nodules, often missed by traditional methods. Similarly, in breast cancer, Artificial Intelligence has proven effective in mammography interpretation, reducing false positives and false negatives, and alleviating the workload of radiologists. Despite its promising potential, the integration of Artificial Intelligence into clinical practice faces several challenges, including issues related to data quality, algorithmic biases, and ethical considerations. The "black box" nature of many Artificial Intelligence systems poses a significant barrier to clinical acceptance, highlighting the need for explainable Artificial Intelligence to provide transparent and interpretable decision-making processes. Furthermore, the successful implementation of Artificial Intelligence in oncology requires robust regulatory frameworks and standardized protocols to ensure patient safety and data security. This review underscores the transformative potential of Artificial Intelligence in revolutionizing cancer care, emphasizing the importance of addressing key challenges to harness its full potential. By enhancing early detection, reducing diagnostic errors, and enabling personalized treatment strategies, Artificial Intelligence has the potential to significantly improve patient outcomes and reduce the global burden of cancer. However, its successful integration into clinical practice will depend on interdisciplinary collaboration, ethical considerations, and a commitment to responsible implementation.

The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making. To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox. A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot. Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model. The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey. A total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, -13.87; 95% CI, -17.38 to -9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, -0.33; 95% CI, -0.50 to -0.17; P < .001). In this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.

Artificial intelligence leverages sophisticated computation and inference to generate insights, enables the system to reason and learn, and empowers decision making of clinicians. Starting from data (medical images, biomarkers, patients’ data) and using powerful tools such as convolutional neural networks, classification, and regression models etc., it aims at creating personalized models, adapted to each patient, which can be applied in real clinical practice as a decision support system to doctors. This chapter discusses the use of AI in medicine, with an emphasis on the classification of patients with carotid artery disease, evaluation of patient conditions with familiar cardiomyopathy, and COVID-19 models (personalized and epidemiological). The chapter also discusses model integration into a cloud-based platform to deal with model testing without any special software needs. Although AI has great potential in the medical field, the sociological and ethical complexity of these applications necessitates additional analysis, evidence of their medical efficacy, economic worth, and the creation of multidisciplinary methods for their wider deployment in clinical practice.

Deep learning analysis of radiological images has the potential to improve diagnostic accuracy of breast cancer, ultimately leading to better patient outcomes. This paper systematically reviewed the current literature on deep learning detection of breast cancer based on magnetic resonance imaging (MRI). The literature search was performed from 2015 to Dec 31, 2022, using Pubmed. Other database included Semantic Scholar, ACM Digital Library, Google search, Google Scholar, and pre-print depositories (such as Research Square). Articles that were not deep learning (such as texture analysis) were excluded. PRISMA guidelines for reporting were used. We analyzed different deep learning algorithms, methods of analysis, experimental design, MRI image types, types of ground truths, sample sizes, numbers of benign and malignant lesions, and performance in the literature. We discussed lessons learned, challenges to broad deployment in clinical practice and suggested future research directions.

Background and AimsThe 12-lead electrocardiogram (ECG) remains a cornerstone of cardiac diagnostics, yet existing artificial intelligence (AI) solutions for automated interpretation often lack generalizability, remain closed source, and are primarily trained using supervised learning (SL), which requires extensive labelled datasets and may limit adaptability across diverse clinical settings. Self-supervised learning (SSL) can potentially overcome these limitations by learning robust representations from unlabelled data. To address these challenges, this study developed and compared two open-source foundational ECG models: DeepECG-SL, a supervised multilabel ECG model, and DeepECG-SSL, a self-supervised model.MethodsBoth models were trained on over 1 million ECGs using a standardized preprocessing pipeline and automated free-text extraction from ECG reports to predict 77 cardiac conditions. DeepECG-SSL leveraged unlabelled data through self-supervised contrastive learning and masked lead modelling before fine-tuning for downstream tasks, while DeepECG-SL was trained directly on labelled diagnostic data in an end-to-end fashion. Performance was evaluated across seven private, multilingual healthcare systems and four public ECG repositories, with assessment of fairness by age and sex, and investigation of privacy vulnerabilities as well as memory and compute requirements.ResultsDeepECG-SSL achieved micro-averaged area under the receiver operating characteristic curves (AUROCs) across all 77 cardiac conditions for ECG interpretation of 0.990 [95% confidence interval (CI): 0.990, 0.990] on the internal dataset (MHI-ds), 0.981 (95% CI: 0.981, 0.981) on external public datasets (UKB, CLSA, MIMIC-IV and PTB), and 0.983 (95% CI: 0.983, 0.983) on external private datasets (UW, UCSF, JGH, NYP, MGH, CSH and CHUM), while DeepECG-SL demonstrated AUROCs of 0.992 (95% CI: 0.992, 0.992), 0.980 (95% CI: 0.980, 0.980), and 0.983 (95% CI: 0.983, 0.984), respectively. Fairness analyses revealed minimal disparities (true-positive rate and false-positive rate difference <0.1) across age and sex groups for both models. DeepECG-SSL demonstrated superior performance on limited-data digital biomarker tasks, with the largest improvements in long QT syndrome (LQTS) genotype classification (AUROC 0.931 vs 0.850, P = .026, n = 127 ECGs) and 5 year atrial fibrillation risk prediction (AUROC 0.742 vs 0.734, P < 0.001, n = 132 050 ECGs), while achieving superior performance in left ventricular ejection fraction ≤40% classification (AUROC 0.926 vs 0.917, P < 0.001, n = 25 252 ECGs) and comparable performance in LQTS detection (AUROC 0.767 vs 0.735, P = 0.117, n = 934 ECGs).ConclusionsThis study establishes SSL as a promising paradigm for ECG analysis, particularly in settings with limited annotated data, enhancing accessibility, generalizability, and fairness in AI-driven cardiac diagnostics. By releasing model weights, preprocessing tools, and validation code, this work aims to support robust, data-efficient AI diagnostics across diverse clinical environments and questions.

The artificial intelligence (AI) community has recently made tremendous progress in developing self-supervised learning (SSL) algorithms that can learn high-quality data representations from massive amounts of unlabeled data. These methods brought great results even to the fields outside of AI. Due to the joint efforts of researchers in various areas, new SSL methods come out daily. However, such a sheer number of publications make it difficult for beginners to see clearly how the subject progresses. This survey bridges this gap by carefully selecting a small portion of papers that we believe are milestones or essential work. We see these researches as the “dots” of SSL and connect them through how they evolve. Hopefully, by viewing the connections of these dots, readers will have a high-level picture of the development of SSL across multiple disciplines including natural language processing, computer vision, graph learning, audio processing, and protein learning.

Breast cancer (BC) is one of the leading causes of malignancy among women, and its prevalence is exponentially rising globally. Early and accurate imaging is critical for early detection, diagnosis, and treatment planning. This comprehensive review explores the current status of BC imaging, from the conventional methods such as mammography, ultrasound (US) and magnetic resonance imaging (MRI) to more advanced techniques including contrast-enhanced imaging, tomosynthesis, and molecular breast imaging (MBI). Conventional imaging remains the foundation for screening, as mammography is the most widely preferred modality. US and MRI are usually employed in dense breasts in highly suspicious cases that are not detected on a mammogram. However, the limitations posed by these traditional techniques can be curtailed using advanced modalities to enhance diagnostic accuracy. These emerging techniques provide faster and earlier detection of malignancy, particularly in high-risk patients, and substantially reduce the burden of missed cases. Emerging technologies, including photoacoustic imaging (PAI) and contrast-enhanced ultrasound (CEUS), show promising potential in visualizing microvascular structures and enhancing diagnostic accuracy. Additionally, artificial intelligence (AI) is revolutionizing BC imaging across all modalities by optimizing interpretation, enhancing sensitivity, and enabling personalized risk assessment.Although technological innovation continues to improve imaging quality and diagnostic precision, challenges such as cost, accessibility, overdiagnosis, and disparities in care remain a concern. Moving forward, a collaborative multimodal strategy that incorporates personalized imaging protocols and equitable access will be crucial for improving BC screening and management. The future of breast imaging lies not in replacing existing modalities but in developing a system where each technology complements the other, leading to earlier detection, more effective treatment, and enhanced outcomes.

18-FDG imaging in breast cancer

PurposeThe purpose of this study was to optimize intra‐voxel incoherent motion (IVIM) measurement in diffusion‐weighted imaging (DWI) of breast cancer by separating perfusion and diffusion effects through the determination of an optimal threshold b‐value, thus benign and cancerous breast tissues can be accurately differentiated using IVIM‐derived diffusion and perfusion parameters.Materials and MethodsTwenty‐eight patients, with biopsy‐confirmed breast cancers, were studied with a 3T MRI scanner, using T1‐weighted dynamic contrast‐enhanced MRI images, and diffusion‐weighted images with nine b‐values, ranging from 0 to 1000 s/mm². IVIM‐derived parameter maps for tissue diffusion coefficients D, perfusion fraction f, and pseudo‐diffusion coefficients D* were computed using the segmented fitting method with optimized threshold b‐value, and the sum of squared residuals (SSR) were calculated for IVIM‐derived parameters in different breast lesions.ResultsThe IVIM analysis method developed in this work can separate perfusion and diffusion effects with the optimal threshold b‐value of 300 s/mm², and the results of diffusion and perfusion parameters from IVIM analysis can be used to differentiate pathological changes in breast tissues. It was found that the averages and standard deviations of the diffusion and perfusion parameters, D, f, D*, are the following, for malignant, benign and normal breast tissues respectively: D (0.813 ± 0.225 × 10−3 mm2/s, 1.437 ± 0.538 × 10−3 mm2/s, 1.838 ± 0.213 × 10−3 mm2/s), f (10.73 ± 3.44%, 7.86 ± 3.70%, 8.92 ± 3.72%), D* (15.23 ± 12.17×10−3 mm²/s, 12.02 ± 3.19 × 10−3 mm2/s, 12.03 ± 7.21 × 10−3 mm2/s).Conclusion IVIM‐derived diffusion and perfusion parameter maps depend highly on the choice of threshold b‐value. Using the methodology developed in this work, and with the optimized threshold b‐value, the diffusion and perfusion parameters of breast tissues can be accurately assessed, making IVIM MRI a technique of choice for differential diagnosis of breast cancer.

Self-supervised learning (SSL) represents a revolutionary AI paradigm which lets machines acquire significant data representations directly from unlabeled information through unsupervised learning approaches. SSL uses contrastive learning and masked data modeling and predictive learning approaches to optimize data efficiency thereby improving model generalization between multiple domains. This paper evaluates the core concepts of SSL alongside its superiority to supervised and unsupervised learning and its usage in different fields such as NLP, computer vision, speech recognition, healthcare, finance and robotics. The paper focuses on analysis of essential techniques and architectures which include SimCLR, MoCo, BERT, MAE, BYOL and approaches combining SSL with reinforcement learning and weak supervision methods. The research analyzes SSL's current challenges including operational expenses and representation degeneration as well as the assessment obstructions while proposing future uses for the method in mixed-data learning and minimal-resource contexts and artificial general intelligence (AGI). The adoption of SSL in real-world AI applications depends on effectively dealing with ethical matters that include bias issues and responsible AI practices and fairness assurance.