Clinician-Driven AI: Code-Free Self-Training on Public Data for Diabetic Retinopathy Referral

Real-World Surveillance of FDA-Cleared Artificial Intelligence Models: Rationale and Logistics.

To enhance the performance of machine learning (ML) models for the post-embolization recanalization of cerebral aneurysms, we evaluated the impact of hemodynamic feature derivation and selection method on six ML algorithms. We utilized computational fluid dynamics (CFD) to simulate hemodynamics in 66 cerebral aneurysms from 65 patients, including 57 stable and nine recanalized aneurysms. We derived a total of 107 features for each aneurysm, encompassing four clinical features, 12 morphological features, and 91 hemodynamic features. To investigate the influence of feature derivation and selection methods on the ML models, we employed two derivation methods, simplified and fully derived, in combination with four selection methods: all features, statistically significant analysis, stepwise multivariate logistic regression analysis (stepwise-LR), and recursive feature elimination (RFE). Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and precision-recall curve (AUPRC) on both the training and testing datasets. The AUROC values on the testing dataset exhibited a wide-ranging spectrum, spanning from 0.373 to 0.863. Fully derived features and the RFE selection method demonstrated superior performance in intra-model comparisons. The multi-layer perceptron (MLP) model, trained with RFE-selected fully derived features, achieved the best performance on the testing dataset, with an AUROC value of 0.863 (95% CI: 0.684- 1.000). Our study demonstrated the importance of feature derivation and selection in determining the performance of ML models. This enabled the development of accurate decision-making models without the need to invade the patient.

Artificial intelligence for diabetic retinopathy screening in Africa

Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study

Background and AimsAccurate differentiation of cardiac amyloidosis (CA) from phenotypic mimics remains challenging using current clinical and echocardiographic techniques. The accuracy of a novel artificial intelligence (AI) screening algorithm for echocardiography-based CA detection was assessed.MethodsUtilizing a multisite, multiethnic dataset (n = 2612, 52% CA), a convolutional neural network was trained to differentiate CA from phenotypic controls using transthoracic apical four-chamber video clips. External validation was conducted globally across 18 sites including 597 CA cases and 2122 controls. Classification accuracy was assessed on the entire external validation dataset, and subgroup analyses were performed both on technetium pyrophosphate scintigraphy referrals, and individuals matched for age, sex, and wall thickness. Model accuracy was also compared with the transthyretin CA score and the increased wall thickness score within a subset of older heart failure with preserved ejection fraction patients with increased wall thickness.ResultsCardiac amyloidosis patients and controls displayed similar age, sex, race, and comorbidities. After the removal of uncertain AI predictions (13%), model discrimination and classification were excellent for the entire external validation dataset [area under the receiver operating characteristic curve (AUROC) 0.93, sensitivity 85%, specificity 93%], irrespective of CA subtype (sensitivity: light-chain = 84%, wild-type transthyretin = 85%, and hereditary transthyretin = 86%). Performance was maintained in subgroup analysis in patients clinically referred for technetium pyrophosphate scintigraphy imaging (AUROC 0.86, sensitivity 77%, specificity 86%) and matched patients (AUROC 0.92, sensitivity 84%, specificity 91%). The AI model (AUROC 0.93) also outperformed transthyretin CA score (AUROC 0.73) and increased wall thickness (AUROC 0.80) scores.ConclusionsThis AI screening model—using only an apical four-chamber view—effectively differentiated CA from other causes of increased left ventricular wall thickness.

Understanding the BRAF alterations preoperatively could remarkably assist in predicting tumor behavior, which leads to a more precise prognostication and management strategy. Recent advances in artificial intelligence (AI) have resulted in effective predictive models. Therefore, for the first time, this study aimed to review the performance of machine learning and deep learning models in predicting the BRAF alterations in low-grade gliomas (LGGs)using imaging data. PubMed/MEDLINE, Embase, and the Cochrane Library were systematically searched for studies published up to June 1, 2024, and evaluated the performance of AI models in predicting BRAF alterations in LGGs using imaging data. Pooled sensitivity, specificity, and area under the receiver operating curve (AUROC) were meta-analyzed. A total of 6 studies with 951 patients were included in this systematic review. The pooled AUROC of internal validation cohorts for their best-performing model was 84.44%, with models detecting BRAF mutation, BRAF fusion, BRAF fusion from mutation, and BRAF wild type producing similar AUROCs of 90.75%, 84.59%, 82.33%, and 82%, respectively. The best-performing models had pooled sensitivities of 80.3%, 87.51%, and 74.14% and pooled specificities of 88.57%, 70.41%, and 83.98% for detection of BRAF fusion from mutation, BRAF fusion, and BRAF mutation, respectively. AI models may perform relatively well in predicting BRAF alterations in LGG using imaging data and appear to be capable of high sensitivities and specificities. However, future studies with larger sample sizes implementing different machine learning or deep learning algorithms are required to reduce imprecision.

Spitz tumors are diagnostically challenging due to overlap in atypical histological features with conventional melanomas. We investigated to what extent artificial intelligence (AI) models, using histological and/or clinical features, can: (1) distinguish Spitz tumors from conventional melanomas; (2) predict the underlying genetic aberration of Spitz tumors; and (3) predict the diagnostic category of Spitz tumors. The AI models were developed and validated using a retrospective cohort from the University Medical Center Utrecht, the Netherlands. The dataset consisted of 393 Spitz tumors and 379 conventional melanomas. Predictive performance was measured using the area under the receiver operating characteristic curve (AUROC) and the accuracy. The performance of the AI models was compared with that of four experienced pathologists in a reader study. Moreover, a simulation experiment was conducted to investigate the impact of implementing AI-based recommendations for ancillary diagnostic testing on the workflow of the pathology department. The best AI model based on UNI features reached an AUROC of 0.95 (95% CI, 0.92–0.98) and an accuracy of 0.86 (95% CI, 0.81–0.91) in differentiating Spitz tumors from conventional melanomas. The genetic aberration was predicted with an accuracy of 0.55 (95% CI, 0.46–0.64) compared to 0.25 for randomly guessing. The diagnostic category was predicted with an accuracy of 0.51 (95% CI, 0.40–0.60), where random chance-level accuracy equaled 0.33. On all three tasks, the AI models performed better than the four pathologists, although differences were not statistically significant for most individual comparisons. Based on the simulation experiment, implementing AI-based recommendations for ancillary diagnostic testing could reduce material costs, turnaround times, and examinations. In conclusion, the AI models achieved a strong predictive performance in distinguishing between Spitz tumors and conventional melanomas. On the more challenging tasks of predicting the genetic aberration and the diagnostic category of Spitz tumors, the AI models performed better than random chance.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Rapid myocardial revascularization in patients with acute myocardial infarction (AMI) is essential to improve clinical outcomes. There is still room for improvement in the timely diagnosis of AMI. This study aimed to develop an artificial intelligence (AI) model using electrocardiograms (ECGs) for detecting AMI needing revascularization. A total of 723 389 ECGs from 300 627 patients in the derivation cohort at a single centre between 2013 and 2020, including 5872 patients with AMI (1.95%) who underwent revascularization, were used for model training and internal testing. A transformer-based deep learning model, initially trained on about one million unlabelled ECGs through self-supervised learning, was fine-tuned for AMI detection. The model's final performance was evaluated in the internal test and the external validation set. The external validation was conducted at an independent centre between 2002 and 2020 using 261 429 ECGs from 259 454 patients, including 1095 patients with AMI (0.42%). By integrating self-supervised learning to train the AI model, we enhanced the AMI detection performance, as demonstrated by an increase in the area under the receiver operating characteristic curve (AUROC) from 0.910 (95% CI, 0.904-0.915) to 0.968 (95% CI, 0.965-0.971) in the external validation set. For ST-elevation myocardial infarction and non-ST-elevation myocardial infarction detection, the AUROCs were 0.991 (95% CI, 0.989-0.993) and 0.947 (95% CI, 0.942-0.952) in the external validation set, respectively. This novel ECG-based AI model may be beneficial for the timely identification of patients with AMI needing revascularization.

Fetal hypoxia is a leading cause of neonatal morbidity and mortality. Cardiotocography (CTG) is widely used to predict fetal hypoxia during labor, but its interpretation remains suboptimal. Artificial intelligence (AI) models have been developed for CTG interpretation, but their clinical utility is limited by two major challenges: demonstrating superiority over human experts and ensuring explainability in real-world settings. A large dataset containing CTG traces from three tertiary hospitals between January 2014 and May 2022 was built for model development. Deep learning architectures, named Cardiotocography Artificial-intelligence Predictors (CAPs), were trained to predict fetal hypoxia from CTG traces based on CNN (CAP-C), Transformer (CAP-T), LSTM (CAP-L), and CfC (CAP-CfC) algorithms. The outcome was fetal hypoxia, determined by either low Apgar score (≤ 7 at 1 or 5 min) or umbilical artery acidemia (grade 1: pH of umbilical artery (pHa) < 7.20; grade 2: pHa < 7.15; grade 3: pHa < 7.10). Model performance was determined by area under the receiver operating characteristic curve (AUROC), evaluated through nationwide AI-human comparison and validated on the CTU-UHB dataset. Gradient-weighted class activation mapping (Grad-CAM) was applied to highlight the CTG regions that contributed most to the model's predictions. A total of 20,780 CTG traces were obtained for model development, and 467 cases were held out for the nationwide AI-human comparison. Among all models, CAP-L achieved highest AUROC in predicting fetal hypoxia (grade 1: 0.758, 95% CI: 0.754-0.761; grade 2: 0.770, 95% CI: 0.764-0.776; grade 3: 0.716, 95% CI: 0.700-0.732). In comparison with 10,571 expert responses, all CAP models achieved higher AUROC (0.757-0.789 vs. 0.715, P values in Delong test < 0.05). On the public CTU-UHB dataset, CAP-L achieved AUROC of 0.709, 0.727, and 0.730 in predicting fetal hypoxia with grade 1, 2, and 3 acidemia. Grad-CAM analysis showed that the CAP models leveraged variable and prolonged decelerations to predict fetal hypoxia, verified by perturbation-based faithfulness test. The CAP algorithms developed in this study showed superior performance in detecting fetal hypoxia from CTG traces compared to human experts, and demonstrated promising explainability, supporting clinical CTG interpretation. Clinical trial registration number: ChiCTR2100045316, ChiCTR2100052695, ChiCTR2400085338.

ObjectiveTo develop and evaluate a deep learning-based artificial intelligence (AI) model for detecting skull fractures on plain radiographs in children.Materials and MethodsThis retrospective multi-center study consisted of a development dataset acquired from two hospitals (n = 149 and 264) and an external test set (n = 95) from a third hospital. Datasets included children with head trauma who underwent both skull radiography and cranial computed tomography (CT). The development dataset was split into training, tuning, and internal test sets in a ratio of 7:1:2. The reference standard for skull fracture was cranial CT. Two radiology residents, a pediatric radiologist, and two emergency physicians participated in a two-session observer study on an external test set with and without AI assistance. We obtained the area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity along with their 95% confidence intervals (CIs).ResultsThe AI model showed an AUROC of 0.922 (95% CI, 0.842–0.969) in the internal test set and 0.870 (95% CI, 0.785–0.930) in the external test set. The model had a sensitivity of 81.1% (95% CI, 64.8%–92.0%) and specificity of 91.3% (95% CI, 79.2%–97.6%) for the internal test set and 78.9% (95% CI, 54.4%–93.9%) and 88.2% (95% CI, 78.7%–94.4%), respectively, for the external test set. With the model’s assistance, significant AUROC improvement was observed in radiology residents (pooled results) and emergency physicians (pooled results) with the difference from reading without AI assistance of 0.094 (95% CI, 0.020–0.168; p = 0.012) and 0.069 (95% CI, 0.002–0.136; p = 0.043), respectively, but not in the pediatric radiologist with the difference of 0.008 (95% CI, -0.074–0.090; p = 0.850).ConclusionA deep learning-based AI model improved the performance of inexperienced radiologists and emergency physicians in diagnosing pediatric skull fractures on plain radiographs.

MP07-06 EVALUATING THE EFFICACY OF ARTIFICIAL INTELLIGENCE IN PREDICTING OUTCOMES FOR PEDIATRIC HYDRONEPHROSIS: A SYSTEMATIC REVIEW

Artificial Intelligence: The Future of Maxillofacial Prognosis and Diagnosis?

Study question Can an AI image-based model be trained to predict whether an endometrium, prior to progesterone, is receptive for successful embryo implantation from ultrasound images? Summary answer An endometrial receptivity AI model, utilizing ultrasound images and clinical features, is predictive of successful implantation and outperforms endometrial thickness as a predictor of implantation. What is known already Despite its importance in successful embryo implantation and subsequent live birth, endometrial receptivity is difficult to measure. The standard of care at some clinics is to cancel an embryo transfer if EMT is measuring less than 7 mm on ultrasound assessment – however, achieving an EMT threshold does not guarantee success, even with euploid blastocysts. Currently available receptivity tests (e.g. endometrial receptivity assays) are unreliable and have not shown significant improvements in reproductive outcomes, necessitating the development of new evaluation methods for endometrial receptivity. AI can potentially address this gap in clinical care and understanding, aiming to increase patient success. Study design, size, duration 79,602 ultrasound images (40,910 patients) of the endometrium, on the day of progesterone start according to patient’s frozen embryo transfer cycle, were retrospectively collected from 4 clinic networks including 70 clinic locations. PGT-A outcomes and blastocyst quality were available for 15,466 and 36,070 blastocysts respectively, with 14,538 having labels for both. Different image samples (based on embryo quality) were selected to train and test several AI models, with implantation (positive beta-hCG) as a binary outcome. Participants/materials, setting, methods Three criteria for negative (implantation) class sample selection were tested for receptivity model development: 1) high- or medium-quality and/or euploid blastocysts; 2) high-quality blastocysts; 3) all blastocysts. Models were ensembles consisting of an image-based deep learning (DL)-model and a feature-based machine learning (ML)-model. Model performance was measured by Area-under-the-curve (AUC), sensitivity, and specificity. Feature importance from the ML model was assessed, while receptivity model performance was compared to the power of EMT to predict implantation. Main results and the role of chance A receptivity model trained on scenario 1) (n = 27,424) achieved AUC 0.613, sensitivity 0.716, and specificity 0.439 on a test set of 9197 samples, in predicting implantation. Another receptivity model trained on scenario 2) (n = 25,625) achieved AUC 0.627, sensitivity 0.637, and specificity 0.543 on a test set of 8642. Whereas a third receptivity model trained on scenario 3) (n = 31,238) was selected for further analysis as it attained the best performance with AUC 0.631, sensitivity 0.628, and specificity 0.556 on a test set of 10,422. The ensemble model also utilizes relevant clinical features, which were ranked by importance to model predictions: EMT; progesterone blood test value; age at transfer; previous total embryo transfers; days between ultrasound date and transfer date; age at retrieval; and oocyte origin. EMT was assessed for its ability to predict implantation at thresholds between 5–20 mm. The threshold that best separates positive and negative implantation was determined by evaluating AUC on a validation set of 5682. A threshold of 8.8 mm achieved an AUC 0.576, sensitivity 0.694, and specificity 0.455 in predicting implantation on a test set of 5686. However, the receptivity model AUC (0.631) was significantly higher (p &amp;lt; 0.001; DeLong test for differences in AUC), outperforming EMT predictability. Limitations, reasons for caution The endometrial receptivity model was built and tested on retrospective frozen embryo transfer data. Results should be corroborated in prospective and non-selection studies, as well as in multiple geographical locations (all clinics included reside in the United States). The dataset requires greater diversification, with a greater representation of EMT (&amp;lt;7mm). Wider implications of the findings This study indicates that an AI model can be built to predict implantation from ultrasound images and clinical features, surpassing current standards - EMT alone. Further experimentation, with additional clinical features, may improve performance to facilitate development of a model that can accurately predict the gold-standard of live birth. Trial registration number Not applicable