Explainable Artificial Intelligence in Medical Decision Support Systems

Real-World Surveillance of FDA-Cleared Artificial Intelligence Models: Rationale and Logistics.

Internet of things is projected to make its way into all spheres of human life in the near future. This has been compounded with the growing demand for contactless solutions in the wake of the recent pandemic. A potential solution could involve a privacy-preserving gesture-based control system that could control a wide range of appliances. Implementing such gesture-based control systems is mainly conducted using opaque box Artificial Intelligence (AI) models. Systems based on such opaque box AI models have shown high-performance metrics on in-distribution data in a lab environment. However, they are prone to failure when exposed to real-world out-of-distribution data where they cannot be tuned or calibrated due to their complexity and opaqueness. Interval Type-2 Fuzzy Logic-based explainable AI models offer an alternative to opaque box models showing comparable performance on lab in-distribution data. In contrast, in the real world, out-of-distribution data, the type-2 fuzzy models could be easily calibrated and tuned (thanks for their explainability) to provide similar performance to those achieved on the lab in-distribution data.

Accurate and interpretable arrhythmia detection is essential for timely diagnosis and intervention, particularly in Medical Decision Support Systems (MDSS). Fuzzy logic, known for its ability to handle uncertainty and enhance interpretability, has emerged as a promising approach. This systematic literature review (SLR) investigates the role of fuzzy logic in advancing arrhythmia detection, focusing on accuracy, interpretability, and integration with computational intelligence. Following the PRISMA guidelines, 13 studies published between 2019 and 2024 were analysed to address four key questions: (Q1) the accuracy and reliability of fuzzy logic systems, (Q2) the effectiveness of hybrid systems combining fuzzy logic with computational intelligence, (Q3) the challenges in developing multi-class fuzzy logic systems, and (Q4) the impact of fuzzy logic on interpretability in MDSS. Techniques such as Adaptive Neural Fuzzy Inference Systems (ANFIS) and hybrid models with neural networks and bio-inspired algorithms were evaluated. ANFIS demonstrated near-perfect accuracy, while hybrid systems enhanced scalability and addressed multi-class classification challenges. Limitations included reliance on benchmark datasets, limited real-world validation, and insufficient focus on explainable artificial intelligence (XAI). Fuzzy logic shows strong potential for developing interpretable and robust MDSS for arrhythmia detection. Future research should prioritise advancing XAI, incorporating diverse datasets, and addressing real-world challenges to improve clinical applicability.

Quantum computing (QC) is not a futuristic notion in agriculture, though its full potential has yet to be realized. QC is an emerging field at the intersection of physics and computer science that holds immense potential to revolutionize various sectors, including agriculture production and artificial intelligence (AI) modeling. While QC is still in the early stages of development and practical applications within agriculture are not yet widespread, researchers are actively exploring its potential benefits in various agricultural domains, including crop optimization, livestock breeding, and environmental monitoring. QC harnesses the principles of quantum mechanics to perform computations using quantum bits or qubits, which can exist in multiple states simultaneously. Unlike classical computers, which rely on binary bits representing 0 or 1, quantum computers exploit phenomena such as superposition and entanglement to process information in parallel, potentially offering exponential speedup for certain types of problems. In agriculture production, particularly in animal science, QC offers promising avenues for optimizing processes and enhancing productivity. Quantum algorithms can analyze vast amounts of genomic data to improve breeding programs, leading to the development of more resilient and productive livestock breeds. Furthermore, QC can facilitate precision farming techniques by modeling complex environmental factors and animal behavior to optimize feeding strategies, disease management, and overall farm management practices. Moreover, QC can significantly benefit AI modeling by accelerating computations and enabling more efficient training of AI models. Quantum algorithms can enhance the performance of AI algorithms in various tasks, including pattern recognition, natural language processing, and predictive analytics. By leveraging quantum-enhanced optimization algorithms, AI models can achieve better convergence and accuracy, leading to more effective decision-making and problem-solving capabilities. While hybrid intelligent models also represent a novel frontier in agriculture, QC has the potential to expedite the merging of mechanistic and AI modeling paradigms, facilitating a more holistic understanding of complex systems in agriculture and beyond. By integrating mechanistic models, which describe the underlying physical processes, with AI models, which learn patterns from data, quantum computing can enable comprehensive simulations and predictions of agricultural systems. This fusion of modeling paradigms can lead to more accurate and robust predictions of crop yields, livestock performance, and environmental impacts, facilitating informed decision-making for farmers and policymakers. The application of QC in agriculture, however, requires interdisciplinary collaborations between physicists, computer scientists, agronomists/animal scientists, and AI researchers. These collaborations can drive the development of quantum algorithms tailored to agricultural applications, the integration of quantum-enhanced AI techniques into existing modeling frameworks, and the deployment of QC resources in real-world agricultural systems. Ultimately, harnessing the power of QC holds the potential to revolutionize agriculture production practices, including regenerative agriculture, and advance AI modeling capabilities, paving the way for a more sustainable and efficient agricultural industry.

This study evaluated if medical doctors could identify more hemorrhage events during chart review in a clinical setting when assisted by an artificial intelligence (AI) model and medical doctors' perception of using the AI model. To develop the AI model, sentences from 900 electronic health records were labeled as positive or negative for hemorrhage and categorized into one of 12 anatomical locations. The AI model was evaluated on a test cohort consisting of 566 admissions. Using eye-tracking technology, we investigated medical doctors' reading workflow during manual chart review. Moreover, we performed a clinical use study where medical doctors read two admissions with and without AI assistance to evaluate performance when using and perception of using the AI model. The AI model had a sensitivity of 93.7% and a specificity of 98.1% on the test cohort. In the use studies, we found that medical doctors missed more than 33% of relevant sentences when doing chart review without AI assistance. Hemorrhage events described in paragraphs were more often overlooked compared with bullet-pointed hemorrhage mentions. With AI-assisted chart review, medical doctors identified 48 and 49 percentage points more hemorrhage events than without assistance in two admissions, and they were generally positive toward using the AI model as a supporting tool. Medical doctors identified more hemorrhage events with AI-assisted chart review and they were generally positive toward using the AI model.

We have recently shown that histological phenotypes focusing on clear and eosinophilic cytoplasm in clear cell renal cell carcinoma (ccRCC) correlated with prognosis and the response to angiogenesis inhibition and checkpoint blockade. This study aims to objectively show the diagnostic utility of clear or eosinophilic phenotypes of ccRCC by developing an artificial intelligence (AI) model using the TCGA-ccRCC dataset and to demonstrate if the clear or eosinophilic predicted phenotypes correlate with pathological factors and gene signatures associated with angiogenesis and cancer immunity. Before the development of the AI model, histological evaluation using hematoxylin and eosin whole-slide images of the TCGA-ccRCC cohort (n=435) was performed by a urologic pathologist. The AI model was developed as follows. First, the highest-grade area on each whole slide image was captured for image processing. Second, the selected regions were cropped into tiles. Third, the AI model was trained using transfer learning on a deep convolutional neural network, and clear or eosinophilic predictions were scaled as AI scores. Next, we verified the AI model using a validation cohort (n=95). Finally, we evaluated the accuracy of the prognostic predictions of the AI model and revealed that the AI model detected clear and eosinophilic phenotypes with high accuracy. The AI model stratified the patients' outcomes, and the predicted eosinophilic phenotypes correlated with adverse clinicopathological characteristics and high immune-related gene signatures. In conclusion, the AI-based histologic subclassification accurately predicted clear or eosinophilic phenotypes of ccRCC, allowing for consistently reproducible stratification for prognostic and therapeutic stratification.

The secondary use of health data for training Artificial Intelligence (AI) models holds immense potential for advancing medical research and healthcare delivery. However, ensuring patient consent for such utilization is paramount to uphold ethical standards and data privacy. Patient informed consent means patients are fully informed about how their data will be collected, used, and protected, and they voluntarily agree to allow their data to be used for AI models. In addition to formal consent frameworks, establishing a social license is critical to foster public trust and societal acceptance for the secondary use of health data in AI systems. This study examines patient consent practices in this domain. In this scoping review, we searched Web of Science, PubMed, and Scopus. We included studies in English that addressed the core issues of interest, namely, privacy, security, legal, and ethical issues related to the secondary use of health data in AI models. Articles not addressing the core issues, as well as systematic reviews, meta-analyses, books, letters, conference abstracts, and study protocols were excluded. Two authors independently screened titles, abstracts, and full texts, resolving disagreements with a third author. Data was extracted using a data extraction form. After screening 774 articles, a total of 38 articles were ultimately included in the review. Across these studies, a total of 178 barriers and 193 facilitators were identified. We consolidated similar codes and extracted 65 barriers and 101 facilitators, which we then categorized into four themes: "Structure," "People," "Physical system," and "Task." We identified notable emphasis on "Legal and Ethical Challenges" and "Interoperability and Data Governance." Key barriers included concerns over privacy and security breaches, inadequacies in informed consent processes, and unauthorized data sharing. Critical facilitators included enhancing patient consent procedures, improving data privacy through anonymization, and promoting ethical standards for data usage. Our study underscores the complexity of patient consent for the secondary use of health data in AI models, highlighting significant barriers and facilitators within legal, ethical, and technological domains. We recommend the development of specific guidelines and actionable strategies for policymakers, practitioners, and researchers to improve informed consent, ensuring privacy, trust, and ethical use of data, thereby facilitating the responsible advancement of AI in healthcare.

A new method to promptly evaluate spatial earthquake probability mapping using an explainable artificial intelligence (XAI) model

Prognosis prediction is crucial for non-small cell lung cancer (NSCLC) treatment planning. While tumor hypoxia significantly impacts patient outcomes, identifying hypoxic genomic markers remains challenging. This study sought to identify hypoxic computed tomography (CT) radiomic features and create an artificial intelligence (AI) model for NSCLC through the integration of multi-modal data. In total, 452 NSCLC patients were enrolled in this study, including patients from The Second Affiliated Hospital of Soochow University (SC, n=112), The Cancer Genome Atlas (TCGA)-NSCLC dataset (n=74), the radiogenomics dataset (n=130), and the Gene Expression Omnibus (GEO) datasets (GSE19188: n=82, and GSE87340: n=54). Hypoxia status was classified using optimized cut-off values of hypoxia enrichment scores, which were calculated through single-sample gene set enrichment analysis (ssGSEA) of hypoxic genes. Radiomic features were extracted using three-dimensional (3D)-Slicer software. The least absolute shrinkage and selection operator (LASSO) algorithm was used to identify hypoxic CT radiomic features. A model named ssuBERT (semantic structured unit embedded in Bidirectional Encoder Representations from Transformers) was developed to analyze electronic health records (EHRs). An AI model for overall survival prediction was constructed by integrating CT radiomic features, ssuBERT features, and clinical data, and evaluated using five-fold cross-validation. Higher hypoxia levels were correlated with worse survival outcomes. Twenty-eight radiomic features showed significant discriminatory power in detecting hypoxia status with an area under the curve (AUC) of 0.8295. The ssuBERT model achieved a weighted accuracy of 0.945 in recognizing semantic structured units in EHRs. The EHR model exhibited superior predictive performance among the single-modal models with an AUC of 0.7662. However, the multi-modal AI model had the highest average AUC of 0.8449 and an F1 score of 0.7557. The AI model demonstrated potential in predicting NSCLC patient prognosis through multi-modal data integration, warranting further validation.

Artificial intelligence driven plaque characterization and functional assessment from CCTA using OCT-based automation: A prospective study.

Spitz tumors are diagnostically challenging due to overlap in atypical histological features with conventional melanomas. We investigated to what extent artificial intelligence (AI) models, using histological and/or clinical features, can: (1) distinguish Spitz tumors from conventional melanomas; (2) predict the underlying genetic aberration of Spitz tumors; and (3) predict the diagnostic category of Spitz tumors. The AI models were developed and validated using a retrospective cohort from the University Medical Center Utrecht, the Netherlands. The dataset consisted of 393 Spitz tumors and 379 conventional melanomas. Predictive performance was measured using the area under the receiver operating characteristic curve (AUROC) and the accuracy. The performance of the AI models was compared with that of four experienced pathologists in a reader study. Moreover, a simulation experiment was conducted to investigate the impact of implementing AI-based recommendations for ancillary diagnostic testing on the workflow of the pathology department. The best AI model based on UNI features reached an AUROC of 0.95 (95% CI, 0.92–0.98) and an accuracy of 0.86 (95% CI, 0.81–0.91) in differentiating Spitz tumors from conventional melanomas. The genetic aberration was predicted with an accuracy of 0.55 (95% CI, 0.46–0.64) compared to 0.25 for randomly guessing. The diagnostic category was predicted with an accuracy of 0.51 (95% CI, 0.40–0.60), where random chance-level accuracy equaled 0.33. On all three tasks, the AI models performed better than the four pathologists, although differences were not statistically significant for most individual comparisons. Based on the simulation experiment, implementing AI-based recommendations for ancillary diagnostic testing could reduce material costs, turnaround times, and examinations. In conclusion, the AI models achieved a strong predictive performance in distinguishing between Spitz tumors and conventional melanomas. On the more challenging tasks of predicting the genetic aberration and the diagnostic category of Spitz tumors, the AI models performed better than random chance.

High-risk breast lesions (HRLs) are associated with future risk of breast cancer. Considering the pathological subtypes, malignancy upgrade rate differs according to each subtype and depends on various factors such as clinical and radiological features and biopsy method. Using artificial intelligence and machine learning models in breast imaging, evaluations can be made in terms of risk estimation in different research areas. This study aimed to develop a machine learning model to distinguish HRL cases requiring surgical excision from lesions with a low risk of accompanying malignancy. A total of 94 patients who were diagnosed with HRL by image-guided biopsy between January 2008 and March 2020 were included in the study. A structured database was created with clinical and radiological characteristics and histopathological results. A machine learning prediction model was created to make binary classifications of lesions as malignant or benign. Random forest, decision tree, K-nearest neighbors, logistic regression, support vector machine (SVM), and multilayer perceptron machine learning algorithms were used. Among these algorithms, SVM was the most successful. The estimations of malignancy for each case detected by artificial intelligence were combined and statistical analyses were performed. Considering all cases, the malignancy upgrade rate was 24.5%. A significant association was observed between malignancy upgrade rate and lesion size (P = 0.004), presence of mammography findings (P = 0.022), and breast imaging-reporting and data system category (P = 0.001). A statistically significant association was also found between the artificial intelligence prediction model and malignancy upgrade rate (P < 0.001). With the SVM model, an 84% accuracy and 0.786 area-underthe- curve score were obtained in classifying the data as benign or malignant. Our artificial intelligence model (SVM) can predict HRLs that can be followed up with a lower risk of accompanying malignancy. Unnecessary surgeries can be reduced, or second line vacuum excisions can be performed in HRLs, which are mostly benign, by evaluating on a case-by-case basis, in line with radiology-pathology compatibility and by using an artificial intelligence model.

Artificial intelligence (AI) models can sense subjective affective states from facial images. Although recent psychological studies have indicated that dimensional affective states of valence and arousal are systematically associated with facial expressions, no AI models have been developed to estimate these affective states from facial images based on empirical data. We developed a recurrent neural network-based AI model to estimate subjective valence and arousal states from facial images. We trained our model using a database containing participant valence/arousal states and facial images. Leave-one-out cross-validation supported the validity of the model for predicting subjective valence and arousal states. We further validated the effectiveness of the model by analyzing a dataset containing participant valence/arousal ratings and facial videos. The model predicted second-by-second valence and arousal states, with prediction performance comparable to that of FaceReader, a commercial AI model that estimates dimensional affective states based on a different approach. We constructed a graphical user interface to show real-time affective valence and arousal states by analyzing facial video data. Our model is the first distributable AI model for sensing affective valence and arousal from facial images/videos to be developed based on an empirical database; we anticipate that it will have many practical uses, such as in mental health monitoring and marketing research.

Virtual Screening (VS) of large compound libraries using Artificial Intelligence (AI) models is a highly effective approach for early drug discovery. Data splitting is crucial for benchmarking the performance of such AI models. Traditional random data splits often result in structurally similar molecules in both training and test sets, which conflict with the reality of VS libraries that typically contain structurally diverse compounds. To tackle this challenge, scaffold split, which groups molecules by shared core structure, and Butina clustering, which clusters molecules by chemotypes, have long been used. However, we show that these methods still introduce high similarities between training and test sets, leading to overestimated model performance. Our study examined four representative AI models across 60 NCI-60 datasets, each comprising approximately 33,000–54,000 molecules tested on different cancer cell lines. Each dataset was split in four ways: random, scaffold, Butina clustering and the more realistic Uniform Manifold Approximation and Projection (UMAP) clustering. Using Linear Regression, Random Forest, Transformer-CNN, and GEM, we trained a total of 8400 models and evaluated under four splitting methods. These comprehensive results show that UMAP split provides more challenging and realistic benchmarks for model evaluation, followed by Butina splits, then scaffold splits and closely after random splits. Consequently, we recommend using UMAP splits instead of overly optimistic Butina splits and especially scaffold splits for molecular property prediction, including VS. Lastly, we illustrate how misaligned ROC AUC is with VS goals, despite its common use. The code and datasets for reproducibility are available at https://github.com/Rong830/UMAP_split_for_VS and archived in https://zenodo.org/records/14736486. Scientific contribution This work advances the field by introducing UMAP clustering as a robust splitting method for molecular datasets, improving over traditional methods like Butina clustering and especially scaffold splits. It offers a new evaluation framework to benchmark AI models under more realistic conditions, fostering progress in molecular property prediction. The findings also show how inappropriate the use of ROC AUC for virtual screening (VS) continues to be, despite its popularity, emphasizing the need for context-specific evaluation metrics.

This study aims at evaluating if ChatGPT-based artificial intelligence (AI) models are effective in interpreting electrocardiograms (ECGs) and determine their accuracy as compared to those of cardiologists. The purpose is therefore to explore if ChatGPT can be employed for clinical setting, particularly where there are no available cardiologists. A total of 107 ECG cases classified according to difficulty (simple, intermediate, complex) were analyzed using three AI models (GPT-ECGReader, GPT-ECGAnalyzer, GPT-ECGInterpreter) and compared with the performance of two cardiologists. The statistical analysis was conducted using chi-square and Fisher exact tests using scikit-learn library in Python 3.8. Cardiologists demonstrated superior accuracy (92.52%) compared to ChatGPT-based models (GPT-ECGReader: 57.94%, GPT-ECGInterpreter: 62.62%, GPT-ECGAnalyzer: 62.62%). Statistically significant differences were observed between cardiologists and AI models (p < 0.05). ChatGPT models exhibited enhanced performance with female patients; however, the differences found were not statistically significant. Cardiologists significantly outperformed AI models across all difficulty levels. When it comes to diagnosing patients with arrhythmia (A) and cardiac structural disease ECG patterns, cardiologists gave the best results though there was no statistical difference between them and AI models in diagnosing people with normal (N) ECG patterns. ChatGPT-based models have potential in ECG interpretation; however, they currently lack adequate reliability beyond oversight from a doctor. Additionally, further studies that would improve the accuracy of these models, especially in intricate diagnoses are needed.