Evaluating explainable artificial intelligence (XAI): algorithmic explanations for transparency and trustworthiness of ML algorithms and AI systems

Abstract. As air pollution is regarded as the single largest environmental health risk in Europe it is important that communication to the public is up to date and accurate and provides means to avoid exposure to high air pollution levels. Long- and short-term exposure to outdoor air pollution is associated with increased risks of mortality and morbidity. Up-to-date information on present and coming days' air quality helps people avoid exposure during episodes with high levels of air pollution. Air quality forecasts can be based on deterministic dispersion modelling, but to be accurate this requires detailed information on future emissions, meteorological conditions and process-oriented dispersion modelling. In this paper, we apply different machine learning (ML) algorithms – random forest (RF), extreme gradient boosting (XGB), and long short-term memory (LSTM) – to improve 1, 2, and 3 d deterministic forecasts of PM10, NOx, and O3 at different sites in Greater Stockholm, Sweden. It is shown that the deterministic forecasts can be significantly improved using the ML models but that the degree of improvement of the deterministic forecasts depends more on pollutant and site than on what ML algorithm is applied. Also, four feature importance methods, namely the mean decrease in impurity (MDI) method, permutation method, gradient-based method, and Shapley additive explanations (SHAP) method, are utilized to identify significant features that are common and robust across all models and methods for a pollutant. Deterministic forecasts of PM10 are improved by the ML models through the input of lagged measurements and Julian day partly reflecting seasonal variations not properly parameterized in the deterministic forecasts. A systematic discrepancy by the deterministic forecasts in the diurnal cycle of NOx is removed by the ML models considering lagged measurements and calendar data like hour and weekday, reflecting the influence of local traffic emissions. For O3 at the urban background site, the local photochemistry is not properly accounted for by the relatively coarse Copernicus Atmosphere Monitoring Service ensemble model (CAMS) used here for forecasting O3 but is compensated for using the ML models by taking lagged measurements into account. Through multiple repetitions of the training process, the resulting ML models achieved improvements for all sites and pollutants. For NOx at street canyon sites, mean squared error (MSE) decreased by up to 60 %, and seven metrics, such as R2 and mean absolute percentage error (MAPE), exhibited consistent results. The prediction of PM10 is improved significantly at the urban background site, whereas the ML models at street sites have difficulty capturing more information. The prediction accuracy of O3 also modestly increased, with differences between metrics. Further work is needed to reduce deviations between model results and measurements for short periods with relatively high concentrations (peaks) at the street canyon sites. Such peaks can be due to a combination of non-typical emissions and unfavourable meteorological conditions, which are rather difficult to forecast. Furthermore, we show that general models trained using data from selected street sites can improve the deterministic forecasts of NOx at the station not involved in model training. For PM10 this was only possible using more complex LSTM models. An important aspect to consider when choosing ML algorithms is the computational requirements for training the models in the deployment of the system. Tree-based models (RF and XGB) require fewer computational resources and yield comparable performance in comparison to LSTM. Therefore, tree-based models are now implemented operationally in the forecasts of air pollution and health risks in Stockholm. Nevertheless, there is big potential to develop generic models using advanced ML to take into account not only local temporal variation but also spatial variation at different stations.

Diabetes is a chronic disease that continues to be a primary and worldwide health concern since the health of the entire population has been affected by it. Over the years, many academics have attempted to develop a reliable diabetes prediction model using machine learning (ML) algorithms. However, these research investigations have had a minimal impact on clinical practice as the current studies focus mainly on improving the performance of complicated ML models while ignoring their explainability to clinical situations. Therefore, the physicians find it difficult to understand these models and rarely trust them for clinical use. In this study, a carefully constructed, efficient, and interpretable diabetes detection method using an explainable AI has been proposed. The Pima Indian diabetes dataset was used, containing a total of 768 instances where 268 are diabetic, and 500 cases are non-diabetic with several diabetic attributes. Here, six machine learning algorithms (artificial neural network (ANN), random forest (RF), support vector machine (SVM), logistic regression (LR), AdaBoost, XGBoost) have been used along with an ensemble classifier to diagnose the diabetes disease. For each machine learning model, global and local explanations have been produced using the Shapley additive explanations (SHAP), which are represented in different types of graphs to help physicians in understanding the model predictions. The balanced accuracy of the developed weighted ensemble model was 90% with a F1 score of 89% using a five-fold cross-validation (CV). The median values were used for the imputation of the missing values and the synthetic minority oversampling technique (SMOTETomek) was used to balance the classes of the dataset. The proposed approach can improve the clinical understanding of a diabetes diagnosis and help in taking necessary action at the very early stages of the disease.

The accurate detection and assessment of stress play a pivotal role in enhancing individual well-being and healthcare outcomes. Traditional methods of stress detection often grapple with limitations in accuracy and scalability. With the advent of machine learning (ML), the potential to revolutionize stress detection has emerged. This paper presents a comprehensive study on the application of ML algorithms for stress detection, with a focus on physiological and behavioral data analysis. Central to our approach is the integration of SHapley Additive exPlanations (SHAP), an Explainable Artificial Intelligence (XAI) technique, to interpret ML models. SHAP provides a novel lens to understand the impact of individual features in the complex decision-making processes of ML models, thereby enhancing the transparency and reliability of stress predictions. We demonstrate how SHAP not only aids in elucidating model decisions but also contributes to refining the models for greater accuracy. Our results highlight the effectiveness of ML in detecting stress and the pivotal role of XAI in making these models more interpretable and trustworthy. This study underscores the synergy between advanced ML techniques and XAI, paving the way for more nuanced and reliable stress detection methodologies that are essential in diverse settings, from healthcare to workplace environments.

Propofol is a widely used sedative-hypnotic agent for critically-ill patients requiring invasive mechanical ventilation (IMV). Despite its clinical benefits, propofol is associated with increased risks of hypertriglyceridemia. Early identification of patients at risk for propofol-associated hypertriglyceridemia is crucial for optimizing sedation strategies and preventing adverse outcomes. Machine learning (ML) models offer a promising approach for predicting individualized patient risks of propofol-associated hypertriglyceridemia. We propose the development of a ML model aimed at predicting the risk of propofol-associated hypertriglyceridemia in ICU patients receiving IMV. The study will utilize retrospective data from four Mayo Clinic sites. Nested cross-validation (CV) will be employed, with a 10-fold inner CV loop for model tuning and selection as well as an outer loop using leave-one-site-out CV for external validation. Feature selection will be conducted using Boruta and LASSO-penalized logistic regression. Data preprocessing steps include missing data imputation, feature scaling, and dimensionality reduction techniques. Six ML algorithms will be tuned and evaluated. Bayesian optimization will be used for hyperparameter selection. Global model explainability will be assessed using permutation importance, and local model explainability will be assessed using SHapley Additive exPlanations (SHAP). The proposed ML model aims to provide a reliable and interpretable tool for clinicians to predict the risk of propofol-associated hypertriglyceridemia in ICU patients. The final model will be deployed in a web-based clinical risk calculator. The model development process and performance measures obtained during nested cross-validation will be described in a study publication to be disseminated in a peer-reviewed journal. The proposed study has received ethics approval from the Mayo Clinic Institutional Review Board (IRB #23-007416). Robust external validation using a nested cross-validation (CV) framework will help assess the generalizability of models produced from the modeling pipeline across different hospital settings.A diverse set of machine learning (ML) algorithms and advanced hyperparameter tuning techniques will be employed to identify the most optimal model configuration.Integration of feature explainability will enhance the clinical applicability of the ML models by providing transparency in predictions, which can improve clinician trust and encourage adoption.Reliance on retrospective data may introduce biases due to inconsistent or erroneous data collection, and the computational intensity of the validation approach may limit replication and future model expansion in resource-constrained settings.

Air pollution in Macau has become a serious problem following the Pearl River Delta’s (PRD) rapid industrialization that began in the 1990s. With this in mind, Macau needs an air quality forecast system that accurately predicts pollutant concentration during the occurrence of pollution episodes to warn the public ahead of time. Five different state-of-the-art machine learning (ML) algorithms were applied to create predictive models to forecast PM2.5, PM10, and CO concentrations for the next 24 and 48 h, which included artificial neural networks (ANN), random forest (RF), extreme gradient boosting (XGBoost), support vector machine (SVM), and multiple linear regression (MLR), to determine the best ML algorithms for the respective pollutants and time scale. The diurnal measurements of air quality data in Macau from 2016 to 2021 were obtained for this work. The 2020 and 2021 datasets were used for model testing, while the four-year data before 2020 and 2021 were used to build and train the ML models. Results show that the ANN, RF, XGBoost, SVM, and MLR models were able to provide good performance in building up a 24-h forecast with a higher coefficient of determination (R2) and lower root mean square error (RMSE), mean absolute error (MAE), and biases (BIAS). Meanwhile, all the ML models in the 48-h forecasting performance were satisfactory enough to be accepted as a two-day continuous forecast even if the R2 value was lower than the 24-h forecast. The 48-h forecasting model could be further improved by proper feature selection based on the 24-h dataset, using the Shapley Additive Explanations (SHAP) value test and the adjusted R2 value of the 48-h forecasting model. In conclusion, the above five ML algorithms were able to successfully forecast the 24 and 48 h of pollutant concentration in Macau, with the RF and SVM models performing the best in the prediction of PM2.5 and PM10, and CO in both 24 and 48-h forecasts.

The purpose of this analysis is to assess the use of machine learning (ML) algorithms in the prediction of postoperative outcomes, including complications, recurrence, and death in transsphenoidal surgery. Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we systematically reviewed all papers that used at least one ML algorithm to predict outcomes after transsphenoidal surgery. We searched Scopus, PubMed, and Web of Science databases for studies published prior to May 12, 2021. We identified 13 studies enrolling 5,048 patients. We extracted the general characteristics of each study; the sensitivity, specificity, area under the curve (AUC) of the ML models developed as well as the features identified as important by the ML models. We identified 12 studies with 5,048 patients that included ML algorithms for adenomas, three with 1807 patients specifically for acromegaly, and five with 2105 patients specifically for Cushing's disease. Nearly all were single-institution studies. The studies used a heterogeneous mix of ML algorithms and features to build predictive models. All papers reported an AUC greater than 0.7, which indicates clinical utility. ML algorithms have the potential to predict postoperative outcomes of transsphenoidal surgery and can improve patient care. Ensemble algorithms and neural networks were often top performers when compared with other ML algorithms. Biochemical and preoperative features were most likely to be selected as important by ML models. Inexplicability remains a challenge, but algorithms such as local interpretable model–agnostic explanation or Shapley value can increase explainability of ML algorithms. Our analysis shows that ML algorithms have the potential to greatly assist surgeons in clinical decision making.

BackgroundDespite the interest in machine learning (ML) algorithms for analyzing real-world data (RWD) in healthcare, the use of ML in predicting time-to-event data, a common scenario in clinical practice, is less explored. ML models are capable of algorithmically learning from large, complex datasets and can offer advantages in predicting time-to-event data. We reviewed the recent applications of ML for survival analysis using RWD in healthcare.MethodsPUBMED and EMBASE were searched from database inception through March 2023 to identify peer-reviewed English-language studies of ML models for predicting time-to-event outcomes using the RWD. Two reviewers extracted information on the data source, patient population, survival outcome, ML algorithms, and the Area Under the Curve (AUC).ResultsOf 257 citations, 28 publications were included. Random survival forests (N = 16, 57%) and neural networks (N = 11, 39%) were the most popular ML algorithms. There was variability across AUC for these ML models (median 0.789, range 0.6–0.950). ML algorithms were predominately considered for predicting overall survival in oncology (N = 12, 43%). ML survival models were often used to predict disease prognosis or clinical events (N = 27, 96%) in the oncology, while less were used for treatment outcomes (N = 1, 4%).ConclusionsThe ML algorithms, random survival forests and neural networks, are mainly used for RWD to predict survival outcomes such as disease prognosis or clinical events in the oncology. This review shows that more opportunities remain to apply these ML algorithms to inform treatment decision-making in clinical practice. More methodological work is also needed to ensure the utility and applicability of ML models in survival outcomes.

Chronic pancreatitis results in pancreatic exocrine insufficiency (PEI), which is treated with pancreatic enzyme replacement therapy (PERT). Despite the clinical benefits, non-compliance to PERT is a frequent problem. Our study aimed at (a) identifying predictors of non-compliance to PERT using machine learning (ML) algorithms and (b) analyzing patient-reported reasons for non-compliance to PERT. A prospective observational study was conducted at a high-volume tertiary care center on two independent cohorts of chronic pancreatitis patients on PERT. In Cohort 1 (1057 patients screened), we used ML algorithms to identify predictors of non-compliance to PERT. The best ML model based on performance metrics was chosen for Shapley Additive explanations (SHAP) analysis. In Cohort 2 (465 patients screened), we conducted detailed interviews to understand patient-reported reasons for non-compliance. In Cohort 1 (751 patients analyzed; median age, 37.5 years; males, 73.1%; idiopathic, 61.9%), 166 (22.1%) patients were non-compliant to PERT. Extreme gradient boosting (XGBoost) exhibited the highest accuracy (area under the curve, AUC = 0.91). The strongest predictors of non-compliance based on SHAP were disease duration, age, fat-restricted diet, rural residence and educational status. A non-compliance (NC) score was developed based on SHAP. In Cohort 2 (129 patients analyzed; mean age, 36 years; males, 72.1%; idiopathic, 52.3%), high treatment costs (34.1%), negligence (19.4%) and adverse effects (10.9%) were the most reported reasons for non-compliance. This study identifies predictors of non-compliance to PERT (disease duration, age, fat-restricted diet, rural habitat, undergraduate education status) in patients with chronic pancreatitis using machine-learning algorithms. The NC score can be a useful tool to identify patients at risk for non-compliance, who can be subjected to targeted counselling. The NC score needs to be validated in large, independent, multi-centre cohorts of patients.

With the evolution of fifth generation (5G) network technologies, network maintenance strategies have become increasingly complex, necessitating the use of predictive analysis enabled by Machine Learning (ML) algorithms. This paper emphasizes exploring how ML algorithms can further enhance predictive maintenance in 5G and future networks. It reviews the current literature on this interdisciplinary topic, identifying key ML models such as Decision Trees, Neural Networks, and Support Vector Machines, and discussing their benefits and limitations. Special attention is given to the methodologies in applying these models, handling of data stages, and the training process. Major challenges in implementing ML in the context of network maintenance, such as data privacy, data gathering, model training, and generalizability, are discussed. Furthermore, the research aims to go beyond predicting maintenance needs to introduce a proactive approach in improving overall network performance and pre-empting potential issues based on ML predictions. The paper also discusses possible future trends including advancements in ML algorithms, Automated Machine Learning (AutoML), Explainable AI, and others. The objective is to provide a comprehensive understanding of the current ML-based predictive maintenance field and outline possibilities for future research. The study finds that the application of ML algorithms continues to show promise in transforming the landscape of network management by improving predictive maintenance and proactive performance enhancement strategies. It remains a challenging yet important area in the context of 5G networks.

The integration of machine learning (ML) algorithms into audit processes represents a significant advancement in the field of auditing, offering substantial benefits in terms of efficiency, accuracy, and risk management. This review examines the transformative potential of ML in auditing, highlighting its key benefits and the challenges that must be addressed to fully leverage its capabilities. Machine learning algorithms, with their ability to analyze large datasets and identify patterns, enhance the accuracy and thoroughness of audits. Traditional auditing methods often rely on sampling and manual checks, which can miss anomalies and fraudulent activities. In contrast, ML algorithms can process entire datasets, uncovering subtle patterns and irregularities that may indicate fraud or errors. This comprehensive analysis reduces the risk of oversight and improves the reliability of audit findings. One of the primary benefits of ML in auditing is its capacity for anomaly detection. ML models can be trained on historical data to understand normal financial behavior and flag deviations that might signify irregularities. This ability to detect anomalies in real-time enables auditors to identify potential issues promptly, reducing the time lag between occurrence and detection of fraud. Predictive analytics, powered by ML, further enhances audit processes by forecasting future risks based on historical data. This proactive approach allows auditors to anticipate and mitigate risks before they materialize, contributing to more robust risk management strategies. Despite these advantages, integrating ML into audit processes presents several challenges. Ensuring data quality and integrity is crucial, as ML algorithms are only as good as the data they analyze. Poor-quality data can lead to inaccurate predictions and conclusions. Additionally, the "black box" nature of some ML algorithms can pose transparency issues, making it difficult for auditors to explain how specific conclusions were reached, which is critical for stakeholder trust and regulatory compliance. Another significant challenge is the potential for algorithmic bias. ML models can inadvertently perpetuate existing biases in the data, leading to unfair or skewed audit outcomes. Continuous monitoring and validation of ML algorithms are necessary to detect and mitigate such biases. In conclusion, while integrating machine learning algorithms into audit processes offers substantial benefits in terms of accuracy, efficiency, and risk management, it also necessitates careful attention to data quality, transparency, and bias mitigation. Addressing these challenges is essential to fully realize the potential of ML in enhancing audit practices. Keywords: Benefits, Challenges, Audit Processes, Algorithms, ML.

12074 Background: Skeletal-related events (SREs) are detrimental clinical events in bone metastasis from advanced cancer, defined by pathologic fracture, spinal cord compression, and inevitable surgical or radiational intervention to the bone. Given their negative impact on quality of life and prognosis and interpatient heterogeneity in the SRE risk, accurate identification of patients with high SRE risk is critical. Methods: The patient-level data from three randomized clinical trials that administered zoledronic acid (ZA) to patients with bone-metastatic breast cancer, castration-resistant prostate cancer (CRPC), and other types of cancer were analyzed. (N = 460, 452, and 315, respectively) Machine learning (ML) models to predict SREs within 18 months (breast cancer), 12 months (CRPC), and 9 months (other cancers) were developed based on more than 40 baseline clinical and laboratory data. Seven ML algorithms and five feature selection methods were utilized to develop multiple models. The ML model with the best performance was identified based on the F1 score and the area under the receiver operating characteristic curve (AUC-ROC) for each cancer type and interrogated for important features with Shapley additive explanations. Lastly, the ML models’ ability to stratify patients by the cumulative SRE risk was evaluated by calculating hazard ratio (HR) with Cox-proportional hazards models. Results: Among the multiple ML models developed with different algorithms and feature selection methods, the model developed utilizing the random forests algorithm and the Boruta method for selecting features demonstrated the best performance in all types of cancer (F1 0.70, 0.67, and 0.67, and AUC-ROC 0.72, 0.68, and 0.73 for breast cancer, CRPC, and other cancers, respectively). In the ML model for breast cancer, performance status (PS), history of SRE, serum alkaline phosphatase (ALP), history of anti-neoplastic surgery, radiation therapy, and pathologic fracture were included as important features. Serum ALP, albumin, sodium, Gleason scores, and geographic regions were shown to be relevant in the CRPC model. For other cancers, serum ALP, albumin, total protein, phosphorus, red blood cell count, white blood cell count, visceral metastases, and history of arthritis were incorporated in the ML model. The ML model prediction successfully stratified the patients for cumulative SRE risk in all three cohorts. (HR with 95% confidence interval: 2.43 [1.86 – 3.18], 1.92 [1.51 – 2.45], and 3.06 [2.29 – 4.09] for breast cancer, CRPC, and other types of cancer, respectively). Conclusions: ML models incorporating baseline clinical and laboratory data can identify patients with bone metastasis on ZA harboring a high SRE risk.

Satellite-based aerosol optical property retrieval over land, especially size-related parameters, is challenging. This study proposed a novel two-stage machine learning (ML) algorithm for retrieving aerosol optical depth (AOD), Ångström exponent (AE), fine mode fraction (FMF), and fine mode AOD (FAOD)) over land using MODIS observed reflectance. The new ML algorithm consists of three steps: (1) first, all samples extracted from AERONET measurements were used to train the ML model, (2) then, to reduce the extreme estimation bias of the model, divided low-value and high-value samples were used to train low-value and high-value ML models, respectively, and (3) finally, the three ML models were integrated into the final retrieval based on the weight interpolation. Independent site network validation results show that the new ML algorithm has a Pearson correlation coefficient (R) of 0.894 (0.638, 0.661, 0.865) and root mean square error (RMSE) of 0.146 (0.258, 0.245, 0.153) for the AOD (AE, FMF, FAOD) retrieval, which significantly outperforms the validation metrics of MODIS operational products, with AOD (AE, FMF, FAOD) RMSE of 0.130-0.156 (0.536-0.569, 0.313, 0.191). The inter-comparison of aerosol products shows that the spatial patterns of AOD, AE, FMF, and FAOD of the new ML algorithm are in good agreement with those of the MODIS and POLDER products. These results illustrate that the new ML algorithm has good performance and transferability and indicate the ability of ML methods to be applied to multispectral instruments (such as MODIS) to retrieve multiple aerosol properties.

This study presents a comprehensive methodology for developing a Fault Detection and Diagnosis (FDD) system for an industrial distillation column using advanced machine learning algorithms. Steady state and dynamic simulations in Aspen Plus® generate extensive datasets under normal and faulty conditions. Feature engineering, using the Minimum Redundancy Maximum Relevance (MRMR) algorithm, selects the most relevant features for fault detection. Various machine learning models, including Decision Trees, Support Vector Machines, k-nearest Neighbours, and Neural Networks, were trained and evaluated based on performance metrics such as accuracy, recall, precision, and F1 score. The top models were integrated into a stacked classifier system with a voting mechanism to enhance fault detection reliability. Explainable Artificial Intelligence (XAI) techniques, such as Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP), were incorporated to improve model interpretability, allowing engineers to understand and validate the FDD system's decision-making process. Simulation results confirm that the proposed methodology accurately identifies and classifies faults. By integrating dynamic simulations, advanced machine learning, and XAI techniques, a robust and scalable solution is achieved for fault detection in distillation columns, improving operational reliability, safety, and reducing downtime. Future work could extend this approach to other industrial processes and explore additional machine learning algorithms to further enhance performance.

Determination of bone age is a critical issue for forensics, surgery, and basic sciences. This study aims to estimate age with high accuracy and precision using Machine Learning (ML) algorithms with parameters obtained from calcaneus x-ray images of healthy individuals. The study was carried out by retrospectively examining the foot X-ray images of 341 people aged 18-65 years. Maximum width of the calcaneus (MW), body width (BW), maximum length (MAXL), minimum length (MINL), facies articularis cuboidea height (FACH), maximum height (MAXH), and tuber calcanei width (TKW) parameters were measured from the images. The measurements were then grouped as 20-45 years of age, 46-64 years of age, 65 and older, and age estimation was made by using these at the input of ML models. As a result of the ML input of the measurements obtained, a 0.85 Accuracy (Acc) rate was obtained with the Extra Tree Classifier algorithm. The accuracy rate of other algorithms was found to vary between 0.78 and 0.82. The contribution of parameters to the overall result was evaluated by using the shapley additive explanations (SHAP) analyzer of Random Forest algorithm and the MAXH parameter was found to have the highest contribution in age estimation. As a result of our study, calcaneus bone was found to have high accuracy and precision in age estimations.

Prediction of a machine’s Remaining Useful Life (RUL) is one of the key tasks in predictive maintenance. The task is treated as a regression problem where Machine Learning (ML) algorithms are used to predict the RUL of machine components. These ML algorithms are generally used as a black box with a total focus on the performance without identifying the potential causes behind the algorithms’ decisions and their working mechanism. We believe, the performance (in terms of Mean Squared Error (MSE), etc.,) alone is not enough to build the stakeholders’ trust in ML prediction rather more insights on the causes behind the predictions are needed. To this aim, in this paper, we explore the potential of Explainable AI (XAI) techniques by proposing an explainable regression framework for the prediction of machines’ RUL. We also evaluate several ML algorithms including classical and Neural Networks (NNs) based solutions for the task. For the explanations, we rely on two model agnostic XAI methods namely Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP). We believe, this work will provide a baseline for future research in the domain.