Explain ability and interpretability in machine learning models

Interpretable vs. noninterpretable machine learning models for data-driven hydro-climatological process modeling

Annual burned areas in the United States have increased twofold during the past decades. With more large fires resulting in more emissions of fine particulate matter, an accurate prediction of fire emissions is critical for quantifying the impacts of fires on air quality, human health, and climate. This study aims to construct a machine learning (ML) model with game-theory interpretation to predict monthly fire emissions over the contiguous US and to understand the controlling factors of fire emissions. By comparing the predicted fire PM2.5 emissions from the interpretable ML model with the Global Fire Emissions Database (GFED) observations and predictions from process-based models in the Fire Modeling Intercomparison Project (FireMIP), the ML model is also used to diagnose the process-based models to inform future development. Results show promising performance for the ML model, Community Land Model (CLM), and Joint UK Land Environment Simulator-Interactive Fire And Emission Algorithm For Natural Environments (JULES-INFERNO) in reproducing the spatial distributions, seasonality, and interannual variability of fire emissions over CONUS. Regional analysis shows that only the ML model and CLM simulate the realistic interannual variability of fire emissions for most of the subregions (r > 0.95 for ML and r = 0.14 ~ 0.70 for CLM), except for Mediterranean California, where all the models perform poorly (r = 0.74 for ML and r < 0.30 for the FireMIP models). Regarding seasonality, most models capture the peak emission in July over western US. However, all models except for the ML model fail to reproduce the bimodal peaks in July and October over Mediterranean California, which may be explained by the coarse spatial resolutions of the processed-based models or atmospheric forcing data or limitations in model parameterizations for capturing the effects of Santa Ana winds on fire activity. Furthermore, most models struggle to capture the spring peak in emissions in southeastern US, probably due to underrepresentation of human effects and the influences of winter dryness on fires in the models. As for extreme events, both the ML model and CLM successfully reproduce the frequency map of extreme emission occurrence but overestimate the number of months with extremely large fire emissions. Comparing the fire PM2.5 emissions from the interpretable ML model with process-based fire models highlights their strengths and uncertainties for regional analysis and prediction and provides useful insights on future directions for model improvements.

This paper analyzes the predictions of image captioning models with attention mechanisms beyond visualizing the attention itself. We develop variants of Layer-wise Relevance Propagation (LRP) and gradient-based explanation methods, tailored to image captioning models with attention mechanisms. We compare the interpretability of attention heatmaps systematically against the explanations provided by explanation methods such as LRP, Grad-CAM, and Guided Grad-CAM. We show that explanation methods provide simultaneously pixel-wise image explanations (supporting and opposing pixels of the input image) and linguistic explanations (supporting and opposing words of the preceding sequence) for each word in the predicted captions. We demonstrate with extensive experiments that explanation methods (1) can reveal additional evidence used by the model to make decisions compared to attention; (2) correlate to object locations with high precision; (3) are helpful to “debug” the model, e.g. by analyzing the reasons for hallucinated object words. With the observed properties of explanations, we further design an LRP-inference fine-tuning strategy that reduces the issue of object hallucination in image captioning models, and meanwhile, maintains the sentence fluency. We conduct experiments with two widely used attention mechanisms: the adaptive attention mechanism calculated with the additive attention and the multi-head attention mechanism calculated with the scaled dot product.

Machine learning models (MLMs) used to influence surgical decision making often require large and complex datasets upon which to train. However, there is a paucity of literature pertaining to the ability to apply standard MLMs to small ICU datasets within resource-constrained institutions. ML models were applied to a prospective cohort of critically ill mechanically ventilated patients from a teaching hospital in rural Kenya. Characteristics included an ICU scoring system specifically for resource-constrained environments (Tropical Intensive Care Score (TropICS)). Outputs included AUC of the ROC and the feature importance table. Python-based MLMs included XGBoost and KNN. AUC of the ROC was calculated to predict mortality as the primary endpoint. There were 294 patients, with an average age of 40.2years, 64.3% male, 23.8% trauma, and an overall mortality of 60.2%. With respect to mortality patients who died were older (43.5 versus 35years; p < 0.001), but with no difference in male gender (64.8% versus 63.8%; p = 0.9), or having been transferred from outside facilities (34% versus 21.5%; p = 0.5). Whilst there was no difference in the rate of tachycardia or acidosis, patients who died were more likely to present with hemodynamic instability (31% versus 6%; p < 0.001) and higher clinical severity scores. In predicting mortality, the ML models performed very well (XGBoost AUC = 0.82). Within MLM feature importance, the Tropical Intensive Care Score (TropICS) performed as well as APACHE-II and the SAPS. ML models can be effectively applied to a small ICU dataset within resource-constrained environments. ML models must demonstrate functionality prior to incorporating within prospective clinical predictive models. Contextualized ICU scoring systems (TropICS) performed well within MLMs.

Aim: The majority of machine learning (ML) models in healthcare are built on retrospective data, much of which is collected without explicit patient consent for use in artificial intelligence (AI) and ML applications. The primary aim of this study was to evaluate whether clinicians and scientific researchers themselves consent to provide their own data for the training of ML models. Materials and Methods: This survey was conducted through an anonymous online survey, utilizing platforms such as Telegram, LinkedIn, and Viber. The target audience comprised specific international groups, primarily Russian, German, and English-speaking, of clinicians and scientific researchers. These participants ranged in their levels of expertise and experience, from beginners to veterans. The survey centered on a singular, pivotal question: “Do You Consent to the Use of Your Biological and Private Data for Training Machine Learning and AI Models?” Respondents had the option to choose from three responses: “Yes” and “No”. Results: The survey was conducted in January 2024. A total of 119 unique and verified individuals participated in the survey. The results revealed that only 50% of respondents (63 persons) expressed consent to provide their own data for the training of ML and AI models. Conclusion: In the development of ML and AI models, particularly open-source ones, it is crucial to ascertain whether participants are willing to provide their private data. While ML algorithms can transform the nature of data, it is important to remember that the primary owner of this data is the individual. Our findings show that in 50% of the cases, even participants from scientific research and clinical backgrounds – individuals typically accountable for ensuring data quality in AI and ML model development – do not consent to the use of their data in AI and ML settings. This highlights the need for more stringent consent processes and ethical considerations in the utilization of personal data in AI and ML research.

Theory-Guided Machine Learning Finds Geometric Structure-Property Relationships for Chemisorption on Subsurface Alloys

This study employs Random Forests (RF) and Artificial Neural Networks (ANN) to model the transient behavior of Ni catalyst deactivation during steam and dry reforming of model biogas containing H2S, with a focus on hydrogen production. Deactivation, induced by carbon deposition and sulfur poisoning, is a complex and transient phenomenon demanding precise kinetic mechanisms for accurately predicting Ni catalyst behavior in biogas reforming. Black-box machine learning (ML) models are developed, incorporating catalyst properties, biogas composition, and operating conditions. Encompassing both dry and steam reforming, the ML models aim to predict catalyst behavior, expressed in terms of packed bed reactor exit mole fractions (H2, CO, CH4, and CO2) and conversions (CH4 and CO2). The ML models are trained and tested across a temperature range of 700–900 C with 0–145 ppm of H2S in model biogas (CH4/CO2 ratio varying from 1.0 to 2.0). RF outperforms the ANN across all performance metrics, including overall R2 and root mean squared error (RMSE). The RF achieves a mean overall R2 of 0.979, with training and testing RMSE equal to 6.7×10−3 and 1.47×10−2 respectively. In contrast, the ANN achieves a mean overall R2 of 0.939, with training and testing RMSE equal to 2.6×10−2 and 2.55×10−2 respectively. Moreover, pre-trained RF models are validated with unseen data of dry reforming of biogas containing 30 ppm of H2S (25 data points). It is suggested that 35 % of this unseen experimental data is required to train the RF model for it to predict catalyst deactivation, achieving a validation R sufficiently2> 0.9. The mean overall R2 values attained by the RF fine-tuned on 35 % of the unseen experiment data for both CH4 and CO2 conversions, as well as for all mole fraction predictions, are 0.952 and 0.948, respectively.

Data reformation – A novel data processing technique enhancing machine learning applicability for predicting streamflow extremes

Background: Coronary Artery Disease (CAD) is one of the biggest causes of mortality worldwide. Risk stratification for early detection is essential for the primary prevention of CAD. QRISK3 is known to overestimate future CAD risk in some populations, resulting in unnecessary preventive treatment and reduced cost-effectiveness and safety. Combining machine learning model with the metaheuristic optimisation approach using the PSO algorithm may outperform QRISK3 in predicting CAD. It improves performance by selecting the best-performing subset of features related to clinical outcomes. Research Question: Does the performance of Machine Learning Models Combined with PSO Algorithm for Feature Selection as a Metaheuristic Optimisation Approach in Predicting Coronary Arterial Disease using the UK Biobank dataset outperform the QRISK3 calculator? Aims: This study is to assess the accuracy of QRISK3 in predicting CAD using the UK Biobank dataset. It aims to evaluate the efficacy of machine learning models on the identical dataset for predicting CAD. The work utilises the PSO algorithm for feature selection to identify the optimal subset of features from the UK Biobank dataset. Methods: This study utilises data from the UK Biobank. The dataset consists of 348,015 participants aged 24-84 with no prior diagnosis of CAD. The performance of both QRISK3 and machine learning models was evaluated separately using ROC analysis. Several machine learning models were employed: Logistic Regression, Decision Tree, Random Forest, Naïve Bayes, and Gradient Boosting. The dataset was split into training and test sets with a ratio of 4:1 for the machine learning models. Each model has been developed by adding a PSO algorithm to enhance the model's classification accuracy. Results: Out of the total 348,015 participants, 23,136 individuals (6.64%) were diagnosed with CAD within 10 years following their first visit, while 324,879 individuals (93.4%) did not develop CAD. The AUC value of the QRISK3 prediction is 0.6113, while the combined machine learning models using the PSO algorithm using the gradient boosting model achieve an AUC of 0.7258, showing better performance. Conclusions: This study shows hybrid machine learning models optimised with the PSO algorithm can better predict CAD than QRISK3. These ML models can effectively identify high-risk CAD patients, allowing for more personalised preventative strategies and supporting policymakers in implementing lifestyle change recommendations.

Background: The air quality of any area depends upon the various PMs (particulate matter) and hazardous gases present in the air. Low-cost PM sensors and gas sensors are present in different target places to monitor the air quality, read the environmental data, and transmit it to local servers through the IoT device. The low-cost sensor is not reliable due to its low sensing capacity; therefore, the read data is calibrated with the meteorological data presented by the nearby meteorological Centre of that particular area. The calibrated reading data sent to the server could be analyzed through some Machine Learning [ML] models. The ML models help to predict the risk of asthma in a particular area. The risk of asthma is directly related to the air quality of the surroundings. It is observed that the air quality of the industrial area is much worse than the non-industrial belt. Air quality monitoring of industrial areas is always a challenging task due to the ununiformed pollution in some segregated places around the industry, emitting pollutants mostly from chimneys. The air quality of any area depends upon the PM (PM), i.e., PM2.5 and PM10.0, as well as the gasses like NO2(Nitrogen Dioxide), NH3 (Ammonia), SO2(Sulfur dioxide), CO(Carbon monoxide), O3(Ozone) and Benzene. These are the most hazardous gases generally emitted by common heavy industries like iron and steel. In this article, the researchers considered the industrial belt of the Asansol- Durgapur region of West Bengal, India, and predicted the risk of asthma attacks for the test dataset. The experiment was carried out on 10 different supervised machine learning [SML] models as well as semi-supervised machine learning (SSML) models. The SML models have been further refined through hyper-parameter tuning, and better results have been obtained in the case of some ML models. The result has been compared with the existing literature considering the same external environment from where the meteorological data was collected, and similar ML models have been used. The research outperformed the existing literature, which is depicted in the result and analysis section of the article. Methods: The study evaluated ML models, both supervised and semi-supervised, to assess pollution levels. Relevant features were selected while less relevant ones were discarded. Accuracy levels of different ML algorithms werecompared in the results. The most effective model for an IoT system was chosen to maximize accuracy. In semi-supervised learning, feature selection followed supervised learning, but testing was akin to unsupervised learning. Results were compared with supervised learning data, enhancing reliability. Results: The result employing various classifiers werepresented across tables after the independent parameter Ozone was removed. Following the output of several classifiers, the results were verified using the k-fold validation method, with k being set to 5 or 10, accordingly. Tables display the best outcome, which is indicated in bold characters. method: In this research work the researcher considered 9 different ML models and used them as supervised as well as semi supervised model to determine the pollution level of the certain area. In this research work the researcher also selected the most relevant features and discarded the less relevant features. In case of SML algorithm, the accuracy level of the different ML algorithm has been determined and depicted in the result analysis section. The most effective ML model has been chosen for the proposed embedded system so that accuracy could be achieved at most. In case of semi supervised algorithm the feature selection is done as per the supervised algorithm. In this case the training is done same as the SML algorithm, but the testing phase is done like unsupervised machine learning algorithm where the decision parameter is predicted and ultimately matched with the previously achieved data of SML algorithm. The reliability of this approach is much more effective than simple SML algorithm. Conclusion: This study focused on predicting asthma risk in the Asansol-Durgapur industrial belt, India, using low-cost PM and gas sensors. Data calibration with meteorological inputs enhanced accuracy. ML models predicted risk and were refined through hyper-parameter tuning. Comparative analysis showed superior performance, emphasizing the importance of precise air quality monitoring. While offering a robust framework for future research, the study’s limitation lies in its area-specific dataset.

The increasing complexity of social science data and phenomena necessitates using advanced analytical techniques to capture nonlinear relationships that traditional linear models often overlook. This chapter explores the application of machine learning (ML) models in social science research, focusing on their ability to manage nonlinear interactions in multidimensional datasets. Nonlinear relationships are central to understanding social behaviors, socioeconomic factors, and psychological processes. Machine learning models, including decision trees, neural networks, random forests, and support vector machines, provide a flexible framework for capturing these intricate patterns. The chapter begins by examining the limitations of linear models and introduces essential machine learning techniques suited for nonlinear modeling. A discussion follows on how these models automatically detect interactions and threshold effects, offering superior predictive power and robustness against noise compared to traditional methods. The chapter also covers the practical challenges of model evaluation, validation, and handling imbalanced data, emphasizing cross-validation and performance metrics tailored to the nuances of social science datasets. Practical recommendations are offered to researchers, highlighting the balance between predictive accuracy and model interpretability, ethical considerations, and best practices for communicating results to diverse stakeholders. This chapter demonstrates that while machine learning models provide robust solutions for modeling nonlinear relationships, their successful application in social sciences requires careful attention to data quality, model selection, validation, and ethical considerations. Machine learning holds transformative potential for understanding complex social phenomena and informing data-driven psychology, sociology, and political science policy-making.

Severe Fever with Thrombocytopenia Syndrome (SFTS) is a severe tick-borne viral infection with high mortality, making the timely prediction of clinical deterioration critical. Current predictive models lack timeliness and generalizability. Therefore, this study aimed to develop and validate an interpretable machine learning model for the early prediction of deterioration in SFTS patients. We retrospectively analyzed 560 SFTS patients from two hospitals. Using clinical and laboratory data from a training set (n = 407), we developed eight ML models. An independent test set (n = 153) was used for external validation. Model performance was assessed via area under the ROC curve (AUC), calibration, decision curve analysis (DCA), and SHapley Additive exPlanations (SHAP). The RF model outperformed others with an external validation AUC of 0.825. Key predictors included viral load, serum creatinine, D-dimer, procalcitonin, platelets, blood urea nitrogen, age, and lymphocytes. SHAP analysis revealed significant interactions, especially between blood urea nitrogen and viral load. The RF model provided reliable risk stratification, with net benefits surpassing no-treatment or all-treatment strategies when the threshold probability was between 0.09 and 0.76. An interactive web-based application was developed for real-time individualized risk prediction. We successfully developed and validated a robust ML model using RF, integrating eight readily measurable clinical variables to predict early deterioration in SFTS patients. This model offers improved timeliness and interpretability, facilitating early clinical interventions. Future multi-center studies will further validate its robustness and reliability, enhancing its clinical utility.

Rapid advances in strategic decision-making in sports analytics, particularly in basketball, have been driven by data-centric computer vision and machine learning (CV-ML) techniques that aim to optimise player performance, reduce injury risk, and support training and competition strategies. This extensive review synthesises recent advancements in ML and deep learning (DL) models applied to strategic decision-making within the context of collegiate basketball. Following PRISMA guidelines, we systematically reviewed 106 peer-reviewed studies published between 2010 and 2025. Each study was assessed across multiple dimensions-model complexity (e.g., classical ML vs. black-box DL models), feature-set exhaustiveness (e.g., only game statistics vs. multimodal data including internal and external stressors), and outcome utility, encompassing both predictive accuracy and the availability of explainable AI (xAI) outputs. The review concluded that athletic performance modelling in collegiate basketball is most effective when interpretable classical ML models are applied to structured and multimodal datasets, and further enhanced through XAI techniques that improve interpretability, build trust, and offer actionable insights for coaching and decision-making.

Improving Clinical Decisions in IR: Interpretable Machine Learning Models for Predicting Ascites Improvement after Transjugular Intrahepatic Portosystemic Shunt Procedures

Chronic pain around the temporomandibular joint (TMJ) and masticatory muscles is a primary symptom of temporomandibular disorders (TMD). However, the clinical significance of magnetic resonance imaging (MRI) features in predicting TMD-related pain remains unclear. This study aimed to develop and interpret machine learning (ML) models based on MRI characteristics for predicting pain severity in patients with TMD. The present retrospective study included 584 patients with TMD between January 2022 and December 2024, yielding a total of 755 TMJ MRI data sets. Pain severity was classified using the visual analogue scale (VAS). Demographic variables (age, sex) and MRI features-including lesion side, disc position, disc morphology, disc signal, disc perforation, bilaminar zone tear, joint space, joint effusion, condylar movement, bony changes and morphology/signal of the lateral pterygoid muscle-were collected. Eleven ML models based on demographic and MRI features were developed: logistic regression (LR), support vector machine (SVM), random forest (RF), extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), adaptive boosting (AdaBoost), gradient boosting classifier (GBC), bagging classifier (BC), extremely randomised trees (ETC), decision tree classifier (DTC) and multilayer perceptron (MLP). Model performance was evaluated using multiple metrics, including the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity and F1 score. Precision-recall (PR) curves and calibration curves were plotted to assess discrimination and model calibration. Decision curve analysis (DCA) was conducted to evaluate the clinical net benefit across a range of threshold probabilities. Model interpretability was enhanced using Shapley Additive Explanations (SHAP), which quantified the contribution of each feature to individual predictions. Feature selection was conducted based on mean SHAP values, and separate LightGBM models were constructed using the Top 3, 5, and 9 most important features, as well as the full-feature set, for performance comparison. The data set was randomly divided into a training set (n = 604) and a test set (n = 151). Among the 11 ML models, the LightGBM model demonstrated the best predictive performance, with an AUC of 0.899, and was therefore identified as the optimal model. SHAP analysis identified age, disc position and condylar movement as the top three contributing features. Feature selection analysis indicated that selecting the top nine SHAP-ranked variables led to the highest diagnostic performance, with an AUC of 0.829. This study developed an interpretable, high-performing MRI-based ML model incorporating SHAP analysis to integrate imaging and clinical features for objective pain assessment, which may help identify high-risk TMD patients and guide personalised treatment strategies.