How do ML practitioners perceive explainability? an interview study of practices and challenges

The accurate detection and assessment of stress play a pivotal role in enhancing individual well-being and healthcare outcomes. Traditional methods of stress detection often grapple with limitations in accuracy and scalability. With the advent of machine learning (ML), the potential to revolutionize stress detection has emerged. This paper presents a comprehensive study on the application of ML algorithms for stress detection, with a focus on physiological and behavioral data analysis. Central to our approach is the integration of SHapley Additive exPlanations (SHAP), an Explainable Artificial Intelligence (XAI) technique, to interpret ML models. SHAP provides a novel lens to understand the impact of individual features in the complex decision-making processes of ML models, thereby enhancing the transparency and reliability of stress predictions. We demonstrate how SHAP not only aids in elucidating model decisions but also contributes to refining the models for greater accuracy. Our results highlight the effectiveness of ML in detecting stress and the pivotal role of XAI in making these models more interpretable and trustworthy. This study underscores the synergy between advanced ML techniques and XAI, paving the way for more nuanced and reliable stress detection methodologies that are essential in diverse settings, from healthcare to workplace environments.

Harmful Algal Blooms (HABs), driven by environmental pollution, pose significant threats to water quality, public health, and aquatic ecosystems. This study aims to enhance the prediction of HABs in Lake Erie, part of the Great Lakes system, by utilizing ensemble machine learning (ML) models coupled with explainable artificial intelligence (XAI) for interpretability. Using water quality data from 2013 to 2020, various physical, chemical, and biological parameters were analyzed to predict chlorophyll-a (Chl-a) concentrations, a proxy for algal blooms. The study employed multiple ensemble ML models, including Random Forest (RF), Deep Forest (DF), Gradient Boosting (GB), and XGBoost, and compared their performance against individual models such as Support Vector Machine (SVM), Decision Tree (DT), and Multi-Layer Perceptron (MLP). The findings reveal that ensemble models, particularly XGBoost and Deep Forest (DF), achieve superior predictive accuracy with R² values of 0.8517 and 0.8544, respectively. The application of SHapley Additive exPlanations (SHAP) provided insights into the relative importance of input features, identifying Particulate Organic Nitrogen (PON), Particulate Organic Carbon (POC), and Total Phosphorus (TP) as critical factors influencing Chl-a concentrations. This research demonstrates the effectiveness of integrating ensemble ML models with XAI to improve HAB prediction accuracy and interpretability. The results support the development of proactive water quality management strategies and highlight the potential of advanced ML techniques in environmental monitoring.

Hydrothermal carbonization (HTC) is a promising process for biomass valorization; however, optimizing HTC conditions and characterizing hydrochar through experimental methods remain costly and time-consuming. Artificial intelligence (AI) and its related machine learning (ML) techniques provide an efficient and cost-effective alternative, enabling efficient optimization and predictive analysis without extensive experimental tests. In this study, an analysis-ready database comprising 544 data points was constructed using the authors' previous research (41) and scattered data compiled from the literature (503). An ensemble of eight diversified machine learning (ML) models was developed using biomass-agnostic properties and process conditions to predict hydrochar properties including elemental analysis, proximate analysis, and hydrochar yield. Tailor-made decision fusion models were developed for each target by merging the outputs of the best-performing models. Furthermore, a set of interpretable machine learning (IML) and explainable AI (XAI) techniques, leveraging feature importance analysis and SHapley Additive exPlanations (SHAP) values, indicated that biomass fixed carbon (FC) content and process temperature are the most influencing features, which were then considered as inputs for the ensemble models. The decision fusion models achieved high accuracy for target prediction, surpassing the currently used models in the literature, with adjusted R² values ranging from 0.98 to 1.0. For outputs that are typically difficult to predict, such as hydrochar yield, the model achieved an adjusted R² of 0.98, representing over a 5% improvement compared to the best-performing model reported in the literature. All predictions were derived from white-box modeling by incorporating XAI into the ensemble-based learning, ensuring greater model transparency and interpretability. • Human-centric AI framework developed to model HTC and predict hydrochar properties. • Eight diverse ML models fused via decision fusion to boost accuracy and robustness. • Models achieved high predictive power (Test R² = 0.98–1.00) for studied properties • SHAP explainability enabled transparent, white-box modeling for HTC process

Partial dependence plots (PDPs) and individual conditional expectation (ICE) plots are visualizations used for explaining the behavior of machine learning (ML) models trained on tabular datasets. They show how the values of a feature or pair of features impact a model's predictions. However, in models with a large number of features, it is impractical for an ML practitioner to analyze all possible plots. To address this, we present new techniques for ranking and filtering PDP and ICE plots and build upon existing strategies for clustering the lines in ICE plots. Together, these techniques aim to help ML practitioners efficiently explore PDP and ICE plots and identify interesting model behavior. We integrate these techniques into PDPilot, a visual analytics tool that runs in Jupyter notebooks. We use PDPilot to study how 7 ML practitioners utilize the ranking, filtering, and clustering techniques to analyze an ML model.

This study utilizes advanced machine learning (ML) algorithms to assess the influence of external weather-related factors on the energy efficiency (EE) of residential buildings. Two deep neural network (DNN) models were developed: a standard DNN and an enhanced one incorporating residual layers and attention mechanisms. The models were trained using a dataset comprising historical meteorological data and energy performance data from residential homes in York, UK. The investigation indicated that energy consumption is substantially influenced by meteorological factors including temperature (denoted as heating degree days (HDD) and cooling degree days (CDD)), humidity, wind velocity, and sun radiation. The random forest model surpassed previous models, attaining root mean square error (RMSE) of 0.0819 and a R 2 score of 1.0000, underscoring its exceptional capacity to represent the intricate interactions between meteorological factors and energy usage.The advanced deep neural network demonstrated favourable outcomes with an RMSE of 4.12 and a R 2 of 0.878, underscoring the significance of employing complicated architectures to represent intricate relationships. The results underscore the need of integrating ML for EE forecasting and identify critical domains for enhancing household energy consumption. The study compared multiple ML models, highlighting their benefits and weaknesses for prediction accuracy and resilience. This is followed by the application of SHAP (SHapley Additive exPlanations) to interpret model outputs and obtain an understanding of the significance of these weather features.

Gu et al. conducted a comprehensive survey on the design and application of electrocatalysts powered by machine learning techniques [1]. They presented a novel approach that utilizes Artificial Neural Networks (ANN) in conjunction with the SHAP (SHapley Additive exPlanations) method to optimize membrane electrode assemblies. The ANN model demonstrated high accuracy in predicting key performance metrics, achieving root mean square error (RMSE) values of 43.536 mW cm−2 for power density and 0.070 gPt kW−1 for platinum utilization. Additionally, the SHAP method was employed to identify the most influential features affecting the target outputs, providing valuable insights into the optimization process [1]. However, this paper raises significant theoretical and empirical concerns regarding the use of ANN in conjunction with SHAP due to the model-specific nature of these techniques, which can lead to erroneous interpretations. It appears that Gu et al. may not fully grasp the fundamental principles underlying machine learning. In supervised machine learning models like ANN, two types of accuracy are crucial: target prediction accuracy and feature importance reliability. While target prediction accuracy can be validated against known ground truth values, the derived feature importances from models lack equivalent ground truth for validation. As a result, achieving high target prediction accuracy does not ensure that the feature importances are also reliable, since there are no established ground truth values for these features. The function call “explain = SHAP(model)” further indicates that SHAP may inherit and potentially amplify any biases present in the feature importances derived from the underlying model (ANN), leading to misleading interpretations of the results [2-5]. This highlights the importance of critically evaluating both the predictions and the interpretability provided by model-agnostic methods like SHAP. In light of these concerns, the paper advocates for a more robust and multifaceted approach utilizing unsupervised machine learning techniques, such as Feature Agglomeration (FA) and Highly Variable Gene Selection (HVGS). FA is a dimensionality reduction technique that aggregates similar features, thereby simplifying the data set and reducing noise, which can enhance the interpretability of the model and the reliability of its predictions. HVGS focuses on selecting a subset of features that exhibit significant variability across samples, ensuring that only the most informative features are retained for further analysis. Following the feature selection process, the authors suggest employing nonlinear nonparametric statistical methods, such as Spearman's correlation, to assess the relationships between features and outcomes. Spearman's correlation evaluates the strength and direction of the association between ranked variables, making it particularly useful in identifying monotonic relationships that do not necessarily follow a linear pattern. The accompanying p-values provide a measure of statistical significance, offering insights into the reliability of these correlations. By leveraging these advanced methods, researchers can achieve a deeper understanding of the key factors influencing electrocatalyst performance, while also mitigating the risks associated with relying solely on model-specific interpretations. Yoshiyasu Takefuji completed this study and wrote this article. The author has nothing to report. The author has nothing to report. The author declares no conflicts of interest. The author has nothing to report.

Interfaces for machine learning (ML), information and visualizations about models or data, can help practitioners build robust and responsible ML systems. Despite their benefits, recent studies of ML teams and our interviews with practitioners (n=9) showed that ML interfaces have limited adoption in practice. While existing ML interfaces are effective for specific tasks, they are not designed to be reused, explored, and shared by multiple stakeholders in cross-functional teams. To enable analysis and communication between different ML practitioners, we designed and implemented Symphony, a framework for composing interactive ML interfaces with task-specific, data-driven components that can be used across platforms such as computational notebooks and web dashboards. We developed Symphony through participatory design sessions with 10 teams (n=31), and discuss our findings from deploying Symphony to 3 production ML projects at Apple. Symphony helped ML practitioners discover previously unknown issues like data duplicates and blind spots in models while enabling them to share insights with other stakeholders.

Explainable artificial intelligence (XAI) for interpreting the contributing factors feed into the wildfire susceptibility prediction model

In the field of network security, botnet attacks pose a significant challenge, exploiting networks of infected devices to launch sophisticated threats. As these attacks evolve, the need for effective detection methods becomes increasingly critical. This study proposes an explainable Machine Learning (ML) model that aims to identify and categorize botnet attacks, and it investigates the efficiency of different Explainable Artificial Intelligence (XAI) techniques for Intrusion Detection Systems (IDS). To train our models, we employed a subset of the NCC-2 dataset, which includes a mix of normal traffic and seven different types of botnet attacks across three sensors. The ML techniques selected for this research are Random Forest (RF), Extra Trees (ET), Decision Tree (DT), and K-Nearest Neighbors (KNN), with a GridSearch cross-validation approach for optimal hyperparameter tuning. We also explored the effects of class balance through Synthetic Minority Oversampling (SMOTE) and Random Undersampling. The models’ performance was rigorously tested using the Matthews Correlation Coefficient (MCC) and Macro F1-score, with the ET model demonstrating superior results of 99% MCC and 97% F1-score, respectively. To enhance the interpretability of the ET model’s decision-making process, we integrated three XAI techniques: SHapley Additive exPlanations (SHAP), Dalex, and Local Interpretable Model-agnostic Explanations (LIME), and evaluated their efficiency. The LIME and Dalex techniques showed efficient construction times for IDS.

This study introduces an efficient methodology for addressing fault detection, classification, and severity estimation in rolling element bearings. The methodology is structured into three sequential phases, each dedicated to generating distinct machine-learning-based models for the tasks of fault detection, classification, and severity estimation. To enhance the effectiveness of fault diagnosis, information acquired in one phase is leveraged in the subsequent phase. Additionally, in the pursuit of attaining models that are both compact and efficient, an explainable artificial intelligence (XAI) technique is incorporated to meticulously select optimal features for the machine learning (ML) models. The chosen ML technique for the tasks of fault detection, classification, and severity estimation is the support vector machine (SVM). To validate the approach, the widely recognized Case Western Reserve University benchmark is utilized. The results obtained emphasize the efficiency and efficacy of the proposal. Remarkably, even with a highly limited number of features, evaluation metrics consistently indicate an accuracy of over 90% in the majority of cases when employing this approach.

Background: Stroke remains a critical global health concern, disproportionately affecting individuals with hypertension which is a well-established, modifiable risk factor. Traditional risk scoring systems often fall short in accurately predicting stroke onset due to their reliance on fixed clinical thresholds and limited variable interaction modeling. As the complexity of health data increases, machine learning (ML) and explainable artificial intelligence (XAI) present powerful tools to uncover hidden patterns and enable precision risk stratification. Objectives: This study proposes a novel, interpretable machine learning framework that uses ensemble learning and SHapley Additive exPlanations (SHAP) to enhance stroke risk prediction in hypertensive patients. The objective is twofold: to improve the predictive power of stroke models and to provide clinically relevant insights that support real-time, data-driven decisions in preventive care. Methods: We utilized a real-world clinical dataset encompassing demographic, physiological, and behavioral variables associated with stroke. Data preprocessing included k-nearest neighbor imputation for missing values, normalization of continuous features, and class balancing via Synthetic Minority Oversampling Technique (SMOTE). A hybrid feature selection pipeline, combining the sparsityenforcing capabilities of LASSO regression with the iterative refinement of Recursive Feature Elimination (RFE), was employed to identify the most salient predictors. Multiple ML models, including logistic regression, deep neural networks, random forests, and gradient boosting machines, were trained and validated using cross-validation. SHAP values were computed post-training to enable individualized, interpretable model outputs. Results: Ensemble models, particularly Gradient Boosting and Random Forest, demonstrated superior discriminative performance, achieving AUC-ROC scores above 0.78 following class balancing. The integrated LASSO-RFE approach revealed age, hypertension status, and average glucose levels as dominant predictors across models. SHAP visualizations confirmed the influence of these features, while also highlighting nuanced interactions involving lifestyle and socioeconomic variables. Logistic Regression, when optimized for recall, achieved the highest balanced accuracy (0.77), reinforcing the clinical utility of simpler models when interpretability is paramount. Conclusion: This study introduces a transparent and high-performing machine learning framework for stroke risk prediction in hypertensive individuals. By integrating ensemble learning, hybrid feature selection, and explainable AI, the framework bridges the gap between predictive modeling and clinical applicability. These findings support the deployment of interpretable ML tools in routine care, enabling proactive interventions, personalized patient education, and ultimately, reduced stroke incidence.

The allure of aesthetic appeal in images captivates our senses, yet the underlying intricacies of aesthetic preferences remain elusive. In this study, we pioneer a novel perspective by utilizing several different machine learning (ML) models that focus on aesthetic attributes known to influence preferences. Our models process these attributes as inputs to predict the aesthetic scores of images. Moreover, to delve deeper and obtain interpretable explanations regarding the factors driving aesthetic preferences, we utilize the popular Explainable AI (XAI) technique known as SHapley Additive exPlanations (SHAP). Our methodology compares the performance of various ML models, including Random Forest, XGBoost, Support Vector Regression, and Multilayer Perceptron, in accurately predicting aesthetic scores, and consistently observing results in conjunction with SHAP. We conduct experiments on three image aesthetic benchmarks, namely Aesthetics with Attributes Database (AADB), Explainable Visual Aesthetics (EVA), and Personalized image Aesthetics database with Rich Attributes (PARA), providing insights into the roles of attributes and their interactions. Finally, our study presents ML models for aesthetics research, alongside the introduction of XAI. Our aim is to shed light on the complex nature of aesthetic preferences in images through ML and to provide a deeper understanding of the attributes that influence aesthetic judgements.

This study presents a comprehensive methodology for developing a Fault Detection and Diagnosis (FDD) system for an industrial distillation column using advanced machine learning algorithms. Steady state and dynamic simulations in Aspen Plus® generate extensive datasets under normal and faulty conditions. Feature engineering, using the Minimum Redundancy Maximum Relevance (MRMR) algorithm, selects the most relevant features for fault detection. Various machine learning models, including Decision Trees, Support Vector Machines, k-nearest Neighbours, and Neural Networks, were trained and evaluated based on performance metrics such as accuracy, recall, precision, and F1 score. The top models were integrated into a stacked classifier system with a voting mechanism to enhance fault detection reliability. Explainable Artificial Intelligence (XAI) techniques, such as Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP), were incorporated to improve model interpretability, allowing engineers to understand and validate the FDD system's decision-making process. Simulation results confirm that the proposed methodology accurately identifies and classifies faults. By integrating dynamic simulations, advanced machine learning, and XAI techniques, a robust and scalable solution is achieved for fault detection in distillation columns, improving operational reliability, safety, and reducing downtime. Future work could extend this approach to other industrial processes and explore additional machine learning algorithms to further enhance performance.

Machine learning-based design of target property-oriented fuels using explainable artificial intelligence

This study aimed to evaluate the performance of machine learning (ML) models in the diagnosis of Hepatitis C Virus (HCV) patients and to identify clinical biomarkers using explainable artificial intelligence (XAI) approaches. Black box algorithms - Random Forest (RF) and Extreme Gradient Boosting (XGBoost) were used in the study, and XAI methods - SHapley Additive Explanations (SHAP) and Accumulated Local Effects (ALE) were applied to increase the interpretability of the models. There were 615 patients in the dataset, and the output variable included various liver disorders including HCV. The results showed that RF and XGBoost models exhibited high performance with 93.75% and 92.38% accuracy rates, respectively. SHAP and ALE analyses revealed the importance and interactions of the factors (ALT, AST, bilirubin, albumin, age) underlying model decisions. This study demonstrates the potential of ML models in early diagnosis of HCV infection and how they can be integrated with XAI methods to make them more reliable in medical applications.