Enhancing Cluster Analysis With Explainable AI and Multidimensional Cluster Prototypes

Explanations based on Item Response Theory (eXirt): A model-specific method to explain tree-ensemble model in trust perspective

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the global behavior of the agent, describing the actions it takes in different states. Other approaches devised local explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a set of world-states that are likely to appear during gameplay. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps, in the form of raw heat maps, did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on during its decision-making, suggesting avenues for future work that can further improve explanations of RL agents.

In a Content Based Image Retrieval (CBIR) system, images are retrieved based on the content of the images such as color, shapes and objects. However, often end users experience a semantic gap between the notion of similarity used by the ranking model versus the users’ perception about image similarity. Explainable AI (XAI) is an emerging research field that attempts to provide transparency of “black box” models, to make AI systems trustworthy and gain user trust. This work aims at building an I mage R etrieval system with T extual EX planations such as “The (global) results are similar to the query by X% due to shape, Y% due to color”. Local explanations are generated using various methods such as comparing images with respect to overlap between low level features such as color, shape and regions (MPEG-7 features). Additionally, high level features such as background-foreground segmentation, deep learned features and major key-points, objects identified (SIFT features) are used to enrich the explanations. We evaluate the quality of rankings on benchmark data-sets such as PASCAL VOC. The XAI facet of user satisfaction and usefulness of the system is evaluated in a lab based user study. Our results show that the semantic gap is better bridged using high level features. While low level features might be better suited for re-ranking of the retrieved images.

To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.

RuleXAI—A package for rule-based explanations of machine learning model

Diabetes is a chronic disease that continues to be a primary and worldwide health concern since the health of the entire population has been affected by it. Over the years, many academics have attempted to develop a reliable diabetes prediction model using machine learning (ML) algorithms. However, these research investigations have had a minimal impact on clinical practice as the current studies focus mainly on improving the performance of complicated ML models while ignoring their explainability to clinical situations. Therefore, the physicians find it difficult to understand these models and rarely trust them for clinical use. In this study, a carefully constructed, efficient, and interpretable diabetes detection method using an explainable AI has been proposed. The Pima Indian diabetes dataset was used, containing a total of 768 instances where 268 are diabetic, and 500 cases are non-diabetic with several diabetic attributes. Here, six machine learning algorithms (artificial neural network (ANN), random forest (RF), support vector machine (SVM), logistic regression (LR), AdaBoost, XGBoost) have been used along with an ensemble classifier to diagnose the diabetes disease. For each machine learning model, global and local explanations have been produced using the Shapley additive explanations (SHAP), which are represented in different types of graphs to help physicians in understanding the model predictions. The balanced accuracy of the developed weighted ensemble model was 90% with a F1 score of 89% using a five-fold cross-validation (CV). The median values were used for the imputation of the missing values and the synthetic minority oversampling technique (SMOTETomek) was used to balance the classes of the dataset. The proposed approach can improve the clinical understanding of a diabetes diagnosis and help in taking necessary action at the very early stages of the disease.

The growing reliance on opaque deep learning models in critical domains has intensified the demand for transparency, accountability, and interpretability of artificial intelligence systems. Explainable Artificial Intelligence (XAI) seeks to address this through methods that clarify how models make predictions, fostering informed oversight and trust. This review synthesizes foundational concepts in XAI, distinguishing interpretability from explainability and contrasting intrinsic with post-hoc methods, as well as local with global explanations. It surveys major techniques such as feature attribution, gradient-based methods, surrogate models, counterfactual reasoning, and inherently interpretable architectures alongside widely used tools, frameworks, and evaluation metrics, including fidelity, human interpretability, robustness, and efficiency. Applications in healthcare, finance, law, and autonomous systems are discussed, highlighting domain-specific interpretability needs. Persistent challenges, including inconsistent terminology, subjective evaluation, and adversarial vulnerabilities, are examined. The review concludes with emerging directions in human-centered, real-time, and multimodal explanations, aiming to guide the design of XAI systems that are both technically sound and aligned with societal expectations.

Explainable artificial intelligence for predicting dengue outbreaks in Bangladesh using eco-climatic triggers.

Explainable Artificial Intelligence (XAI) is a key concept in building trustworthy machine learning models. Local explainability methods seek to provide explanations for individual predictions. Usually, humans must check these explanations manually. When large numbers of predictions are being made, this approach does not scale. We address this deficiency for a rooftop classification problem specifically with ExplainabilityAu-dit, a method that automatically evaluates explanations generated by a local explainability toolkit and identifies rooftop images that require further auditing by a human expert. The proposed method utilizes explanations generated by the Local Interpretable Model-Agnostic Explanations (LIME) framework as the most important superpixels of each validation rooftop image during the prediction. Then a bag of image patches is extracted from the superpixels to determine their texture and evaluate the local explanations. Our results show that 95.7% of the cases to be audited are detected by the proposed system.

1. Guo F, Wu T, Jin X (2020) An Efficient Method Based on Region-adjacent Embedding for Text Classification of Chinese Electronic Medical Records. Proceedings - 2020 5th International Conference on Computational Intelligence and Applications, ICCIA 183-187. 2. Lee CH, Wang C, Fan X, Li F, Chen CH (2023) Artificial intelligence-enabled digital transformation in elderly healthcare field: Scoping review Adv Eng Inform 55: 1474-0346. 3. Obaido G, Mienye ID, Egbelowo OF, Emmanuel ID, Ogunleye A, et al. (2024) Supervised machine learning in drug discovery and development: Algorithms, applications, challenges, and prospects Mach Learn Appl 17: 2666-8270, 4. Lesley U, Kuratomi Hernández A (2024) Improving XAI explanations for clinical decision-making – Physicians’ perspective on local explanations in healthcare Lecture notes in computer science, 0302-9743, 9783031665349, Springer Nature Switzerland 296-312. 5. Khalifa M, Albadawy M (2024) AI in diagnostic imaging: Revolutionising accuracy and efficiency Comput Methods Programs Biomed Update 5: 2666-9900 6. Lu SC, Swisher CL, Chung C, Jaffray D, Sidey Gibbons C (2023) On the importance of interpretable machine learning predictions to inform clinical decision making in oncology Front Oncol 10.3389/fonc.2023.1129380Google Scholar.7. Hulsen T (2023) Explainable artificial intelligence (XAI): Concepts and challenges in healthcare AI 4: 652-666. 8. Heath Goodrum, Kirk Roberts, Elmer V (2020) Bernstam Automatic classification of scanned electronic health record documents https://www.sciencedirect.com/science/article/abs/pii/S1386505620309977, https://www.researchgate.net/publication/387290255_Medical_Document_Classification_using_NLP_Techniques.9. Al Doulat A, Obaidat I, Lee M (2019) Unstructured medical text classification using linguistic analysis: A supervised deep learning approach. Proceedings of IEEE/ACS International Conference on Computer Systems and Applications, AICCSA, 2019-Novem 1-7. 10. Aldhoayan M, Zhou L (2016) An accurate and customizable text classification algorithm: Two applications in healthcare. 2016 IEEE 6th International Conference on Computational Advances in Bio and Medical Sciences, ICCABS 2016. https://ieeexplore.ieee.org/document/7802778/metrics#metrics. 11. Hughes M, Li I, Kotoulas S, Suzumura T (2017) Medical Text Classification Using Convolutional Neural Networks. Studies in Health Technology and Informatics 235: 246-250.

Necessity and sufficiency are the building blocks of all successful explanations. Yet despite their importance, these notions have been conceptually underdeveloped and inconsistently applied in explainable artificial intelligence (XAI), a fast-growing research area that is so far lacking in firm theoretical foundations. In this article, an expanded version of a paper originally presented at the 37th Conference on Uncertainty in Artificial Intelligence (Watson et al., 2021), we attempt to fill this gap. Building on work in logic, probability, and causality, we establish the central role of necessity and sufficiency in XAI, unifying seemingly disparate methods in a single formal framework. We propose a novel formulation of these concepts, and demonstrate its advantages over leading alternatives. We present a sound and complete algorithm for computing explanatory factors with respect to a given context and set of agentive preferences, allowing users to identify necessary and sufficient conditions for desired outcomes at minimal cost. Experiments on real and simulated data confirm our method’s competitive performance against state of the art XAI tools on a diverse array of tasks.

This study presents a comprehensive methodology for developing a Fault Detection and Diagnosis (FDD) system for an industrial distillation column using advanced machine learning algorithms. Steady state and dynamic simulations in Aspen Plus® generate extensive datasets under normal and faulty conditions. Feature engineering, using the Minimum Redundancy Maximum Relevance (MRMR) algorithm, selects the most relevant features for fault detection. Various machine learning models, including Decision Trees, Support Vector Machines, k-nearest Neighbours, and Neural Networks, were trained and evaluated based on performance metrics such as accuracy, recall, precision, and F1 score. The top models were integrated into a stacked classifier system with a voting mechanism to enhance fault detection reliability. Explainable Artificial Intelligence (XAI) techniques, such as Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP), were incorporated to improve model interpretability, allowing engineers to understand and validate the FDD system's decision-making process. Simulation results confirm that the proposed methodology accurately identifies and classifies faults. By integrating dynamic simulations, advanced machine learning, and XAI techniques, a robust and scalable solution is achieved for fault detection in distillation columns, improving operational reliability, safety, and reducing downtime. Future work could extend this approach to other industrial processes and explore additional machine learning algorithms to further enhance performance.

This work focuses on applying explainable artificial intelligence (AI) techniques to improve the interpretability and reliability of models in two crucial scenarios: detecting collusion in public procurement auctions and identifying corruption in public contracts in Mexico. While these challenges are specific, the need for transparency and explainability in AI systems is universal, especially when it comes to high-impact government issues. Our analysis is based on case studies of these particular applications, highlighting how interpretability techniques such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) can be used to provide crucial insights into decisions of the algorithms involved. In doing so, our paper seeks to contribute to a broader understanding of how explainable AI can be applied in diverse government contexts. Additionally, we explore the ethical and practical implications of explainable AI in governance, highlighting its ability to promote greater transparency, accountability, and trust in automated decisions. By the end of this study, we hope to provide a more comprehensive view on how explainable AI can be a valuable tool in improving public governance, not only in relation to cases of collusion and corruption, but also in a wide range of critical government applications.

Post-hoc explainability is essential for understanding black-box machine learning models. Surrogate-based techniques are widely used for local and global model-agnostic explanations but have significant limitations. Local surrogates capture non-linearities but are computationally expensive and sensitive to parameters, while global surrogates are more efficient but struggle with complex local behaviors. In this paper, we present ILLUME, a flexible and interpretable framework grounded in representation learning, that can be integrated with various surrogate models to provide explanations for any black-box classifier. Specifically, our approach combines a globally trained surrogate with instance-specific linear transformations learned with a meta-encoder to generate both local and global explanations. Through extensive empirical evaluations, we demonstrate the effectiveness of ILLUME in producing feature attributions and decision rules that are not only accurate but also robust and computationally efficient, thus providing a unified explanation framework that effectively addresses the limitations of traditional surrogate methods.

Image recognition with prototypes is considered an interpretable alternative for black box deep learning models. Classification depends on the extent to which a test image “looks like” a prototype. However, perceptual similarity for humans can be different from the similarity learned by the classification model. Hence, only visualising prototypes can be insufficient for a user to understand what a prototype exactly represents, and why the model considers a prototype and an image to be similar. We address this ambiguity and argue that prototypes should be explained. We improve interpretability by automatically enhancing visual prototypes with quantitative information about visual characteristics deemed important by the classification model. Specifically, our method clarifies the meaning of a prototype by quantifying the influence of colour hue, shape, texture, contrast and saturation and can generate both global and local explanations. Because of the generality of our approach, it can improve the interpretability of any similarity-based method for prototypical image recognition. In our experiments, we apply our method to the existing Prototypical Part Network (ProtoPNet). Our analysis confirms that the global explanations are generalisable, and often correspond to the visually perceptible properties of a prototype. Our explanations are especially relevant for prototypes which might have been interpreted incorrectly otherwise. By explaining such ‘misleading’ prototypes, we improve the interpretability and simulatability of a prototype-based classification model. We also use our method to check whether visually similar prototypes have similar explanations, and are able to discover redundancy. Code is available at https://github.com/M-Nauta/Explaining_Prototypes.