Understanding, Idealization, and Explainable AI

Computer Vision, and hence Artificial Intelligence-based extraction of information from images, has increasingly received attention over the last years, for instance in medical diagnostics. While the algorithms' complexity is a reason for their increased performance, it also leads to the "black box" problem, consequently decreasing trust towards AI. In this regard, "Explainable Artificial Intelligence" (XAI) allows to open that black box and to improve the degree of AI transparency. In this paper, we first discuss the theoretical impact of explainability on trust towards AI, followed by showcasing how the usage of XAI in a health-related setting can look like. More specifically, we show how XAI can be applied to understand why Computer Vision, based on deep learning, did or did not detect a disease (malaria) on image data (thin blood smear slide images). Furthermore, we investigate, how XAI can be used to compare the detection strategy of two different deep learning models often used for Computer Vision: Convolutional Neural Network and Multi-Layer Perceptron. Our empirical results show that i) the AI sometimes used questionable or irrelevant data features of an image to detect malaria (even if correctly predicted), and ii) that there may be significant discrepancies in how different deep learning models explain the same prediction. Our theoretical discussion highlights that XAI can support trust in Computer Vision systems, and AI systems in general, especially through an increased understandability and predictability.

Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in 'black box' models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the 'user'. Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders needs, ensuring better implementation and operation in a healthcare context.

AI as a concept has been around since the 1950's. With the recent advancements in machine learning technology, and the availability of big data and large computing processing power, the scene is set for AI to be used in many more systems and applications which will profoundly impact society. The current deep learning based AI systems are mostly in black box form and are often non-explainable. Though it has high performance, it is also known to make occasional fatal mistakes. This has limited the applications of AI, especially in mission critical problems such as decision support, command and control, and other life-critical operations. This talk focuses on explainable AI, which holds promise in helping humans to better understand and interpret the decisions made by black-box AI models. Current research efforts towards explainable multimedia AI center on two parts of solution. The first part focuses on better understanding of multimedia content, especially video. This includes dense annotation of video content from not just object recognition, but also relation inference. The relation includes both correlation and causality relations, as well as common sense knowledge. The dense annotation enables us to transform the level of representation of video towards that of language, in the form of relation triplets and relation graphs, and permits in-depth research on flexible descriptions, question-answering and knowledge inference of video content. A large scale video dataset has been created to support this line of research. The second direction focuses on the development of explainable AI models, which are just beginning. Existing works focus on either the intrinsic approach, which designs self-explanatory models, or post-hoc approach, which constructs a second model to interpret the target model. Both approaches have limitations on trade-offs between interpretability and accuracy, and the lack of guarantees about the explanation quality. In addition, there are issues of quality, fairness, robustness and privacy in model interpretation. In this talk, I present current state-of-the arts approaches in explainable multimedia AI, along with our preliminary research on relation inference in videos, as well as leveraging prior domain knowledge, information theoretic principles, and adversarial algorithms to achieving interpretability. I will also discuss future research towards quality, fairness and robustness of interpretable AI.

Explainable AI (XAI) is discussed as an important feature of AI systems that is required for professional, ethical and commercial reasons. XAI is particularly needed in AI systems that remain relatively closed, i.e. black box, on how to make decisions whether for complexity or commercial sensitivity. Still, many factors play an important role on whether people perceive a need for XAI. The personal relevance of the AI system might play a role, but also individual differences in personality traits or cognitive functions. In order to test these assumptions, 215 participants (129 female) underwent an online experiment testing their objective working memory capacity and completed the Intellect facet of the Big Five Personality traits (as a proxy for self-assessed cognitive capacity). Further, all participants were presented three AI systems scenarios of relatively different degrees of (personal) relevance and, hence, personal investment: skin cancer detection (i), the personalization of social media feeds (ii), and the amount to pay for a car insurance (iii). We observed that people reported a higher need for XAI in scenarios where they were more personally invested (health, payment for insurance>social media); for signficant differences between these scenarios see the result section. Interestingly, working memory capacity was not associated with XAI needs across the three domains, but self-reported Intellect showed a mild positive association with need for an AI explanation in the context of the insurance story. The present study represents a first attempt to investigate objective working memory capacity in the context of need for XAI. Beyond that, the study indicates that personal investment is crucial for determining the need for XAI.

Artificial intelligence (AI) is revolutionizing the world we live in. It holds significant promise for improving performance across various sectors. Researchers worldwide have been working on developing solutions that leverage this technology for application by diverse user groups. A key category of AI is healthcare AI; smart solutions designed to address the needs of various stakeholders in the sector, from medical practitioners to patients. Crucial for any AI deployment are good adoption rates from the intended users and support from key stakeholders. Therefore, understanding the factors that influence its adoption is essential. This paper aims to provide insights through a systematic literature review of research related to healthcare AI adoption. Specifically, it reports on research works from ASEAN countries published over the past decade, between 2014 and 2024, identified through a systematic search of the Lens.org database. The search string focused on terms such as “Artificial Intelligence,” “Medical,” “Healthcare,” “Explainable,” “Transparency,” “Trust,” and “Adoption.” The motivation for this search stems from developments in Explainable Artificial Intelligence (XAI). XAI is vital for fostering trust and facilitating the widespread adoption of AI-driven clinical decision support systems (CDSS) in healthcare. While AI has the potential to augment clinical decision-making, the lack of transparency in traditional “black box” models can undermine trust and hinder adoption. XAI techniques aim to make AI systems more interpretable by providing explanations and visualizations of the decision-making process. This transparency is essential for overcoming skepticism and encouraging the adoption of CDSS in clinical practice. The findings of this review will offer insights into the current research landscape in the region and inform future research on the adoption and use of healthcare AI. These issues are particularly relevant in the ASEAN context, where varying levels of digital literacy and healthcare infrastructure necessitate explainable and transparent AI to build user trust and support effective integration.

Explainable Artificial Intelligence (XAI) aims to bring transparency to AI systems by translating, simplifying, and visualizing its decisions. While society remains skeptical about AI systems, studies show that transparent and explainable AI systems result in improved confidence between humans and AI. We present preliminary results from a study designed to assess two presentation-order methods and three AI decision visualization attribution models to determine each visualization’s impact upon a user’s cognitive load and confidence in the system by asking participants to complete a visual decision-making task. The results show that both the presentation order and the morphological clarity impact cognitive load. Furthermore, a negative correlation was revealed between cognitive load and confidence in the AI system. Our findings have implications for future AI systems design, which may facilitate better collaboration between humans and AI.

The AAAI 2025 Bridge on “Explainable AI, Energy and Critical Infrastructure Systems” was held at the Pennsylvania Convention Centre, Philadelphia, Pennsylvania, USA, on February 25, 2025. The bridge gathered researchers and practitioners, bringing together innovation research across explainable AI, energy and critical infrastructure systems so they can enhance each other. The Bridge featured five keynote presentations by experts, one tutorial, poster presentations by authors who contributed their research findings, and three breakout sessions to discuss new challenges arising at the intersection of these exciting disciplines.

PurposeInscrutable machine learning (ML) models are part of increasingly many information systems. Understanding how these models behave, and what their output is based on, is a challenge for developers let alone non-technical end users.Design/methodology/approachThe authors investigate how AI systems and their decisions ought to be explained for end users through a systematic literature review.FindingsThe authors’ synthesis of the literature suggests that AI system communication for end users has five high-level goals: (1) understandability, (2) trustworthiness, (3) transparency, (4) controllability and (5) fairness. The authors identified several design recommendations, such as offering personalized and on-demand explanations and focusing on the explainability of key functionalities instead of aiming to explain the whole system. There exists multiple trade-offs in AI system explanations, and there is no single best solution that fits all cases.Research limitations/implicationsBased on the synthesis, the authors provide a design framework for explaining AI systems to end users. The study contributes to the work on AI governance by suggesting guidelines on how to make AI systems more understandable, fair, trustworthy, controllable and transparent.Originality/valueThis literature review brings together the literature on AI system communication and explainable AI (XAI) for end users. Building on previous academic literature on the topic, it provides synthesized insights, design recommendations and future research agenda.

The robot middleware is a key concept in developing complex robot systems even in geographically distributed environment. Giving a handy API, reusable standardized components and communication channels together with some automatism, the robot middleware helps the user to build easily reconfigurable systems. The behavior of the components and the manner of interaction among them are standardized by the Robotic Technology Component (RTC) specification. One implementation of the RTC specification is the OpenRTM-aist. It is a well written and convenient modular system built upon the Common Object Request Broker Architecture (CORBA). The first version of CORBA is released in 1991. Nowadays the CORBA is getting more and more out of date. Our ultimate goal is developing a new robot middleware in which we apply a modern distributed framework instead of CORBA. As a first step of the substitution we suggest some practical extensions of the OpenRTM-aist by the adaptation of the Internet Communication Engine (ICE) for component communication and the introduction of the web application concept for system editing and control. Having these structural modifications the resulting system became more powerful than the original system has been. The chapter introduces some structural and implementation details of the OpenRTM-aist together with the results of the experiments done for the performance comparison of the original and extended system.

Hybrid ML-driven simulation exploring the fate of environmentally relevant nanoplastics in complex aqueous matrices: Mechanistic insight into colloidal behavior via explainable AI.

Feature visualization within an automated design assessment leveraging explainable artificial intelligence methods

With increasing deployment of complex and opaque machine learning algorithms (black boxes) to make decisions in areas that profoundly affect individuals such as underwriting, judicial sentencing, and robotic driving, are increasing calls for explanations of how they make these decisions to assure that they are accurate, objective, and fair. Black boxes are also being increasingly explored and deployed in real world pharmacovigilance, so an understandable question is whether such explainability is important in this domain as well. “Explainable artificial intelligence (AI)” refers to a set of tools that aim to provide understandable approximations, hypothesis, or more precise traces of the “inner thoughts” of black boxes. We consider whether and how general arguments made for explainable AI, such as building trust and gaining scientific insights, apply to pharmacovigilance including an explication of the limitations in these arguments and the methods themselves. Given the multiple application domains within pharmacovigilance, the answer to the question is situation dependent. If the field of explainable AI advances to the point of consistently providing high quality explanations as testable hypothesis, explainable artificial intelligence should be a credible addition to the pharmacovigilance toolkit for model development, signal management and clinical pharmacovigilance. Its incremental contribution to broad trust building per se, though widely touted in general, is a dubious argument for pharmacovigilance. This article is protected by copyright. All rights reserved.

With the explosive growth of computer science and big data, quantitative finance is undergoing a huge shift from traditional econometrics to data-driven Artificial Intelligence (AI). This study aims to review the applications of Machine Learning (ML), Deep Learning (DL), and Reinforcement Learning (RL) in vital areas such as asset pricing, algorithmic trading and risk management. Research shows that Deep Neural Networks (DNNs) capture non-linear market patterns, Natural Language Processing (NLP) analyses unstructured data while providing superior sentiment signals compared to generic alternatives. Deep Reinforcement Learning (DRL) can effectively automate execution in dynamic market environments, which optimizes trading decisions. Similarly, Temporal Fusion Transformers (TFT) have emerged as a dominant architecture for multi-horizon time series forecasting, offering superior interpretability and accuracy over standard recurrent networks. These ML models greatly outperform traditional methods like CAPM and ARIMA in prediction accuracy and handling complex data. However, there are still challenges about the lack of interpretability ("black box") and overfitting. In the future, quantitative finance lies in combining "Explainable AI" (XAI).

The growing adoption of Artificial Intelligence (AI) and Machine Learning (ML) in critical decision-making areas such as healthcare, finance, and autonomous systems has raised concerns regarding the interpretability of these models. While deep learning and other advanced ML models deliver high accuracy, their "black box" nature makes it difficult to explain their decision-making processes. Explainable AI (XAI) aims to bridge this gap by introducing methods that enhance transparency without significantly compromising performance. This paper explores key advancements in XAI, including model-agnostic and model-specific interpretability techniques, and evaluates their effectiveness in balancing model explainability and performance. We conduct an empirical analysis on commonly used XAI techniques, present a case study on AI-assisted healthcare diagnostics, and analyze stakeholder perspectives through a structured questionnaire. Our findings suggest that while XAI methods improve interpretability and stakeholder trust, they often come with computational and accuracy trade-offs. The study also highlights the challenges and opportunities in integrating XAI into real-world applications and provides recommendations for future research.

BackgroundGenerative artificial intelligence (AI) is rapidly transforming healthcare, but its adoption introduces significant ethical and practical challenges. Algorithmic bias, ambiguous liability, lack of transparency, and data privacy risks can undermine patient trust and create health disparities, making their resolution critical for responsible AI integration.ObjectivesThis systematic review analyzes the generative AI landscape in healthcare. Our objectives were to: (1) identify AI applications and their associated ethical and practical challenges; (2) evaluate current data-centric, model-centric, and regulatory solutions; and (3) propose a framework for responsible AI deployment.MethodsFollowing the PRISMA 2020 statement, we conducted a systematic review of PubMed and Google Scholar for articles published between January 2020 and May 2025. A multi-stage screening process yielded 54 articles, which were analyzed using a thematic narrative synthesis.ResultsOur review confirmed AI’s growing integration into medical training, research, and clinical practice. Key challenges identified include systemic bias from non-representative data, unresolved legal liability, the “black box” nature of complex models, and significant data privacy risks. Proposed solutions are multifaceted, spanning technical (e.g., explainable AI), procedural (e.g., stakeholder oversight), and regulatory strategies.DiscussionCurrent solutions are fragmented and face significant implementation barriers. Technical fixes are insufficient without robust governance, clear legal guidelines, and comprehensive professional education. Gaps in global regulatory harmonization and frameworks ill-suited for adaptive AI persist. A multi-layered, socio-technical approach is essential to build trust and ensure the safe, equitable, and ethical deployment of generative AI in healthcare.ConclusionsThe review confirmed that generative AI has a growing integration into medical training, research, and clinical practice. Key challenges identified include systemic bias stemming from non-representative data, unresolved legal liability, the “black box” nature of complex models, and significant data privacy risks. These challenges can undermine patient trust and create health disparities. Proposed solutions are multifaceted, spanning technical (such as explainable AI), procedural (like stakeholder oversight), and regulatory strategies.