A mental models approach for defining explainable artificial intelligence

Artificial intelligence (AI) is revolutionizing the world we live in. It holds significant promise for improving performance across various sectors. Researchers worldwide have been working on developing solutions that leverage this technology for application by diverse user groups. A key category of AI is healthcare AI; smart solutions designed to address the needs of various stakeholders in the sector, from medical practitioners to patients. Crucial for any AI deployment are good adoption rates from the intended users and support from key stakeholders. Therefore, understanding the factors that influence its adoption is essential. This paper aims to provide insights through a systematic literature review of research related to healthcare AI adoption. Specifically, it reports on research works from ASEAN countries published over the past decade, between 2014 and 2024, identified through a systematic search of the Lens.org database. The search string focused on terms such as “Artificial Intelligence,” “Medical,” “Healthcare,” “Explainable,” “Transparency,” “Trust,” and “Adoption.” The motivation for this search stems from developments in Explainable Artificial Intelligence (XAI). XAI is vital for fostering trust and facilitating the widespread adoption of AI-driven clinical decision support systems (CDSS) in healthcare. While AI has the potential to augment clinical decision-making, the lack of transparency in traditional “black box” models can undermine trust and hinder adoption. XAI techniques aim to make AI systems more interpretable by providing explanations and visualizations of the decision-making process. This transparency is essential for overcoming skepticism and encouraging the adoption of CDSS in clinical practice. The findings of this review will offer insights into the current research landscape in the region and inform future research on the adoption and use of healthcare AI. These issues are particularly relevant in the ASEAN context, where varying levels of digital literacy and healthcare infrastructure necessitate explainable and transparent AI to build user trust and support effective integration.

Purpose. Digital transformation of education determines the need to rethink the goals and meanings of the educational process and, at present, is associated with the problems of our time, the expansion of the role of artificial intelligence. The paper is devoted to the presentation of the taxonomy of educational goals from the standpoint of a mental model of thinking, which allows designing new educational results in the digital transformation of education. Methodology and methods. Unlike Bloom’s taxonomy, the main goal of modern education is the formation and development of computational thinking. The basis for constructing the taxonomy of educational goals is a mental approach, which is a set of principles and strategies for organizing the educational process aimed at forming and developing the cognitive abilities of the student. To identify the essence of computational thinking, a mental model of thinking is used, in which cognitive functions are determined by the mechanisms of perception, memorization, structuring and retrieval of information. In this model, knowledge is a structural set of mental images; mental schemes and mental models based on the first, second, and third signal systems. Results. Consideration of the mental characteristics of a modern person in a digital society, definition of the 3rd signal system in the “person + ICT + artificial intelligence” link, in which mental schemes and mental models play the main role, made it possible to formulate the stages of the educational process target settings in the form of the taxonomy of educational goals. The taxonomy of educational goals begins with the assignment of mental entropy - as a subject field of the future professional activity of a specialist, containing questions, tasks and problems. They can be formulated by traditional educational goals in the form of: know, be able to, master. The specification of these goals leads to target guidelines for the composition, volume and content of subject mental images, mental schemes and mental models. The set of mental images, schemes and models represent a knowledge block. In parallel, an ascending target block of universal cognitive operations is set, that a modern specialist must master. The pinnacle of the taxonomy is computational thinking. Based on the mental model of thinking, the essence of computational thinking is revealed and a set of universal cognitive operations is compiled. The basic composition of mental models that determine human cognitive behavior in a digital society is formed. Computational thinking is a type of thinking with the activation of conceptual-abstract and conceptual-machine superstructures of mental images, mental schemes and models in the circuit of the third signal system. Conclusion. The proposed mental taxonomy of educational goals determines the vector of the direction of digital transformation of education to achieve the main result - computational thinking. The stages of formation and the quality of computational thinking are determined by the formation of a set of universal cognitive operations, the volume and content of mental images, mental schemes and mental models of the subject area.

Artificial intelligence (AI) has become nearly ubiquitous in modern society, from components of mobile applications to medical support systems, and everything in between. In societally impactful systems imbued with AI, there has been increasing concern related to opaque AI, that is, artificial intelligence where it is unclear how or why certain decisions are reached. This has led to a recent boom in research on “explainable AI” (XAI), or approaches to making AI more explainable and understandable to human users. In the military domain, numerous bodies have argued that autonomous and AI-enabled weapon systems ought not incorporate unexplainable AI, with the International Committee of the Red Cross and the United States Department of Defense both explicitly including explainability as a relevant factor in the development and use of such systems. In this article, I present a cautiously critical assessment of this view, arguing that explainability will be irrelevant for many current and near-future autonomous systems in the military (which do not incorporate any AI), that it will be trivially incorporated into most military systems which do possess AI (as these generally possess simpler AI systems), and that for those systems with genuinely opaque AI, explainability will prove to be of more limited value than one might imagine. In particular, I argue that explainability, while indeed a virtue in design, is a virtue aimed primarily at designers and troubleshooters of AI-enabled systems, but is far less relevant for users and handlers actually deploying these systems. I further argue that human–machine teaming is a far more important element of responsibly using AI for military purposes, adding that explainability may undermine efforts to improve human–machine teamings by creating a prima facie sense that the AI, due to its explainability, may be utilized with little (or less) potential for mistakes. I conclude by clarifying that the arguments are not against XAI in the military, but are instead intended as a caution against over-inflating the value of XAI in this domain, or ignoring the limitations and potential pitfalls of this approach.

Formal analyses using human mental models have been effective in identifying how automation behavior can cause mode confusion and associated human error. However, state-machine-based mental model formalisms are unrealistically precise. To address this, we introduce Fuzzy Mental Models (FMMs). FMMs combine state machines with fuzzy logic to precisely represent vagueness in mental models. We define how FMMs integrate into larger formal system models, where the state of the automation and FMM co-evolve. We identify patterns within a system model that indicate mode confusion. These include new conditions that emerge only through the vagueness captured by FMMs. We also present a model-based analysis capable of discovering the mode confusion patterns in system models containing FMMs. Our approach is demonstrated with two realistic case studies: a modern automobile gear shifter and a cloud storage file system. We conclude by discussing the implications of our contributions and directions for future research.

Explainable artificial intelligence received great research attention in the past few years during the widespread of Black-Box techniques in sensitive fields such as medical care, self-driving cars, etc. Artificial intelligence needs explainable methods to discover model biases. Explainable artificial intelligence will lead to obtaining fairness and Transparency in the model. Making artificial intelligence models explainable and interpretable is challenging when implementing black-box models. Because of the inherent limitations of collecting data in its raw form, data fusion has become a popular method for dealing with such data and acquiring more trustworthy, helpful, and precise insights. Compared to other, more traditional-based data fusion methods, machine learning's capacity to automatically learn from experience with nonexplicit programming significantly improves fusion's computational and predictive power. This paper comprehensively studies the most explainable artificial intelligent methods based on anomaly detection. We proposed the required criteria of the transparency model to measure the data fusion analytics techniques. Also, define the different used evaluation metrics in explainable artificial intelligence. We provide some applications for explainable artificial intelligence. We provide a case study of anomaly detection with the fusion of machine learning. Finally, we discuss the key challenges and future directions in explainable artificial intelligence.

BackgroundGenerative artificial intelligence (AI) is rapidly transforming healthcare, but its adoption introduces significant ethical and practical challenges. Algorithmic bias, ambiguous liability, lack of transparency, and data privacy risks can undermine patient trust and create health disparities, making their resolution critical for responsible AI integration.ObjectivesThis systematic review analyzes the generative AI landscape in healthcare. Our objectives were to: (1) identify AI applications and their associated ethical and practical challenges; (2) evaluate current data-centric, model-centric, and regulatory solutions; and (3) propose a framework for responsible AI deployment.MethodsFollowing the PRISMA 2020 statement, we conducted a systematic review of PubMed and Google Scholar for articles published between January 2020 and May 2025. A multi-stage screening process yielded 54 articles, which were analyzed using a thematic narrative synthesis.ResultsOur review confirmed AI’s growing integration into medical training, research, and clinical practice. Key challenges identified include systemic bias from non-representative data, unresolved legal liability, the “black box” nature of complex models, and significant data privacy risks. Proposed solutions are multifaceted, spanning technical (e.g., explainable AI), procedural (e.g., stakeholder oversight), and regulatory strategies.DiscussionCurrent solutions are fragmented and face significant implementation barriers. Technical fixes are insufficient without robust governance, clear legal guidelines, and comprehensive professional education. Gaps in global regulatory harmonization and frameworks ill-suited for adaptive AI persist. A multi-layered, socio-technical approach is essential to build trust and ensure the safe, equitable, and ethical deployment of generative AI in healthcare.ConclusionsThe review confirmed that generative AI has a growing integration into medical training, research, and clinical practice. Key challenges identified include systemic bias stemming from non-representative data, unresolved legal liability, the “black box” nature of complex models, and significant data privacy risks. These challenges can undermine patient trust and create health disparities. Proposed solutions are multifaceted, spanning technical (such as explainable AI), procedural (like stakeholder oversight), and regulatory strategies.

Advancing AI in healthcare: A comprehensive review of best practices

Australian providers of mental health services and support for young people include private and public allied health providers, government initiatives (eg, headspace), nongovernment organizations (eg, Kids Helpline), general practitioners (GPs), and the hospital system. Over 20 years of research has established that many young people prefer to seek mental health support online; however, clear client pathways within and between online and offline mental health services are currently lacking. The authors propose a Digital Mental Health Practice model of care for youth to assist with digital mental health service mapping. The proposed model offers accessible pathways for a client to engage with digital mental health services, provides clear navigation to access support for individual needs, and facilitates a seamless connection with offline mental health services using a transferable electronic health records system. This future-looking model also includes emerging technologies, such as artificial intelligence and the metaverse, which must be accounted for as potential tools to be leveraged for digital therapies and support systems. The urgent need for a user-centered Digital Mental Health Practice model of care for youth in Australia is discussed, highlighting the shortcomings of traditional and existing online triage models evident during the COVID-19 pandemic, and the complex challenges that must be overcome, such as the integration of diverse mental health care providers and establishment of a robust electronic health records system. Potential benefits of such a model include reduced pressure on emergency rooms, improved identification of immediate needs, enhanced referral practices, and the establishment of a cost-efficient national digital mental health care model with global applicability. The authors conclude by stressing the consequences of inaction, warning that delays may lead to more complex challenges as new technologies emerge and exacerbate the long-term negative consequences of poor mental health management on the economic and biopsychosocial well-being of young Australians.

New research focuses on creating artificial intelligence (AI) solutions for network intrusion detection systems (NIDS), drawing its inspiration from the ever-growing number of intrusions on networked systems, increasing its complexity and intelligibility. Hence, the use of explainable AI (XAI) techniques in real-world intrusion detection systems comes from the requirement to comprehend and elucidate black-box AI models to security analysts. In an effort to meet such requirements, this paper focuses on applying and evaluating white-box XAI techniques (particularly LRP, IG, and DeepLift) for NIDS via an end-to-end framework for neural network models, using three widely used network intrusion datasets (NSL-KDD, CICIDS-2017, and RoEduNet-SIMARGL2021), assessing its global and local scopes, and examining six distinct assessment measures (descriptive accuracy, sparsity, stability, robustness, efficiency, and completeness). We also compare the performance of white-box XAI methods with black-box XAI methods. The results show that using white-box XAI techniques scores high in robustness and completeness, which are crucial metrics for IDS. Moreover, the source codes for the programs developed for our XAI evaluation framework are available to be improved and used by the research community.

This study investigates the integration of Artificial Intelligence (AI) as a co-pilot in aviation. As aviation increasingly embraces automation, concerns about safety, pilot roles, and ethical implications grow. The research emphasizes the necessity for human-AI collaboration in which AI enhances pilot capabilities without replacing them. Using a mixed-methods approach that includes literature review, case analysis, and stakeholder input, the study provides a model for safely integrating AI into flight operations. Key findings support that AI should assist rather than replace pilots, maintaining human oversight in high-stakes environments. The thesis concludes with policy and design recommendations for AI integration in aviation. This comprehensive study explores the transformative potential of Artificial Intelligence (AI) in modern commercial aviation, specifically emphasizing the concept of AI as a co-pilot. With rising operational complexity, global pilot shortages, and increasing demand for enhanced safety and efficiency, AI presents itself as a promising technological innovation. It has the capacity to provide real-time flight data interpretation, system anomaly alerts, autonomous recommendations, and task automation that collectively enhance situational awareness and reduce pilot workload. However, the expansion of AI usage in aviation also introduces critical concerns around pilot-AI interaction, explainability of AI systems, cybersecurity risks, and ethical responsibility in cases of failure. The research argues for a balanced integration model where AI supports—rather than supplants—human pilots. The idea of a "co-pilot AI" is conceptualized to highlight systems that work in tandem with human decision-makers, offering real-time insights while respecting human authority. Human-in-the-loop (HITL) design frameworks, explainable AI (XAI) principles, and trust-calibration models were central to the theoretical foundation of this research. A mixed-methods approach was adopted to strengthen the analysis, combining literature reviews, international aviation policy reviews, expert interviews, and comparative incident analysis. The findings of the study reveal a general consensus among professionals that while AI significantly enhances operational responsiveness and data processing capabilities, human oversight remains non-negotiable for moral judgment and adaptive decision-making during complex flight conditions. The research further identifies that system transparency, pilot familiarity, and regulatory clarity are essential for cultivating pilot trust in AI tools. It also highlights training gaps in existing aviation programs, where pilots are inadequately prepared to interface effectively with AI technologies. From a managerial and policy standpoint, the study recommends a strategic shift in training curricula to include AI-system literacy, greater investment in explainable interfaces, cross-industry collaboration for best practices, and harmonization of international regulatory standards. These measures, if implemented, will not only prevent technological overdependence but also build a more resilient and adaptive aviation ecosystem.

In recent years, there has been a surge of interest in Artificial Intelligence (AI) systems that can provide human-centric explanations for decisions or predictions. No matter how good and efficient an AI model is, users or practitioners find it difficult to trust it if they cannot understand the AI model or its behaviours. Incorporating explainability that is human-centric in event detection systems is significant for building a decision-making process that is more trustworthy and sustainable. Human-centric and semantics-based explainable event detection will achieve trustworthiness, explainability, and reliability, which are currently lacking in AI systems. This paper provides a survey on human-centric explainable AI, explainable event detection, and semantics-based explainable event detection by answering some research questions that bother on the characteristics of human-centric explanations, the state of explainable AI, methods for human-centric explanations, the essence of human-centricity in explainable event detection, research efforts in explainable event solutions, and the benefits of integrating semantics into explainable event detection. The findings from the survey show the current state of human-centric explainability, the potential of integrating semantics into explainable AI, the open problems, and the future directions which can guide researchers in the explainable AI domain.

The problem of assessing the situation in the control of anthropocentric objects using situational awareness models is considered. In general, the term “situational awareness” is used to describe the degree of adequate assessment of external and internal factors that affect the decision-making of a human operator in the process of controlling a complex object. This problem is not easy due to the complexity and the large number of factors that need to be taken into account to make effective decisions. As a result, situational awareness in the control of anthropocentric objects has a hierarchical structure. The levels of formation of situational awareness in the human operator are given. The place of the situational awareness model in the control system is noted and the relationship of the situation models with human mental models is analyzed. An approach to the formation of situational awareness based on biological principles of constructing mental models and taking into account the functional state of a human operator is proposed.KeywordsHuman operatorAnthropocentric objectsSituational awarenessSituational modelsMental models

The rapid proliferation of artificial intelligence (AI) technologies across various industries and decision-making processes has undeniably transformed the way of approaching complex problems and tasks. AI systems have proven their prowess in areas such as healthcare, finance, and autonomous systems, revolutionizing how decisions are made. Nevertheless, this proliferation of AI has raised critical concerns regarding the transparency, accountability, and fairness of these systems, as many of the state-of-the-art AI models often resemble complex black boxes. These intricate models, particularly deep learning neural networks, harbor non-linear relationships that are difficult for human users to decipher, thereby raising concerns about bias, fairness, and overall trustworthiness in AI-driven decisions. The urgency of this issue is underscored by the realization that AI should not merely be accurate; it should also be interpretable. Explainable AI (XAI) has emerged as a vital field of research, emphasizing the development of models and techniques that render AI systems comprehensible and transparent in their decision-making processes. This paper investigates into the relevance and significance of XAI across various domains, including healthcare, finance, and autonomous systems, where the ability to understand the rationale behind AI decisions is paramount. In healthcare, where AI assists in diagnosis and treatment, the interpretability of AI models is crucial for clinicians to make informed decisions. In finance, applications like credit scoring and investment analysis demand transparent AI to ensure fairness and accountability. In the realm of autonomous systems, transparency is indispensable to guarantee safety and compliance with regulations. Moreover, government agencies in areas such as law enforcement and social services require interpretable AI to maintain ethical standards and accountability. This paper also highlights the diverse array of research efforts in the XAI domain, spanning from model-specific interpretability methods to more general approaches aimed at unveiling complex AI models. Interpretable models like decision trees and rule-based systems have gained attention for their inherent transparency, while integrating explanation layers into deep neural networks strives to balance accuracy with interpretability. The study emphasizes the significance of this burgeoning field in bridging the gap between AI's advanced capabilities and human users' need for comprehensible AI systems. It seeks to contribute to this field by exploring the design, development, and practical applications of interpretable AI models and techniques, with the ultimate goal of enhancing the trust and understanding of AI-driven decisions.

This paper examines how explanations related to the adverse outcomes of Artificial Intelligence (AI) contribute to the development of causal evidentiary explanations in disputes surrounding AI liability. The study employs a dual approach: first, it analyzes the emerging global caselaw in the field of AI liability, seeking to discern prevailing trends regarding the evidence and explanations considered essential for the fair resolution of disputes. Against the backdrop of those trends, the paper evaluates the upcoming legislation in the European Union (EU) concerning AI liability, namely the AI Liability Directive (AILD) and Revised Product Liability Directive (R-PLD). The objective is to ascertain whether the systems of evidence and procedural rights outlined in this legislation, particularly the right to request the disclosure of evidence, enable litigants to adequately understand the causality underlying AI-related harms. Moreover, the paper seeks o determine if litigants can effectively express their views before dispute-resolution authorities based on that understanding. An examination of the AILD and R-PLD reveals that their evidence systems primarily support ad hoc explanations, allowing litigants and courts to assess the extent of the defendants' compliance with the standards enshrined in regulatory instruments, such as the AI Act. However, the paper contends that, beyond ad hoc explanations, achieving fair resolution in AI liability disputes necessitates post-hoc explanations. These should be directed at unveiling the functionalities of AI systems and the rationale behind harmful automated decisions. The paper thus suggests that ‘full’ explainable AI (XAI) that is, both ad hoc and post hoc, is necessary so that the constitutional requirements associated with the right to a fair trial (access to courts, equality of arms, contradictory debate) can be effectively met. Keywords: AI, Causation, Explainability, Fair Trial, Procedural Fairness, Equality of Arms, Effective Participation, AI liability, Product Liability, AI Act, AI Liability Directive, Product Liability Directive

Current robot designs often reflect an anthropomorphic approach, apparently aiming to convince users through an ideal system, being most similar or even on par with humans. The present paper challenges human-likeness as a design goal and questions whether simulating human appearance and performance adequately fits into how humans think about robots in a conceptual sense, i.e., human's mental models of robots and their self. Independent of the technical possibilities and limitations, our paper explores robots' attributed potential to become human-like by means of a thought experiment. Four hundred eighty-one participants were confronted with fictional transitions from human-to-robot and robot-to-human, consisting of 20 subsequent steps. In each step, one part or area of the human (e.g., brain, legs) was replaced with robotic parts providing equal functionalities and vice versa. After each step, the participants rated the remaining humanness and remaining self of the depicted entity on a scale from 0 to 100%. It showed that the starting category (e.g., human, robot) serves as an anchor for all former judgments and can hardly be overcome. Even if all body parts had been exchanged, a former robot was not perceived as totally human-like and a former human not as totally robot-like. Moreover, humanness appeared as a more sensible and easier denied attribute than robotness, i.e., after the objectively same transition and exchange of the same parts, the former human was attributed less humanness and self left compared to the former robot's robotness and self left. The participants' qualitative statements about why the robot has not become human-like, often concerned the (unnatural) process of production, or simply argued that no matter how many parts are exchanged, the individual keeps its original entity. Based on such findings, we suggest that instead of designing most human-like robots in order to reach acceptance, it might be more promising to understand robots as an own “species” and underline their specific characteristics and benefits. Limitations of the present study and implications for future HRI research and practice are discussed.