The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder, Early-stage Deliberations Around Public Sector AI Proposals

Human-Centered AI prioritizes end-users’ needs like transparency and usability. This is vital for applications that affect people’s everyday lives, such as social assessment tasks in the public sector. This paper discusses our pioneering effort to involve public sector AI users in XAI application design through a co-creative workshop with unemployment consultants from Estonia. The workshop’s objectives were identifying user needs and creating novel XAI interfaces for the used AI system. As a result of our user-centered design approach, consultants were able to develop AI interface prototypes that would support them in creating success stories for their clients by getting detailed feedback and suggestions. We present a discussion on the value of co-creative design methods with end-users working in the public sector to improve AI application design and provide a summary of recommendations for practitioners and researchers working on AI systems in the public sector.

Lived experiences fundamentally shape how individuals interact with AI systems, influencing perceptions of safety, trust, and usability. While prior research has focused on developing techniques to emulate human preferences, and proposed taxonomies to categorize risks (such as psychological harms and algorithmic biases), these efforts have provided limited systematic understanding of lived human experiences or actionable strategies for embedding them meaningfully into the AI development life-cycle. This work proposes a framework for meaningfully integrating lived experience into the design and evaluation of AI systems. We synthesize interdisciplinary literature across lived experience philosophy, human-centered design, and human-AI interaction, arguing that centering lived experience can lead to models that more accurately reflect the retrospective, emotional, and contextual dimensions of human cognition. Drawing from a wide body of work across psychology, education, healthcare, and social policy, we present a targeted taxonomy of lived experiences with specific applicability to AI systems. To ground our framework, we examine three application domains— (i) education, (ii) healthcare, and (iii) cultural alignment—illustrating how lived experience informs user goals, system expectations, and ethical considerations in each context. We further incorporate insights from AI system operators and human-AI partnerships to highlight challenges in responsibility allocation, mental model calibration, and long-term system adaptation. We conclude with actionable recommendations for developing experience-centered AI systems that are not only technically robust but also empathetic, context-aware, and aligned with human realities. This work offers a foundation for future research that bridges technical development with the lived experiences of those impacted by AI systems.

One of the sectors for which Artificial Intelligence applications have been considered as exceptionally promising is the healthcare sector. As a public-facing sector, the introduction of AI applications has been subject to extended news coverage. This article conducts a quantitative and qualitative data analysis of English news media articles covering AI systems that allow the automation of tasks that so far needed to be done by a medical expert such as a doctor or a nurse thereby redistributing their agency. We investigated in this article one particular framing of AI systems and their agency: the framing that positions AI systems as (1a) replacing and (1b) outperforming the human medical expert, and in which (2) AI systems are personified and/or addressed as a person. The analysis of our data set consisting of 365 articles written between the years 1980 and 2019 will show that there is a tendency to present AI systems as outperforming human expertise. These findings are important given the central role of news coverage in explaining AI and given the fact that the popular frame of ‘outperforming’ might place AI systems above critique and concern including the Hippocratic oath. Our data also showed that the addressing of an AI system as a person is a trend that has been advanced only recently and is a new development in the public discourse about AI.

PurposeInscrutable machine learning (ML) models are part of increasingly many information systems. Understanding how these models behave, and what their output is based on, is a challenge for developers let alone non-technical end users.Design/methodology/approachThe authors investigate how AI systems and their decisions ought to be explained for end users through a systematic literature review.FindingsThe authors’ synthesis of the literature suggests that AI system communication for end users has five high-level goals: (1) understandability, (2) trustworthiness, (3) transparency, (4) controllability and (5) fairness. The authors identified several design recommendations, such as offering personalized and on-demand explanations and focusing on the explainability of key functionalities instead of aiming to explain the whole system. There exists multiple trade-offs in AI system explanations, and there is no single best solution that fits all cases.Research limitations/implicationsBased on the synthesis, the authors provide a design framework for explaining AI systems to end users. The study contributes to the work on AI governance by suggesting guidelines on how to make AI systems more understandable, fair, trustworthy, controllable and transparent.Originality/valueThis literature review brings together the literature on AI system communication and explainable AI (XAI) for end users. Building on previous academic literature on the topic, it provides synthesized insights, design recommendations and future research agenda.

Explainable Artificial Intelligence (XAI) aims to bring transparency to AI systems by translating, simplifying, and visualizing its decisions. While society remains skeptical about AI systems, studies show that transparent and explainable AI systems result in improved confidence between humans and AI. We present preliminary results from a study designed to assess two presentation-order methods and three AI decision visualization attribution models to determine each visualization’s impact upon a user’s cognitive load and confidence in the system by asking participants to complete a visual decision-making task. The results show that both the presentation order and the morphological clarity impact cognitive load. Furthermore, a negative correlation was revealed between cognitive load and confidence in the AI system. Our findings have implications for future AI systems design, which may facilitate better collaboration between humans and AI.

We introduce an advanced computer vision-based AI system that offers people with vision impairments (VI) dynamic, in-situ access to information about the location, identity and gaze-direction of other people nearby. Our AI system utilizes the camera technology of a head-worn HoloLens device, which captures a near 180° field-of-view surrounding the person who is wearing it. Captured images are then processed by multiple state-of-the-art perception algorithms whose outputs are integrated into a real-time tracking model of all people that the system detected. Users can receive information about those people acoustically (via spatialized audio) using a wrist-worn input controller. Having such dynamic access to information, through AI system interactions, enables people with VI to: develop their communication skills; more easily focus on others; and be more confident in their social interactions. Thus, our work explores how AI systems can serve as a useful resource for humans, helping expand their agency to develop new or extend existing skills.

Criminal liability for acts committed by AI systems has recently become a hot legal topic. This paper includes three different contributions. The first contribution is an analysis of the extent to which an AI system can satisfy the requirements for criminal liability: accomplishing an actus reus, having the corresponding mens rea, possessing the cognitive capacities needed for responsibility. The second contribution is a discussion of criminal activity accomplished by an AI entity, with reference to a recent case involving an online bot, the Random Darknet Shopper. This discussion will provide the context for the analysis of commonalities and differences between criminal activities by humans and by artificial systems. The third contribution concerns the evaluation of different ways of addressing criminal activities by AI systems in a regulatory perspective.

We propose, Trustworthy Explainability Acceptance metric to evaluate explainable AI systems using expert-in-the-loop. Our metric calculates acceptance by quantifying the distance between the explanations generated by the AI system and the reasoning provided by the experts based on their expertise and experience. Our metric also evaluates the trust of the experts to include different groups of experts using our trust mechanism. Our metric can be easily adapted to any Interpretable AI system and be used in the standardization process of trustworthy AI systems. We illustrate the proposed metric using the high-stake medical AI application of Predicting Ductal Carcinoma in Situ (DCIS) Recurrence. Our metric successfully captures the explainability of AI systems in DCIS recurrence by experts.

This chapter gives an AI overview by starting to list and compare different terms and definitions in Sect. 2.1. It shows, how important it is to clarify terms beforehand, as there are not always unique definitions and understanding by humans about AI. It highlights that an AI system does not represent a robot or physical machine. The software can be operated on different kinds of hardware systems or platforms but does not require to have any physical shape. Section 2.2 gives an overview of different AI technology aspects. It shows the differences in data computation and progress between humans and computer systems. Section 2.3 describes several AI technologies and highlights the challenge of generated bias. If the AI algorithm gets trained with an already biased data set whether that can happen with consciousness or not, it will not generate a “better human.” The next Sect. 2.4 lists some AI challenges. A current main challenge is that humans still see AI systems as robots or machines and think of physical threats or opportunities. As AI systems can be only software modules, it is recommended to shift the broad society discussions from an attacking Terminator robot to broader ethics and responsibility discussion. Every citizen should take responsibility and proactive actions to shape “Friendly AI” systems. Like every technology, also AI can get abused and used in a not general friendly way. Section 2.5 describes opportunities and risks of AI technology. As humans stopped being goal driven, Section 2.6 highlights the scenario, where future AI systems might overtake the definition of goals for humans. To generate a broad awareness within society to develop and implement AI systems in daily life, it needs acceptance, transparency, and understanding. Section 2.7 gives an overview of a humans-based classification of evolution.

Practical Aspects of Designing a Human-centred AI System in Manufacturing

The growing concern over bias in AI systems, particularly in high-impact areas like healthcare, hiring, criminal justice and education emphasizes the need for ethical implementations to ensure fairness. To address bias, indicators such as demographic parity, equalized odds, calibration, and disparate impact measurement are crucial in monitoring and reducing biased outcomes across different demographic groups. In a scenario of university students taking online courses, where the exams were evaluated by an AI system, raising concerns about potential bias in the evaluation process. The key problem identified is ensuring that AI-driven assessments fairly evaluate students from diverse backgrounds, without disproportionately favoring or penalizing certain groups. Bias detection and mitigation efforts are essential to foster trust, fairness, and consistency in AI-based evaluations.

The ability to discern between true and false information is essential to making sound decisions. However, with the recent increase in AI-based disinformation campaigns, it has become critical to understand the influence of deceptive systems on human information processing. In experiment (N=128), we investigated how susceptible people are to deceptive AI systems by examining how their ability to discern true news from fake news varies when AI systems are perceived as either human fact-checkers or AI fact-checking systems, and when explanations provided by those fact-checkers are either deceptive or honest. We find that deceitful explanations significantly reduce accuracy, indicating that people are just as likely to believe deceptive AI explanations as honest AI explanations. Although before getting assistance from an AI-system, people have significantly higher weighted discernment accuracy on false headlines than true headlines, we found that with assistance from an AI system, discernment accuracy increased significantly when given honest explanations on both true headlines and false headlines, and decreased significantly when given deceitful explanations on true headlines and false headlines. Further, we did not observe any significant differences in discernment between explanations perceived as coming from a human fact checker compared to an AI-fact checker. Similarly, we found no significant differences in trust. These findings exemplify the dangers of deceptive AI systems and the need for finding novel ways to limit their influence human information processing.

AI systems are complex and difficult to understand, which makes it essential to establish a basis of trust between the user and the system to enable comfortable interactions. AI systems present both opportunities and risks, hence developers should endeavor to minimize risks as much as possible. Users, particularly in the realm of safety-critical systems, require sufficient trust in any new technology to consider its usage. Uncertainty and missing understanding about the field of AI and algorithms in general hinders and stops its usage. Therefore, transparency and comprehensibility should be goals in the development of AI systems. These factors can be divided into three groups: Technical factors, social factors, and evaluation factors. Each of these groups consists of five factors that can either enhance or diminish trust in an AI system. Based on these factors, a reference model is created to classify these three groups of factors within the overall context of trust formation. A study was conducted with 258 participants using a questionnaire to examine and subsequently analyze and evaluate the previously formulated reference model. An attempt is made to hierarchize the individual factors within the three groups and to identify any differences related to safety-critical and non-safety-critical systems. Keywords: artificial intelligence, trustworthy ai, responsible ai

The convergence of explainable artificial intelligence and robust AI systems represents one of the most critical challenges in contemporary machine learning research. This paper presents a unified theoretical framework that addresses the fundamental tension between model interpretability and adversarial robustness, two properties traditionally viewed as conflicting objectives in AI system design. Through a comprehensive analysis of the mathematical foundations underlying both explainability and robustness, we demonstrate that these characteristics can be synergistically integrated rather than traded off against each other. The proposed framework establishes theoretical connections between explanation quality metrics and adversarial vulnerability measures, revealing that well-explained models can actually exhibit enhanced robustness when properly constructed. Our approach introduces novel methodologies for simultaneously optimizing interpretability and security in AI systems, with particular emphasis on applications in critical domains where both transparency and reliability are essential. The framework provides practical guidelines for developing AI systems that maintain high performance while offering meaningful explanations and demonstrating resilience against adversarial attacks. This work contributes to the emerging field of trustworthy AI by providing both theoretical foundations and practical methodologies for building systems that are simultaneously explainable, robust, and reliable.

Simion and Kelp defend a non-anthropocentric account of trustworthy AI, based on the idea that the obligations of AI systems should be sourced in purely functional norms. In this commentary, I highlight some pressing counterexamples to their account, involving AI systems that reliably fulfil their functions but are untrustworthy because those functions are antagonistic to the interests of the trustor. Instead, I outline an alternative account, based on the idea that AI systems should not be considered primarily as tools but as technological participants in social practices. Specifically, I propose to source the obligations of an AI system in the norms that should govern the role it plays within the social practices it participates in, taking into account any changes to the social practices that its participation may bring about. This proposal is anthropocentric insofar as it ascribes obligations to AI systems that are similar to those of human participants in social practices, but only modestly so, as it does not require trustworthy AI to have contentious anthropomorphic capacities (e.g. for consciousness or moral responsibility).