As artificial intelligence (AI), machine learning (ML), and other forms of advanced automation are increasingly considered for deployment in safety-critical industries, there is an urgent need for evaluation methods which reliably identify risks of deployment prior to people being harmed. In this narrative review, we discuss the benefits and drawbacks of 11 major methodological decisions underpinning evaluations of AI-infused technologies from the perspective of cognitive systems engineering (CSE) and naturalistic decision making (NDM). These methodological decisions are organized around four aspirations central to the perspective of CSE and NDM: evaluations of AI-infused technologies should be (1) integrated, (2) naturalistic, (3) grounded, and (4) pattern-centered. We use these aspirations to interpret common human-AI evaluation methods and discuss new evaluation challenges for emerging AI-infused technologies. This narrative review is meant to guide both current methods and future research toward safe and effective strategies for evaluating AI-infused technologies, especially in safety-critical settings.

This study investigated whether humans can be efficiently trained in human-agent teams (HATs) teamwork competencies to improve HAT collaboration. In HATs, humans and artificially intelligent (AI) agents collaborate on shared tasks, which requires teamwork. However, human and agent approaches to teamwork differ, posing challenges in HATs. These challenges raise the need to train humans to develop teamwork competencies that they can effectively apply in HAT settings. The cooperative video game, KeyWe, was used as a testbed, in which human participants completed tasks with a scripted agent. A HAT training intervention that took less than 30 minutes was developed to train humans on seven teamwork competencies. The training was not associated with the KeyWe game task itself. Half of the participants received the training, and half did not. Participants who received the training delegated a higher percentage of tasks to the agent and more often assigned tasks to the agent by defining strategies than participants who did not receive teamwork training. Trained teams demonstrated resilience by achieving higher task performance when the game difficulty increased. This study demonstrated that training humans to develop teamwork competencies, independent from task training, can enhance collaboration and performance in HATs.

Incorporating advanced autonomous systems, such as intelligent tactical autopilots, onboard manned military fighter aircraft will allow pilots to focus on managing additional air combat tasks while their autonomous counterparts maneuver the aircraft. One key challenge of maximizing the mission’s overall success is that the pilots must optimally divide their attention between monitoring the autonomy, to ensure safety, and other necessary tasks. The research conducted here explores methods for calibrating attention and affecting human trust in an autonomy-aided fighter cockpit while minimizing pilot workload. An experiment was performed to measure pilot eye-gaze, head position, workload, task performance, and trust while flying with autonomous agents that were simulated to be in control of the aircraft during live aerial combat maneuvers. During the combat maneuvers, the participants completed non-flying tasks directly related to the combat scenario, with and without a novel attention calibration system in the loop. With the addition of an attention calibration system in the loop, overall pilot task performance and trust increased, and cognitive workload decreased.

High-consequence domains such as medical triage for critical casualties often involve decisions for which there is no single right answer, and for which expert decision makers will disagree. These decisions thus represent a particular challenge for trustworthy and ethical AI in these domains, as simply training AI to make “the right decision” is not possible. In the current research, we offer a framework for aligning AI systems to the key, domain-specific attributes of decision-makers in order to produce trust in human experts. In Study 1, we investigate six possible attributes and find that the degree of alignment predicts rated trust ratings and delegation using short vignettes for assessment and delegation. In Study 2, we extend these results to demonstrate that alignment predicts trust in decisions from two different AI systems across two distinct attributes. The results thus offer a dimensionality-reducing approach to AI trustworthiness in high-consequence domains.

Those responsible in organisations for the design and delivery of training require a practical method for the analysis and prediction of skills retention. To address this, a taxonomy of nine psychological domains was developed specifically to provide a finer-grained approach to analysis of the skills required in the performance of trained tasks. An extant predictive model relevant to five of the domains was applied to produce a set of domain retention curves for physical/lower-order cognitive skills. These curves informed the development of a novel Competence Retention Analysis Technique (CRA-T) that incorporates a simple ‘traffic light’ approach indicating workforce proficiency, following a period without practice. CRA-T simplifies the process of understanding skill retention for practitioners by providing an alternative to separate empirical studies. By identifying the psychological domains involved in task performance insights can be gained into the acquisition and retention of these components, allowing the determination of those most at risk of decay. CRA-T is suitable for the analysis of a range of physical/cognitive tasks across sectors, where systematic approaches to training analysis/design for skill retention optimisation are required. CRA-T considers complex cognitive skills, but as no predictive models currently exist, longitudinal research is required to define their retention levels.

The increasing popularity of Artificial Intelligence (AI) emphasizes the need for a deeper understanding of the factors that can influence its use. Providing accurate explanations may be a critical factor to help establish trust and increase the likelihood of the operator following an AI agent’s recommendations. However, the impact of explanations on human performance, trust, mental workload (MW) and situation awareness (SA), and confidence are not fully understood. In this paper, we investigate the effects of providing explanations about the agent’s recommendations (of varying levels of accuracy) on human operator’s performance, trust, MW, SA, and confidence compared to an agent that provides no explanations. Thirty individuals were divided into three groups and randomly assigned to an agent accuracy level (high, medium, low). Each participant completed two sessions using the AI agent: one with explanations, one without explanations. In each session, the user’s performance was measured objectively (number of anomalies correctly diagnosed and time to diagnosis), trust, MW, SA, and confidence in their response were measured using surveys. Participants’ performance, trust, SA, and confidence were higher in the sessions when the agent provided explanations for its recommendations. Accuracy had a significant effect on performance, MW and user confidence but not on trust, or SA.

Envisioning new kinds of operations requires systematically developing architectures, work procedures, and artifacts to support human and machine agents in coordinating within dynamic environments. Accurately predicting how envisioned operations will unfold is challenging as (1) early design-phase descriptions of architectures, work procedures, and artifacts are often underspecified, and (2) key outcomes of interest emerge from interactions between cognitive work and environmental dynamics. This paper discusses how computational simulation of work can serve as a discovery tool for envisioning future operations. We introduce a three-phase approach using the Work Models that Compute (WMC) framework, which involves converting paper-based representations of work into computational models, developing scenarios and test conditions, and simulating work dynamics to analyze emergent behaviors. We illustrate this approach through a case study on developing contingency management procedures for envisioned air transport operations, specifically Urban Air Mobility (UAM). The case study demonstrates how computational simulation can (1) reveal the need for clearer design specifications, (2) uncover interactions and emergent behavior that may lead to undesirable outcomes, such as coordination surprises, and (3) identify trade-offs between multiple design options. Insight from simulation can complement other cognitive systems engineering methods to refine and enhance the feasibility and robustness of envisioned operations.

The potential to create autonomous teammates that work alongside humans has increased with continued advancements in AI and autonomous technology. Research in human–AI teams and human–autonomy teams (HATs) has seen an influx of new and diverse researchers from human factors, computing, and teamwork, yielding one of the most interdisciplinary domains in modern research. However, the HAT domain’s interdisciplinary nature can make the design of research, especially experiments, more complex, and new researchers may not fully grasp the numerous decisions required to perform high-impact HAT research. To aid researchers in designing high-impact experiments, this article itemizes four initial decision points needed to form a HAT experiment: deciding on a research question, deciding on a team composition, deciding on a research environment, and deciding on data collection. For each decision point, this article discusses these decisions in practice, providing related works to guide researchers toward different options available to them. These decision points are then synthesized through actionable recommendations to guide future researchers. The contribution of this article will increase the impact and knowledge of HAT experiments.

Automation is becoming increasingly complex, playing a larger role in driving and expanding its operational design domain to dynamic urban roads. Explainable AI (XAI) research in computer science aims to craft explanations of automation that help people understand the behavior of complex algorithms. However, many XAI approaches rely on fixed-format explanations, which may not effectively support drivers with varying levels of automation knowledge and tasks with different timescales. Maintaining common ground is a multilevel process, in which individuals and automation must adjust communication format and abstraction based on knowledge and time constraints. We first draw on existing research to suggest that common ground is a shared understanding between drivers and automation that requires constant maintenance. We applied the abstraction hierarchy (AH) modeling method, which describes complex systems across multiple abstraction levels to match drivers’ cognitive capacity. We modified it to translate vehicle and traffic data into multilevel explanations of automation behavior. We expanded the model into the abstraction–decomposition space, naming it the Driver–Automation Teaming model, designed to generate explanations that account for task timescale. With this modified model, we developed three human–machine interface concepts to demonstrate how it can improve XAI’s support for driver–automation collaboration.

Overconfidence in situation awareness (SA) can lead to various detrimental consequences, including risky behaviors. This study investigates the influence of stress and attention allocation on SA in manufacturing. Specifically, this study aims to (1) examine the effects of stress on objective SA, (2) explore the relationship between SA levels and SA overconfidence, and (3) investigate the potential alleviation of SA overconfidence through attention allocation to more challenging SA levels. These findings demonstrate that stress impairs the comprehension aspect of SA. Moreover, SA overconfidence increases with the SA level. In exploring strategies to mitigate SA overconfidence, allocating attention specifically to the most challenging SA level—SA projection—proved effective in alleviating overconfidence. These findings contribute to future SA research by clarifying the relationship between stress and SA, uncovering the link between SA levels and SA overconfidence, and exploring methods to alleviate overconfidence and improve decision accuracy in manufacturing systems.