Using Comparative Cognitive Work Analysis to Identify Design Priorities in Complex Socio-Technical Systems

Cognitive work analysis: An influential legacy extending beyond human factors and engineering

This thesis demonstrates that many large, socio-technical systems fail to meet the goal of their sponsors and that one reason for this failure is that socio-technical systems are complex, co-evolutionary systems and that once such systems reach a particular level of complexity, they become as unpredictable as eddies in a stream. Technology based systems are built by engineers; systems engineers, software engineers, telecommunication engineers, mechanical engineers, etc. Engineering is goal-directed and based on the laws of physics. There are goals to achieve and requirements to be met. Thus engineering, and systems engineering particularly, is concerned with plans for the future. These plans are based on predicting the global state description of the system of interest, or in other words the behaviour, at specified times in the future. In a truly complex, large socio-technical system, this planned predictability can break down into a morass of unanticipated behaviour. This unpredictability comes from the propagation of the effects of change through coevolving subsystems. The effect of a change in one subsystem is often deterministic and in general predictable but when influencing links connect many subsystems, the effects become complex. The propagating changes originating from two or more initial changes interfere and alter the outcome. The subsystems' new states apply conflicting pressures for change on each other inducing them to enter a cyclic or chaotic set of states. These changes to individual or small groups of are local in nature but their aggregate effect is evident in the emergent behaviour of the system. The holistic system behaviour is unpredictable because it is dependent on a multitude of local effects. At the local level the pressure one subsystem has on another is dependent on the degree of coupling between the subsystems. Within a subsystem, there is again conflict, the magnitude of alteration in one dimension being traded off against that of another so that the aggregate effect affects the pressure for change. The threshold computation and the propagation of change can be brought together in a protocol, i.e. a set of rules, that is described in the thesis. This protocol relates the pressure for change, the subsystem aims, the risks and costs of change, and the control of change. It also describes the effects of the propagation of change. The ANZAC warship is used to illustrate the findings of the research. Each ship is a very complex socio-technical system so changes must be carefully considered in order to achieve the required result. Furthermore, each ship has an unending list of planned changes as capability requirements change. A tool was needed so that multiple changes could be simulated concurrently or successively and the implications shown. The thesis describes a number of concepts that could be the basis for such a tool. The tool, named SeeChange, was designed and built as part of the research. In a socio-technical system, the non-deterministic effect of the behaviour of humans also needs to be considered in depth. Human behaviour makes it extremely difficult to foresee the effect of changing one variable in a complex scenario. Building a large information system, for example, is a complex undertaking and the introduction of human subsystems raises the level of complexity. Often when change is contrived to achieve a particular effect the implementer is bewildered when the change achieves a different effect. This thesis draws attention to the presence of chaotic states within relatively simple complex systems and extends that concept to real world systems. It highlights the certainty of change, the uncertainty in foreseeing the effects of change, indicates why some systems may fail and offers some suggestions that may help to alleviate the problem of complex socio-technical systems failure. In summary, the thesis demonstrates that the effect of changes propagating within a complex, socio-technical system is inherently unpredictable and that there is a strong probability of the system failing to meet expectations. Therefore, there is not and, importantly, cannot be a single solution for preventing chaotic, socio-technical system failure, however, the SeeChange program coupled with the alleviative statements suggest a mechanism for improving the success rate and turning the nature of change to advantage.

Over the past decade, Cognitive Work Analysis (CWA) has been one of the popular human factors approaches for complex systems evaluation and design applications. This is reflected by a diverse range of applications across safety critical domains. The book brings together a series of CWA applications and discussions from world-leading human factors researchers and practitioners. It begins with an overview of the CWA framework, including its theoretical underpinnings, the methodological approaches involved (including practical guidance on each phase), and previous applications of the framework. The core of the book is a series of CWA applications, undertaken in a wide range of safety critical domains for a range of purposes. These serve to demonstrate the contribution that CWA can make to real-world projects and provide readers with inspiration for how such analyses can be practically carried out. Following this, a series of applications in which new approaches or adaptations have been added to the framework are presented. These show how practical applications feedback into the theories/approaches underpinning CWA. The closing chapter then speculates on future applications of the framework and on a series of new research directions required in order to enhance its utility. In emphasising the practical realities of performing CWA, and the real-world impacts it can provide, the book tackles several common misconceptions in a constructive and persuasive way. It provides a welcome demonstration of how CWA can be a powerful ally in tackling complexity-related problems that afflict systems in all areas.

Traditional approaches to cyber-security resilience, assuring the overall socio-technical system is secure from immediate known attacks and routes to potential future attacks, have relied on three pillars of people, process, and technology.In any complex socio-technical system, human behaviour can disrupt the secure and efficient running of the system with risk accumulating through individual and system-wide errors and compromised security behaviours that may be exploited by actors with malicious intent.Practitioners’ experience and use of different assessment methods and approaches to establish cyber-security vulnerabilities and risk are evaluated. Qualitative and quantitative methods and data are used for different stages of investigations in order to derive risk assessments and access contextual experience for further analyses. Organisational security culture and development approaches along with safety assessment methods are discussed in this case study to understand how well the people, the system, and the organisation interact.Cyber-security Human Factors practice draws on other application areas such as safety, usability, behaviours and culture to progressively assess security posture; the benefits of each approach are discussed.This study identifies the most effective methods for vulnerability identification and risk assessment, with focus on modelling large, dynamic and complex socio-technical systems, to be those which identify cultural factors with impact on human-system interactions.KeywordsHuman factorsCyber-securityBehavioural scienceOrganisational cultureSecurity cultureCyber- resilienceSocio-technicalSafety assessmentClimateBehavioralOrganizational cultureSocio-behaviouralSociotechnical

A Qualitative-Quantitative Resilience Assessment Approach for Socio-technical Systems

The article “Situation Awareness Misconceptions and Misunderstandings” (Endsley, 2015) discusses common fallacies noted in the literature in the interpretation of Endsley’s 1995 model of situation awareness (SA; Endsley, 1995). The clarifications presented in the article provide a more complete and comprehensible explanation of the model. As in the complex domains that we study, our attention suffers from limitations as we simplify each other’s work in our eager and well-intentioned quest to attack new and exciting problems. Dr. Endsley is to be thanked for giving us pause to rebuild our own SA of SA in her thoughtful article. SA has, for many years, been a powerful and influential construct. In our own work in cognitive work analysis (CWA), we have viewed SA as a complementary framework that challenges and drives CWA. Without doubt, the output of a CWA-based design process should be the design of a system that promotes better SA and performance (Burns et al., 2008). CWA and goaldirected task analysis may organize the world in slightly different dimensions, but the overall intent is the same: to create systems that support human decision making as well as we can. Indeed, it is this common intent that unites us in our field. To advance cognitive engineering, there are times when we must challenge each other, critique each other’s models, hunt for flaws, and identify promising new directions. Assuredly, this helps us progress, strengthen our methods, and deepen our understanding. This article clearly identifies such activity and responds to it. We are all better for this exercise, as it challenges both SA and all our approaches to grow and deepen. Acknowledging this, we would like to change our perspective to a larger one and discuss challenges facing cognitive engineering as a whole. In these challenges, our existing methods, SA, CWA, and other approaches must adapt and grow. Advancements in intelligent systems and automation have increased in the amount of data produced by information systems and yet placed the human in new roles. In many cases, these roles are partially in the loop and partially out of the loop and may involve supervisory control or may have the human working in systems with very little supervisory control at all because the automation is largely nontransparent. We outline three core areas of challenges: self-awareness and self-regulation, memory failures or performance with incorrect SA, and design for unstructured environments.

As design discourse continues to grapple with complexity in sociotechnical systems, new methodologies have coalesced under the term — systemic design. These approaches rely heavily on systems mapping through co-design to synthesize stakeholder perspectives on a system-of-interest into a visual representation of the system — a system map. Subsequent visual abductive reasoning then becomes the primary meaning-making process and method of creating insight and consensus. While system maps are useful in many ways and contexts, they also have limitations that stem from their status as static visual artifacts. In other systems disciplines, system maps are frequently used as inputs into more dynamic modeling processes, like simulation. Systemic design has yet to adopt simulation widely, due to the large expense, specific expertise, and tremendous effort needed to create such models of complex systems. We offer a Playful Systems approach that uses game co-design as a more flexible, familiar, and functional, way of capturing and representing the dynamics of systems from varying stakeholder perspectives. We outline the advantages of our Playful Systems approach in terms of five qualities of iterative game design: dynamics, intervention, freedom, embodiment, and persistence and associate each with a collection of research questions that together create an agenda for design researchers who want to explore the unique power of games and game design for managing complexity in sociotechnical systems.

Sociotechnical systems, risk management, and public health: comparing the North Battleford and Walkerton outbreaks

We are witnessing a significant expansion and penetration of wireless appliances, sensors, smart phones, and PDAs in a wide range of domains such as hospitals, healthcare, smart homes, and elderly home care. Healthcare applications are mission-critical, real-time, and are used in a complex socio-technical system that requires collaboration, communication and coordination between many actors (doctors, patients, nurses, lab technicians, administration, etc.) and devices in the system, depending on their context. This paper introduces a framework for supporting context awareness, which is built from available open source components. The paper illustrates how the framework can be used in the healthcare domain with the help of an emergency department application.

This paper describes the application of requirements engineering concepts to support the analysis of the impact of new software systems on system-wide goals. Requirements on a new or revised software component of a socio-technical system not only have implications on the goals of the subsystem itself, but they also impact upon the goals of the existing integrated system. In industries such as air traffic management and healthcare, impacts need to be identified and demonstrated in order to assess concerns such as risk, safety, and accuracy. A method called PiLGRIM was developed which integrates means-end relationships within goal modelling with knowledge associated with the application domain. The relationship between domain knowledge and requirements, as described in a satisfaction argument, adds traceability rationale to help determine the impacts of new requirements across a network of heterogeneous actors. We report procedures that human analysts follow to use the concepts of satisfaction arguments in a software tool for i* goal modelling. Results were demonstrated using models and arguments developed in two case studies, each featuring a distinct socio-technical system—a new controlled airspace infringement detection tool for NATS (the UK’s air navigation service provider), and a new version of the UK’s HIV/AIDS patient reporting system. Results provided evidence towards our claims that the conceptual integration of i* and satisfaction arguments is usable and useful to human analysts, and that the PiLGRIM impact analysis procedures and tool support are effective and scalable to model and analyse large and complex socio-technical systems.

Hierarchical task analysis (HTA) and cognitive work analysis (CWA) were applied to laparoscopic surgery to identify critical technical, cognitive, and social challenges. Our purpose was to understand the complexity of laparoscopy and use this knowledge to develop a surgical training system. Five laparoscopic procedures were analyzed using the HTA methodology, while generic laparoscopic approaches were analyzed by applying the five phases of the CWA framework. The results from these analyses were applied to a surgical training system. The focus was on the application of partial-task training to areas that could be more technically challenging. Additionally, practical, social, and cognitive training aspects were included into the training system, based on our findings from the analyses. Practical Relevance: The use of digital training systems is becoming increasingly important in complex work systems, especially when it comes to simulations that can have serious consequences in reality. This paper presents the development of a surgical training system.

This is the second part of a two part article. In Part I, a social systems theory was applied to the analysis of hazardous technology and socio-technical systems, their complex dynamics, and risky dimensions and likelihood of accidents. It identified many of the diverse human risk factors associated with complex technologies and socio-technical systems, thus contributing knowledge toward preventing — or minimizing the likelihood of — accidents or catastrophes. This second part of the article will systematically address the broader issues of risk conceptions, analysis, and management in contemporary society including policy and other practical aspects. The social systems perspective and its derivations are contrasted to such impressionistic conceptions as those of Ulrich Beck. Section 1 of the paper introduces the topic of risk as a discursive concept in contemporary society. Our point of departure is the social system approach introduced in Part I, which is contrasted to that of Ulrich Beck, who eschews systematic theorizing at the same time that he denigrates empirical sociology. The section stresses that contemporary society is not so much threatened by high risks all around (as in Ulrich Beck’s “risk society”) but is more characterized by its developed risk discourses (a great deal owing to Beck himself), risk consciousness, risk theorizing, and risk management. What is truly characteristic of modern society are discretionary powers to determine dimensions, levels, and regulation of risk, that is, choices can be made whether or not to develop a technology, whether or not to not to tightly regulate it, for instance limiting or banning its use or whether or not to allow its widespread application, and under what conditions. Section 2 provides a brief review of our social systems framework, actor-system-dialectics (ASD) theory. Section 3 treats risk and risk analysis in a systems perspective, emphasizing the limitations of risk assessment and the risk management of complex, hazardous systems. Section 4 considers several principles which may serve to guide policy-making and regulation with respect to the hazards and risks of complex technologies and socio-technical systems.

Understanding human performance in sociotechnical systems – Steps towards a generic framework

Most business and social organisations can be seen nowadays as complex sociotechnical systems (STSs), including three components: technical artifacts, social artifacts, and humans. Within social artifacts, a special role have norms, which largely influence the overall system's behavior. However, norms need to be understood, interpreted, negotiated, and actuated by humans, who may of course deviate from them, or even decide to change them. STSs are therefore essentially prone to failure: critical situations are part of STS’s life, and may sometimes lead to tragic outcomes. That’s why resilience to failure must be built into such systems, and is a crucial parameter to determine their quality. We argue in this paper that, to achieve a high level of resilience, transparency is the key: actors within the system need to take a reflective stance toward the system itself. In other words, an STS must be open to its actors, which by observing and understanding its dynamics can take the appropriate initiatives in presence of unforeseen problems, possibly modifying the system at run time. Ontological models can play a crucial role in this context. However, we need to make a radical change in our modelling approach, shifting the focus of analysis from ontology-driven information systems to ontology-driven sociotechnical systems.KeywordsLegal NormTechnical ArtifactSociotechnical SystemUnforeseen ProblemSocial SubsystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Towards safe electronic health records: A socio-technical perspective and the need for incident reporting