Application of a Continuous Personal Development Methodology

This review provides the state of crew resource management (CRM) training evaluations since the E. Salas, C. S. Burke, C. A. Bowers, and K. A. Wilson (2001) review and extends it to areas beyond aviation cockpits. Some critical evaluation needs in CRM training are also covered. Because of the purported success of CRM training in aviation, other high-consequence domains have begun to implement CRM training for their workforces. However, the true impact of CRM training in aviation and these other domains has yet to be determined. Using D. L. Kirkpatrick's (1976) framework for evaluating training (i.e., reactions, learning, behavior, and organizational impact), we reviewed 28 published accounts of CRM training to determine its effectiveness within aviation, medicine, offshore oil production and maintenance, shipping/maritime, and nuclear power domains. Findings indicate that CRM training generally produced positive reactions from trainees; however, the impact of training on learning and behavioral changes suggest mixed results across and within domains. Furthermore, and as was found by Salas, Burke, et al. in 2001, we cannot ascertain whether CRM has had an impact on the organization's bottom line (i.e., safety). Based on the results, there are several critical needs that the CRM training community must address before CRM training can have the desired impact on safety: a mandate, access to data, and resources. As CRM training expands to organizations beyond aviation, it is critical that its impact be understood such that it can be improved and achieve the intended results.

Chapter 19 - The Military Perspective

One challenge for applied training research is to effectively balance the need to deliver a product to meet operational requirements with the need for scientific rigor. In this article we document an applied research effort and offer lessons learned from balancing science and operational necessity while meeting naval aviation's crew resource management (CRM) training requirements. Specifically, in 1990, 60%-80% of all accidents in naval aviation were due to human error. In an effort to reduce accidents attributable to poor aircrew coordination, a program of training research was conducted. Research involving empirical and field investigations was performed and the result was strategies, methods, and tools that supported CRM training. The success of this applied training research resulted in a methodology for the design, development, and evaluation of CRM training programs for naval aviation. The methodology was used as the basis for required annual CRM training throughout naval aviation. This article discusses the (a) operational need, (b) research effort, (c) operational transition, and (d) lessons learned.

Chapter 12 - The Migration of Crew Resource Management Training

The US Navy's Crew Resource Management (CRM) training programme has not been evaluated within the last decade. Reactions were evaluated by analysing 51,570 responses to an item pertaining to CRM that is part of a safety climate survey. A total of 172 responses were obtained on a knowledge test. The attitudes of 553 naval aviators were assessed using an attitudes questionnaire. The CRM mishap rate from 1997 until 2007 was evaluated. It was found that naval aviators appear to think than CRM training is useful, are generally knowledgeable of, and display positive attitudes towards, the concepts addressed in the training. However, there is a lack of evidence to support the view that CRM training is having an effect on the mishap rate. As the next generation of highly automated aircraft becomes part of naval aviation, there is a need to ensure that CRM training evolves to meet this new challenge.

Due to the success of CRM training in aviation at improving the knowledge, attitudes, and skills of trainees, CRM training has been adapted and implemented in a number of domains beyond aviation. In this paper, we chose to focus on the health care community's efforts because of the high risk and frequency of errors in this domain. Therefore, the purpose of this paper is threefold. First, we turned to the literature to identify and gather the training evaluation efforts being conducted in the health care community. Fifteen studies were identified. Next, we analyzed the research to determine the effectiveness of the CRM training efforts. As the effectiveness of training was not as positive as hoped, we conclude with several observations that may be hindering the effectiveness of their training efforts.

Training teams and leaders to reduce resuscitation errors and improve patient outcome

Excellence in performance and stress reduction during two different full scale simulator training courses: A pilot study

Surgery in Safe Hands

Il y a le feu au bloc opératoire !

Errors are frequent in healthcare, but Emergency Departments are among the highest risk areas due to frequent changes in team composition, complexity and variety of cases, and difficulties encountered in managing multiple patients simultaneously.Crisis resource management (CRM) training has been associated with decreased error rates in the aviation industry as well as in certain areas of acute medical care, such as anesthesia and emergency medicine. In this study, we assessed whether a single day CRM training, combining didactic and simulation sessions, improves nontechnical skills (NTS) of interprofessional emergency medical teams.Seventy health professionals with different qualifications, working in an emergency department, were enrolled in the study. Twenty individual interprofessional teams were created. Each team was assessed before and after the training, through 2 in situ simulated exercises. The exercises were videotaped and were evaluated by 2 assessors who were blinded as to whether it was the initial or the final exercise. They used a new tool designed specifically for the assessment of emergency physicians’ NTS. The intervention consisted of one-day training, combining didactic and simulation sessions, followed by an instructor facilitated debriefing. All participants went through this training after the initial assessment exercises.A significant improvement (P < 0.05) was shown for all the NTS assessed, in all professional categories involved, regardless of the duration of prior work experience in the Emergency Department.This study shows that even a short intervention, such as a single day CRM training, can have a significant impact in improving NTS, and can potentially improve patient safety.

Research has shown that lack of non-technical skills (also called, Crew Resource Management – CRM) contributes to most personal and process safety incidents in high-risk industries. Situational awareness in the most influential non-technical skill. Authors developed a disruptive program to generate situational awareness competencies and improve non-technical skills of front-line employees. The program is called: Permanent Attention. It combines best practices adapted from high-risk industries and industrial psychology science to generate statistically significant improvements in situational awareness competencies through a sustainable approach. The paper shows real case applications from two organizations where the Permanent Attention program has generated significant improvement in the participants’ situational awareness competency and non-technical skills (CRM) acquisition and retention. The program is executed with the learner at the centre of the process. The Situation Awareness Global Assessment Technique (SAGAT, Endsley, and Garland, 2000) is used to measure the levels of situational awareness competency before and after the training, to determine the impact achieved through the program. The ‘Repeated measures – ANOVA’ statistic model is used to scientifically demonstrate the significant impact of the process in the three elements of situational awareness: Attention, Comprehension and Anticipation. The training sessions in-between SAGATs contain theoretical information and practical exercises that equip the participant(s) with fundamentals. Exercises for each element of the situational awareness have been designed to obtain optimal skills. To complement the practice of non-technical skills, employees are exposed to simulations of critical operational situations, so they can demonstrate their leadership, communications, teamwork, decision-making skills in group and learn how to manage stress in such situations. The levels of situational awareness competencies are periodically monitored via practical field-based SAGAT exercises after the training, to ensure a sustained improvement in the participants’ competencies. The feedback provided immediately after each SAGAT is key to let the participant focus on the element(s) of situational awareness that require improvement. Once the feedback is received, the participant is requested to self-evaluate her level on each element of situational awareness. This reinforces self-consciousness and encourages a sense of commitment to improve by conviction, not by imposition. This novel methodology gives the organization a clear metric on the return on the investment the program offers and allows very specific interventions to improve individual and group level non-technical skills across the organization. A truly disruptive method. The paper shows how the Permanent Attention instructional design combines Endsley's research from the aviation and nuclear industries, with Knowles’ andragogy (2015) theory to achieve competency acquisition and retention. Novel approaches such as the SAGAT, are used with scenarios that are familiar to the target population. Participants’ levels of situational awareness competency are measured before and after the training to determine whether a statistically significant impact was achieved through the program. Permanent Attention has been implemented in organizations with a wide range of maturity. Case-studies are presented to illustrate the process, results, and limitations in two leading drilling companies, in which 60 to 70% of incidents had situational awareness as a contributing factor, over last five years. The implementation of the program has brought statistically significant improvements in the competency of employees and the pilot in Colombia has shown early signs of accident reduction.

Problems in the application of data from accidents and incidents in the design of crew training are discussed. An alternative approach, monitoring normal operations, is proposed and described. Criticisms of the usefulness of crew resource management (CRM) training based on accidents are considered, and the importance of normal process monitoring for evaluation of CRM efficacy is discussed.

This research is a meta-analysis of studies on Crew Resource Management (CRM)/Incident Command System implementation in the fire and emergency services. After a thorough literature review, four sets of results were analyzed to determine if CRM training was effective. An aggregate total of 283 test scores were evaluated. The data indicated that CRM training was effective in all studies analyzed. Fixed and random effects models indicated significance as well. The studies had a high degree of heterogeneity probably due to different training and testing procedures used. The data support the use of CRM training in the fire and emergency services. There is evidence for the need for ongoing CRM training as well. Recommendations include designing CRM training with both initial and recurring sessions to ensure internalization of CRM concepts. Future research should also focus on studies with course outcome measures such as pre and post test scores.

In their paper, Marshal et al. sought to measure the effect of a checklist on the technical and non-technical skills of doctors during a simulated airway problem 1. While the Anaesthetists’ Non-Technical Skills (ANTS) instrument they used has been validated in simulated practice 2, 3, we are not aware of any published data relating to routine clinical practice. This is surprising, since the assessment of non-technical skills is now a routine part of anaesthetic training, and failure to demonstrate appropriate non-technical skills may lead to a failure to progress through anaesthesia training 4. In contrast, the UK aviation industry introduced standardised non-technical skills training instruction in 1998, in the form of ‘crew resource management’, with later development into NOTECHS (non-technical markers). With institutional research and ethical approval, we have recently compared the non-technical skills of 21 volunteer non-consultant anaesthetists (seven female, 16 male) in three hospitals, measured using both normal subjective clinical assessment and ANTS assessment. All clinical supervisors had completed formal, summative assessments on a regular basis, but were not formally trained in the use of the ANTS tool. Out of a maximum score of 16, the median (IQR [range]) overall rating was 14 (12–16 [12–16]). Nine anaesthetists scored 16, with all supervisors scoring at least one subject as 16. Within the four main categories of task management, teamworking, situation awareness and decision making, all ratings were either good or acceptable with no ratings of poor or marginal. There was a median (IQR [range]) of 4 (0-5 [0-20]) positive and 0 (0–1 [0–4]) negative comments per case. We did not expect ANTS scores to be so uniformly high, and believe it unlikely that almost half of study anaesthetists performed at an ‘exemplar’ level with no subject exhibiting poor behaviour. While it is possible that the ANTS scoring system is invalid or that the raters failed to observe correctly, the literature suggests this is unlikely 2, 3, 5. However, that an ‘acceptable’ rating was provided for six anaesthetists despite their failure to complete the mandatory surgical safety checklist, and for 21 anaesthetists despite negative comments’ being offered by assessors, suggests a lack of clarity among assessors in this study about how to apply the standards objectively within the close social interaction that exists between student and assessor 6. These data reinforce the absolute necessity for the assessment of non-technical skills to be undertaken by properly trained assessors 3, 5.