Validating Human Performance Models: Building a House without Bricks

Human performance modeling (HPM) has been effective for determining crew designs. Crew design includes determining the number of operators needed, the role of automation, and member task responsibilities required to operate a system. Crew models are built for target systems under development to better inform the design and the needs of each operator’s user interface. This is a Verification and Validation (V&V) challenge—it is difficult to obtain metrics to validate a model when the completed system does not yet exist for obtaining sample measures. This paper addresses the challenge with an implementation of the V&V approach applying it to an HPM that successfully received V&V Accreditation for a large United States Air Force (USAF) command and control system. This paper then compares the results of the validated model with NASA TLX workload metrics and discusses the V&V effectiveness. Finally, it recommends better methods for V&V for human performance models.

ChemInformVolume 19, Issue 33 Physical Organic Chemistry ChemInform Abstract: Ionic Structure and Interactions in 1-Methyl-3-ethylimidazolium Chloride-AlCl3 Molten Salts K. M. DIETER, K. M. DIETER Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorC. J. JUN. DYMEK, C. J. JUN. DYMEK Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorN. E. HEIMER, N. E. HEIMER Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorJ. W. ROVANG, J. W. ROVANG Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorJ. S. WILKES, J. S. WILKES Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this author K. M. DIETER, K. M. DIETER Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorC. J. JUN. DYMEK, C. J. JUN. DYMEK Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorN. E. HEIMER, N. E. HEIMER Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorJ. W. ROVANG, J. W. ROVANG Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this authorJ. S. WILKES, J. S. WILKES Frank J. Seiler Res. Lab., US Air Force Acad., Colorado Springs, CO 80840-6528, USASearch for more papers by this author First published: August 16, 1988 https://doi.org/10.1002/chin.198833051Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat No abstract is available for this article. Volume19, Issue33August 16, 1988 RelatedInformation

Human performance plays an important role in system performance and is influenced by a wide set of internal, organizational, and environmental factors. Human–robot systems are being developed for a large number of domains, including in-home care, military deployments, emergency response, industrial robots, and therapeutic applications. Human performance modeling provides the opportunity to investigate multiple system prototypes and theories regarding human performance that can be further evaluated in human-in-the-loop experiments, to quickly adapt to rapidly changing robotic technology, and to represent human behavior in extreme conditions. Interacting with robots also influences human performance, and models of the human–robot system provide an avenue to investigate the impact on human performance that the human–robot teaming elicits. This chapter provides an overview of examples representing common methods for assessing human performance in human–robot systems. The topics are summarized in Table 3.2 and include a description of human performance modeling, a summary of commonly used human performance modeling tools, examples of human performance modeling with a focus on its application in human–robot systems, information regarding the validation of models, and guidelines for implementing human performance modeling techniques for robotics.

WHAT WITH delivering thousands of babies a year, caring for family members with physical and mental disabilities, rehabilitating adults with alcohol and/or drug abuse problems, counseling couples with marital woes, and dealing with the various ailments and injuries that can afflict a basically healthy young population, it's cause for wonder that peacetime personnel at the US Army and US Air Force medical centers in West Germany are able effectively to carry out the other half of their mission: being prepared to provide medical support in the event of war. Visits to the two centers under the Heidelberg-based Seventh Medical Command, Frankfurt Army Regional Medical Center (FARMC) and Landstuhl Army Regional Medical Center (LARMC), and to the Wiesbaden Air Force Regional Medical Center give some idea how they do it. The task is clearly immense. All three are responsible for primary and tertiary care in their own approximately 13 000 km

We evaluated the effectiveness of the US Air Force Suicide Prevention Program (AFSPP) in reducing suicide, and we measured the extent to which air force installations implemented the program. We determined the AFSPP's impact on suicide rates in the air force by applying an intervention regression model to data from 1981 through 2008, providing 16 years of data before the program's 1997 launch and 11 years of data after launch. Also, we measured implementation of program components at 2 points in time: during a 2004 increase in suicide rates, and 2 years afterward. Suicide rates in the air force were significantly lower after the AFSPP was launched than before, except during 2004. We also determined that the program was being implemented less rigorously in 2004. The AFSPP effectively prevented suicides in the US Air Force. The long-term effectiveness of this program depends upon extensive implementation and effective monitoring of implementation. Suicides can be reduced through a multilayered, overlapping approach that encompasses key prevention domains and tracks implementation of program activities.

: The Air Operations Center (AOC) is the centerpiece of the Air Force's new command and control (C2) system for prosecuting theater conventional war. The AOC is a direct outgrowth of the Tactical Air Control Center (TACC). In Vietnam, the TACC mirrored the divided command structure of the US military establishment in Southeast Asia. This resulted in a C2 system that allowed air power to be responsive to the needs of traditional land campaigns, and yet require extensive planning time for deep interdiction and strategic attacks. Additionally, since a land strategy dictated targeting priorities in South Vietnam, the Air Force's measure of effectiveness in the South was its ability to strike targets requested by ground commanders efficiently. Similarly, agencies other than the Air Force selected and approved deep interdiction and strategic targets during Rolling Thunder. Thus, the Air Force's measure of effectiveness in the North likewise became its efficiency of attacking targets there. Assessment thereby became disconnected from the political and military objectives.

The objective of this research is to evaluate a new methodology, data envelopment analysis (DEA), as a tool for measuring and evaluating the operational efficiency of US Air Force organizations. This study involves the application of DEA to locate possible inefficiencies in the performance of US Air Force real-property maintenance activities. Dynamic behaviour, as exhibited via window analyses, is reviewed along with static counterparts. The testing was done in close coordination with Air Force officials, who reviewed the results for accuracy, validity and relevance. We conclude that this type of efficiency analysis does have value for the Air Force, where it can serve as a guide to auditors, budget programmers, managers and others in measuring, evaluating and enhancing operational efficiency.

Fitness tests for military personnel require primarily weight-bearing activity and are consequently disadvantageous to members with greater body mass. Recent research has developed correction factors for adjusting actual fitness scores in order to remove body mass bias. PURPOSE: To validate these published correction factors for use with US Air Force fitness test components. This study also developed regression models removing all bias due to body weight, body mass index (BMI), and abdominal circumference (AC) measurements. METHODS: Subjects consisted of 533 male and female US Air Force personnel (35.6 ± 6.9 yrs) who completed all 4 components of the Air Force fitness test, consisting of a 1.5 mile run, a one-minute push-up test, a one-minute crunch test, and AC measurements. Published correction factors were used to adjust each fitness test component for body mass. As an alternative to the correction factor method, multiple regression models were developed for each fitness component. The validity of the published correction factors as well as the regression models to remove bias based on body weight, BMI, and AC was determined using the PPM correlation coefficient. RESULTS: For the 1.5 mile run, the corrected run times were not significantly correlated with BMI or AC. However there was a small yet significant correlation between corrected 1.5 mile run times and body weight in both males (-.13) and females (-.14). In females, there was no significant correlation between the corrected push-up or crunch scores and any of the body mass measurements. In males, there were small yet significant correlations between corrected push-up scores and AC (-.15), and between corrected crunch scores and body weight (.17) and BMI (.12). The multiple regression models successfully removed all body mass bias from each fitness component. CONCLUSION: The published correction factors were effective for eliminating body mass bias in push-up and crunch scores in females; however, these correction factors appear to over-correct for 1.5 mile run times in both genders. These correction factors appear to be useful for assessing muscular endurance independent of body mass in US Air Force personnel. For the 1.5 mile run, the developed regression models in this study appear to be an effective alternative method for correction.

Risk factors for severe outcomes among members of the United States military hospitalized with pneumonia and influenza, 2000–2012

To date, the vast majority of developmental programs have had to be satisfied with assessing students’ learning within the confines of their programs. Although there were opportunities for indirect assessments, with graduate surveys or surveys of supervisors/employers, response rates tended to be low. Even when responses were received, they often came from people with little expertise in learning, making it difficult for educators to translate the feedback into actionable program improvements. The United States (US) Air Force has, however, decided to attempt what other learning institutions have not – arguably cannot. It is attempting to leverage its competency-based approach to performance assessment to enhance program evaluation. Because of the closed nature of this learning environment, where students graduate from US Air Force programs and go on to work in US Air Force jobs, there is a unique opportunity to assess learning as it is actually being applied in the workplace. Leveraging a competency-based force-development paradigm, the US Air Force has constructed a “foundational competencies” model for common knowledge, skills and abilities, complemented by occupational competencies specific to individual jobs, positions and specialties. These competencies are mapped to performance descriptions that determine levels of mastery. These performance descriptions are, in turn, mapped to rubrics that guide performance evaluations. Shortcomings in performance can be identified, quantified, and fed back to the developmental program for potential program modification or, for special updates to the force, to ensure Airmen perform their duties properly and achieve desired levels of mastery. This approach has been approved but is not yet fully implemented. The new Foundational Competencies Model has been developed and validated through a criterion-sampling methodology. In addition, many technical careers have created “occupational competencies” models delineated the specific knowledge, skills and abilities specific to their jobs. The full range of supporting assessment mechanisms, however, are still in the process of being designed and dispatched to facilitate ease of use and centralize data-gathering. This article overviews the US Air Force’s approach and its progress to date.

The US Air Force EarthRadar system is a multi-purpose sensor designed and constructed using radar principles. This technology was originally developed for the US Air Force to detect buried unexploded ordnance (UXO). Bakhtar Associates developed the US Air Force EarthRadar technology under the DoD SBIR program. The system is capable of detecting buried metallic and non-metallic objects, including glass vials. In addition, the manner in which system hardware, signal processing, and the integrated high resolution global positioning system (GPS) are configured makes it ideal for applications such as mapping subsurface geological features, locating cavities and collapse features, and identifying contaminated ground.

: The United States Air Force is at a critical time in its history. Since the end of World War II, the Air Force has enjoyed qualitative technology superiority over its adversaries. With the development of the cyber age, this technology advantage gained by the Air Force has been continuously under assault. The rapid advance of cyberspace operations is driving an imperative to evolve Intelligence, Surveillance and Reconnaissance (ISR) for the Air Force. Within this context, ISR can be the impetus for proactive defense within the cyberspace domain. The existing Air Force ISR capability for support to defensive cyberspace operations has to operate in an environment of global adversaries. The effectiveness of Air Force defensive cyber strategy will depend on long range trend analysis of adversary capabilities and intent. An evolution of ISR for cyber defense can improve protection of key Air Force command and control functions, as well as best preserve the Air Force's qualitative technology advantage against adversary network reconnaissance and attack activities. This paper provides several recommendations to advance ISR for cyber defense. The Air Force should develop a robust ISR Processing, Exploitation and Dissemination (PED) capability devoted to cyberspace. Additionally, the Air Force should conduct an in-depth study to determine resources required for the National Air and Space Intelligence Center to grow capacity for more robust analysis of adversary cyber capabilities. Next, a stronger cyber defensive strategy, enabled by ISR, will require additional intelligence resources or realignment of existing resources in the Air Force ISR Agency and 24th Air Force. ISR capabilities will be the catalyst for cyber defense of critical assets to more fully protect commanders' air, space and cyber operations.

<div class="section abstract"><div class="htmlview paragraph">Evaluation of integrated human-machine systems depends on having accurate human performance models. However, such models often provide only instantaneous snapshots of cognitive state and fail to account for ongoing dynamics. We argue that generative AI solutions can be used to alleviate this problem. Generative AI tools have been successful when applied to problems that have repeatable structure captured by a low-dimensional lexicon and associated with large amounts of training data. These properties apply to human performance modeling as well. Here, we introduce our Generative Cognitive Modeling Tool, a prototype human performance model developed using strategies from the generative AI community. We demonstrate the utility of our approach using simulated driving data. Our results show that cognitive states associated with driving errors are not randomized events but rather the outcome of continuous dynamics and are predictable up to 25 secs prior to the error event. We also find that the voluntary utilization of autonomous driving aids can be predicted, in part, by the disruption of ongoing dynamics. Overall, this underscores the importance of ongoing dynamics for human performance modeling and establish that generative AI approaches can provide one way to account for such factors.</div></div>

: The study of pilot personality has a long and controversial history. Personality characteristics are fairly poor predictors of training completion, but are probably better predictors of operational performance. Personality characteristics are also important considerations in clinical psychological assessment. The current paper describes the personality characteristics of 1301 US Air Force students pilots based on the NEO Personality Inventory (NEO-PI-R). Compared to male adult norms, male student pilots had higher levels of extraversion and lower levels of agreeableness. Compared to female adult norms, female student pilots had higher levels of extraversion and higher levels of openness as well as lower levels of agreeableness. Percentile tables for the five domain scores and 30 facet scales are provided and discussed for clinical use. A case study is also provided as an example of the clinical utility of these US Air Force norms.

United States Air Force survey of dental needs, 1977: methodology and summary of findings