Modeling Human Performance for Human–Robot Systems

<div class="htmlview paragraph">The major goal of Space Human Factors program is to provide the technology which will enable safe and productive human performance throughout and after long duration space flights and planetary/lunar missions. The Human Factors element has three major sub-goals:</div> <div class="htmlview paragraph"> <ol class="list nostyle"> <li class="list-item"> <span class="li-label">1.</span> <div class="htmlview paragraph">Human Performance Models, Data and Tools - provide the basic understanding of human's activities in the physical, mental, and perceptual domains and to identify important lessons learned about human activities from prior space and lunar missions. To this end, Johnson Space Center is using a laser-based human body mapping system to measure different anthropometric sizes and shapes for incorporation into a large database. This database is employed, along with a spatial computer image graphics system, to analyze human motion in zero gravity conditions as the hypothetical, human bodysized figure performs a variety of new procedures in Space Station Freedom or in the Shuttle orbiter. At Ames Research Center, human performance models for computer-aided engineering, a program initially developed for aeronautical applications, are being modified for space applications. The value of the dynamic human performance models lies in the versatility in which various management decisions can be evaluated as to their impact on mission effectiveness, personnel skill and knowledge allocation, workload, workstation and cockpit design, and overall system design.</div> </li> <li class="list-item"> <span class="li-label">2.</span> <div class="htmlview paragraph">Crew Support and Enhancement - address the development of technology and its applications for the evaluation of the crew's living and working activities in the spacecraft and planetary/lunar habitat and on planetary/lunar surfaces. During the year, a number of projects were completed. At Johnson Space Center, new computer graphics were developed, particularly new methods for portraying procedures which reduce human error. Also, advances in information management methods for application to Shuttle maintenance operations were developed at Ames Research Center. Data from Kennedy Space Center operations is being used as a testbed for managing the information processing in maintenance.</div> </li> <li class="list-item"> <span class="li-label">3.</span> <div class="htmlview paragraph">Human-Automation-Robotic Systems - determine the means by which humans and automated or semi-autonomous systems can effectively work together. A major project at Ames Research Center is the development of the Virtual Workstation as a user interface device for exploring lunar or planetary surfaces. The effort this year has been to obtain a Mar-like data base and begin to employ it in simulated missions. At Johnson Space Center, effort was directed toward identifying the best location of visual sensors for teleoperations.</div> </li> </ol> </div>

Verification and validation (V&V) for human performance models (HPMs) can be likened to building a house with no bricks, since it is difficult to obtain metrics to validate a model when the system is still in development. HPMs are effective for performing trade-offs between the human system designs factors including number of operators needed, the role of automated tasks versus operator tasks, and member task responsibilities required to operate a system. On a recent government contract, our team used a human performance model to provide additional analysis beyond traditional trade studies. Our team verified the contractually mandated staff size for using the system. This task demanded that the model have sufficient fidelity to provide information for high confidence staffing decisions. It required a method for verifying and validating the model and its results to ensure that it accurately reflected the real world. The situation caused a dilemma because there was no actual system to gather real data to use to validate the model. It is a challenge to validate human performance models, since they support design decisions prior to system. For example, crew models are typically inform the design, staffing needs, and the requirements for each operator’s user interface prior to development. This paper discusses a successful case study for how our team met the V&V challenges with the US Air Force model accreditation authority and successfully accredited our human performance model with enough fidelity for requirements testing on an Air Force Command and Control program.

Validation is critically important when human performance models are used to predict the effect of future system designs on human performance. A model of flight deck operations was validated using a rigorous, iterative, model validation process. The process included the validation of model inputs (task trace and model input parameters), process models (workload, perception, and visual attention) and model outputs of human performance measures (including workload and visual attention). This model will be used to evaluate proposed changes to flight deck technologies and pilot procedures in the NextGen Closely Spaced Parallel Operations concept.KeywordsVisual AttentionVisual FixationModel Input ParameterFederal Aviation AdministrationOutput ValidationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The Requirements and Technical Concepts for Aviation, Inc. (RTCA) has recently proposed a new concept known as “free flight” for guiding the separation of aircraft in the National Airspace System (NAS). “Free flight” in the United States is a Federal Aviation Administration (FAA) strategic goal for system capacity and for Air Traffic Services to improve accessibility, flexibility, and predictability in the national airspace in order to reduce flight times, crew resources, maintenance, and fuel costs. The scenarios in the current experiment were used to explore the farthest out parameters of “free flight” anticipated by RTCA in the year 2025. An evaluation of predicted scenario generation structures associated with “current day” and “free flight” operations under a specific experimental scenario was performed using two integrated human performance modeling tools, Air Man-machine Integration Design and Analysis System (Air MIDAS) and the Integrated Performance Modeling Environment (IPME). In analyses of a common scenario, each tool predicted a different Point of Closest Approach (PCA) distance under “current day” operations ( M Air MIDAS = 5.86 nm, M IPME = 16.67 nm) as compared with “free flight” operations ( M Air MIDAS = 6.65 nm, M IPME = 4.94 nm). This indicates that differences in the proximity between the aircraft existed between the two software tools as well as the contextually sensitive variables. This provides evidence that more focus needs to be brought to the effects of contextual effects that impact the virtual human operator of the NAS. A validation effort of these contextual findings with human-in-the-loop (HITL) data is anticipated and required prior to drawing any definite conclusions.

We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for high-quality surface generation in minutes. It marries the implicit volumetric rendering of the truncated signed distance field (TSDF) with multi-resolution hash encoding. We further propose a hybrid neural tracker to generate animated meshes, which combines explicit non-rigid tracking with implicit dynamic deformation in a self-supervised framework. The former provides the coarse warping back into the canonical space, while the latter implicit one further predicts the displacements using the 4D hash encoding as in our reconstructor. Then, we discuss the rendering schemes using the obtained animated meshes, ranging from dynamic texturing to lumigraph rendering under various bandwidth settings. To strike an intricate balance between quality and bandwidth, we propose a hierarchical solution by first rendering 6 virtual views covering the performer and then conducting occlusion-aware neural texture blending. We demonstrate the efficacy of our approach in a variety of mesh-based applications and photo-realistic free-view experiences on various platforms, i.e., inserting virtual human performances into real environments through mobile AR or immersively watching talent shows with VR headsets.

<div class="section abstract"><div class="htmlview paragraph">Evaluation of integrated human-machine systems depends on having accurate human performance models. However, such models often provide only instantaneous snapshots of cognitive state and fail to account for ongoing dynamics. We argue that generative AI solutions can be used to alleviate this problem. Generative AI tools have been successful when applied to problems that have repeatable structure captured by a low-dimensional lexicon and associated with large amounts of training data. These properties apply to human performance modeling as well. Here, we introduce our Generative Cognitive Modeling Tool, a prototype human performance model developed using strategies from the generative AI community. We demonstrate the utility of our approach using simulated driving data. Our results show that cognitive states associated with driving errors are not randomized events but rather the outcome of continuous dynamics and are predictable up to 25 secs prior to the error event. We also find that the voluntary utilization of autonomous driving aids can be predicted, in part, by the disruption of ongoing dynamics. Overall, this underscores the importance of ongoing dynamics for human performance modeling and establish that generative AI approaches can provide one way to account for such factors.</div></div>

Human factors professionals strive to create effective system design by ensuring that the capabilities and limitations of the human operator are a primary design consideration. By demonstrating the effect of various design decisions concerning the human operator on system performance early in the development cycle, resulting recommendations may be adopted to enhance system performance. To achieve this goal, human performance modeling tools are used. One of the common misconceptions about modeling is that a model alone will impact design. A model is merely a building block within an analysis. An influential analysis incorporates the principles of experimental design, the power of predictive modeling and the varying system design requirements to produce results that empower program managers to make informed decisions regarding system design. Analysts at the U.S. Army Research Laboratory have successfully influenced the conceptual design of several manned ground vehicles by recognizing this difference.

Modeling critical performance elements can assist the human performance technology practitioner in better understanding the often complex, messy, and fuzzy real world of human performance. As illustrated in the embedded case study, generically developed models that, in turn, can be adapted to context-specific applications often have the greatest practical value. The key in developing any good model is to first identify those key variables—the critical few—that truly account for the greatest variance in performance.

Human performance modeling in temporary segmentation Chinese character handwriting recognizers

Providing an authoritative representation of human behavior has recently been recognized as essential for modeling and simulation of system performance. Even earlier; however, the Human Research & Engineering Directorate of the U.S. Army Research Laboratory was working to do just that in a tool called IMPRINT, (the Improved Performance Research Integration Tool). IMPRINT enables a quantitative portrayal of human performance. Task network modeling is implemented with embedded data and sound psychological methods in an easy-to-use interface, all of which has been subjected to intensive verification and validation. IMPRINT is being used today to influence system design and acquisition decisions. Several examples of applications to Army systems are discussed, with implications ranging from detailed equipment design to force-on-force effectiveness. Of course, “more research needs to be done,” but in the meantime, human performance modeling is ready for use.

: The Air Force Research Laboratory's Human Performance Model Integration (HPMI) program is exploring the merit and feasibility of combining multiple human performance models and tools to create hybrid models of performance that address application-specific requirements for model fidelity while controlling cost. The first exploration of HPMI feasibility involves development of a human performance model using task network modeling and ACT-R cognitive modeling. Under the Combat Automation Requirements Testbed program, a detailed task network model of strike fighter pilot performance was developed and integrated with a flight simulator. In the initial implementation the task network model managed prioritization of objects and made decisions about the order in which the targets would be examined. This original task network model of prioritization then was replaced with one implemented in ACT-R (Anderson and Lebeire, 1998). The cognitive model provided improved representations of decision-making and memory, as well as confusion and errors committed by humans. We will describe the task domain, model and interface, as well as, an informal verification study that compared the performance of the hybrid model with that of humans.

In probabilistic risk assessment (PRA) of nuclear power plants (NPPs), human reliability analysis (HRA) is conducted to identify potential human failure events that could contribute to risk scenarios and estimate human error probabilities. Lessons learned from the 2011 Fukushima Daiichi NPP accident underscored that for PRA, it is critical to model external control room (Ex-CR) human actions. The state-of-practice HRA methods, historically developed for the main control room HRA, are limited in capturing the unique nature of Ex-CR human actions, such as location dependence (in addition to the time dependence) of human actions and spatiotemporal interactions of human performance with the surrounding physical environments, for instance, hazard propagation. To advance the Ex-CR HRA in the context of the fire PRA for NPPs, the authors’ team developed a simulation-based fire crew performance model using an agent-based modeling (ABM) technique. The ABM fire crew simulation was coupled with a fire progression model through a spatiotemporal interface using a geographic information system. This paper focuses on the validation of the ABM simulation, which is the key requirement for the simulation-based Ex-CR human performance model to be utilized in PRA. The existing validation approach, initially developed for physical models in the fire PRA of NPPs, is extended for validation of the simulation-based Ex-CR human performance model. Model uncertainty is used as a measure of model validity, which facilitates the incorporation of the validation result into the PRA. The degree of the model uncertainty is characterized by a lognormal error model whose parameters are quantified based on a pairwise comparison between empirical data and model predictions. The proposed validation approach is demonstrated using a case study of the fire PRA of NPPs. This study makes two research contributions: (1) it is the first to validate the simulation-based Ex-CR human performance model against empirical human performance data and incorporate the validation result into the PRA of NPPs, and (2) this study, for the first time, conducts a controlled experimental test to collect empirical data for fire crew performance at NPPs.

A framework to develop an analysis agent for evaluating human performance in manufacturing systems

Volumetric video represents a transformative advancement in visual media, enabling users to freely navigate immersive virtual experiences and narrowing the gap between digital and real worlds. However, the need for extensive manual intervention to stabilize mesh sequences and the generation of excessively large assets in existing workflows impedes broader adoption. In this paper, we present a novel Gaussian-based approach, dubbed DualGS , for real-time and high-fidelity playback of complex human performance with excellent compression ratios. Our key idea in DualGS is to separately represent motion and appearance using the corresponding skin and joint Gaussians. Such an explicit disentanglement can significantly reduce motion redundancy and enhance temporal coherence. We begin by initializing the DualGS and anchoring skin Gaussians to joint Gaussians at the first frame. Subsequently, we employ a coarse-to-fine training strategy for frame-by-frame human performance modeling. It includes a coarse alignment phase for overall motion prediction as well as a fine-grained optimization for robust tracking and high-fidelity rendering. To integrate volumetric video seamlessly into VR environments, we efficiently compress motion using entropy encoding and appearance using codec compression coupled with a persistent codebook. Our approach achieves a compression ratio of up to 120 times, only requiring approximately 350KB of storage per frame. We demonstrate the efficacy of our representation through photo-realistic, free-view experiences on VR headsets, enabling users to immersively watch musicians in performance and feel the rhythm of the notes at the performers' fingertips. Project page: https://nowheretrix.github.io/DualGS/.

Human performance modeling provides a tool to facilitate the development of well-designed systems that support their users. However, models need to be based on and compared against empirical human-in-the-loop data to ensure that the human performance predictions are correct. This paper describes an approach to gathering already-existing empirical data to support the development of a model of astronaut performance in long-duration space missions.