Comparative Evaluation of Performance Assessment and Modeling Method for Software Architecture

Introduction In 2011 the Consensus Statement on Performance Assessment was published in Medical Teacher. That paper was commissioned by AMEE (Association for Medical Education in Europe) as part of the series of Consensus Statements following the 2010 Ottawa Conference. In 2019, it was recommended that a working group be reconvened to review and consider developments in performance assessment since the 2011 publication. Methods Following review of the original recommendations in the 2011 paper and shifts in the field across the past 10 years, the group identified areas of consensus and yet to be resolved issues for performance assessment. Results and discussion This paper addresses developments in performance assessment since 2011, reiterates relevant aspects of the 2011 paper, and summarises contemporary best practice recommendations for OSCEs and WBAs, fit-for-purpose methods for performance assessment in the health professions.

Assessment and monitoring is an important step to ensure and facilitate better performance for any activity. Likewise, for the Occupational Health, Safety and Environment Management (HSE), there are different performance assessment parameters, monitoring methods, procedures and the related Management aspects. Some of these are of general type while others specific. Selection of the assessment parameters may serve to establish the required benchmarking or to ascertain performance level leading to better loss control management or these may be some means to satisfy the statutory compliance etc. In an integrated upstream E&P oil/gas company where activities like exploration, well completion, production, maintenance, pipeline network and all associated services are available and undertaken in-house, the selection of the performance parameters and assessment system adopted, are required to be more elaborate and comprehensive. Each organization has a HSE policy and set of defined objectives to be accomplished. This necessitates a better coordinated approach for the selection of performance parameters, assessment and monitoring methods with a view to ensure that the company objectives are met and sustained along with desired future trends. In this paper, different performance assessment parameters and the monitoring systems have been discussed. While dealing with the subject and the concerned issues, effort has been made to lay emphasis on strengthening the systems making it more system-centric instead of data-centric. This paper is more analytical and practical in approach and systematically covers to a large extent the whole gamut of issues involved regarding performance parameters, assessment and monitoring in the area of Occupational Health, Safety and Environment management. Nature of the work upon which paper is based; the field study, literature survey and the professional experience. The conclusions are that while finalizing the performance parameters assessment, monitoring methods, procedures and the related management aspects, it is necessary to view it objectively, detailing the purpose and to ensure minimum acceptable HSE operating standards. Besides, standards of performance are an important tool ensuring Management by Objectives. The area of interest in the paper is primarily based on sharing of experience.

Exploration of design alternatives and estimation of their key performance metrics such as latency and energy consumption is essential for making the proper design decisions in the early phases of system development. Often, high-level models of the dynamic behavior of the system are used for the analysis of design alternatives. Our work presents a blueprint for building efficient and re-usable models for this purpose. It builds on the well-known Y-chart pattern in that it gives more structure for the proper modeling of interaction on shared resources that plays a prominent role in software-intensive embedded systems. We show how the blueprint can be used to model a small yet illustrative example system with the Uppaal tool, and with the Java general-purpose programming language, and reflect on their respective strengths and weaknesses. The Java-based approach has resulted in a very flexible and fast discrete-event simulator with many re-usable components. It currently is used by TNO-ESI and Oce-Technologies B.V. for early model-based performance analysis that supports the design process for professional printing systems.

Natural analogues and validation of performance assessment models

The maritime industry is considered to be a backbone of the global economy. It is therefore imperative to ensure that maritime operations run safely and efficiently. Assessment of maritime performance is necessary for designers and engineers to be able to pinpoint the weakest links in the system and make impactful system improvements. The current article presents a systematic quantitative literature review of research on performance assessment in the maritime industry with the goal of establishing an understanding of accuracy and consistency in the development of methods used to assess performance. The review focuses on four major segments within the industry—port logistics, ship handling, safety and environmental research—and investigates their uses in developing accurate and consistent performance assessment methods. After the completion of an exclusion process, 62 articles published in a wide range of academic journals were used in the analysis. Two important conclusions were drawn from the analysis. First, performance assessment is generally consistent throughout the maritime industry; most papers used accurate and consistent approaches to develop the methods (n = 43). A subsequent bivariate analysis revealed a call for increased attention to the development of assessment methods within the maritime segment of ship handling. The current study suggests and discusses certain directions with regard to assessment research in the maritime industry.

New sources of ancillary services are expected in the power system. For large and conventional generation units the dynamic response is well understood and detailed individual measurement is feasible, which factors in to the straightforward performance requirements applied today. For secure power system operation, a reliable service delivery is required, yet it may not be appropriate to apply conventional performance requirements to new technologies and methods. The service performance requirements and assessment methods therefore need to be generalized and standardized in order to include future ancillary service sources. This paper develops a modeling method for ancillary services performance requirements, including performance and verification indices. The use of the modeling method and the indices is exemplified in two case studies.

In Germany, a geological system is to be created for the final disposal of high-level radioactive waste. Classical methods for assessing human reliability consider people performing tasks in an existing system. This paper is intended to show which reliability aspects need to be addressed to humans in earlier phases of a system development based on typical phases. Also, a method is described for measuring human reliability in the phase of system analysis and system conception for complex and novel research and modeling activities. These are typical of the search for the best possible site for high-level nuclear waste. For the development of reliable systems, reliable human actions are crucial, especially in the early phases of system development. In the later phases, adjustments to the system become increasingly expensive and time-consuming.

Most people in the world now live in urban areas and their shared quest for better cities is embodied in several Sustainable Development Goals of the United Nations. These indicate that successful cities need jobs, adequate housing stock, effective governance, and other support systems. At the most basic level, they need a basket of core public works services like clean water and efficient transit, among others. These must be provided to improve public trust in government by addressing equity and affordability while also improving operational and cost efficiency. These targets are moving as transitions are occurring from stove-piped to integrated services, even while social contracts between government and the private sector are also shifting. Essential tools to improve cities include urban planning and infrastructure development, but applying them effectively faces challenges like climate change, inequality, social disorder, and even armed conflicts. This paper focuses on seven core public works services for drinking water, wastewater, stormwater, trash collection, mass transit, streets and traffic control, and disaster management. It reviews how these have evolved in the US, how they are organized under the federalism system, and how the goal of integrated management is being pursued. Challenges to integrated approaches include increasing responsibilities but lack of funding, political stress, and rule-driven and internally oriented management. Methods for performance assessment are explained under legacy systems based on methods like indicators and benchmarking applied to public works systems. Current methods focus on regulatory targets and the details; information has been shallow and not always timely. This paper projects how the performance assessment of core public works systems can be broadened to address goals like those of the SDGs and assesses why it is difficult to rate major systems. Examples of the activities of NGOs are given and an example of how progress toward SDG6 is included to show why performance management of integrated management applied to linked systems is needed. Performance dashboards with open government are currently the most common pathways, but emerging methods based on data analytics and visualization offer new possibilities. Reviewing the status of public works management shows that it is an important branch of the field of public administration, and it can be presented as a professional field with its own identity. The findings will support educators and researchers as well as provide policy insights into public works and stakeholder engagement.

BackgroundThere are several frameworks that attempt to address the challenges of evaluation of health information systems by offering models, methods, and guidelines about what to evaluate, how to evaluate, and how to report the evaluation results. Model-based evaluation frameworks usually suggest universally applicable evaluation aspects but do not consider case-specific aspects. On the other hand, evaluation frameworks that are case specific, by eliciting user requirements, limit their output to the evaluation aspects suggested by the users in the early phases of system development. In addition, these case-specific approaches extract different sets of evaluation aspects from each case, making it challenging to collectively compare, unify, or aggregate the evaluation of a set of heterogeneous health information systems.ObjectivesThe aim of this paper is to find a method capable of suggesting evaluation aspects for a set of one or more health information systems—whether similar or heterogeneous—by organizing, unifying, and aggregating the quality attributes extracted from those systems and from an external evaluation framework.MethodsOn the basis of the available literature in semantic networks and ontologies, a method (called Unified eValuation using Ontology; UVON) was developed that can organize, unify, and aggregate the quality attributes of several health information systems into a tree-style ontology structure. The method was extended to integrate its generated ontology with the evaluation aspects suggested by model-based evaluation frameworks. An approach was developed to extract evaluation aspects from the ontology that also considers evaluation case practicalities such as the maximum number of evaluation aspects to be measured or their required degree of specificity. The method was applied and tested in Future Internet Social and Technological Alignment Research (FI-STAR), a project of 7 cloud-based eHealth applications that were developed and deployed across European Union countries.ResultsThe relevance of the evaluation aspects created by the UVON method for the FI-STAR project was validated by the corresponding stakeholders of each case. These evaluation aspects were extracted from a UVON-generated ontology structure that reflects both the internally declared required quality attributes in the 7 eHealth applications of the FI-STAR project and the evaluation aspects recommended by the Model for ASsessment of Telemedicine applications (MAST) evaluation framework. The extracted evaluation aspects were used to create questionnaires (for the corresponding patients and health professionals) to evaluate each individual case and the whole of the FI-STAR project.ConclusionsThe UVON method can provide a relevant set of evaluation aspects for a heterogeneous set of health information systems by organizing, unifying, and aggregating the quality attributes through ontological structures. Those quality attributes can be either suggested by evaluation models or elicited from the stakeholders of those systems in the form of system requirements. The method continues to be systematic, context sensitive, and relevant across a heterogeneous set of health information systems.

This paper briefly discusses the role of human factors engineering in the early phases of system development. The concept of system functions is discussed, along with a description of the human factors processes of function analysis and function allocation and how the inclusion of human engineering representation on the design team can enhance achieved design. Finally, an example of system enhancement through function re-allocation is provided. >

Traditional means of gathering performance data are tracing, which is limited by the available storage, and profiling, which has limited accuracy. Performance modeling is often used to interpret the tracing data and generate performance predictions. We aim to complement the traditional data collection mechanisms with online performance modeling, a method that generates performance models while the application is running. This allows us to greatly reduce the storage overhead while still producing accurate predictions. We present PEMOGEN, our compilation and modeling framework that automatically instruments applications to generate performance models during program execution. We demonstrate the ability of PEMOGEN to both reduce storage cost and improve the prediction accuracy compared to traditional techniques such as least squares fitting. With our tool, we automatically detect 3,370 kernels from fifteen NAS and Mantevo applications and model their execution time with a median coefficient of variation (R2) of 0.81. These automatically generated performance models can be used to quickly assess the scaling and potential bottlenecks with regards to any input parameter and the number of processes of a parallel application.

Cascade control is a commonly used control strategy in chemical engineering processes. Compared with the single loop configuration, cascade control has obvious advantages in improving dynamic response performance and anti-interference. Although the technique of single loop performance assessment based on minimum variance theory is relatively mature, the research concerning cascade loop performance assessment is still rarely. Aiming at overcoming the drawback of excessive manipulator, weak robustness and huge computation burden caused by minimum variance performance index, this paper innovatively proposes a method of cascade control system performance assessment based on Naive Bayesian Classification(NBC), which owns the ability of predicting the performance of the control system in advance. Based on the above work, identification and retuning process of the unreasonable parameters are employed afterwards to further enhances the control quality of industry process. Finally, the feasibility and effectiveness of the proposed method are demonstrated by simulation experiments.

The Dundee Endoscopic Psychomotor Tester (DEPT) was developed as a means of obtaining an objective assessment of endoscopic performance using a single hand. This paper evaluates the reliability of the system. Twenty medical students were tested on the DEPT for four runs using the dominant hand. Test-retest reproducibility and coefficient alpha to measure the internal consistency were used as indexes of the reliability of the DEPT system. A positive correlation was found between different runs in all outcome measures of the DEPT. Coefficient a was > 0.7 for all parameters. DEPT provides a reliable method for the objective assessment of single-handed endoscopic performance in terms of aiming and force exerted on the target plate.

The Command Center Processing and Display System-Replacement (CCPDS-R) Performance Model (CPM) has enjoyed an expanded role in the CCPDS-R project. The performance model has influenced not only the system design, but also system allocations of resources. CCPDS-R performance analysis and requirements compliance insight have been substantially enhanced over the early phase of development. The CCPDS-R program has maximized the use of an ADPE performance model during all phases of a systems design. It also provided an approach for its use in concert with the Ada process model. With the effective use of the CPM and the Ada process model, one can maximize the efficiency of both efforts. The CPM is a complementary tool in the Ada process model for building a computer system which meets its requirements.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In sports science, the automation of performance analysis and assessment is urgently required to increase the evaluation accuracy and decrease the performance analysis time of a subject. Existing methods of performance analysis and assessment are either performed manually based on human experts’ opinions or using motion analysis software, i.e., biomechanical analysis software, to assess only one side of a subject. Therefore, we propose an automated system for performance analysis and assessment that can be used for any human movement. The performance of any skill can be described by a curve depicting the joint angle over the time required to perform a skill. In this study, we focus on only 14 body joints, and each joint comprises three angles. The proposed system comprises three main stages. In the first stage, data are obtained using motion capture inertial measurement unit sensors from top professional fighters/players while they are performing a certain skill. In the second stage, the collected sensor data obtained are input to the biomechanical software to extract the player’s joint angle curve. Finally, each joint angle curve is processed using a continuous wavelet transform to extract the main curve points (i.e., peaks and valleys). Finally, after extracting the joint curves from several top players, we summarize the players’ curves based on five statistical indicators, i.e., the minimum, maximum, mean, and mean ± standard deviation. These five summarized curves are regarded as standard performance curves for the joint angle. When a player’s joint curve is surrounded by the five summarized curves, the performance is considered acceptable. Otherwise, the performance is considered unsatisfactory. The proposed system is evaluated based on four different karate skills. The results of the proposed system are identical to the decisions of the expert panels and are thus suitable for real‐time decisions.