Benchmark exercise on expert judgment techniques in PSA Level 2

When physical and data sciences are not sufficient to support models and/or decisions, expert judgment is a recognized approach to quantify the uncertainties around specific issues. Among different expert judgment methods, Structured Expert Judgment (SEJ) employs a formalized, documented procedure for obtaining probabilistic belief statements from a group of experts about unknown quantities or parameters. This provides an attractive approach for performing assessments at volcanoes characterized by large knowledge gaps by integrating diverse kinds of information. Uncertainties are likely to be large in these cases, and SEJ can quantify these uncertainties to provide scientists and decision-makers with indications of the reliability of the assessments. The final goal of this approach is to obtain the group’s synthesized uncertainty distribution (representing a new “virtual expert” often called Decision Maker - DM) around specific items (or “target questions”), that result from combining elicited judgments of all the experts.More specifically, performance-based expert elicitation relies on validating expert probability assessments through an impartial empirical trial. Thus, in a performance-based elicitation, experts are tasked with providing their estimations of probability distributions for a set of known quantities, often referred to as “seed items,” which, jointly, serve as calibration benchmarks for expert performance. Experts’ responses provide the basis for performance scoring using the Classical Model algorithm (Cooke, 1991), determined empirically on the individual expert’s attainment, jointly in terms of two separate metrics: “statistical accuracy” and “informativeness”, which are evaluated on the set of seed items overall.Traditionally, the management of a performance-based expert elicitation is time consuming and involves the collection of tens of questionnaires, often manually copied from hard-copies of the responses or email attachments. The performance-calibration algorithms and the production of standard outputs, including statistical samples of the DM responses, are necessary steps every time an elicitation is conducted, but relied on different scripts and independent pieces of software.In this study we present the latest version of the recently released software ELICIPY (de’Michieli Vitturi et al. 2024), which allows organizing and managing performance-based expert elicitation sessions in a partially automated way, resulting in a significantly enhanced efficiency. This new version includes, among other improvements, an “agreement index” to quantify the level of agreement among experts, a continuous version of the Classical Model to compute expert weights, the possibility to import weights from an external file, new plot options, an interactive dashboard for results exploration.ELICIPY enables the smooth conduction of elicitations with a relatively large number of questions and/or experts. Moreover, the management of multiple elicitation sessions, with questionnaire modifications and response updates are made easier. Some of these improvements are demonstrated on the case study of a hazard and risk assessment for the active Kolumbo submarine Volcano (Aegean Sea, Greece), which was made of 20 Seed Items and 64 Target Items in total, the latter structured into 17 subject matter groups (Bevilacqua et al., 2025).

Workshop report

Uncertainty quantification of reservoir performance using machine learning algorithms and structured expert judgment

The ‘PROCESO’ index: a new methodology for the evaluation of operational safety in the chemical industry

Early childhood education needs to develop mathematics. The process of mathematical development in children does not escape the guidance and role of parents. Factors supporting parental guidance include education and parental occupation. This study aims to describe the effect of parents' educational background and occupation on the ability to guide children's mathematics. This study uses a descriptive correlational quantitative method with 186 parents of kindergarten children in the Koto Tangah sub-district as respondents. Analysis of research data was carried out using the percentage of multiple regression with dummy variables using SPSS 26. Data collection from the study used a questionnaire validated by expert judgement techniques and reliability test using Cronbach's alpha formula with the results obtained that is 0.90. The research findings are (1) in terms of parental education, it is in a good category in encouraging children's development, (2) in terms of parents' work, it is in a good category in improving the ability of teachers to monitor children, (3) the effect on increasing mathematical abilities. ,5%, (4) the effect of work on the ability to encourage mathematics by 7.1%, (5) gender does not affect the ability to develop, (6) work and gender together on the ability to develop mathematics by 8 ,3%. (7) overall education, occupation and gender contribute or influence the ability to guide children's mathematics.

Today the experience we provide to customers is very important for the company, a good shopping experience allows for a happy customer, generate benefits for the company such as increased sales. Customer satisfaction is the way in which we have been treated as consumers when making a purchase, this depends both on the treatment and on the product that the consumer receives. Maintain that "Multiple studies indicate a significant relationship between the perception of the quality of services and customer satisfaction." On the other hand, when a customer is in a state of "customer satisfaction" they tend to repeat their purchasing behavior, generating a state of "customer loyalty". Therefore, customer satisfaction and loyalty maintain a strong link. Getting customers to feel happy with the company's products and for them to return to make new purchases requires time and dedication, since the sale does not end when the product is delivered, quite the contrary, the customer must be followed up. after the sale, this will allow the customer to be satisfied and expand that list of customers in the company. As indicated above, services or products are intangible entities, for which it is important to know customer satisfaction in order to analyze them. The general objective of the research was to propose a Differentiation Strategy to improve the level of Customer Satisfaction in the Industrial Processes Company. The research was descriptive and the non-experimental design, descriptive cross-sectional, it was used as a technique to the survey and as an instrument the questionnaire that measured Customer Satisfaction through the Likert Scale. Crombach's alpha test was used to determine the validity and reliability of the instrument. Likewise, an interview guide was applied to the business administrator to carry out the qualitative analysis of customer satisfaction. The theoretical precedents that support the present investigation are based on the theories that, according to Lovelock and Patterson, maintain that service is any act, performance or experience that one party can offer to another and that it is essentially intangible and does not result in ownership. you're welcome, but nevertheless create value for the recipient. Its production may or may not be linked to a physical product. essful in any business. The results allowed to know the level of customer satisfaction, which is at a high level which is equivalent to 37.71%, which means that the company has acceptable strengths regarding the level of customer satisfaction such as variety of products, prices according to the market, among others, it was also possible to design the differentiation strategy, validated through the expert judgment technique.

This study aims to calibrate the instrument of argumentation skills on the subject of Fluid Statics using item response theory (IRT). The blueprint instrument developed based on the aspects and sub-aspects of argumentation skills, totaling 20 items. The scoring uses the Partial Credit Model (PCM) principle in 4 categories. Subjects involved in the study were 206 undergraduate students at two different universities. The validation of the instrument using expert judgment techniques, which consisted of linguists, physicists, and educational measurement experts. Instrument calibration for argumentation skills is carried out by (1) analysis of item fit with PCM by testing goodness of fit items, (2) testing prerequisite analysis, (3) reliability, (4) difficulty level, and (5) Standard Error of Measurement (SEM). The test developed is valid with Aiken V index values in the range 0.93 to 1. The analysis of IRT assumptions uses two parameters, the adequacy test of the sample and the factor analysis. The Chi-Square value in the Bartlett test is 2202.67 with df 190, and a p-value value (Sig.) <0.05 indicates that the sample size used has met the unidimensional requirements. The instrument developed has met the criteria for scoring the Polytomus items, based on a) the MNSQ INFIT value is in the range of 0.70 to 1.30, and b) the difficulty index test is in the range -2.00 <b <2.00. The results of the reliability estimate on the test equipment obtained a reliability value of 0.86, which means that the reliability of the test is in the high category. The graph of the information function and SEM converge on the ability scales of -2.8 and +1.5, meaning that the skills test kit argues it is suitable for testees in the low to moderate ability range.

La investigación está enmarcada bajo una metodología de un estudio descriptivo, de campo bajo el enfoque cuantitativo. La población y muestra está constituida por las docentes de Educación Inicial de la Escuela U.E.E. “Argelia Laya” Municipio Santiago Mariño del estado Aragua. Para la recolección de datos se aplicó como instrumento la encuesta que consta de 4 preguntas con varias alternativas de repuestas determinando su validez de contenido a través de la técnica de Juicio de Expertos y la confiabilidad se obtuvo mediante Alpha de Combrach resultando 0,87. Esto permitió conocer la disposición que tienen los docentes de aplicar la estrategia propuesta. Los resultados obtenidos demostraron que aun cuando los docentes poseen los conocimientos sobre el proceso constructivo de la enseñanza de la escritura, estos continúan dejando de lado dicho proceso de aprendizaje en la etapa inicial.

Education is one of those affected by Covid-19, so students carry out the learning process from home and the role of parents is demanded more so that they can develop science in children. The socioeconomic status of parents is very important in supporting facilities for learning and developing children's science. This study aims to describe the contribution of parents' socioeconomic background during the Covid-19 period to children's scientific development. This study uses a descriptive correlational quantitative method with 94 respondents as parents of children in Kindergarten, Bukik Barisan District. Data analysis was performed using the regression method using dummy variables with SPSS 26 software. The instrument was validated using expert judgment techniques and the reliability test showed Cronbach's Alpha 0.872. The research findings show that learning outcomes based on (1) parental occupation and income are in the good category, (2) the contribution of parents' work to the development of science is 3.7%, (3) the contribution of parents' income to the development of science is 9.3%, (4) the contribution of work and income of parents to the development of science is 14.7%, (5) overall, work and income contribute to the development of children's science.

Abstrak: Validitas LKPD dengan Variasi Gamification berbantuan Aplikasi Wordwall: Metode Expert Judgement. Pendidikan harus meningkatkan upaya penyelarasan teknologi dan manusia untuk membuka peluang baru yang lebih kreatif dan inovatif. Pembelajaran di SMKN 1 Jombang masih menunjukkan kurangnya pemanfaatan bahan ajar sehingga mengakibatkan peserta didik kurang minat dan berfokus belajar. Penelitian ini dilakukan untuk mengungkap validitas produk hasil pengembangan bahan ajar Lembar Kerja Peserta Didik (LKPD) dengan variasi gamification berbantuan aplikasi Wordwall. Jenis penelitian pengembangan ini dilakukan dengan metode ADDIE, terfokus pada bagaimana penilaian kelayakan bahan ajar LKPD oleh para ahli, yang terdiri dari ahli materi, ahli bahasa, dan ahli media. Hasil penelitian menunjukkan bahwa bahan ajar LKPD dinilai sangat layak oleh para ahli dengan nilai rata-rata 89%. Penilaian ahli materi 96%, penilaian ahli bahasa 79%, dan penilaian ahli media 92%. Dengan demikian, dapat dikatakan bahwa bahan ajar LKPD layak digunakan peserta didik. Bahan ajar yang valid dan layak menurut ahli diharapkan dapat membantu peserta didik mencapai tujuan pembelajaran yang ditetapkan.Kata kunci: Expert Judgement; Gamifikasi; LKPD; Wordwall; Abstract: Validity of LKPD with Gamification Variations Assisted by Wordwall Application: Expert Judgement Method. Educational institutions must enhance their efforts to integrate technology with human endeavors to foster innovation. At SMKN 1 Jombang, inadequate use of teaching materials has led to a decline in student interest and focus. This study aimed to evaluate the effectiveness of developing Learner Worksheets (LKPD) with gamification features using the Wordwall application. The research employed the ADDIE method to assess the feasibility of these materials from the perspectives of material experts, linguists, and media experts. The findings revealed that the LKPD teaching materials were highly feasible, with an overall expert rating of 89%. Specifically, material experts gave a rating of 96%, linguists 79%, and media experts 92%. The study concluded that the LKPD teaching materials are suitable for student use. These validated and expert-approved materials are expected to help students achieve the desired learning objectives by increasing engagement and focus.Keyword: Expert Judgement; Gamification; LKPD; Wordwall;

Uncertainty Quantification of Reservoir Performance Using Artificial Neural Network and Structured Expert Judgment

BackgroundAccording to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary.ObjectiveWe present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries.MethodsWe used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Performance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation.ResultsIndividual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection.ConclusionsCooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.

Effects of Increases in IQ in India on the Present Value of Lifetime Earnings: A Structured Expert Judgment Study

Illnesses transmitted by food and water cause a major disease burden in the United States despite advancements in food safety, water treatment, and sanitation. We report estimates from a structured expert judgment study using 48 experts who applied Cooke’s classical model of the proportion of disease attributable to 5 major transmission pathways (foodborne, waterborne, person-to-person, animal contact, and environmental) and 6 subpathways (food handler–related, under foodborne; recreational, drinking, and nonrecreational/nondrinking, under waterborne; and presumed person-to-person-associated and presumed animal contact-associated, under environmental). Estimates for 33 pathogens were elicited, including bacteria such as Salmonella enterica, Campylobacter spp., Legionella spp., and Pseudomonas spp.; protozoa such as Acanthamoeba spp., Cyclospora cayetanensis, and Naegleria fowleri; and viruses such as norovirus, rotavirus, and hepatitis A virus. The results highlight the importance of multiple pathways in the transmission of the included pathogens and can be used to guide prioritization of public health interventions.

Developing therapies for rare diseases is challenging due to limited evidence and high degrees of uncertainty regarding the value of new treatments. Clinical expert judgment can inform modeling assumptions and address areas of uncertainty in reimbursement submissions. As current protocols do not adequately address the challenges faced in rare diseases, this research aimed to generate recommendations for the collection and reporting of clinical expert judgment in rare diseases. An international group of industry, payer, and patient experts with a background in rare diseases participated in a roundtable meeting, which aimed to identify practical challenges in and solutions for gathering clinical insights to aid reimbursement decisions for rare disease therapies. Recommendations were cocreated through iterative discussions and group agreement. Developers should proactively identify uncertainties that expert judgment can address, in parallel with early evidence generation planning. Expert judgment method(s) depend on the uncertainties, with those key to decision-making requiring more robust and time-intensive methods. For highly complex and uncertain topics, methods should facilitate consensus building and expression of diverse views. Given the scarcity of rare disease experts, a high time burden falls on a few experts. Developers should engage diverse stakeholder groups to integrate broader clinical perspectives and reduce reliance on specific individuals while approaching conflicts of interest pragmatically and transparently. These recommendations create a blueprint for developers of rare disease therapies to conduct high-quality clinical expert judgment studies. Hence, developers can present more robust evidence to inform key areas of uncertainty in reimbursement decisions, where empirical evidence is unavailable.