Software-driven structured expert judgment: modern tools to efficiently synthesize scientific knowledge for uncertainty quantification in volcanic hazard assessment

Benchmark exercise on expert judgment techniques in PSA Level 2

Workshop report

Developing therapies for rare diseases is challenging due to limited evidence and high degrees of uncertainty regarding the value of new treatments. Clinical expert judgment can inform modeling assumptions and address areas of uncertainty in reimbursement submissions. As current protocols do not adequately address the challenges faced in rare diseases, this research aimed to generate recommendations for the collection and reporting of clinical expert judgment in rare diseases. An international group of industry, payer, and patient experts with a background in rare diseases participated in a roundtable meeting, which aimed to identify practical challenges in and solutions for gathering clinical insights to aid reimbursement decisions for rare disease therapies. Recommendations were cocreated through iterative discussions and group agreement. Developers should proactively identify uncertainties that expert judgment can address, in parallel with early evidence generation planning. Expert judgment method(s) depend on the uncertainties, with those key to decision-making requiring more robust and time-intensive methods. For highly complex and uncertain topics, methods should facilitate consensus building and expression of diverse views. Given the scarcity of rare disease experts, a high time burden falls on a few experts. Developers should engage diverse stakeholder groups to integrate broader clinical perspectives and reduce reliance on specific individuals while approaching conflicts of interest pragmatically and transparently. These recommendations create a blueprint for developers of rare disease therapies to conduct high-quality clinical expert judgment studies. Hence, developers can present more robust evidence to inform key areas of uncertainty in reimbursement decisions, where empirical evidence is unavailable.

BackgroundAccording to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary.ObjectiveWe present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries.MethodsWe used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Performance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation.ResultsIndividual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection.ConclusionsCooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.

Abstrak: Validitas LKPD dengan Variasi Gamification berbantuan Aplikasi Wordwall: Metode Expert Judgement. Pendidikan harus meningkatkan upaya penyelarasan teknologi dan manusia untuk membuka peluang baru yang lebih kreatif dan inovatif. Pembelajaran di SMKN 1 Jombang masih menunjukkan kurangnya pemanfaatan bahan ajar sehingga mengakibatkan peserta didik kurang minat dan berfokus belajar. Penelitian ini dilakukan untuk mengungkap validitas produk hasil pengembangan bahan ajar Lembar Kerja Peserta Didik (LKPD) dengan variasi gamification berbantuan aplikasi Wordwall. Jenis penelitian pengembangan ini dilakukan dengan metode ADDIE, terfokus pada bagaimana penilaian kelayakan bahan ajar LKPD oleh para ahli, yang terdiri dari ahli materi, ahli bahasa, dan ahli media. Hasil penelitian menunjukkan bahwa bahan ajar LKPD dinilai sangat layak oleh para ahli dengan nilai rata-rata 89%. Penilaian ahli materi 96%, penilaian ahli bahasa 79%, dan penilaian ahli media 92%. Dengan demikian, dapat dikatakan bahwa bahan ajar LKPD layak digunakan peserta didik. Bahan ajar yang valid dan layak menurut ahli diharapkan dapat membantu peserta didik mencapai tujuan pembelajaran yang ditetapkan.Kata kunci: Expert Judgement; Gamifikasi; LKPD; Wordwall; Abstract: Validity of LKPD with Gamification Variations Assisted by Wordwall Application: Expert Judgement Method. Educational institutions must enhance their efforts to integrate technology with human endeavors to foster innovation. At SMKN 1 Jombang, inadequate use of teaching materials has led to a decline in student interest and focus. This study aimed to evaluate the effectiveness of developing Learner Worksheets (LKPD) with gamification features using the Wordwall application. The research employed the ADDIE method to assess the feasibility of these materials from the perspectives of material experts, linguists, and media experts. The findings revealed that the LKPD teaching materials were highly feasible, with an overall expert rating of 89%. Specifically, material experts gave a rating of 96%, linguists 79%, and media experts 92%. The study concluded that the LKPD teaching materials are suitable for student use. These validated and expert-approved materials are expected to help students achieve the desired learning objectives by increasing engagement and focus.Keyword: Expert Judgement; Gamification; LKPD; Wordwall;

Uncertainty quantification of reservoir performance using machine learning algorithms and structured expert judgment

Employees are valuable to a company, so hiring the right ones is crucial. However, "Kenapa Creative" startup struggles with recruiting, as 179 applicants overwhelm their staff. Manual data management makes it hard to get the best results. To tackle this, a Decision Support System (DSS) is developed. The goal of this research is to create an employee recruitment DSS to help startup managers make hiring decisions more easily. It begins with collecting recruitment-related data through interviews and literature review. System design starts from activity diagrams to application with the Analytical Hierarchy Process (AHP) method for prioritizing criteria and sub-criteria, while the Weighted Product (WP) method is used for alternative ranking. AHP is considered capable of decomposing complex problems using hierarchies, making a problem more structured and systematic. WP is deemed to have simple complexity, making computation time faster and addressing recruitment problems requiring quick selection. DSS testing uses Black Box, Usability, SPK validity, and Expert Judgment methods. The research results in an employee recruitment DSS. The system can obtain the best candidate to be accepted. The system features work according to black box testing. SUS testing yields scores of 78.5 from the applicant side and 84.5 from the admin side, categorizing the system as Acceptable. SPK validity testing results in an 80% percentage from the comparison between manual calculations and the system. Expert Judgment testing shows that the system produces alternative rankings consistent with expert views.

ABSTRACTThis study evaluates five scoring rules, or measures of statistical accuracy, for assessing uncertainty estimates from expert judgment studies and model forecasts. These rules — the Continuously Ranked Probability Score (), Kolmogorov‐Smirnov (), Cramer‐von Mises (), Anderson Darling (), and chi‐square test — were applied to 6864 expert uncertainty estimates from 49 Classical Model (CM) studies. We compared their sensitivity to various biases and their ability to serve as performance‐based weight for expert estimates. Additionally, the piecewise uniform and Metalog distribution were evaluated for their representation of expert estimates because four of the five rules require interpolating the experts' estimates. Simulating biased estimates reveals varying sensitivity of the considered test statistics to these biases. Expert weights derived using one measure of statistical accuracy were evaluated with other measures to assess their performance. The main conclusions are (1) overlooks important biases, while chi‐square and behave similarly, as do and . (2) All measures except agree that performance weighting is superior to equal weighting with respect to statistical accuracy. (3) Neither distributions can effectively predict the position of a removed quantile estimate. These insights show the behavior of different scoring rules for combining uncertainty estimates from expert or models, and extent the knowledge for best‐practices.

Uncertainty Quantification of Reservoir Performance Using Artificial Neural Network and Structured Expert Judgment

Effects of Increases in IQ in India on the Present Value of Lifetime Earnings: A Structured Expert Judgment Study

Illnesses transmitted by food and water cause a major disease burden in the United States despite advancements in food safety, water treatment, and sanitation. We report estimates from a structured expert judgment study using 48 experts who applied Cooke’s classical model of the proportion of disease attributable to 5 major transmission pathways (foodborne, waterborne, person-to-person, animal contact, and environmental) and 6 subpathways (food handler–related, under foodborne; recreational, drinking, and nonrecreational/nondrinking, under waterborne; and presumed person-to-person-associated and presumed animal contact-associated, under environmental). Estimates for 33 pathogens were elicited, including bacteria such as Salmonella enterica, Campylobacter spp., Legionella spp., and Pseudomonas spp.; protozoa such as Acanthamoeba spp., Cyclospora cayetanensis, and Naegleria fowleri; and viruses such as norovirus, rotavirus, and hepatitis A virus. The results highlight the importance of multiple pathways in the transmission of the included pathogens and can be used to guide prioritization of public health interventions.

The qualitative functioning of any medical institution depends on both professional competence of its staff and document management. The contemporary society is moving towards e-document management. The topicality of investigation stems from necessity to give scientific rationale and put into practice modern information systems and electronic technologies in the activities of head nurses. The article considers realization issues of the Decree of the President of the Russian Federation of May 7, 2018 No. 204 "On national goals and strategic objectives of the development of the Russian Federation for the period until 2024". Among other "national development goals of the Russian Federation for the period until 2024, the Decree sets the goal to ensure the accelerated digital enabling in the economy and social sphere". Therefore, it is no coincidence that the Federal project "Creating a single digital circuit in health care based on unified state health information system" takes central position in the National Health Care Project. The study used content analysis, expert judgment, sociological and statistical methods. The introduction of the AWP SMS program into professional activities of senior medical nurses allows optimizing documents processing. Its effectiveness was evaluated using mathematical modeling methodology.

The study investigated the effect of play therapy on school adjustment on attention deficit hyperactivity disorder pupils in Port Harcourt Metropolis, Rivers State. This study was guided by three research questions and three null hypotheses. Quasi-experimental research design within which; the Solomon four group design was employed. A purposive sampling technique was used to draw the sample for the study. A sample of 60 primary three pupils who met the criteria for ADHD were randomly assigned into the four groups which was made up of two experimental groups and two control groups. Two valid instruments titled: Conners' Rating Scales-Revised (CRS-R) for assessment of Attention Deficit Hyperactive Disorder and School Adjustment Scale (SAS) were used for collection of data in the study. The validity of instruments was determined using both expert judgment and multivariate statistical method of factor analysis. To ensure the reliability of the instruments for internal consistency, Cronbach alpha was used and reliability coefficients of .78 and .510 were gotten for CRS-R and SAS respectively. Data obtained was analyzed using mean, standard deviation, dependent sample t-test, 2, Analysis of Covariance (ANCOVA) and using One-way analysis of Variance (one-way ANOVA). The results revealed that play therapy (toy and object and role play techniques) is significantly effective in the improvement of school adjustment of attention deficit hyperactivity pupils.

As global competition for dominance in advance technologies continues to grow, international research and development (R&D) collaboration has become crucial to secure national competitiveness. For effective collaboration, selecting joint research areas and formulating cooperation strategies based on technological readiness is essential. Existing approaches have primarily relied on expert judgement and qualitative methods for strategic planning. While expert-based methods offer contextual insights, such approaches often lack objectivity and scalability. To address the limitations, this study proposes a data-driven framework that incorporates topic modeling to identify collaboration areas and an index-based quantitative technology readiness assessment to evaluate national capabilities. Applying the research framework to analyze the Korea-United Kingdom cooperation strategies, net-zero building was identified as the priority research area. Furthermore, the net-zero building technology was divided into specific sub-technologies, and collaboration strategies were formulated based on the comparative technological readiness of each country. The proposed framework enhances transparency, objectivity, and reproducibility for strategic international collaboration.

Renewable energy communities are emerging as key players in the sustainable energy transition, yet there is a lack of systematic approaches for evaluating the digital tools that support their development and operation. This study proposes a comprehensive methodology for assessing tools for supporting renewable energy communities, based on a system of key performance indicators and the multi-criteria decision analysis framework method. Twenty-three specific sub-criteria were defined and scored for each tool, and a weighted sum model was applied to aggregate performance. To ensure robust comparison, criteria weights were derived using both expert judgement (pairwise comparisons of ranking and analytical hierarchy process) and objective data-driven methods (the entropy-based method and the criteria importance through intercriteria correlation weighting method). The framework was applied to a diverse sample of contemporary renewable energy community’s tools, including open-source, commercial, and European Union project tools. Key findings indicate that some of the tools have shown noticeable rank shifts between expert-weighted and data-weighted evaluations, reflecting that expert opinions emphasize technical and operational features while objective variability elevates environmental and economic criteria. This assessment enables stakeholders to compare energy community tools based on structured criteria, offering practical guidance for tool selection and highlighting areas for future improvement.