Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Abstract: The development of Situational Judgment Tests (SJTs) is a multi-step procedure that often requires substantial resources and time. ChatGPT as a large language model has already proven helpful for many complex tasks, including the development of psychological questionnaires. However, ChatGPT has so far not been tested for its ability to develop SJT items, which could speed up SJT development and thus contribute to the dissemination of psychometrically well-designed SJTs in practice and also support research requiring specific SJTs or SJT versions. Thus, the current study ( N = 419) examined whether ChatGPT (2023, Feb 13 version using ChatGPT-3.5) can create SJT items, assessing a facet of personality (i.e., gregariousness), that show similar psychometric properties (examined through confirmatory factor, reliability, and correlational analyses) as human-created SJT items. Results revealed that the measurement model for all eleven SJT items that were created by ChatGPT showed a poor model fit, while a reduced set of eight items yielded an acceptable fit. Reliability estimates as well as convergent and discriminant validity evidence were similar for the human-created and the ChatGPT-created SJT. Implications as well as potential boundary conditions of SJT development through ChatGPT are discussed.

A Situational Judgment Test to Assess Students’ Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4

Azerbaijan is updating its civil service recruitment by transitioning to a competency based HRM model to better align with the rapid changes in the labour market. This shift fosters a more flexible, transparent, and validated system, emphasising individual skills and organisational goals. The State Examination Centre (SEC) is instrumental in this transformation, integrating broader HRM strategies for career mobility and talent management. The new model includes significant updates in assessment methods, focusing on cognitive abilities, competencies, and reducing organisational burdens, aimed at enhancing efficiency and inclusiveness in civil service practices. Incorporating Situational Judgment Tests (SJT), general mental abilities (GMA) assessments, structured interviews, and assessment centre (AC), the new approach seeks to provide a more nuanced evaluation of candidates, ensuring a fit that supports both individual career development and national talent management strategies, marking a significant evolution from traditional methods to a comprehensive, competency-driven recruitment process. Key words: Competency-based recruitment, competency-based HRM, generic competencies, technical competencies, situational judgement test (SJT), general mental abilities (GMA)

Abstract. The development of a scoring key for the situational judgment test often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran–Weiss–Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS index. Next, we demonstrate how the CWS index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found that the CWS index was associated with reflective thinking and intuition avoidance.

Widening access promotes student diversity and the appropriate representation of all demographic groups. This study aims to examine diversity-related benefits of the use of situational judgement tests (SJTs) in the UK Clinical Aptitude Test (UKCAT) in terms of three demographic variables: (i) socio-economic status (SES); (ii) ethnicity, and (iii) gender. Outcomes in medical and dental school applicant cohorts for the years 2012 (n=15581) and 2013 (n=15454) were studied. Applicants' scores on cognitive tests and an SJT were linked to SES (parents' occupational status), ethnicity (White versus Black and other minority ethnic candidates), and gender. Firstly, the effect size for SES was lower for the SJT (d=0.13-0.20 in favour of the higher SES group) than it was for the cognitive tests (d=0.38-0.35). Secondly, effect sizes for ethnicity of the SJT and cognitive tests were similar (d=~0.50 in favour of White candidates). Thirdly, males outperformed females on cognitive tests, whereas the reverse was true for SJTs. When equal weight was given to the SJT and the cognitive tests in the admission decision and when the selection ratio was stringent, simulated scenarios showed that using an SJT in addition to cognitive tests might enable admissions boards to select more students from lower SES backgrounds and more female students. The SJT has the potential to appropriately complement cognitive tests in the selection of doctors and dentists. It may also put candidates of lower SES backgrounds at less of a disadvantage and may potentially diversify the student intake. However, use of the SJT applied in this study did not diminish the role of ethnicity. Future research should examine these findings with other SJTs and other tests internationally and scrutinise the causes underlying the role of ethnicity.

Lievens and Motowidlo (2016) present a case for situational judgment tests (SJTs) to be conceptualized as measures of general domain knowledge, which the authors define as knowledge of the effectiveness of general domains such as integrity, conscientiousness, and prosocial behaviors in different jobs. This argument comes from work rooted in the use of SJTs as measures of implicit trait policies (Motowidlo & Beier, 2010; Motowidlo, Hooper, & Jackson, 2006), measured with a format described as a “single response SJT” (Kell, Motowidlo, Martin, Stotts, & Moreno, 2014; Motowidlo, Crook, Kell, & Naemi, 2009). Given evidence that SJTs can be used as measures of general domain knowledge, the focal article concludes with a suggestion that general knowledge can be measured not only by traditional text-based or paper-and-pencil SJTs but also through varying alternate formats, including multimedia SJTs and interactive SJTs.

Reports an error in "Insights from an updated personnel selection meta-analytic matrix: Revisiting general mental ability tests' role in the validity-diversity trade-off" by Christopher M. Berry, Filip Lievens, Charlene Zhang and Paul R. Sackett (Journal of Applied Psychology, 2024[Oct], Vol 109[10], 1611-1634; see record 2024-80266-001). In the original article, the adverse impact ratios for the dotted lines in Figure 1 are incorrect. The error does not impact the conclusions of this research, and the corrected figure is shown in the erratum. (The following abstract of the original article appeared in record 2024-80266-001). General mental ability (GMA) tests have long been at the heart of the validity-diversity trade-off, with conventional wisdom being that reducing their weight in personnel selection can improve adverse impact, but that this results in steep costs to criterion-related validity. However, Sackett et al. (2022) revealed that the criterion-related validity of GMA tests has been considerably overestimated due to inappropriate range restriction corrections. Thus, we revisit the role of GMA tests in the validity-diversity trade-off using an updated meta-analytic correlation matrix of the relationships six selection methods (biodata, GMA tests, conscientiousness tests, structured interviews, integrity tests, and situational judgment tests) have with job performance, along with their Black-White mean differences. Our results lead to the conclusion that excluding GMA tests generally has little to no effect on validity, but substantially decreases adverse impact. Contrary to popular belief, GMA tests are not a driving factor in the validity-diversity trade-off. This does not fully resolve the validity-diversity trade-off, though: Our results show there is still some validity reduction required to get to an adverse impact ratio of .80, although the validity reduction is less than previously thought. Instead, it shows that the validity-diversity trade-off conversation should shift from the role of GMA tests to that of other selection methods. The present study also addresses which selection methods now emerge as most valid and whether composites of selection methods can result in validities similar to those expected prior to Sackett et al. (2022). (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Although situational judgment tests have a long history in the psychological assessment literature and continue to be frequently used in employment contexts, there has been virtually no summarization of this literature. The purpose of this article is to review the history of such tests and present the results of a meta-analysis on criterion-related and construct validity. On the basis of 102 coefficients and 10,640 people, situational judgment tests showed useful levels of validity (rho = .34) that were generalizable. A review of 79 correlations between situational judgment tests and general cognitive ability involving 16,984 people indicated that situational judgment tests typically evidence relationships with cognitive ability (rho = .46). On the basis of the literature review and meta-analytic findings, implications for the continued use of situational judgment tests are discussed, particularly in terms of recent investigations into tacit knowledge.

BackgroundThis study evaluated whether two selection tests previously validated for primary care General Practice (GP) trainee selection could provide a valid shortlisting selection method for entry into specialty training for the secondary care specialty of radiology.MethodsWe conducted a retrospective analysis of data from radiology applicants who also applied to UK GP specialty training or Core Medical Training. The psychometric properties of the two selection tests, a clinical problem solving (CPS) test and situational judgement test (SJT), were analysed to evaluate their reliability. Predictive validity of the tests was analysed by comparing them with the current radiology selection assessments, and the licensure examination results taken after the first stage of training (Fellowship of the Royal College of Radiologists (FRCR) Part 1).ResultsThe internal reliability of the two selection tests in the radiology applicant sample was good (α ≥ 0.80). The average correlation with radiology shortlisting selection scores was r = 0.26 for the CPS (with p < 0.05 in 5 of 11 shortlisting centres), r = 0.15 for the SJT (with p < 0.05 in 2 of 11 shortlisting centres) and r = 0.25 (with p < 0.05 in 5 of 11 shortlisting centres) for the two tests combined. The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both). The SJT did not correlate with either component of the examination.ConclusionsThe current CPS test may be an appropriate selection method for shortlisting in radiology but would benefit from further refinement for use in radiology to ensure that the test specification is relevant. The evidence on whether the SJT may be appropriate for shortlisting in radiology is limited. However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes. Further validation work (e.g. with non-academic outcome variables) is required to evaluate whether an SJT will add value in recruitment for radiology specialty training and will further inform construct validity of SJTs as a selection methodology.

This paper presents a systematic review of the emerging international research evidence for the use of situational judgement tests (SJTs) for testing important non-academic attributes (such as empathy, integrity and resilience) in selection processes. Several databases (e.g. MEDLINE, PsycINFO, Web of Science) were searched to retrieve empirical studies relating to SJTs published between 1990 and 2010. Personal contact with experts in the field was made to identify any unpublished research or work in progress to obtain the most current material. Finally, reference lists were checked to access other relevant journal articles and further research. All research studies were required to meet specific inclusion criteria selected by two independent reviewers. Over 1000 citations were identified during the initial literature search; following the review of abstracts, full-text copies of 76 articles were retrieved and evaluated. A total of 39 articles that adequately met the inclusion criteria were included in the final review. The research evidence shows that, compared with personality and IQ tests, SJTs have good levels of reliability, predictive validity and incremental validity for testing a range of professional attributes, such as empathy and integrity. SJTs can be designed to test a broad range of non-academic constructs depending on the selection context. As a relatively low-fidelity assessment, SJTs are a cost-efficient methodology compared with high-fidelity assessments of non-academic attributes, such as those used in objective structured clinical examinations. In general, SJTs are found to demonstrate less adverse impact than IQ tests and are positively received by candidates. Further research is required to explore theoretical developments and the underlying construct validity of SJTs.

BackgroundThis study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.MethodsMedical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.ResultsApplicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants – but not ethnic majority applicants – showed greater preference for should-do than would-do instructions. Additionally, first-generation university students – but not non-first-generation university students – were more favourable of rating formats than of pick-one formats.ConclusionsFindings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

: This paper discusses the Situational Judgment Test (SJT) methodology for developing selection measures, and provides a brief review of some key research on this type of test. SJTs have been used as employee selection tools for several decades, but in recent years the situational judgment approach has become increasingly popular. These tests present realistic, job-related situations, usually described in writing. Examinees are typically asked to indicate, in a multiple choice format, what should be done to handle each situation effectively. These responses are often scored according to relative level of effectiveness, rather than simply right or wrong. The most common use of SJTs is for selecting managers and supervisors (e.g., Motowidlo, Dunnette, & Carter, 1990). However, SJTs have also been developed to predict success in other types of jobs, including insurance agent, police, and sales positions. This sort of test has become increasingly popular for selecting employees for work in customer service positions as well (e.g., Motowidlo & Tippins, 1993). The military has used SJTs for years (e.g., Helme, 1968), and recently there seems to be an increase in military interest in this type of measure (e.g., Arad & Borman, in preparation; Hanson & Borman, 1995; Hedge, Hanson, Borman, Bruskiewicz, & Logan, 1997; Legree, 1995).

Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process. Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance. A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01). Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.

Abstract: Situational judgment tests (SJTs) are popular instruments in selection and assessment. However, the application of SJTs to non-Western cultural contexts remains scarce. In this study, we investigated whether an SJT on personal initiative, developed in Germany and translated into Cuban Spanish, had similar psychometric properties in Cuba. Second, there is an ongoing debate about the extent to which the situation description plays an important role for SJTs. We supposed that the impact of situation descriptions might depend on test takers’ familiarity with the culture in which the SJT was developed. Hence, we tested whether the omission of situation descriptions had larger effects in a Cuban than in a German sample. We applied a 2 (with vs. without situation description in the item stem) × 2 (cultural background: Cuba vs. Germany) between-subjects design ( NCuba = 192, NGermany = 213). The results revealed similar psychometric properties between Cuban and German test takers concerning measurement invariance, construct-related validity, and reliability. In addition, we examined whether samples differ regarding applicant perceptions: Notably, for four of six applicant perception scales, the Cuban sample reported a more positive view of the SJT. Furthermore, we found that the effect of situation availability on SJT performance did not substantially depend on the test takers’ cultural background. Implications for cross-cultural generalizability are discussed.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test