Situational judgment tests: An overview of current research

Widening access promotes student diversity and the appropriate representation of all demographic groups. This study aims to examine diversity-related benefits of the use of situational judgement tests (SJTs) in the UK Clinical Aptitude Test (UKCAT) in terms of three demographic variables: (i) socio-economic status (SES); (ii) ethnicity, and (iii) gender. Outcomes in medical and dental school applicant cohorts for the years 2012 (n=15581) and 2013 (n=15454) were studied. Applicants' scores on cognitive tests and an SJT were linked to SES (parents' occupational status), ethnicity (White versus Black and other minority ethnic candidates), and gender. Firstly, the effect size for SES was lower for the SJT (d=0.13-0.20 in favour of the higher SES group) than it was for the cognitive tests (d=0.38-0.35). Secondly, effect sizes for ethnicity of the SJT and cognitive tests were similar (d=~0.50 in favour of White candidates). Thirdly, males outperformed females on cognitive tests, whereas the reverse was true for SJTs. When equal weight was given to the SJT and the cognitive tests in the admission decision and when the selection ratio was stringent, simulated scenarios showed that using an SJT in addition to cognitive tests might enable admissions boards to select more students from lower SES backgrounds and more female students. The SJT has the potential to appropriately complement cognitive tests in the selection of doctors and dentists. It may also put candidates of lower SES backgrounds at less of a disadvantage and may potentially diversify the student intake. However, use of the SJT applied in this study did not diminish the role of ethnicity. Future research should examine these findings with other SJTs and other tests internationally and scrutinise the causes underlying the role of ethnicity.

Lievens and Motowidlo (2016) present a case for situational judgment tests (SJTs) to be conceptualized as measures of general domain knowledge, which the authors define as knowledge of the effectiveness of general domains such as integrity, conscientiousness, and prosocial behaviors in different jobs. This argument comes from work rooted in the use of SJTs as measures of implicit trait policies (Motowidlo & Beier, 2010; Motowidlo, Hooper, & Jackson, 2006), measured with a format described as a “single response SJT” (Kell, Motowidlo, Martin, Stotts, & Moreno, 2014; Motowidlo, Crook, Kell, & Naemi, 2009). Given evidence that SJTs can be used as measures of general domain knowledge, the focal article concludes with a suggestion that general knowledge can be measured not only by traditional text-based or paper-and-pencil SJTs but also through varying alternate formats, including multimedia SJTs and interactive SJTs.

Abstract: The development of Situational Judgment Tests (SJTs) is a multi-step procedure that often requires substantial resources and time. ChatGPT as a large language model has already proven helpful for many complex tasks, including the development of psychological questionnaires. However, ChatGPT has so far not been tested for its ability to develop SJT items, which could speed up SJT development and thus contribute to the dissemination of psychometrically well-designed SJTs in practice and also support research requiring specific SJTs or SJT versions. Thus, the current study ( N = 419) examined whether ChatGPT (2023, Feb 13 version using ChatGPT-3.5) can create SJT items, assessing a facet of personality (i.e., gregariousness), that show similar psychometric properties (examined through confirmatory factor, reliability, and correlational analyses) as human-created SJT items. Results revealed that the measurement model for all eleven SJT items that were created by ChatGPT showed a poor model fit, while a reduced set of eight items yielded an acceptable fit. Reliability estimates as well as convergent and discriminant validity evidence were similar for the human-created and the ChatGPT-created SJT. Implications as well as potential boundary conditions of SJT development through ChatGPT are discussed.

The Five-Factor Model (FFM) is the most widely used framework for understanding personality, typically assessed through self-report inventories such as the IPIP-NEO-120. Although these tools show strong psychometric properties, reliance on self-reports introduces vulnerabilities to bias, including social desirability, response distortion, and faking, particularly in high-stakes contexts like personnel selection. To address such limitations, researchers have increasingly explored behavior-oriented measures, notably Situational Judgment Tests (SJTs). When analyzed using Cognitive Diagnostic Models (CDMs), SJTs can yield multidimensional mastery profiles that offer a nuanced representation of personality. However, limited evidence exists regarding the convergence between CDM-based SJT profiles and traditional self-reports. This study applied Robust Canonical Correlation Analysis (RCCA) with a Spearman rank correlation matrix to examine structural alignment between CDM-based SJT scores and IPIP-NEO-120 domain scores in a sample of 289 participants. RCCA was chosen to handle common violations of multivariate assumptions, such as non-normality and outliers. Results showed a moderate canonical correlation in the first root, supporting convergent validity while highlighting distinct trait representations across methods. Subsequent roots yielded weaker and less stable associations. Overall, findings indicate that RCCA is a viable approach for evaluating cross-method validity and emphasize the complementary insights of situational and self-report approaches in personality assessment.

Abstract. During the past 20 years, Situational Judgment Tests (SJTs) have developed into a viable tool in personnel selection. Despite their growing popularity, research examining the extent of measurement error is widely lacking. Using reliability generalization, the aim of this article was twofold: (1) establish an estimate for an average coefficient alpha of SJT scores across studies and (2) examine the influence of essential SJT features and selected study variables on score reliability. To handle potential dependent observations a three-level hierarchical linear model was used. The results indicate that the reliability of SJT scores is typically rather low and below recommended levels for high-stakes applications. Additionally, both SJT and study characteristics affect score reliability. Implications for practitioners and researchers are provided to guide an appropriate use of SJTs and to initiate future research.

BackgroundThis study evaluated whether two selection tests previously validated for primary care General Practice (GP) trainee selection could provide a valid shortlisting selection method for entry into specialty training for the secondary care specialty of radiology.MethodsWe conducted a retrospective analysis of data from radiology applicants who also applied to UK GP specialty training or Core Medical Training. The psychometric properties of the two selection tests, a clinical problem solving (CPS) test and situational judgement test (SJT), were analysed to evaluate their reliability. Predictive validity of the tests was analysed by comparing them with the current radiology selection assessments, and the licensure examination results taken after the first stage of training (Fellowship of the Royal College of Radiologists (FRCR) Part 1).ResultsThe internal reliability of the two selection tests in the radiology applicant sample was good (α ≥ 0.80). The average correlation with radiology shortlisting selection scores was r = 0.26 for the CPS (with p < 0.05 in 5 of 11 shortlisting centres), r = 0.15 for the SJT (with p < 0.05 in 2 of 11 shortlisting centres) and r = 0.25 (with p < 0.05 in 5 of 11 shortlisting centres) for the two tests combined. The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both). The SJT did not correlate with either component of the examination.ConclusionsThe current CPS test may be an appropriate selection method for shortlisting in radiology but would benefit from further refinement for use in radiology to ensure that the test specification is relevant. The evidence on whether the SJT may be appropriate for shortlisting in radiology is limited. However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes. Further validation work (e.g. with non-academic outcome variables) is required to evaluate whether an SJT will add value in recruitment for radiology specialty training and will further inform construct validity of SJTs as a selection methodology.

This paper presents a systematic review of the emerging international research evidence for the use of situational judgement tests (SJTs) for testing important non-academic attributes (such as empathy, integrity and resilience) in selection processes. Several databases (e.g. MEDLINE, PsycINFO, Web of Science) were searched to retrieve empirical studies relating to SJTs published between 1990 and 2010. Personal contact with experts in the field was made to identify any unpublished research or work in progress to obtain the most current material. Finally, reference lists were checked to access other relevant journal articles and further research. All research studies were required to meet specific inclusion criteria selected by two independent reviewers. Over 1000 citations were identified during the initial literature search; following the review of abstracts, full-text copies of 76 articles were retrieved and evaluated. A total of 39 articles that adequately met the inclusion criteria were included in the final review. The research evidence shows that, compared with personality and IQ tests, SJTs have good levels of reliability, predictive validity and incremental validity for testing a range of professional attributes, such as empathy and integrity. SJTs can be designed to test a broad range of non-academic constructs depending on the selection context. As a relatively low-fidelity assessment, SJTs are a cost-efficient methodology compared with high-fidelity assessments of non-academic attributes, such as those used in objective structured clinical examinations. In general, SJTs are found to demonstrate less adverse impact than IQ tests and are positively received by candidates. Further research is required to explore theoretical developments and the underlying construct validity of SJTs.

BackgroundThis study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.MethodsMedical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.ResultsApplicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants – but not ethnic majority applicants – showed greater preference for should-do than would-do instructions. Additionally, first-generation university students – but not non-first-generation university students – were more favourable of rating formats than of pick-one formats.ConclusionsFindings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

: This paper discusses the Situational Judgment Test (SJT) methodology for developing selection measures, and provides a brief review of some key research on this type of test. SJTs have been used as employee selection tools for several decades, but in recent years the situational judgment approach has become increasingly popular. These tests present realistic, job-related situations, usually described in writing. Examinees are typically asked to indicate, in a multiple choice format, what should be done to handle each situation effectively. These responses are often scored according to relative level of effectiveness, rather than simply right or wrong. The most common use of SJTs is for selecting managers and supervisors (e.g., Motowidlo, Dunnette, & Carter, 1990). However, SJTs have also been developed to predict success in other types of jobs, including insurance agent, police, and sales positions. This sort of test has become increasingly popular for selecting employees for work in customer service positions as well (e.g., Motowidlo & Tippins, 1993). The military has used SJTs for years (e.g., Helme, 1968), and recently there seems to be an increase in military interest in this type of measure (e.g., Arad & Borman, in preparation; Hanson & Borman, 1995; Hedge, Hanson, Borman, Bruskiewicz, & Logan, 1997; Legree, 1995).

Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process. Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance. A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01). Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.

Abstract: Situational judgment tests (SJTs) are popular instruments in selection and assessment. However, the application of SJTs to non-Western cultural contexts remains scarce. In this study, we investigated whether an SJT on personal initiative, developed in Germany and translated into Cuban Spanish, had similar psychometric properties in Cuba. Second, there is an ongoing debate about the extent to which the situation description plays an important role for SJTs. We supposed that the impact of situation descriptions might depend on test takers’ familiarity with the culture in which the SJT was developed. Hence, we tested whether the omission of situation descriptions had larger effects in a Cuban than in a German sample. We applied a 2 (with vs. without situation description in the item stem) × 2 (cultural background: Cuba vs. Germany) between-subjects design ( NCuba = 192, NGermany = 213). The results revealed similar psychometric properties between Cuban and German test takers concerning measurement invariance, construct-related validity, and reliability. In addition, we examined whether samples differ regarding applicant perceptions: Notably, for four of six applicant perception scales, the Cuban sample reported a more positive view of the SJT. Furthermore, we found that the effect of situation availability on SJT performance did not substantially depend on the test takers’ cultural background. Implications for cross-cultural generalizability are discussed.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test

This article was migrated. The article was marked as recommended. Introduction There is little evidence on tools that can be used to identify individuals most at risk for remediation and attrition in postgraduate surgical training. We explore the extent to which a situational judgment test (SJT) can predict which trainees are likely to require remediation or leave the program. Methods Postgraduate trainees in a single general surgery residency program in the United States completed a 50-item SJT. Data regarding remediation actions and attrition were retrieved from the program for the two years following completion of the assessment. United States Medical Licensing Examination (USMLE) scores were also included to examine their ability to predict remediation and attrition. Results Complete data was available from fifty-two of 56 (92%) residents in the program. Twenty-one percent (11/52) of residents required at least one remediation intervention within the one (for PGY5s) to two years after completing the SJT. Mann-Whitney U tests revealed a significant difference in SJT performance between those needing remediation versus those who did not require remediation, such that those requiring remediation performed worse on the assessment (p Conclusions These data demonstrate that SJTs can be created to effectively identify surgery residents most at risk for remediation across a two-year timeframe. These data provide an additional layer of validity evidence to support the role of SJTs in surgical education and align with other studies failing to find linkages between USMLE scores and residency performance criterion.

Introduction: One of the leadership philosophies that has been getting more and more attention lately is digital leadership. However, measuring the level of digital leadership has not yet been widely discussed. Furthermore, one tool for gathering data is the Situational Judgement Test (SJT). In the recent discussion, the level of digital leadership between primary and secondary teachers has hardly been discussed. Measuring and comparing digital leadership among teachers aligns with the overarching goal of improving education quality, which is a key aspect of Sustainable Development Goal (SDG) 4: Quality Education. Objective: This study intends to determine whether there are significant differences in nine digital leadership dimensions between primary and secondary teachers utilising the SJT as a data collecting tool. Method: This study involved 2,830 respondents from Malaysia. The respondents were teachers who were selected using stratified random sampling. All 10 assumptions are met before the one-way MANOVA analysis is conducted. Results and Discussion: The results indicate that there is no statistically significant difference in the mean score of digital leadership in terms of (1) student engagement, learning and outcomes, (2) learning environment and spaces, (3) professional growth and learning, (4) communication, (5) public relations, (6) branding, (7) opportunities, (8) empowered professionals, and (9) learning catalyst between primary and secondary teachers. Therefore, no further analysis is required. Research Implications: This study contributes to the measurement aspect by using a different assessment tool to collect data, namely the SJT. Originality/Value: The study's findings highlight the relationship between the nature of SJT and digital leadership dimensions. Moreover, this study measures the level of digital leadership based on a combination of two sources as a reference for the constructs. Therefore, it is recommended that SJT be used as an assessment tool in future research on various styles of leadership.

This study analyzes the abilities of pre-service physics teachers in Indonesia using the Situational Judgment Test (SJT) and Pedagogical Content Knowledge (PCK) approaches. Data were collected from 1,090 respondents (77.1% female, average age 26 years) across three regions (West, Central, East) using PCK competency tests and SJT instruments. The data were analyzed with one-sample t-tests to evaluate mean differences against reference values. Results show that the average pedagogical score was 59.8 (SD = 6.5) and the average SJT score was 8.17 (SD = 0.50). Regional disparities were evident, with the Central region scoring highest in both domains (PCK = 60.1; SJT = 8.19), while the East region scored lowest (PCK = 55.3; SJT = 8.12). These findings reveal that although most pre-service teachers demonstrate adequate situational judgment, nearly 80% showed weaknesses in integrating pedagogical strategies with physics concepts. The study suggests incorporating explicit PCK scaffolding and SJT-based training modules into teacher education curricula to strengthen analytical and pedagogical skills. Such targeted interventions are crucial for ensuring equitable and high-quality physics teacher preparation across Indonesia.