Creating a Psychological Test in a Few Seconds

It is common practice to rely on a convenience sample of subject matter experts (SMEs) when developing scoring keys for situational judgment tests (SJTs). However, the defining characteristics of what constitutes a SME are often ambiguous and inconsistent. Sampling SMEs can also impose considerable costs. Other research fields have adopted crowdsourcing methods to replace or reproduce judgments thought to require subject matter expertise. Therefore, we conducted the current study to compare crowdsourced scoring keys to SME‐based scoring keys for three SJTs designed for three different job domains: Medicine, Communication, and Military. Our results indicate that scoring keys derived from crowdsourced samples are likely to converge with keys based on SME judgment, regardless of test content ( r = .88 to .94 between keys). We observed the weakest agreement among individual MTurk and SME ratings for the Medical SJT (classification consistency = 61%) relative to the Military and Communication SJTs (80% and 85%). Although general mental ability and conscientiousness were each related to greater expert similarity among MTurk raters, the average crowd rating outperformed nearly all individual MTurk raters. Using randomly‐drawn bootstrapped samples of MTurk ratings in each of the three samples, we found that as few as 30–40 raters may provide adequate estimates of SME judgments of most SJT items. These findings suggest the potential usefulness of crowdsourcing as an alternative or supplement to SME‐generated scoring keys.

Abstract. The development of a scoring key for the situational judgment test often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran–Weiss–Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS index. Next, we demonstrate how the CWS index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found that the CWS index was associated with reflective thinking and intuition avoidance.

This study used principles underlying item generation theory to posit competing perspectives about which features of situational judgment tests might enhance or impede consistent measurement across repeat test administrations. This led to 3 alternate-form development approaches (random assignment, incident isomorphism, and item isomorphism). The effects of these approaches on alternate-form consistency, mean score changes, and criterion-related validity were examined in a high-stakes context (N = 3,361). Generally, results revealed that even small changes in the context of the situations presented resulted in significantly lower alternate-form consistency. Conversely, placing more constraints on the alternate-form development process proved beneficial. The contributions, implications, and limitations of these results for the development of situational judgment tests and high-stakes testing are discussed.

This article outlines an active learning project that gives students hands-on experience in developing an undergraduate situational judgment test. The five-part activity models the process for constructing a situational judgment test—a tool commonly used for employee selection in organizations. The project is designed to help students assimilate concepts associated with the development of and decision to use different selection tools. Student reactions indicate that the exercise promotes student attention and engagement, stimulates learning about employee selection tools and concepts, and helps students understand the process of constructing and validating selection methods. Student development is a positive side effect of the exercise as constructing undergraduate situational judgment test items requires participants to explore and evaluate solutions to the problems faced by the typical undergraduate.

The Five-Factor Model (FFM) is the most widely used framework for understanding personality, typically assessed through self-report inventories such as the IPIP-NEO-120. Although these tools show strong psychometric properties, reliance on self-reports introduces vulnerabilities to bias, including social desirability, response distortion, and faking, particularly in high-stakes contexts like personnel selection. To address such limitations, researchers have increasingly explored behavior-oriented measures, notably Situational Judgment Tests (SJTs). When analyzed using Cognitive Diagnostic Models (CDMs), SJTs can yield multidimensional mastery profiles that offer a nuanced representation of personality. However, limited evidence exists regarding the convergence between CDM-based SJT profiles and traditional self-reports. This study applied Robust Canonical Correlation Analysis (RCCA) with a Spearman rank correlation matrix to examine structural alignment between CDM-based SJT scores and IPIP-NEO-120 domain scores in a sample of 289 participants. RCCA was chosen to handle common violations of multivariate assumptions, such as non-normality and outliers. Results showed a moderate canonical correlation in the first root, supporting convergent validity while highlighting distinct trait representations across methods. Subsequent roots yielded weaker and less stable associations. Overall, findings indicate that RCCA is a viable approach for evaluating cross-method validity and emphasize the complementary insights of situational and self-report approaches in personality assessment.

This chapter reviews key research on the similarities and differences in leadership and management across different regions of the world. It also looks at similarities and differences on other relevant aspects, that is, commitment, work values, personality and emotional intelligence. Research has tended to focus on drawing out the differences as that appears to be worthy of news and attracts interest. We also report on the types of errors in research which might actually make real differences appear much larger. The reality is that what we find is a great deal of similarity in leadership and management behaviour across the different regions of the world. Given these similarities, can we develop a management level Situational Judgment Test (SJT) that can be used effectively across different world regions? We believe this can be achieved by identifying SJT items that work consistently across world regions and then assembling a bias-free test with robust psychometric properties.

Aim/Purpose: The study investigates the factors influencing the acceptance and utilisation of large language models (LLMs) (predictor variables of LLM usage), such as ChatGPT, in Learning design by instructional designers and university-teaching academics from various countries. Background: Large language models (LLMs) have exploded onto the scene, transforming the landscape of learning design. Instructional designers and university teaching academics have been overburdened with content creation for their teaching programmes, and the arrival of LLM models will help in this regard by developing more interactive content that drives student engagement and, in turn, contributes to student success. Since LLMs are a relatively new phenomenon, little is known about the factors influencing their acceptance in learning design; therefore, this research is needed, as learning design principles are the bedrock of student engagement and success. Methodology: A cross-sectional correlational quantitative study was employed. Data was collected using an online questionnaire posted on social media, including LinkedIn, from 203 instructional designers and university teaching academics. Purposive and snowball sampling methods were used to target instructional designers and university teaching academics at colleges and universities worldwide. Participants were asked to share the survey link with fellow instructional designers and university-teaching academics in their communities. The factor structure of the data was determined using exploratory factor analysis. Nonetheless, the factor structure derived from the LLMs did not entirely reflect the original configuration of the Unified Theory of Acceptance and Use of Technology (UTAUT3), as certain predictors appeared to coalesce, indicating LLMs’ unique nature in learning design. Confirmatory factor analysis was used to verify the fit of the data on the measurement model. First-order and second-order structural modelling were used to identify the structural relationships among the variables. Contribution: The study determines significant factors for the acceptance of LLMs by instructional designers and academic teaching staff in learning design, enabling possible opportunities for best practices in the field through interventions to optimize LLM usage. The study applies the technology acceptance model to the emerging LLM technology and extends the technology acceptance model by adding the trust construct as a predictor variable. Findings: The structural analysis results indicated that the ingrained LLM practices, LLM peer-driven expectations, innovative propensity towards LLM adoption, reliability and provider trust in LLMs, and ease of use and support influenced perceived LLM benefits and usage, but community standards and infrastructure had no influence. The second-order structural equation modelling indicated that perceived LLM benefits and usage and ingrained LLM habits contributed most to the learning design. Recommendations for Practitioners: Teaching academics and instructional designers must use LLMs in designing content, assessments, and interactive learning activities, and attend LLM training workshops on prompting and best practices in integrating LLMs into learning and teaching to see their benefits; hence, regular use of LLMs will then lead to trust and innovation in LLMs usage, enhancing learning design and improving student learning outcomes. Recommendation for Researchers: Researchers must use mixed methods approaches to have a deeper understanding of the factors influencing LLMs. Since habit and perceived LLM benefits and usage contributed the most variance to learning design, researchers must investigate strategies that optimise these factors in learning design, such as effective intervention strategies that can help form positive LLM habits. In addition, the findings provide researchers with a starting point for future research. Further researchers must investigate interventions that optimise the influence of personal innovativeness and trust that contributed the least variance to learning design, hence unlocking the potential of LLMs in learning design through innovation, responsible, and ethical use. Impact on Society: The use of LLMs in learning design has a high possibility of transforming education, specifically the learning design landscape. Using LLMs will free up more time for teaching academics and instructional designers so that they spend more time on higher-order thinking skill demands. Consequently, the students will be exposed to more engaging and interactive content, resulting in improved learning outcomes. Future Research: Future research must include context-derived external variables in technology acceptance models, such as levels of prompting competencies, to provide a deeper understanding of LLMs. In addition, future research must be based on the application and impact of LLMs on student engagement and success, and their attainment of 21st-century skills.

Objective: This study aimed to examine the validity and reliability of the Multidimensional Assessment of Interoceptive Awareness (MAIA) in a Colombian community sample. Methods: A cross-sectional study was conducted among 487 Colombian people. Participants completed an online survey that included the MAIA and International Fitness Scale (IFIS). Confirmatory factor analysis was conducted to verify the scale's internal structure, reliability analyses (Cronbach's alpha, McDonald's omega, Guttman's lambda 6, and Greatest Lower Bound) were performed at both the overall and factor levels, and test-retest reliability. Correlations with perceived physical condition indicators (as evidence of convergent validity) were also examined. Results: Confirmatory factor analysis supported the dimensional structure of the scale, with adequate goodness-of-fit indices (CFI = .991; GFI = .990; NFI = .988; RMSEA = .070). Intercorrelations among the MAIA dimensions provided evidence of construct reliability, with high values above .80 for six of the eight factors, acceptable values around .50 for factor 2, and low values only for factor 3. Test-retest reliability was high (rho = .717, p < .001). Direct correlations were found between perceived physical condition and some dimensions of the MAIA, as evidence of convergent validity. Conclusion: The MAIA demonstrated adequate evidence of internal structural validity, convergent and discriminant validity, and adequate reliability for measuring interoceptive awareness in a healthy Colombian population. The use of the MAIA to assess interoceptive awareness is recommended in future research as a tool for the prevention of mental and physical health conditions. Further studies are recommended to analyze its metric quality in other populations.

Large Language Models (LLMs) have garnered increasing attention in neurosurgery and possess significant potential to improve the field. However, the breadth and performance of LLMs across diverse neurosurgical tasks have not been systematically examined, and LLMs come with their own challenges and unique terminology. We seek to identify key models, establish reporting guidelines for replicability, and highlight progress in key application areas of LLM use in the neurosurgical literature. We searched PubMed and Google Scholar using terms related to LLMs and neurosurgery ("large language model" OR "LLM" OR "ChatGPT" OR "GPT-3" OR "GPT3" OR "GPT-3.5" OR "GPT3.5" OR "GPT-4" OR "GPT4" OR "LLAMA" OR "MISTRAL" OR "BARD") AND "neurosurgery". The final set of articles was reviewed for publication year, application area, specific LLM(s) used, control/comparison groups used to evaluate LLM performance, whether the article reported specific LLM prompts, prompting strategy types used, whether the LLM query could be reproduced in its entirety (including both the prompt used and any adjoining data), measures of hallucination, and reported performance measures. Fifty-one articles met inclusion criteria, and were categorized into six application areas, with the most common being Generation of Text for Direct Clinical Use (n = 14, 27.5%), Answering Standardized Exam Questions (n = 12, 23.5%), and Clinical Judgement and Decision-Making Support (n = 11, 21.6%). The most frequently used LLMs were GPT-3.5 (n = 30, 58.8%), GPT-4 (n = 20, 39.2%), Bard (n = 9, 17.6%), and Bing (n = 6, 11.8%). Most studies (n = 43, 84.3%) used LLMs directly out-of-the-box, while 8 studies (15.7%) conducted advanced pre-training or fine-tuning. Large language models show advanced capabilities in complex tasks and hold potential to transform neurosurgery. However, research typically addresses basic applications and overlooks enhancing LLM performance, facing reproducibility issues. Standardizing detailed reporting, considering LLM stochasticity, and using advanced methods beyond basic validation are essential for progress.

Teachers’ social-emotional competence is considered important in order to master the social and emotional challenges inherent in their profession and to build positive teacher-student relationships. In turn, this is key to both teachers’ occupational well-being and positive student development. Nonetheless, an instrument assessing the profession-specific knowledge and skills that teachers need to master the social and emotional demands in the classroom is still lacking. Therefore, we developed the Test of Regulation in and Understanding of Social Situations in Teaching (TRUST), which is a theory-based situational judgment test measuring teachers’ knowledge about strategies for emotion regulation and relationship management in emotionally and socially challenging situations with students. Results from three studies (N = 166 in-service teachers, N = 73 in-service teachers, N = 107 pre-service teachers) showed satisfactory internal consistency for both the emotion regulation and relationship management subtests. Furthermore, confirmatory factor analyses supported the differentiation between the two facets of social-emotional competence. Regarding convergent validity, results from Study 3 revealed a positive association between the profession-specific TRUST and pre-service teachers’ general emotional intelligence. Furthermore, small to moderate correlations with the Big Five personality traits provided evidence for the discriminant validity of TRUST. In Studies 1 and 2, we found evidence for a correlation with external criteria, that is, teachers with higher test scores reported providing more emotional support for students and having better teacher-student relationships. For teachers’ occupational well-being, we found a link with symptoms of depersonalization and job satisfaction, but none for emotional exhaustion. We will discuss the use of TRUST in research, for the evaluation of interventions, in teacher education, and professional development and will illustrate ideas for enhancing the tool.

This article reports the results of several psychometric analyses that were conducted to provide evidence of construct validity for scores on a measure of psychological empowerment, the Psychological Empowerment Scale (PES), for parents of children with a disability. Confirmatory factor analyses were conducted to evaluate the internal structure of the PES and the reliability of its scores. The results of the confirmatory factor analyses provided evidence of convergent and discriminant validity for the scores from the four subscales underlying the PES: (a) attitudes of control and competence, (b) cognitive appraisals of critical skills and knowledge, (c) formal participation in organizations, and (d) informal participation in social systems and relationships. Reliability coefficients for the subscale scores and total scale score ranged from .90 to .97. In addition, the PES scores were correlated with other empowerment-related measures. The results of these correlational analyses and group discrimination analyses provided additional evidence of convergent and discriminant validity of the PES subscale and total scale scores.

Background: Despite the widespread application of the Chinese version of Symptom Check List 90 (SCL-90) amongst undergraduate students, researchers have not thoroughly evaluated its psychometric features with representative samples of Chinese undergraduate students.Aim: To evaluate its psychometric properties based on a large, representative sample of Chinese undergraduate students.Methods: A total of 4456 Chinese undergraduate students (age range = 17–25 years) completed the SCL-90. Correlation analysis, confirmatory factor analysis and structural equation modeling were used to evaluate its reliability and validity.Results: The evidence for item-total correlations, internal consistency reliability, intraclass correlation coefficients and composite reliability, and the hypothesized factor structure was satisfactory. The evidence for convergent validity was acceptable; however, the evidence for discriminant validity was not satisfactory.Conclusion: Based on a large, representative sample of Chinese undergraduate students, the Chinese version of the SCL-90 revealed both strengths and limitations, suggesting the need for further research.

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

BackgroundThis study aimed to validate a comprehensive and psychometrically soundinstrument—the Propensity to Cheat Scale (PCS)—designed tomeasure undergraduate students’ propensity toward academic dishonestyin Ethiopian universities. Based on Ajzen’s Theory of PlannedBehavior, the PCS was validated to assess students’ attitudes,subjective norms, and perceived behavioral control related to various formsof cheating, including cheating on tests and examinations, cheating onassignments, cheating on research work (plagiarism), and theft andmutilation of library materials.MethodsThe present study employed an explanatory research design using aquestionnaire based on the Propensity to Cheat Scale (PCS). Thequestionnaire was administered to 500 university students (male = 367[73.4%]; female = 133 [26.6%]) selected from three Ethiopian publicuniversities between November and January 2022. In order to measure theunderlying variables of propensity towards cheating, a factor model isdeveloped using exploratory factor analysis (EFA), and confirmatory factoranalysis was employed to validate the students’ perceived PTC. Theinternal consistency of the PTC scale was assessed using reliabilityanalysis, and validity evaluations were conducted to confirm thescale’s discriminant and convergent validity.ResultsConfirmatory factor analysis (CFA) results revealed a good fit to the data,and the internal consistency of the PCS was found to be strong, providing areliable measure of students’ propensity for cheating. Validityevaluations, including discriminant validity and convergent validity,confirmed the validity of the scale. The average variance extracted (AVE)and composite reliability values also supported the scale’sconvergent validity. The multidimensional concept of the PTC was supportedby a four-factor solution consisting of 26 reliable and valid items.ConclusionThe findings of the study demonstrate that the scale has also providedsufficient evidence of convergent and discriminant validity. By establishingdiscriminant and convergent validity, as well as reliability, throughdifferent validation procedures, the study has provided strong evidence forthe effectiveness of the PCS as an instrument for determining whetheruniversity students are likely to engage in cheating behavior.

BackgroundExisting measures of injection drug users' quality of life have focused primarily on health and health-related factors. Clearly, however, quality of life among injection drug users is impacted by a range of unique cultural, socioeconomic, medical, and geographic factors that must also be considered in any measure. The Injection Drug User Quality of Life (IDUQOL) scale was designed to capture the unique and individual circumstances that determine quality of life among injection drug users. The overall purpose of the present study was to examine the validity of inferences made from the IDUQOL by examining the (a) dimensionality, (b) reliability of scores, (c) criterion-related validity evidence, and (d) both convergent and discriminant validity evidence.MethodsAn exploratory factor analysis using principal axis factoring in SPSS 12.0 was conducted to determine whether the use of a total score on the IDUQOL was advisable. Reliability of scores from the IDUQOL was obtained using internal consistency and one-week test-retest reliability estimates. Criterion-related validity evidence was gathered using variables such as stability of housing, sex trade involvement, high-risk injection behaviours, involvement in treatment programs, emergency treatment or overdose over the previous six months, hospitalization and emergency treatment over the subsequent six month period post data collection. Convergent and discriminant validity evidence was gathered using measures of life satisfaction, self-esteem, and social desirability.ResultsThe sample consisted of 241 injection drug users ranging in age from 19 to 61 years. Factor analysis supports the use of a total score. Both internal consistency (alpha = .88) and one-week test-retest reliability (r = .78) for IDUQOL total scores were good. Criterion-related, convergent, and discriminant validity evidence supports the interpretation of IDUQOL total scores as measuring a construct consistent with quality of life.ConclusionThe findings from this study provide initial evidence to support the use of the IDUQOL total score. The results of the study also suggest the IDUQOL could be further strengthened with additional attention to how some IDUQOL domains are described and satisfaction is measured.