Consequences of Ignoring Guessing Effects on Measurement Invariance Analysis.

Women with multi-gender attractions (i.e. bisexual+ women) experience alarmingly high rates of sexual violence (SV), which puts women at risk for posttraumatic stress disorder (PTSD). Bisexual+ women experience higher rates of PTSD compared to heterosexual and lesbian/gay people. The PTSD Checklist for DSM-5 (PCL-5) is a widely used self-report measure of PTSD symptoms. We conducted an item response theory (IRT) analysis to evaluate the item difficulty and discrimination of PCL-5 items in assessing PTSD symptoms related to adult SV among young bisexual+ women. We combined two samples of women: sample 1 participants were recruited via MTurk (N = 267; Mage = 25.9, range 18–35; 82.4% White) and sample 2 participants recruited via online ads (N = 133; Mage = 22.0, range 18–25, 85.0% White). Item difficulty and discrimination was evaluated using IRT with a graded response model. At clinically significant levels of PTSD, the most discriminating items were: flashbacks, nightmares, reckless behavior, inability to experience positive emotions, and intrusive memories, whereas the least discriminating items were: internal avoidance, hypervigilance, difficulty with sleep, external avoidance, and exaggerated startle response. The most difficult items were: reckless behavior, flashbacks, traumatic amnesia, nightmares, and irritability, whereas the least difficult items were: internal avoidance, cued emotional distress, external avoidance, hypervigilance, and strong negative emotions. Results suggest that symptoms related to flashbacks, nightmares, and reckless behavior are important for assessing clinically significant SV-related PTSD symptoms among young bisexual+ women, whereas symptoms related to internal and external avoidance and hypervigilance may be relatively less informative.

Background: Rasch analysis has the advantage of placing both the items and the person along a single ratio scale and calibrates person ability and item difficulty onto an interval scale by logits. Therefore, Rasch analysis has been recommended as a better method for evaluating functional outcome questionnaires than traditional analyses. Objects: The aim of current study was to investigate item fit, item difficulty, rating scale, and separation index of the Korean version of the Fullerton Advanced Balance (KFAB) scale using Rasch analysis. Methods: In total, 93 patients with stroke (male=58, female=35) participated in this study. To investigate the item fit, difficulty, rating scale, and separation index of the KFAB scale, Rasch analysis was completed by the Winsteps software program. Results: In this study, all items of the KFAB scale were included in the Rasch model. The most difficult item was ‘standing with feet together and eyes closed’, and the easiest item was ‘two-footed jump’. The rating scale was a 4-point scale instead of the original 5-point scale. Person and item separation indices showed high values that can identify a person with a wide range of balance ability. Conclusion: The KFAB scale appears to be a reliable and valid tool to assess balance function in patients with stroke. Furthermore, the scale was found to discriminate among stroke patients of varying balance abilities.

Development and Psychometric Evaluation of an Evidence-Based Practice Questionnaire for a Chiropractic Curriculum

The research on using "none-of-the-above" (N-O-T-A) as a test item option continues to find contradictory results on item difficulty and item discrimination. The argument for not using N-O-T-A is that this option is considered more difficult and less discriminating than items with one correct format (regular).The purpose of this study was to conduct a meta-analysis on the difficulty and discrimination of the N-O-T-A test option. A total of twelve articles yielding 20 effect sizes was examined for difficulty, and a total of seven studies yielding eleven effect sizes was examined for discrimination. The meta-analysis indicated nonsignificant effect sizes of .01 for discrimination and -.17 for item difficulty. These findings indicate that using "none-of-the-above"as a test item option does not result in items of lesser quality than in items not using this option.

ABSTRACTThis study examined the relationship of differential item functioning (DIF) to item difficulty on the Scholastic Aptitude Test (SAT). The data comprise verbal and mathematical item statistics from nine recent administrations of the SAT. In general, item difficulty is related to DIF. The nature of that relationship appears to be independent of the choice of DIF index (either the Mantel‐Haenszel or the standardization approach) as well as of test form. However, the relationship was dependent on the particular group comparison and on both the test sections and the item type being analyzed.The relationship was strong for each of the racial and ethnic group contrasts—in which black, Hispanic, and Asian American examinees were compared in turn with white examinees—but was weak for the female and male examinee contrast. The relationship also appeared stronger on the verbal sections than on the mathematical sections. The relationship is such that more difficult items tended to exhibit positive DIF (DIF favored the focal group over the white reference group). On the verbal sections, only the reading comprehension item type (with the smallest observed range in item difficulty) failed to exhibit a strong relationship.Another index, the standardized difference in percentage omit (DIFPOM), correlated very highly (negatively) with DIF. Differential omission refers to a relative difference in omit rates between groups matched in ability. In fact, DIFPOM was consistently a better predictor of DIF in most models than was item difficulty. The relationship between DIF and DIFPOM held up across all four comparisons, including gender. It was also present in the mathematical sections with nearly the same magnitude exhibited in the verbal sections.Although DIF and DIFPOM are mathematically dependent measures, it was proposed that DIFPOM may be partly responsible for the relationship between DIF and item difficulty. To what extent DIF is a consequence of differential omission and to what extent differential omission is a manifestation of DIF is problematic. Nonetheless, the presence of differential omission on a test has the potential to influence DIF indices and therefore should be an important concern especially for formula‐scored tests, where omission occurs often on difficult items.Among other findings is that Hispanic and black focal groups tended to omit differentially less than did the white reference groups. For Asian American examinees, the reverse holds. For females and males, the direction depends on the test sections. In general, groups that omitted differentially less experienced a relative advantage (high‐positive DIF values) on the more difficult items, as measured by the DIF indices studied here (which treat omits as wrong in their calculation).

Background: Item analysis ensures valid and reliable test construction in educational measurement. In postgraduate physical education particularly in sports biomechanics it is important to maintain the right balance between how challenging an assessment is and how well it distinguishes between different levels of student understanding. Objective: This study assessed item difficulty and discrimination indices of objective test items in sports biomechanics for M.P.Ed. students, validating their scientific soundness and educational relevance. Methods: Thirty-three (N=33) M.P.Ed. students (Batch 2024–26) enrolled in the “Fundamentals of Sports Biomechanics” course at IGIPESS, University of Delhi, participated in the study. A 17‑item knowledge test, comprising multiple choice, fill in the blank, and matching type questions, was administered under standardized conditions. Item difficulty (p‑value) and discrimination index (D) were calculated, and reliability was assessed using the odd–even method with Pearson correlation. Results: Analysis revealed that 47% of items were easy (p ≥ 0.70), 41% moderate (0.30 ≤ p < 0.70), and 12% difficult (p < 0.30). Discrimination indices showed that 29% of items were poor (<0.15), 29% acceptable (0.15–0.24), and 41% excellent (>0.34). Reliability testing demonstrated significant internal consistency (r = 0.599, p < 0.01), confirming the stability of the instrument. Conclusion: The assessment demonstrated balanced item difficulty, strong discriminatory power, and acceptable reliability, establishing its validity as a tool for evaluating biomechanics knowledge at the postgraduate level. Systematic item analysis strengthens test construction and ensures alignment with curricular objectives, supporting evidence‑based teaching in physical education. Keywords: Item Difficulty, Discrimination Index, Reliability, Sports Biomechanics, Knowledge Evaluation, Test Construction.

Artificial intelligence, specifically large language models such as ChatGPT, offers valuable potential benefits in question (item) writing. This study aimed to determine the feasibility of generating case-based multiple-choice questions using ChatGPT in terms of item difficulty and discrimination levels. This study involved 99 fourth-year medical students who participated in a rational pharmacotherapy clerkship carried out based-on the WHO 6-Step Model. In response to a prompt that we provided, ChatGPT generated ten case-based multiple-choice questions on hypertension. Following an expert panel, two of these multiple-choice questions were incorporated into a medical school exam without making any changes in the questions. Based on the administration of the test, we evaluated their psychometric properties, including item difficulty, item discrimination (point-biserial correlation), and functionality of the options. Both questions exhibited acceptable levels of point-biserial correlation, which is higher than the threshold of 0.30 (0.41 and 0.39). However, one question had three non-functional options (options chosen by fewer than 5% of the exam participants) while the other question had none. The findings showed that the questions can effectively differentiate between students who perform at high and low levels, which also point out the potential of ChatGPT as an artificial intelligence tool in test development. Future studies may use the prompt to generate items in order for enhancing the external validity of the results by gathering data from diverse institutions and settings.

Item analysis is an important procedure to determine the quality of the items. The purpose of this study is to assess two important indices in item analysis procedure, namely (1) item difficulty (p) and (2) item discrimination (D) as well as a correlation between them. The study involves ten 40-item multiple-choice mathematics tests. A total of 1243 Form 2 students from public schools in Penang, Kedah, Perak, and Pahang are employed as sample for this study. Both indices are calculated based on classical test theory framework because of the advantage provided for teachers. On average, only 67 % (min = 50 %, max = 87.5 %) of the items are considered of good quality that can be kept for future testing. There is incomprehensible result regarding the correlation between the two indices. Implication in terms of teachers’ competency in test development is also discussed.

BackgroundFailure to adhere to standard item-writing guidelines may render examination questions easier or more difficult than intended. Item complexity describes the cognitive skill level required to obtain a correct answer. Higher cognitive examination items promote critical thinking and are recommended to prepare students for clinical training. This study evaluated faculty-authored examinations to determine the impact of item-writing flaws and item complexity on the difficulty and discrimination value of examination items used to assess third year veterinary students.MethodsThe impact of item-writing flaws and item complexity (cognitive level I-V) on examination item difficulty and discrimination value was evaluated on 1925 examination items prepared by clinical faculty for third year veterinary students.ResultsThe mean (± SE) percent correct (83.3 % ± 17.5) was consistent with target values in professional education, and the mean discrimination index (0.18 ± 0.17) was slightly lower than recommended (0.20). More than one item-writing flaw was identified in 37.3 % of questions. The most common item-writing flaws were awkward stem structure, implausible distractors, longest response is correct, and responses are series of true-false statements. Higher cognitive skills (complexity level III-IV) were required to correctly answer 38.4 % of examination items. As item complexity increased, item difficulty and discrimination values increased. The probability of writing discriminating, difficult examination items decreased when implausible distractors and all of the above were used, and increased if the distractors were comprised of a series of true/false statements. Items with four distractors were not more difficult or discriminating than items with three distractors.ConclusionPreparation of examination questions targeting higher cognitive levels will increase the likelihood of constructing discriminating items. Use of implausible distractors to complete a five-option multiple choice question does not strengthen the discrimination value.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-016-0773-3) contains supplementary material, which is available to authorized users.

Large language models (LLMs) have the potential to scale assessment in medical education, but their psychometric equivalence to expert-written items and the detectability of their origin remain uncertain. In a preregistered, single-center, blinded observational, within-subject comparison, we evaluated 24 GPT-4o-generated versus 24 human-authored topic-matched multiple-choice questions (MCQs) across radiation oncology, radiology, and nuclear medicine. Medical students (n = 82) and physicians (n = 46) completed an identical 48-item formative mock examination, with item origin masked. Item difficulty (human: mean 0.65 [SD 0.22] vs LLM: 0.67 [0.20]) and discrimination (0.27 [0.12] vs 0.29 [0.12]) did not differ significantly; participants did not identify item origin above chance (0.50). Expert ratings of appropriateness and didactic quality showed low interrater agreement (ICC = 0.07-0.18). In this expert-reviewed, human-in-the-loop workflow, the item difficulty and discriminatory power of MCQs generated with GPT-4o did not differ significantly from those of expert-authored items, and were not reliably recognized as AI-generated by examinees. These findings delineate a feasible pathway for responsibly scaling formative assessment content in imaging-focused medical education, while underscoring the need for explicit educational policies regarding oversight, transparency, and fairness.

The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with the descriptors "easy," "moderate," or "hard" and classified them according to whether they tested knowledge, comprehension, or application. Overall analysis showed a statistically significant, but relatively low, correlation between the intended item difficulty and actual student scores (ρ = -0.19, P < 0.01), indicating that, as intended item difficulty increased, the resulting student scores on items tended to decrease. Although this expected inverse relationship was detected, faculty members were correct only 48% of the time when estimating difficulty. There was also significant individual variation among faculty members in the ability to predict item difficulty (χ(2) = 16.84, P = 0.02). With regard to the cognitive level of items, no significant correlation was found between the item cognitive level and either actual student scores (ρ = -0.09, P = 0.14) or item discrimination (ρ = 0.05, P = 0.42). Despite the inability of faculty members to accurately predict item difficulty, the examinations were of high quality, as evidenced by reliability coefficients (Cronbach's α) of 0.70-0.92, the rejection of only 4 of 300 items in the postexamination review, and a mean item discrimination (point biserial) of 0.37. In conclusion, the effort of assigning annotations describing intended difficulty and cognitive levels to multiple-choice items is of doubtful value in terms of controlling examination difficulty. However, we also report that the process of annotating questions may enhance examination validity and can reveal aspects of the hidden curriculum.

High-quality multiple-choice (MC) items are essential for creating efficient, valid assessments. This research aims to test the impact of these two guidelines on item difficulty (p), item discrimination (r), and test reliability for mathematics items empirically. The research findings have revealed that item discrimination and test reliability were not statistically different across MC items with four options, three options, and NOTA options while the means of item difficulty of four-option MC items was not statistically different from those of three-option and NOTA-option MC items. However, the mean of item difficulty of NOTA-option MC items was statistically lower than those of three-option.

Introduction: Simulation-based training (SBT) is a key method for teaching bronchoscopy skills to pulmonology residents. A theoretical foundation can enhance SBT efficiency. This study developed and evaluated the validity of an anatomy and theoretical bronchoscopy exam using Kane’s validity framework. Methods: Nineteen anatomy and 58 theoretical exam questions, developed by pulmonology experts, were assessed through two Delphi rounds. Both exams were then taken by 53 prepared pulmonology residents. The theoretical exam was also taken by three unprepared groups: novices, intermediates, and experts. Using the residents’ data, scoring evidence for the theoretical exam was evaluated using item difficulty and item discrimination indices, and generalization evidence was assessed using Cronbach’s alpha. Extrapolation evidence was obtained by comparing theoretical exam scores across the different groups. Implications evidence for both exams was gathered by evaluating residents’ preparedness, based on exam performance and instructor feedback. Results: The Delphi procedure resulted in 19 anatomy and 31 theoretical questions. Item difficulty values predominantly ranged from 0.85 to 1.0; item discrimination indices mostly ranged from 0.0 to 0.25. Cronbach’s alpha was 0.55. While scores appeared to correlate with experience, no significant differences were observed between the four groups. Most residents passed both exams on their first attempt, and instructors rated their anatomical knowledge as good. Conclusion: Expert involvement and acceptable item difficulty, item discrimination, and internal consistency supported the exams’ validity. The exams also effectively motivated residents to prepare for SBT. These findings highlight the value of pre-SBT exams in enhancing residents’ preparation, allowing more time to focus on mastering procedural skills.

The research aimed to validate the Raven Intelligence Test (RIT) in the Jordanian context using a sample of 414 students selected from secondary schools in Amman, Jordan aged between 15 and 17 years. The study yielded several findings. Notably, in terms of validity, the study assessed predictive validity through school achievement, concurrent validity using the School and College Ability Test (SCAT), construct validity by comparing scores of normal and gifted students, discrimination power through extreme score differences, and factorial validity via confirmatory factor analysis (CFA). The reliability of the test was evaluated using Cronbach’s (α), split-halves, test-retest method, and the correlation between each item and the total score. We also calculated item difficulty and item discrimination in order to evaluate the quality of the test as well as its ability to discriminate among the intelligence levels. These favorable outcomes provide additional support for the reliability and validity of the test in measuring cognitive abilities which may further prove useful in contexts such as talent identification and recruitment.

This study aimed to refine a dimensional scale for measuring psychosocial adjustment in African youth using item response theory (IRT). A 60-item scale derived from qualitative data was administered to 667 war-affected adolescents (55% female). Exploratory factor analysis (EFA) determined the dimensionality of items based on goodness-of-fit indices. Items with loadings less than 0.4 were dropped. Confirmatory factor analysis (CFA) was used to confirm the scale's dimensionality found under the EFA. Item discrimination and difficulty were estimated using a graded response model for each subscale using weighted least squares means and variances. Predictive validity was examined through correlations between IRT scores (θ) for each subscale and ratings of functional impairment. All models were assessed using goodness-of-fit and comparative fit indices. Fisher's Information curves examined item precision at different underlying ranges of each trait. Original scale items were optimized and reconfigured into an empirically-robust 41-item scale, the African Youth Psychosocial Assessment (AYPA). Refined subscales assess internalizing and externalizing problems, prosocial attitudes/behaviors and somatic complaints without medical cause. The AYPA is a refined dimensional assessment of emotional and behavioral problems in African youth with good psychometric properties. Validation studies in other cultures are recommended.