R Packages for Item Response Theory Analysis: Descriptions and Features

An examination of the psychometric properties of the physical self-description questionnaire using a polytomous item response model

Psychometric properties of the Psy-flex scale: A validation study in a community sample in Korea

[Correction Notice: An Erratum for this article was reported in Vol 26(3) of Psychological Assessment (see record 2014-16017-001). The mean, standard deviation and alpha coefficient originally reported in Table 1 should be 74.317, 10.214 and .802, respectively. The validity coefficients in the last column of Table 4 are affected as well. Correcting this error did not change the substantive interpretations of the results, but did increase the mean, standard deviation, alpha coefficient, and validity coefficients reported for the Honesty subscale in the text and in Tables 1 and 4. The corrected versions of Tables 1 and Table 4 are shown in the erratum.] Item response theory (IRT) models were applied to dichotomous and polytomous scoring of the Self-Deceptive Enhancement and Impression Management subscales of the Balanced Inventory of Desirable Responding (Paulhus, 1991, 1999). Two dichotomous scoring methods reflecting exaggerated endorsement and exaggerated denial of socially desirable behaviors were examined. The 1- and 2-parameter logistic models (1PLM, 2PLM, respectively) were applied to dichotomous responses, and the partial credit model (PCM) and graded response model (GRM) were applied to polytomous responses. For both subscales, the 2PLM fit dichotomous responses better than did the 1PLM, and the GRM fit polytomous responses better than did the PCM. Polytomous GRM and raw scores for both subscales yielded higher test-retest and convergent validity coefficients than did PCM, 1PLM, 2PLM, and dichotomous raw scores. Information plots showed that the GRM provided consistently high measurement precision that was superior to that of all other IRT models over the full range of both construct continuums. Dichotomous scores reflecting exaggerated endorsement of socially desirable behaviors provided noticeably weak precision at low levels of the construct continuums, calling into question the use of such scores for detecting instances of "faking bad." Dichotomous models reflecting exaggerated denial of the same behaviors yielded much better precision at low levels of the constructs, but it was still less precision than that of the GRM. These results support polytomous over dichotomous scoring in general, alternative dichotomous scoring for detecting faking bad, and extension of GRM scoring to situations in which IRT offers additional practical advantages over classical test theory (adaptive testing, equating, linking, scaling, detecting differential item functioning, and so forth).

This article investigates how measurement models and statistical procedures can be applied to estimate the accuracy of proficiency classification in language testing. The paper starts with a concise introduction of four measurement models: the classical test theory (CTT) model, the dichotomous item response theory (IRT) model, the testlet response theory (TRT) model, and the polytomous item response theory (Poly-IRT) model. Following this, two classification procedures are presented: the Livingston and Lewis method for CTT and the Rudner method for the three IRT-based models. The utility of these models and procedures are then evaluated by examining the accuracy of classifying 5000 language test takers from a large-scale language certification examination into two proficiency categories. The most important finding is that the testlet format (multiple questions based on one prompt), which language tests usually rely on, has a great impact on the proficiency classification. All testlets in this study show a strong testlet effect. Hence, the TRT model is recommended for proficiency classification. Using the standard IRT model would inflate the classification accuracy due to the underestimated measurement error. Meanwhile, using the Poly-IRT model would give slightly less accurate classification results. Concerning the CTT model, while its classification accuracy is comparable to that of the TRT, there exists considerable inconsistency between their classification results.

ABSTRACTDetecting item fit for common dichotomous item response theory (IRT) models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. This paper employs the posterior predictive model checking method (Guttman, 1967; Rubin, 1981, 1984), a popular Bayesian model checking tool, to examine item fit for common dichotomous item response theory models. An item fit plot, comparing the observed and predicted proportion correct scores of examinee groups (with groups based on raw scores), promises to be useful in real applications. This paper also suggests how to obtain posterior predictive p‐values for the χ2‐type test statistics of Orlando and Thissen (2000) comparing the observed and predicted proportion correct scores for different raw score groups. A number of simulation studies and real data applications examine the effectiveness of the suggested item fit diagnostics. The suggested p‐values seem to have low Type I error rate, low false alarm rate, and adequate power, indicating that researchers might find the methods more acceptable than the existing item fit measures.

The purpose of this study is to carry out differential item functioning (DIF) analysis for content areas of a reading comprehension subtest using four area indices within Item Response Theory (IRT) framework. The differences in the magnitudes of the area indices were compared based on the subject areas. The DIF analysis was carried out across gender groups only. The item level data of the English reading comprehension subtest were gathered from English Language Achievement exam done in School of Foreign Languages, Ege University, Turkey, in 2013. A sample of 2,117 examinees (1,011 males and 1,116 females) was randomly selected. For the DIF analysis, (a) an IRT model for the item characteristic curves was specified; (b) model-data-fit was investigated for the selected IRT model;(c) Item characteristic curves were separately computed for each group on a common scale; finally, (d) indices indicating the degree of DIF on each item were computed. The results of the study indicated that both un-weighted and weighted area indices showed non-uniformity in DIF in the item characteristic curves in reading comprehension subtest in most cases. A significant correlation was observed between un-weighted and weighted area indices. Key words: Item characteristic curve, item bias, differential item functioning (DIF).

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord's Wald test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

In contrast to dichotomous item response theory (IRT) models, most well-known polytomous IRT models do not imply stochastic ordering of the latent trait by the total test score (SOL). This has been thought to make the ordering of respondents on the latent trait using the total test score questionable and throws doubt on the justifiability of using nonparametric polytomous IRT models for ordinal measurement. We show that a broad class of polytomous IRT models has a weaker form of SOL, denoted weak SOL, and argue that weak SOL justifies ordering respondents on the latent trait using the total test score and, therefore, the use of nonparametric polytomous IRT models for ordinal measurement.

Item response theory (IRT) models provide an appropriate alternative to the classical ordinal confirmatory factor analysis (CFA) during the development of patient-reported outcome measures (PROMs). Current literature has identified the assessment of IRT model fit as both challenging and underdeveloped (Sinharay & Johnson, 2003; Sinharay, Johnson, & Stern, 2006). This study evaluates the performance of Ordinal Bayesian Instrument Development (OBID), a Bayesian IRT model with a probit link function approach, through applications in two breast cancer-related instrument development studies. The primary focus is to investigate an appropriate method for comparing Bayesian IRT models in PROMs development. An exact Bayesian leave-one-out cross-validation (LOO-CV) approach (Vehtari & Lampinen, 2002) is implemented to assess prior selection for the item discrimination parameter in the IRT model and subject content experts’ bias (in a statistical sense and not to be confused with psychometric bias as in differential item functioning) toward the estimation of item-to-domain correlations. Results support the utilization of content subject experts’ information in establishing evidence for construct validity when sample size is small. However, the incorporation of subject experts’ content information in the OBID approach can be sensitive to the level of expertise of the recruited experts. More stringent efforts need to be invested in the appropriate selection of subject experts to efficiently use the OBID approach and reduce potential bias during PROMs development.

Many causes of abortion in livestock are due to zoonotic pathogens that pose serious infection risks for humans. Carefully designed, empirical One Health research allows to untangle the complexity around these risks and guides the development of practical health education guidelines and best prevention practices for veterinary public health interventions. To support this, the study presented here aimed at understanding knowledge, attitudes, and practices (KAP) on zoonotic risks from livestock birth products among rural communities in Ethiopia. From July 2018 to February 2019, a cross-sectional study design was conducted with 327 randomly selected farmers and pastoralists in five districts in three regions in Ethiopia. The structured questionnaire consisted of 48 items to evaluate knowledge (24), attitude (9), and prevention practices (15) related to zoonotic diseases risks from livestock birth products. A unidimensional two-parameter logistic (2-PL) Item Response Theory (IRT) model was used for zoonotic disease risk KAP scale construction and evaluation. The 2-PL IRT model was fitted to determine the probability of a person to appropriately respond to an item with a provided zoonotic disease KAP level. We then examined differential item functioning (DIF) concerning to five important covariates. The attitude subscale had the highest total mean score (37.3, ± 28.92%) and the knowledge subscale had the lowest mean score (22.4, ± 33.6%) among the three subscales. The mixed model regression analysis indicated that region was the only apparent factor explaining differences in zoonotic diseases knowledge, attitude, and practice total mean scores. The knowledge and attitude subscales had good internal consistency with a Cronbach's α at 0.83 and 0.81, respectively, whereas the practice subscale had lower internal consistency with 0.51. There was a positive association between responding to knowledge questions correctly and a positive attitude (r2 = 0.44, p < 0.0001) and self-reported good practice (r2 = 0.307, p < 0.0001). The differential item functioning test showed that 19 of 37 (51.35%) and 12 of 37 (32.43%) items of the retained KAPs survey items had non-uniform and uniform DIF linked to at least one covariate respectively and all the covariates were related with DIF in at least one item. This study found substantial knowledge gaps, a low level of the desired attitude, and high-risk behavioural practices regarding zoonotic disease from livestock birth products. Consequently, livestock keepers are likely exposed to pathogens and thus these practices are an important contributing factor for zoonotic disease infection in people.

The validity of inferences from patient-reported outcome measure (PROM) scores can be confounded by differential item functioning (DIF). DIF occurs when there is heterogeneity in how patients respond to and interpret questions about their health, despite having the same underlying health status. Ignoring the effects of DIF could lead to inaccurate interpretations and misinformed clinical decisions resulting in compromised healthcare delivery. Tree-based item response theory (IRT) models are recommended as an alternative class of methods for analyzing PROMs because they offer a robust approach for identifying DIF when covariates associated with DIF are unknown a priori. This paper introduces a web application developed using R Shiny, which enables users to implement tree-based IRT models for DIF assessment in potentially heterogeneous populations. The app provides flexible model specifications, visualization tools, and customizable settings to accommodate various data types and research needs. A practical tutorial is included, guiding users through the application interface, data preparation, model selection, and interpretation of results. The web application (https://ucalgary-pcma-lab.shinyapps.io/tree_based_dif_analysis/) offers interactive data upload in .CSV and .XLSX data formats. Recommendations are provided for selecting model parameters within the app based on the results of previous simulation studies. The web app tests for DIF on dichotomous- and polytomous-scored items. The coefficients, item parameters, and plots provide insights into potential sources of DIF. This web application provides a user-friendly, interactive, innovative, easily accessible, and valuable tool for clinicians, applied health researchers, and analysts seeking to understand sample heterogeneity due to DIF in PROM data.

Applying single-level statistical models to multilevel data typically produces underestimated standard errors, which may result in misleading conclusions. This study examined the impact of ignoring multilevel data structure on the estimation of item parameters and their standard errors of the Rasch, two-, and three- parameter logistic models in item response theory (IRT) to demonstrate the degree of such underestimation in IRT. Also, the Lord’s chi-square test using the underestimated standard errors was used to test differential item functioning (DIF) to show the impact of such underestimation on the practical applications of IRT. The results of simulation studies showed that, in the most severe case of multilevel data, the standard error estimate from the standard single-level IRT models was about half of the minimal asymptotic standard error, and the type I error rate of the Lord’s chi-square test was inflated up to .35. The results of this study suggest that standard single-level IRT models may seriously mislead our conclusions in the presence of multilevel data, and therefore multilevel IRT models need to be considered as alternatives.

We derive formulas for the differential item functioning (DIF) measures that two routinely used DIF statistics are designed to estimate. The DIF measures that match on observed scores are compared to DIF measures based on an unobserved ability (theta or true score) for items that are described by either the one‐parameter logistic (1PL) or two‐parameter logistic (2PL) item response theory (IRT) model. We use two different weighting schemes (uniform weights and item discrimination weights) to construct the observed score matching variable. Our results show that (a) under the 1PL item response model, the observed score‐based DIF measures always approximate the true score‐based DIF measures very closely; (b) under the 2PL model, when the observed score is the simple sum score, the observed score‐based DIF measures underestimate or overestimate the true score‐based DIF measures under the null hypothesis of no DIF when the groups are different in ability, and this bias is related to the degree to which the average discrimination parameter underestimates or overestimates the studied discrimination parameter; and (c) under the 2PL model, when the item discrimination weights are used to define the observed score, the observed score‐based DIF measures always approximate the true score‐based DIF measures very closely. These results will hold for any sets of item responses that are described by either the 1PL or 2PL IRT model.

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

ABSTRACTThis paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.