Numerical Differentiation Methods for Computing Error Covariance Matrices in Item Response Theory Modeling

This study investigates the impact of item parameter drift (IPD) on parameter and ability estimation when the underlying measurement model fits a mixture distribution, thereby violating the item invariance property of unidimensional item response theory (IRT) models. An empirical study was conducted to demonstrate the occurrence of both IPD and an underlying mixture distribution using real-world data. Twenty-one trended anchor items from the 1999, 2003, and 2007 administrations of Trends in International Mathematics and Science Study (TIMSS) were analyzed using unidimensional and mixture IRT models. TIMSS treats trended anchor items as invariant over testing administrations and uses pre-calibrated item parameters based on unidimensional IRT. However, empirical results showed evidence of two latent subgroups with IPD. Results also showed changes in the distribution of examinee ability between latent classes over the three administrations. A simulation study was conducted to examine the impact of IPD on the estimation of ability and item parameters, when data have underlying mixture distributions. Simulations used data generated from a mixture IRT model and estimated using unidimensional IRT. Results showed that data reflecting IPD using mixture IRT model led to IPD in the unidimensional IRT model. Changes in the distribution of examinee ability also affected item parameters. Moreover, drift with respect to item discrimination and distribution of examinee ability affected estimates of examinee ability. These findings demonstrate the need to caution and evaluate IPD using a mixture IRT framework to understand its effects on item parameters and examinee ability.

This article examines calibration designs, which maximize the determinant of Fisher’s information matrix on the item parameters (D-optimal), for sets of polytomously scored items. These items were analyzed using a number of item response theory (IRT) models, which are members of the “divide-by-total” family, including the nominal categories model, the rating scale model, the unidimensional polytomous Rasch model and the partial credit model. We extend the known results for dichotomous items, both singly and in tests to polytomous items. The structure of Fisher’s information matrix is examined in order to gain insights into the structure of D-optimal calibration designs for IRT models. A theorem giving an upper bound for the number of support points for such models is proved. A lower bound is also given. Finally, we examine a set of items, which have been analyzed using a number of different models. The locally D-optimal calibration design for each analysis is calculated using an exact numerical and a sequential procedure. The results are discussed both in general and in relation to each other.

Item response theory (IRT) is increasingly being applied to health-related quality of life instrument development and refinement. This article discusses results obtained using categorical confirmatory factor analysis (CCFA) to check IRT model assumptions and the application of IRT in item analysis and scale evaluation. To demonstrate the value of CCFA and IRT in examining a health-related quality of life measure in children and adolescents. This illustration uses data from 10,241 children and their parents on items from the 4 subscales of the PedsQL 4.0 Generic Core Scales. CCFA was applied to confirm domain dimensionality and identify possible locally dependent items. IRT was used to assess the strength of the relationship between the items and the constructs of interest and the information available across the latent construct. CCFA showed generally strong support for 1-factor models for each domain; however, several items exhibited evidence of local dependence. IRT revealed that the items generally exhibit favorable characteristics and are related to the same construct within a given domain. We discuss the lessons that can be learned by comparing alternate forms of the same scale, and we assess the potential impact of local dependence on the item parameter estimates. This article describes CCFA methods for checking IRT model assumptions and provides suggestions for using these methods in practice. It offers insight into ways information gained through IRT can be applied to evaluate items and aid in scale construction.

Patient navigation (PN) is a promising intervention to eliminate cancer health inequities. Patient navigators play a critical role in the navigation process. Patients' satisfaction with navigators is important in determining the effectiveness of PN programs. We applied item response theory (IRT) analysis to establish item-level psychometric properties for the Patient Satisfaction with Interpersonal Relationship with Navigators (PSN-I). We conducted a confirmatory factor analysis (CFA) to establish unidimensionality of the 9-item PSN-I in 751 cancer patients (68% female) between 18 and 86years old. We fitted unidimensional IRT models-unconstrained graded response model (GRM) and Rasch model-to PSN-I data, and compared model fit using likelihood ratio (LR) test and information criteria. We obtained item parameter estimates (IPEs), item category/operating characteristic curves, and item/test information curves for the better fitting model. CFA with diagonally weighted least squares confirmed that the one-factor model fit the data (RMSEA = 0.047, 95% CI = 0.033-0.060, and CFI ≈ 1). Responses to PSN-I items clustered into the 4th and 5th categories. We aggregated the first three response categories to provide stable parameter estimates for both IRT models. The GRM fit the data significantly better than the Rasch model (LR = 80.659, df = 8, p < 0.001). Akaike's information coefficient (6384.978 vs. 6320.319) and Bayesian information coefficient (6471.851 vs. 6443.771) were lower for the GRM. IPEs showed substantial variation in items' discriminating power (1.80-3.35) for GRM. This IRT analysis confirms the latent structure of the PSN-I and supports its use as a valid and reliable measure of latent satisfaction with PN.

In applications of item response theory (IRT) models, it is known that empirical violations of the local independence (LI) assumption can significantly bias parameter estimates. To address this issue, we propose a threshold-autoregressive item response theory (TAR-IRT) model that additionally accounts for order dependence among the item responses of each examinee. The TAR-IRT approach also defines a new family of IRT models for polytomous item responses under both unidimensional and multidimensional frameworks, with order-dependent effects between item responses and relevant dimensions. The feasibility of the proposed model was demonstrated by an empirical study using a polytomous response data. A simulation study for polytomous item responses with order effects of different magnitude in an education context shows that the TAR modeling framework could provide more accurate ability estimation than the partial credit model when order effect exists.

ABSTRACTMany standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet‐based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large‐scale English‐language test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.

A ranking pattern approach is proposed to build item response theory (IRT) models for forced-choice (FC) items. This new approach is an addition to the two existing approaches, sequential selection and Thurstone's law of pairwise comparison. A new dominance IRT model, the multidimensional generalized partial preference model (MGPPM), is proposed for FC items with any number (greater than 1) of statements. The maximum marginal likelihood estimation using an expectation-maximization algorithm (MML-EM) and Markov chain Monte Carlo (MCMC) estimation are developed. A simulation study is conducted to show satisfactory parameter recovery on triplet and tetrad data. The relationships between the newly proposed approach/model and the existing approaches/models are described, and the MGPPM, Thurstonian IRT (TIRT) model, and Triplet-2PLM are compared when applied to simulated and real triplet data. The new approach offers more flexible IRT modeling than the other two approaches under different assumptions, and the MGPPM is more statistically elegant than the TIRT and Triple-2PLM.

One theoretical feature that makes item response theory (IRT) models those of choice for many psychometric data analysts is parameter invariance, the equality of item and examinee parameters from different examinee populations or measurement conditions. In this article, using the well-known fact that item and examinee parameters are identical only up to a set of linear transformations specific to the functional form of a given IRT model, violations of these transformations for unidimensional IRT models are investigated using analytical, numerical, and visual tools. Because item parameter drift (IPD) constitutes a lack of invariance (LOI) at the individual item level or item set level, the magnitudes and effects of IPD on examinee response probabilities and true scores are algebraically derived and connected to empirical results from a recent simulation study. Thus, this article facilitates a deeper understanding of the exact statistical formulation of parameter invariance as a fundamental property of latent variable measurement and explicates some practical consequences of LOI for decision making.

The PlotIRT set of R functions facilitates plotting a variety of curves associated with item response theory (IRT). These functions were developed so that researchers using IRT could have a simple, yet flexible method for creating graphics associated with item parameters. These graphics can be customized in numerous ways to produce attractive figures that can be used in manuscripts or presentations. The R software environment was chosen so that users could produce IRT graphics via free software. The plots that are available in PlotIRT are item and test characteristic curves, item and test information curves, expected item score functions, and posterior density distributions. The R functions are applicable to the two-parameter logistic, three-parameter logistic, graded, nominal, and multiple-choice models. The functions work with both individual items or with groups of items, which do not need to be based on the same IRT model. If the user chooses, the graphics can be customized with a number of options. These options include varying the color of the lines by item, response category, or test; making the lines a solid, dashed, or other pattern; labeling the response category curves; specifying the number of items per plot and the number of plots per window; labeling the plot and axes; and adjusting the font and its size. The posterior distribution plot requires the user to specify a response pattern for the items, and it may be produced along with the item characteristic curves and the underlying y distribution. Basic R functions can be used to read in an assortment of item parameter files, or item parameters can be typed directly into R arrays. As the function is submitted, plots appear in windows within the R interface. An option is included in the function to save the plots as any file type supported by R, including postscript and portable document format.

The estimation of high-dimensional latent regression item response theory (IRT) models is difficult because of the need to approximate integrals in the likelihood function. Proposed solutions in the literature include using stochastic approximations, adaptive quadrature, and Laplace approximations. We propose using a second-order Laplace approximation of the likelihood to estimate IRT latent regression models with categorical observed variables and fixed covariates where all parameters are estimated simultaneously. The method applies when the IRT model has a simple structure, meaning that each observed variable loads on only one latent variable. Through simulations using a latent regression model with binary and ordinal observed variables, we show that the proposed method is a substantial improvement over the first-order Laplace approximation with respect to the bias. In addition, the approach is equally or more precise to alternative methods for estimation of multidimensional IRT models when the number of items per dimension is moderately high. Simultaneously, the method is highly computationally efficient in the high-dimensional settings investigated. The results imply that estimation of simple-structure IRT models with very high dimensions is feasible in practice and that the direct estimation of high-dimensional latent regression IRT models is tractable even with large sample sizes and large numbers of items.

Background: The Multiple Sclerosis Walking Scale-12 (MSWS-12) measures walking ability from the patients’ perspective. We examined the quality of the MSWS-12 using an item response theory model, the graded response model (GRM). Methods: A total of 625 unique Dutch multiple sclerosis (MS) patients were included. After testing for unidimensionality, monotonicity, and absence of local dependence, a GRM was fit and item characteristics were assessed. Differential item functioning (DIF) for the variables gender, age, duration of MS, type of MS and severity of MS, reliability, total test information, and standard error of the trait level (θ) were investigated. Results: Confirmatory factor analysis showed a unidimensional structure of the 12 items of the scale, explaining 88% of the variance. Item 2 did not fit into the GRM model. Reliability was 0.93. Items 8 and 9 (of the 11 and 12 item version respectively) showed DIF on the variable severity, based on the Expanded Disability Status Scale (EDSS). However, the EDSS is strongly related to the content of both items. Conclusion: Our results confirm the good quality of the MSWS-12. The trait level (θ) scores and item parameters of both the 12- and 11-item versions were highly comparable, although we do not suggest to change the content of the MSWS-12.

Questionnaires are widely used in clinical research and mental health studies, but their responses are usually more subjective than clinical biomarkers. Researchers may use latent variable models such as the item response theory (IRT) model to characterize individual ability, but response instability and small sample sizes can lead to increased uncertainty in estimation. Historical data from similar questionnaires may help reduce variation, but they must be incorporated adaptively. The challenge is further compounded by the need to ensure computational efficiency in IRT models with a large number of parameters. In this work, we develop a Bayesian framework for adaptive borrowing in IRT models based on an approximated normalized power prior (NPP) that treats the borrowing weight as a random parameter. The proposed method makes NPP feasible for general IRT models and is coupled with a full Bayesian data augmentation strategy that enables joint estimation of ability and item parameters through an efficient Gibbs sampler. In simulations, the approach adaptively increases borrowing when historical and current data are compatible and automatically downweights conflicting information. Relative to analyzes without borrowing, the method reduces variance and mean squared error while maintaining coverage across a range of test lengths, item discrimination profiles, and historical-current concordance scenarios. We illustrate the method by integrating historical mental health surveys into current assessments, yielding more precise ability estimates with preserved calibration. An efficient implementation is provided in our updated package NPP available on the Comprehensive R Archive Network.

The novelty of this study is to propose multidimensional item response theory (IRT) models for the investigation of heterogeneity in residents’ perceptions and attitudes toward the tourism industry. By following the social exchange theory, a sample of residents in the Romagna area (Italy) were invited to declare their agreement to some aspects related to benefits and costs of the tourism industry in 2010. The analysis suggests that the additive IRT model with two kind of latent variables (viz. the general latent trait measuring attitude toward tourism and two specific latent traits measuring perceptions on benefits and costs of tourism) fits the data adequately. Moreover, the estimated scores show that the tourism features are perceived differently by the residents, supporting the hypothesis that perceptions and attitudes change with regard to the individuals’ characteristics. Finally, the analysis proposes how the parameters of the IRT models can be interpreted in this new context.

Two common aspects of symptom measurement are 1) the occurrence or presence of symptoms, and 2) the intensity or severity of symptoms when they occur. We adopt a latent trait perspective based on item response theory (IRT), using both unidimensional and multidimensional IRT models. We demonstrate how to (a) prepare data for analysis, (b) specify, estimate, and compare models, (c) interpret model parameters, (d) compare scores from models, and (e) visualize analysis results. We develop the relevant sequential IRT model, noting its flexibility, congruence with the theorized data generating process for symptom measures, and its promise for facilitating additional research and practical applications. The sequential model is less frequently used than other IRT models for polytomous data such as the generalized partial credit or graded response models. However, estimation of the sequential model can be readily accomplished with standard latent variable modeling and IRT software for binary indicators that allows constraints on the discrimination parameters. We compare the sequential model to other modeling options. We provide discussion of future research directions.

Although item response theory (IRT) models such as the bifactor, two-tier, and between-item-dimensionality IRT models have been devised to confirm complex dimensional structures in educational and psychological data, they can be challenging to use in practice. The reason is that these models are multidimensional IRT (MIRT) models and thus are highly parameterized, making them only suitable for data provided by large samples. Unfortunately, many educational and psychological studies are conducted on a small scale, leaving the researchers without the necessary MIRT models to confirm the hypothesized structures in their data. To address the lack of modeling options for these researchers, we present a general Bayesian MIRT model based on adaptive informative priors. Simulations demonstrated that our MIRT model could be used to confirm a two-tier structure (with two general and six specific dimensions), a bifactor structure (with one general and six specific dimensions), and a between-item six-dimensional structure in rating scale data representing sample sizes as small as 100. Although our goal was to provide a general MIRT model suitable for smaller samples, the simulations further revealed that our model was applicable to larger samples. We also analyzed real data from 121 individuals to illustrate that the findings of our simulations are relevant to real situations.