A State Space Approach to Modeling IRT and Population Parameters from a Long Series of Test Administrations

Tests used for such purposes as determining educational quality, defining educational needs, hiring an employee, student selection and placement and performing guidance and clinic services have an important place in education and psychology. Of course, they should have certain psychometric features related to test scores' validity and reliability. Various test theories have helped to create more valid and reliable measurements and, as a result, to make better decisions regarding individuals. In education and psychology, Classical Test Theory (CTT) and Item Response Theory (IRT) are both widely used. CTT assumes that an individual's observed score is the total of the true score and the error score, while IRT estimates an individual's ability or latent trait from responses to test items (Embretson & Reise, 2000).When IRT assumptions and model-data fit are ensured, item and ability parameters' invariance occurs; this is known as the most important advantage IRT has over CTT. Item and ability parameters' invariance means estimating ability parameters independently of item sample and estimating item parameters independently of ability sample. IRT's invariance feature makes it very practicable in many applications, for instance, test development, computerized adaptive testing, bias studies, test equating and item mapping (Hambleton & Swaminathan, 1985). IRT is classified under two main categories as parametric IRT (PIRT) and nonparametric IRT (NIRT) (Olivares, 2005; Sijtsma & Molenaar, 2002).To analyse ordered items, such as Likert-type attitude items, partial credit cognitive items or not ordered graded items such as multiple-choice test items, item response models are developed towards polytomous items in IRT (Ostini & Nering, 2006). In these models developed for polytomous items, a non-linear relationship between an individual's latent trait and the possibility of choosing a certain category of item answer is explained (Embretson & Reise, 2000). Graded Response Model (GRM), part of IRT models developed for polytomous items, is often preferred by researchers for applications since it is more useful in presentations, portfolios, essays and Likert-type items with ordered item categories (DeMars, 2010; Ostini & Nering, 2006). To scale tests that consist of polytomous items by making true estimates according to GRM, evaluating PIRT's assumptions and model-data fit is necessary. And to provide these assumptions and model-data fit, large samples are needed. At this point, NIRT models draw attention because they provide a practical advantage in determining psychometric properties of tests with fewer items and respondents (Stout, 2001).NIRT models are defined as statistical scaling methods that require fewer assumptions than PIRT models for measuring persons and items (Stochl, 2007). With their wide application area, NIRT models are used in ordinal scales, applied research areas, sociology, marketing research and health research on quality of life (Sijtsma, 2005). The literature reveals that two models, namely, the Mokken model and nonparametric regression estimation models, are employed. These two models are themselves divided into sub-models. The Mokken model consists of the sub-models Monotone Homogeneity Model (MHM) and the Double Monotonicity Model (DMM). Nonparametric regression estimation models consist of such sub-models as the Kernel Smoothing Approach Model (KSAM), the Isotonic Regression Estimation and the Smoothed Isotonic Regression Estimation models (Lee, 2007; Sijtsma & Molenaar, 2002). Along with theoretical studies being conducted, new sub-models are being added to nonparametric regression estimation models.As a NIRT model, MHM requires unidimensionality, local independence and monotonicity assumptions, and it defines the relationship that latent variables and items with homogeneous (unidimensional) and monotone item characteristic curve (ICC) have (Meijer & Baneke, 2004; Sijtsma & Molenaar, 2002). …

This study investigates the impact of item parameter drift (IPD) on parameter and ability estimation when the underlying measurement model fits a mixture distribution, thereby violating the item invariance property of unidimensional item response theory (IRT) models. An empirical study was conducted to demonstrate the occurrence of both IPD and an underlying mixture distribution using real-world data. Twenty-one trended anchor items from the 1999, 2003, and 2007 administrations of Trends in International Mathematics and Science Study (TIMSS) were analyzed using unidimensional and mixture IRT models. TIMSS treats trended anchor items as invariant over testing administrations and uses pre-calibrated item parameters based on unidimensional IRT. However, empirical results showed evidence of two latent subgroups with IPD. Results also showed changes in the distribution of examinee ability between latent classes over the three administrations. A simulation study was conducted to examine the impact of IPD on the estimation of ability and item parameters, when data have underlying mixture distributions. Simulations used data generated from a mixture IRT model and estimated using unidimensional IRT. Results showed that data reflecting IPD using mixture IRT model led to IPD in the unidimensional IRT model. Changes in the distribution of examinee ability also affected item parameters. Moreover, drift with respect to item discrimination and distribution of examinee ability affected estimates of examinee ability. These findings demonstrate the need to caution and evaluate IPD using a mixture IRT framework to understand its effects on item parameters and examinee ability.

In educational measurement, Item Response Theory (IRT) provides a means of quantifying student knowledge. Specifically, IRT models the probability of a student answering a particular item correctly as a function of the student’s continuous-valued latent abilities Θ (e.g. add, subtract, multiply, divide) and parameters associated with the item (e.g. difficulty). Given a group of students’ binary responses (correct/incorrect) to an assessment, parameter estimation techniques are used to infer the students’ abilities Θ to better evaluate student performance. But as the number of students, items, and dimension of Θ increases, traditional parameter estimation methods which rely on numerical integration or MCMC techniques become infeasible. In this thesis, a novel modification to a variational autoencoder (VAE), an unsupervised learning method, is presented which incorporates multiple pieces of domain knowledge into the VAE architecture. Specifically, an expert-annotated Q-matrix detailing the association between items and abilities is used to constrain the trainable weights in the VAE decoder. This directly links the generative posterior distribution of a VAE to IRT models, and allows interpretation of trainable weights as item parameters and a hidden neural layer as student ability estimates. The use of a neural network in this application allows for efficient estimation of parameters in high-dimensional datasets. An additional alteration to the VAE encoder allows for modeling of correlated, rather than independent, latent abilities. This architecture is also of interest to areas outside of education where the probability distribution of the VAE latent code is known. The proposed method, titled ML2P-VAE, achieves accuracy similar to traditional parameter estimation methods on smaller datasets and provides quality estimates on educational assessments with a large number of students, items, and latent abilities, where traditional methods struggle. Modifications used in ML2P-VAE are extended to integrate IRT into deep knowledge tracing models, which use time-dependent neural networks such as long short-term memory networks and Transformers. In an online environment where many items are available, the task of knowledge tracing is to track a student’s learning dynamically as they progress through an assessment. This task has received more attention recently due to the emergence of online learning and AI tutoring systems, where following student’s knowledge acquisition can help facilitate automated curriculum in real time. The inclusion of IRT in the knowledge tracing framework presents a trade-off between explainability and accuracy, while remaining competitive with state-of-the-art methods.

This article examines calibration designs, which maximize the determinant of Fisher’s information matrix on the item parameters (D-optimal), for sets of polytomously scored items. These items were analyzed using a number of item response theory (IRT) models, which are members of the “divide-by-total” family, including the nominal categories model, the rating scale model, the unidimensional polytomous Rasch model and the partial credit model. We extend the known results for dichotomous items, both singly and in tests to polytomous items. The structure of Fisher’s information matrix is examined in order to gain insights into the structure of D-optimal calibration designs for IRT models. A theorem giving an upper bound for the number of support points for such models is proved. A lower bound is also given. Finally, we examine a set of items, which have been analyzed using a number of different models. The locally D-optimal calibration design for each analysis is calculated using an exact numerical and a sequential procedure. The results are discussed both in general and in relation to each other.

The PlotIRT set of R functions facilitates plotting a variety of curves associated with item response theory (IRT). These functions were developed so that researchers using IRT could have a simple, yet flexible method for creating graphics associated with item parameters. These graphics can be customized in numerous ways to produce attractive figures that can be used in manuscripts or presentations. The R software environment was chosen so that users could produce IRT graphics via free software. The plots that are available in PlotIRT are item and test characteristic curves, item and test information curves, expected item score functions, and posterior density distributions. The R functions are applicable to the two-parameter logistic, three-parameter logistic, graded, nominal, and multiple-choice models. The functions work with both individual items or with groups of items, which do not need to be based on the same IRT model. If the user chooses, the graphics can be customized with a number of options. These options include varying the color of the lines by item, response category, or test; making the lines a solid, dashed, or other pattern; labeling the response category curves; specifying the number of items per plot and the number of plots per window; labeling the plot and axes; and adjusting the font and its size. The posterior distribution plot requires the user to specify a response pattern for the items, and it may be produced along with the item characteristic curves and the underlying y distribution. Basic R functions can be used to read in an assortment of item parameter files, or item parameters can be typed directly into R arrays. As the function is submitted, plots appear in windows within the R interface. An option is included in the function to save the plots as any file type supported by R, including postscript and portable document format.

ABSTRACTMany standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet‐based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large‐scale English‐language test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.

One theoretical feature that makes item response theory (IRT) models those of choice for many psychometric data analysts is parameter invariance, the equality of item and examinee parameters from different examinee populations or measurement conditions. In this article, using the well-known fact that item and examinee parameters are identical only up to a set of linear transformations specific to the functional form of a given IRT model, violations of these transformations for unidimensional IRT models are investigated using analytical, numerical, and visual tools. Because item parameter drift (IPD) constitutes a lack of invariance (LOI) at the individual item level or item set level, the magnitudes and effects of IPD on examinee response probabilities and true scores are algebraically derived and connected to empirical results from a recent simulation study. Thus, this article facilitates a deeper understanding of the exact statistical formulation of parameter invariance as a fundamental property of latent variable measurement and explicates some practical consequences of LOI for decision making.

Questionnaires are widely used in clinical research and mental health studies, but their responses are usually more subjective than clinical biomarkers. Researchers may use latent variable models such as the item response theory (IRT) model to characterize individual ability, but response instability and small sample sizes can lead to increased uncertainty in estimation. Historical data from similar questionnaires may help reduce variation, but they must be incorporated adaptively. The challenge is further compounded by the need to ensure computational efficiency in IRT models with a large number of parameters. In this work, we develop a Bayesian framework for adaptive borrowing in IRT models based on an approximated normalized power prior (NPP) that treats the borrowing weight as a random parameter. The proposed method makes NPP feasible for general IRT models and is coupled with a full Bayesian data augmentation strategy that enables joint estimation of ability and item parameters through an efficient Gibbs sampler. In simulations, the approach adaptively increases borrowing when historical and current data are compatible and automatically downweights conflicting information. Relative to analyzes without borrowing, the method reduces variance and mean squared error while maintaining coverage across a range of test lengths, item discrimination profiles, and historical-current concordance scenarios. We illustrate the method by integrating historical mental health surveys into current assessments, yielding more precise ability estimates with preserved calibration. An efficient implementation is provided in our updated package NPP available on the Comprehensive R Archive Network.

The estimation of high-dimensional latent regression item response theory (IRT) models is difficult because of the need to approximate integrals in the likelihood function. Proposed solutions in the literature include using stochastic approximations, adaptive quadrature, and Laplace approximations. We propose using a second-order Laplace approximation of the likelihood to estimate IRT latent regression models with categorical observed variables and fixed covariates where all parameters are estimated simultaneously. The method applies when the IRT model has a simple structure, meaning that each observed variable loads on only one latent variable. Through simulations using a latent regression model with binary and ordinal observed variables, we show that the proposed method is a substantial improvement over the first-order Laplace approximation with respect to the bias. In addition, the approach is equally or more precise to alternative methods for estimation of multidimensional IRT models when the number of items per dimension is moderately high. Simultaneously, the method is highly computationally efficient in the high-dimensional settings investigated. The results imply that estimation of simple-structure IRT models with very high dimensions is feasible in practice and that the direct estimation of high-dimensional latent regression IRT models is tractable even with large sample sizes and large numbers of items.

In item response theory (IRT) modeling, the item parameter error covariance matrix plays a critical role define? in statistical inference procedures. When item parameters are estimated using the EM algorithm, the parameter error covariance matrix is not an automatic by-product of item calibration. Cai proposed the use of Supplemented EM algorithm for computing the item parameter error covariance matrix. This method has been subsequently implemented in commercial IRT software programs such as IRTPRO and flexMIRT. Jamshidian and Jennrich noted that Supplemented EM is among a class of methods based on numerically differentiating the EM map, and they proposed noniterative alternatives, such as forward difference and Richardson extrapolation, that are mathematically simpler and may lead to a reduction in computational burden when compared with Supplemented EM. However, the relative merits of the various numerical differentiation methods have not been evaluated in the context of IRT modeling. We perform such an evaluation, using both simulated and empirical data. It is found that the accuracy of the simpler noniterative alternatives is heavily dependent on the choice of the numerical differentiation perturbation constants. On the other hand, Supplemented EM consistently maintains accuracy and does not require the selection of perturbation constants. Furthermore, when implemented with an adaptive iteration scheme, an updated Supplemented EM algorithm can be as computationally efficient as the alternatives. The expected (Fisher) information matrix, while accurate, requires too heavy computation for realistic test lengths. Therefore, we recommend the routine use of the updated Supplemented EM algorithm in IRT applications.

The novelty of this study is to propose multidimensional item response theory (IRT) models for the investigation of heterogeneity in residents’ perceptions and attitudes toward the tourism industry. By following the social exchange theory, a sample of residents in the Romagna area (Italy) were invited to declare their agreement to some aspects related to benefits and costs of the tourism industry in 2010. The analysis suggests that the additive IRT model with two kind of latent variables (viz. the general latent trait measuring attitude toward tourism and two specific latent traits measuring perceptions on benefits and costs of tourism) fits the data adequately. Moreover, the estimated scores show that the tourism features are perceived differently by the residents, supporting the hypothesis that perceptions and attitudes change with regard to the individuals’ characteristics. Finally, the analysis proposes how the parameters of the IRT models can be interpreted in this new context.

Joint maximum likelihood (JML) estimation is one of the earliest approaches to fitting item response theory (IRT) models. This procedure treats both the item and person parameters as unknown but fixed model parameters and estimates them simultaneously by solving an optimization problem. However, the JML estimator is known to be asymptotically inconsistent for many IRT models, when the sample size goes to infinity and the number of items keeps fixed. Consequently, in the psychometrics literature, this estimator is less preferred to the marginal maximum likelihood (MML) estimator. In this paper, we re-investigate the JML estimator for high-dimensional exploratory item factor analysis, from both statistical and computational perspectives. In particular, we establish a notion of statistical consistency for a constrained JML estimator, under an asymptotic setting that both the numbers of items and people grow to infinity and that many responses may be missing. A parallel computing algorithm is proposed for this estimator that can scale to very large datasets. Via simulation studies, we show that when the dimensionality is high, the proposed estimator yields similar or even better results than those from the MML estimator, but can be obtained computationally much more efficiently. An illustrative real data example is provided based on the revised version of Eysenck's Personality Questionnaire (EPQ-R).

It is not uncommon to use unidimensional item response theory (IRT) models to estimate ability in multidimensional data. Therefore it is important to understand the implications of summarizing multiple dimensions of ability into a single parameter estimate, especially if effects are confounded when applied to computerized adaptive testing (CAT). Previous studies have investigated the effects of different IRT models and ability estimators by manipulating the relationships between item and person parameters. However, in all cases, the maximum information criterion was used as the item selection method. Because maximum information is heavily influenced by the item discrimination parameter, investigating a-stratified item selection methods is tenable. The current Monte Carlo study compared maximum information, a-stratification, and a-stratification with b blocking item selection methods, alone, as well as in combination with the Sympson-Hetter exposure control strategy. The six testing conditions were conditioned on three levels of interdimensional item difficulty correlations and four levels of interdimensional examinee ability correlations. Measures of fidelity, estimation bias, error, and item usage were used to evaluate the effectiveness of the methods. Results showed either stratified item selection strategy is warranted if the goal is to obtain precise estimates of ability when using unidimensional CAT in the presence of two-dimensional data. If the goal also includes limiting bias of the estimate, Sympson-Hetter exposure control should be included. Results also confirmed that Sympson-Hetter is effective in optimizing item pool usage. Given these results, existing unidimensional CAT implementations might consider employing a stratified item selection routine plus Sympson-Hetter exposure control, rather than recalibrate the item pool under a multidimensional model.

Two common aspects of symptom measurement are 1) the occurrence or presence of symptoms, and 2) the intensity or severity of symptoms when they occur. We adopt a latent trait perspective based on item response theory (IRT), using both unidimensional and multidimensional IRT models. We demonstrate how to (a) prepare data for analysis, (b) specify, estimate, and compare models, (c) interpret model parameters, (d) compare scores from models, and (e) visualize analysis results. We develop the relevant sequential IRT model, noting its flexibility, congruence with the theorized data generating process for symptom measures, and its promise for facilitating additional research and practical applications. The sequential model is less frequently used than other IRT models for polytomous data such as the generalized partial credit or graded response models. However, estimation of the sequential model can be readily accomplished with standard latent variable modeling and IRT software for binary indicators that allows constraints on the discrimination parameters. We compare the sequential model to other modeling options. We provide discussion of future research directions.

Although item response theory (IRT) models such as the bifactor, two-tier, and between-item-dimensionality IRT models have been devised to confirm complex dimensional structures in educational and psychological data, they can be challenging to use in practice. The reason is that these models are multidimensional IRT (MIRT) models and thus are highly parameterized, making them only suitable for data provided by large samples. Unfortunately, many educational and psychological studies are conducted on a small scale, leaving the researchers without the necessary MIRT models to confirm the hypothesized structures in their data. To address the lack of modeling options for these researchers, we present a general Bayesian MIRT model based on adaptive informative priors. Simulations demonstrated that our MIRT model could be used to confirm a two-tier structure (with two general and six specific dimensions), a bifactor structure (with one general and six specific dimensions), and a between-item six-dimensional structure in rating scale data representing sample sizes as small as 100. Although our goal was to provide a general MIRT model suitable for smaller samples, the simulations further revealed that our model was applicable to larger samples. We also analyzed real data from 121 individuals to illustrate that the findings of our simulations are relevant to real situations.