Measured Progress

Repeatedly using items in high-stake testing programs provides a chance for test takers to have knowledge of particular items in advance of test administrations. A predictive checking method is proposed to detect whether a person uses preknowledge on repeatedly used items (i.e., possibly compromised items) by using information from secure items that have zero or very low exposure rates. Responses on the secure items are first used to estimate a person's proficiency distribution, and then the corresponding predictive distribution for the person's responses on the possibly compromised items is constructed. The use of preknowledge is identified by comparing the observed responses to the predictive distribution. Different estimation methods for obtaining a person's proficiency distribution and different choices of test statistic in predictive checking are considered. A simulation study was conducted to evaluate the empirical Type I error and power rate of the proposed method. The simulation results suggested that the Type I error of this method is well controlled, and this method is effective in detecting preknowledge when a large proportion of items are compromised even with a short secure section. An empirical example is also presented to demonstrate its practical use.

In this article we present results of a Differential Item Functioning (DIF) study using Shealy and Stout's (1993) multidimensionality-based DIF analysis framework. In this framework, differences in test score distributions across different groups of examinees may be a result of multidimensionality if secondary dimensions (not the primary dimension measured by the test) differentially affect examinee performance. Thus, this framework requires both statistical and substantive judgments for identifying potential DIF items and substantiating the causes of DIF, which, as a result, will enhance a comprehensive construct validity argument. In this article, we illustrate step-by-step procedures of multidimensionality-based DIF analyses using LanguEdge reading comprehension test data. Qualitative data from think-aloud verbal protocols were used to generate DIF hypotheses about differential functioning of vocabulary items between two groups of Indo-European and non-Indo-European L2 learners. Statistical Simultaneous Item Bias Test (SIBTEST; Shealy & Stout, 1993) was used to test the DIF hypotheses. The DIF results supported the hypotheses by flagging four uniform DIF items and one crossing DIF item. Post-hoc analyses of the DIF-flagged items were performed by visual inspection of group differences using TestGraf (Ramsay, 2001) and revisiting the qualitative verbal data and analyses of cognate types. The results showed that DIF items with large effect sizes were associated with the following: (a) translation-equivalent cognates; (b) word meaning determined independent of context, and (c) less frequent, more difficult distracter words than those in stems. In light of empirical evidence, the article discusses implications for test development and validation processes.

Several types of parameterizations of attribute correlations in cognitive diagnosis models use the reduced reparameterized unified model. The general approach presumes an unconstrained correlation matrix with K(K−1)/2 parameters, whereas the higher order approach postulates K parameters, imposing a unidimensional structure on the correlation matrix between the latent skills. This article investigates the differences in performance between the correlational structure parameterizations (a general structure, a higher order single-factor structure, and a baseline uniform distributional approach constraining the attributes to be independent) across a wide variety of simulated multidimensional attribute spaces. Results suggest that the correlational approaches perform equally well with respect to classification and item parameter estimation accuracy, regardless of the violations of the assumptions of the higher order model. Findings suggest the general robustness of the higher order model and the associated estimation procedure. The three approaches are also used to analyze a real-world test; results suggest that such tests can be analyzed effectively by the higher order algorithm.

The development of the DETECT procedure marked an important advancement in nonparametric dimensionality analysis. DETECT is the first nonparametric technique to estimate the number of dimensions in a data set, estimate an effect size for multidimensionality, and identify which dimension is predominantly measured by each item. The efficacy of DETECT critically depends on accurate, minimally biased estimation of the expected conditional covariances of all the item pairs. However, the amount of bias in the DETECT estimator has been studied only in a few simulated unidimensional data sets. This is because the value of the DETECT population parameter is known to be zero for this case and has been unknown for cases when multidimensionality is present. In this article, integral formulas for the DETECT population parameter are derived for the most commonly used parametric multidimensional item response theory model, the Reckase and McKinley model. These formulas are then used to evaluate the bias in DETECT by positing a multidimensional model, simulating data from the model using a very large sample size (to eliminate random error), calculating the large‐sample DETECT statistic, and finally calculating the DETECT population parameter to compare with the large‐sample statistic. A wide variety of two‐ and three‐dimensional models, including both simple structure and approximate simple structure, were investigated. The results indicated that DETECT does exhibit statistical bias in the large‐sample estimation of the item‐pair conditional covariances; but, for the simulated tests that had 20 or more items, the bias was small enough to result in the large‐sample DETECT almost always correctly partitioning the items and the DETECT effect size estimator exhibiting negligible bias.