A UNIFIED APPROACH TO IRT SCALE LINKING AND SCALE TRANSFORMATIONS

In educational testing, it is a common practice to produce test forms under a nonequivalent groups with anchor test (NEAT) design. In this design, the two test forms share a certain number of common items, while the populations who take the test forms might not be equivalent to each other. Test linking is conducted to establish the equivalency of ability scales from separate item response theory (IRT) calibrations of two test forms. Among existing linking methods, two approaches are related to the study in this paper: IRT model-based linking (Loyd & Hoover, 1980; Marco, 1977; Stocking & Lord, 1983) and equipercentile linking (Kolen & Brennan, 1995; von Davier, Holland, & Thayer, 2004b).

An examination of the psychometric properties of the physical self-description questionnaire using a polytomous item response model

Psychometric properties of the Psy-flex scale: A validation study in a community sample in Korea

ABSTRACTAbout 45 R packages to analyze data using item response theory (IRT) have been developed over the last decade. This article introduces these 45 R packages with their descriptions and features. It also describes possible advanced IRT models using R packages, as well as dichotomous and polytomous IRT models, and R packages that contain applications such as differential item functioning and equating are also introduced. Thus, this article helps researchers who plan to use IRT-based analysis to decide on the type of IRT analysis and choose the appropriate R packages.

Kane, Mroch, Suh, and Ripkey (2009) describe what they call five linear equating methods for the nonequivalent groups with anchor test (NEAT) design. The authors embed these methods within a two-di...

MH4: APPLICATION OF THE RASCH MODEL TO THE SF36 MENTAL HEALTH 5 ITEM SCALE (MH5)

This article investigates how measurement models and statistical procedures can be applied to estimate the accuracy of proficiency classification in language testing. The paper starts with a concise introduction of four measurement models: the classical test theory (CTT) model, the dichotomous item response theory (IRT) model, the testlet response theory (TRT) model, and the polytomous item response theory (Poly-IRT) model. Following this, two classification procedures are presented: the Livingston and Lewis method for CTT and the Rudner method for the three IRT-based models. The utility of these models and procedures are then evaluated by examining the accuracy of classifying 5000 language test takers from a large-scale language certification examination into two proficiency categories. The most important finding is that the testlet format (multiple questions based on one prompt), which language tests usually rely on, has a great impact on the proficiency classification. All testlets in this study show a strong testlet effect. Hence, the TRT model is recommended for proficiency classification. Using the standard IRT model would inflate the classification accuracy due to the underestimated measurement error. Meanwhile, using the Poly-IRT model would give slightly less accurate classification results. Concerning the CTT model, while its classification accuracy is comparable to that of the TRT, there exists considerable inconsistency between their classification results.

In contrast to dichotomous item response theory (IRT) models, most well-known polytomous IRT models do not imply stochastic ordering of the latent trait by the total test score (SOL). This has been thought to make the ordering of respondents on the latent trait using the total test score questionable and throws doubt on the justifiability of using nonparametric polytomous IRT models for ordinal measurement. We show that a broad class of polytomous IRT models has a weaker form of SOL, denoted weak SOL, and argue that weak SOL justifies ordering respondents on the latent trait using the total test score and, therefore, the use of nonparametric polytomous IRT models for ordinal measurement.

A developmental scale for the North Carolina End‐of‐Grade Mathematics Tests was created using a subset of identical test forms administered to adjacent grade levels. Thurstone scaling and item response theory (IRT) techniques were employed to analyze the changes in grade distributions across these linked forms.Three variations of Thurstone scaling were examined, one based on Thurstone's 1925 procedure and two based on Thurstone's 1938 procedure. The IRT scaling was implemented using both BiMain and Multilog.All methods indicated that average mathematics performance improved from Grade 3 to Grade 8, with similar results for the two IRT analyses and one version of Thurstone's 1938 method.The standard deviations of the IRT scales did not show a consistent pattern across grades, whereas those produced by Thurstone's 1925 procedure generally decreased; one version of the 1938 method exhibited slightly increasing variation with increasing grade level, while the other version displayed inconsistent trends.

The goal of this study is to investigate how features of a rating scale developed for English-speaking populations interact with Spanish-speaking respondents' response styles and functional categories of judgment. A sample of 400 Spanish-speaking students took a translated scale and a scaling task developed to measure response sets and functional categories of judgment, respectively. Three response set models—extreme response, central tendency, and acquiescence—under two conditions—base and revised with respondents' functional categories—were studied with item response theory and multidimensional scaling methods. Revising the number of scale categories with the number of salient functional categories statistically improved fit of the base models. Multidimensional scaling results showed scale content features interacting with response styles and functional categories. Translation of rating scales requires adapting scale features to characteristics of target languages, such as salient response styles and respondents' functional categories of judgment.

Applying single-level statistical models to multilevel data typically produces underestimated standard errors, which may result in misleading conclusions. This study examined the impact of ignoring multilevel data structure on the estimation of item parameters and their standard errors of the Rasch, two-, and three- parameter logistic models in item response theory (IRT) to demonstrate the degree of such underestimation in IRT. Also, the Lord’s chi-square test using the underestimated standard errors was used to test differential item functioning (DIF) to show the impact of such underestimation on the practical applications of IRT. The results of simulation studies showed that, in the most severe case of multilevel data, the standard error estimate from the standard single-level IRT models was about half of the minimal asymptotic standard error, and the type I error rate of the Lord’s chi-square test was inflated up to .35. The results of this study suggest that standard single-level IRT models may seriously mislead our conclusions in the presence of multilevel data, and therefore multilevel IRT models need to be considered as alternatives.

[Correction Notice: An Erratum for this article was reported in Vol 26(3) of Psychological Assessment (see record 2014-16017-001). The mean, standard deviation and alpha coefficient originally reported in Table 1 should be 74.317, 10.214 and .802, respectively. The validity coefficients in the last column of Table 4 are affected as well. Correcting this error did not change the substantive interpretations of the results, but did increase the mean, standard deviation, alpha coefficient, and validity coefficients reported for the Honesty subscale in the text and in Tables 1 and 4. The corrected versions of Tables 1 and Table 4 are shown in the erratum.] Item response theory (IRT) models were applied to dichotomous and polytomous scoring of the Self-Deceptive Enhancement and Impression Management subscales of the Balanced Inventory of Desirable Responding (Paulhus, 1991, 1999). Two dichotomous scoring methods reflecting exaggerated endorsement and exaggerated denial of socially desirable behaviors were examined. The 1- and 2-parameter logistic models (1PLM, 2PLM, respectively) were applied to dichotomous responses, and the partial credit model (PCM) and graded response model (GRM) were applied to polytomous responses. For both subscales, the 2PLM fit dichotomous responses better than did the 1PLM, and the GRM fit polytomous responses better than did the PCM. Polytomous GRM and raw scores for both subscales yielded higher test-retest and convergent validity coefficients than did PCM, 1PLM, 2PLM, and dichotomous raw scores. Information plots showed that the GRM provided consistently high measurement precision that was superior to that of all other IRT models over the full range of both construct continuums. Dichotomous scores reflecting exaggerated endorsement of socially desirable behaviors provided noticeably weak precision at low levels of the construct continuums, calling into question the use of such scores for detecting instances of "faking bad." Dichotomous models reflecting exaggerated denial of the same behaviors yielded much better precision at low levels of the constructs, but it was still less precision than that of the GRM. These results support polytomous over dichotomous scoring in general, alternative dichotomous scoring for detecting faking bad, and extension of GRM scoring to situations in which IRT offers additional practical advantages over classical test theory (adaptive testing, equating, linking, scaling, detecting differential item functioning, and so forth).

ABSTRACTThe original purpose of this study was to address the test‐disclosure‐related need to introduce more Graduate Record Examinations (GRE) General Test editions each year than formerly, in a context of stable, or possibly declining examinee volume. The legislative conditions that created this initial concern regarding test equating have abated. However, several of the test equating models considered in this research might provide other advantages to the GRE Program. These potential advantages are listed in the body of the report.Equating can be considered to consist of three parts: (1) a data collection design, (2) an operational definition of the equating transformation, and (3) the specific statistical estimation techniques used to obtain the equating transformation. Currently, the GRE General Test collects data using an equivalent groups design. Typically, a linear equating method is used, and the specific estimation technique is setting means and standard deviations equal.For this research, two other data collection designs were studied: nonrandom group, external anchor test, and random group, preoperational section. Both item response theory (IRT) and linear equating definitions were used. IRT true score equating was based on item statistics for the three‐parameter logistic model as estimated using LOGIST. Linear models included section pre‐equating using the EM algorithm, Tucker's observed score model, and several true score models developed by Tucker and Levine. For each of the three GRE measures, verbal, quantitative and analytical, all equating methods were assessed for bias and root mean squared error by equating a test edition to itself through a chain with six equating links.Bias and root mean squared error were extremely large for equating the verbal and analytical measures using section pre‐equating or IRT equating with data based on the random group preoperational section data collection design. For the quantitative measure, this data collection design produced a small amount of bias, but moderate amount of root mean squared error.Using the nonrandom group, external anchor test data collection design, quantitative equatings had moderate amounts of both bias and root mean squared error. Verbal nonrandom group, external anchor test equatings showed relatively small amounts of bias and root mean squared error, with the Tucker observed score model performing particularly well. Bias was small for the analytical anchor test equatings, and root mean squared error ranged from small to moderate.All nonrandom group, external anchor test methods worked about as well in practice for the verbal measure as the currently used random group method does in theory. The current random group method, however, has never been subjected to an empirical check comparable to that used in this study for the experimental equating methods. Two anchor test methods, Tucker 2 True and Levine, appear to have worked as well in practice for the analytical measure as the random group method does in theory.A possible explanation for the generally poor results for the random group, preoperational section data collection design based equatings was the constant use of the last section of the test to collect equating data. It may be, now that the sections of the GRE General Test are administered in various orders in different editions of the test, that the extreme bias found in this study for the verbal and analytical random group preoperational section equatings will disappear or at least be substantially diminished.

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items.

PurposeIn clinical ophthalmology as in other fields, measuring patient-reported outcomes imposes a burden on patients. To decrease that burden, we used item-response theory (IRT) to develop and test a short version of the National Eye Institute's Visual Function Questionnaire (VFQ).MethodsWe analyzed VFQ data from 276 adults in Japan. Most of them had glaucoma, cataract, or macular degeneration. Their visual acuity (Snellen fraction) averaged 20/120 (range: 20/13 to 20/2000) for the better eye, and 20/200 (range: 20/13 to 20/2000) for the worse eye. We used a polytomous IRT model, the Generalized Partial Credit Model as implemented in software for parameter scaling of rating data (PARSCALE). To select items for inclusion in the short version we examined each item's location on the latent-trait continuum, its slope, and its frequency of missing data. We also ensured representation of all 7 domains that are important in Japan. To examine the characteristics of the resulting scale, we computed its test information (an index of precision that can vary with the value of the latent trait), and carried out validation testing.ResultsFrom 32 of the original VFQ items, we selected 11. The scale comprising those 11 items (the VFQ-J11) had test information greater than 9 for values of the latent trait between −2.0 and +0.8. The item thresholds were well-targeted for patients with vision problems. Scores on the VFQ-J11 correlated strongly and in the expected direction with measures of visual field and corrected visual acuity. As expected for a valid measure, those scores also improved by a large amount (almost one standard deviation) after cataract surgery.ConclusionThis 11-item instrument can provide reliable and the valid data on visual functioning in patients with ophthalmic problems. It is expected to be less of a burden on respondents, while it maintains good psychometric properties.