Evaluating the Robustness of Graded Response Model and Classical Test Theory Parameter Estimates to Deviant Items

Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual change assessment. In this study, the authors compared the CTT and IRT methods with respect to their Type I error and detection rates. Preliminary results revealed that IRT is indeed superior to CTT in individual change detection, provided that the tests consist of at least 20 items. For shorter tests, however, CTT is generally better at correctly detecting change in individuals. The results and their implications are discussed.

ABSTRACTNomophobia, or the anxiety of being separated from one's mobile phone, is an emerging psychological condition in our digital age. The 20‐item Nomophobia Questionnaire (NMP‐Q) was developed to assess nomophobia symptoms. This study utilized item response theory (IRT) and classical test theory (CTT) methods to evaluate the NMP‐Q's psychometric properties. Data were collected from 5087 adults who completed the NMP‐Q. The CTT analyses included the computation of correlation coefficients, including McDonald's ω, Cronbach's α, Guttman's λ2, Guttman's λ6, greatest lower bound, average interitem correlation, Feldt–Gilmer coefficient, Feldt–Brennan coefficient, and Raju's β. IRT analyses included graded response modeling of items, test information function, differential item functioning, and reliability. CTT analysis revealed high reliability coefficients across nine metrics and a four‐factor model with good fit indices. IRT analysis showed a strong item fit to the polytomous graded model, indicating the questionnaire's robustness. Based on IRT (both uni‐ and multidimensional IRT) analyses, we propose a 10‐item short version (NMP‐Q Short), demonstrating high correlation (r = 0.97) with the original NMP‐Q. Our findings provide very good support for the NMP‐Q's reliability and construct validity. The items displayed good discrimination and difficulty parameters. The second 10 items of the NMP‐Q appeared to be the most informative. The proposed NMP‐Q Short offers a valuable option for efficient/rapid and accurate screening. However, additional validation and sensitivity testing are warranted to further establish its usefulness.

Background Item response theory (IRT), particularly the graded response model, provides a latent trait (θ) that reflects balance ability and offers a complementary framework for psychometric evaluation. Although IRT is recommended for estimating clinical thresholds, its application to cutoff determination for stroke populations remains limited. We hypothesized that IRT-based approaches will yield cutoff values that differ from those derived using classical test theory and provide additional methodological insight for cutoff determination. Purpose To estimate and compare cutoff values for walking independence in inpatients with stroke using classical test theory and IRT. Methods Data from 165 inpatients with stroke were analyzed. Balance was assessed using the Brief Balance Evaluation Systems Test (Brief-BESTest). Cutoff scores were calculated using receiver operating characteristic (ROC), IRT combined with ROC (IRT+ROC), prevalence-based IRT method (old IRT), and embedded state-item IRT method (new IRT). Discriminative performance was evaluated using area under the ROC curve (AUC), sensitivity, specificity, likelihood ratios, and accuracy. Results The cutoff values for walking independence were 12.37 (ROC), 11.46 (IRT+ROC), 10.54 (old IRT), and 7.90 (new IRT). All methods demonstrated comparable performance, with AUC values ranging from 0.769 to 0.794, indicating good discriminative ability of the Brief-BESTest for identifying walking independence. The new IRT method achieved the highest AUC and sensitivity; ROC and IRT+ROC methods achieved similar AUCs. Although cutoff values and classification metrics differed slightly across methods, their confidence intervals overlapped. Conclusion To our knowledge, this is the first study to directly compare cutoff estimation methods incorporating IRT in stroke rehabilitation, suggesting that IRT-based approaches can yield cutoff values with stability comparable to that of classical test theory and providing methodological insight for future research and clinical assessment scale development.

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items.

Anxiety disorders in youth are frequently underdiagnosed and untreated, partly due to a lack of screening in primary care. The Generalized Anxiety Disorder 7-item (GAD-7) scale is a brief self-report measure designed to screen for anxiety in primary care settings. However, little is known about the psychometrics of this scale with adolescents. Participants included 579 youth age 11 to 17 years who received screening for depression in a primary care setting through a web-based application, VitalSign6, over a 4-year period. Psychometric analyses were completed based on classical test theory (CTT) and item response theory (IRT). Using CTT and IRT methods, the GAD-7 has a unidimensional structure with good psychometric properties. In addition, the IRT analysis demonstrates that items 1 and 2 are strongly associated with the total score, and thus are good choices as a 2-item screening tool. Convergent validity was demonstrated, with high correlations between the GAD-7 and other measures of anxiety, and discriminant validity was also demonstrated, with low correlations to measures of other psychological states. This psychometric evaluation of the GAD-7 provides support for the utility of this measure with adolescents. The GAD-2 is a good estimate of GAD-7 total score.

In order to develop valid and reliable instruments, psychometric validation should be conducted as an iterative process that “requires a multi-method assessment” (Schimmack, 2019, p. 4). In this study, a multi-method psychometric approach was applied to a recently developed and validated scale, the Affinity for Technology Interaction (ATI) scale (Franke, Attig, & Wessel, 2018). The dataset (N ​= ​240) shared by the authors of the scale (Franke et al., 2018) was used. Construct validity of the ATI was explored by means of hierarchical clustering on variables, and its psychometric properties were analysed in accordance with an extended psychometric protocol (Dima, 2018) by methods of Classical Test Theory (CTT) and Item Response Theory (IRT). The results showed that the ATI is a unidimensional scale (homogeneity H ​= ​0.55) with excellent reliability (ω ​= ​0.90 [0.88-0.92]) and construct validity. Suggestions for further improvement of the ATI scale and the psychometric protocol were made.

In view of ICD-11 revision, we evaluate whether the 18 DSM-IV diagnostic items retained by DSM-5 could be further improved (i) in predicting ADHD 'caseness' and 'impairment' and (ii) discriminating ADHD without CD (ADHD - CD) cases from ADHD with CD (ADHD + CD) cases. In a multi-centre study sample consisting of 1497 ADHD probands and 291 unaffected subjects, 18 diagnostic items were examined for redundancy; then each item was evaluated for association with caseness, impairment and CD status using Classical Test Theory, Item-Response Theory and logistic regression methods. First, all 18 DSM-IV items contributed significantly and independently to the clinical diagnosis of ADHD. Second, not all the DSM-IV items carried equal weighting. "Often loses things", "forgetfulness" and "difficulty sustaining attention" mark severity for Inattentiveness (IA) items and "often unduly noisy", "exhibits a persistent pattern of restlessness", "leaves seat in class" and "often blurts out answers" for Hyperactivity/Impulsivity (HI) items. "Easily distracted", "inattentive to careless mistakes", "often interrupts" and "often fidgets" are associated with milder presentations. In the IA domain, "distracted" yields most information in the low-severity range of the latent trait, "careless" in the mid-severity range and "loses" in the high-severity range. In the HI domains, "interrupts" yields most information in the low-severity range and "motor" in the high-severity range. Third, all 18 items predicted impairment. Fourth, specific ADHD items are associated with ADHD + CD status. The DSM-IV diagnostic items were valid and not redundant; however, some carried more weight than others. All items were associated with impairment.

Classical Test Theory (CTT), also known as the true score theory, refers to the analysis of test results based on test scores. The statistics produced under CTT include measures of item difficulty, item discrimination, measurement error and test reliability. The term “Classical” is used in contrast to “Modern” test theory which usually refers to item response theory (IRT). The fact that CTT was developed before IRT does not mean that CTT is outdated or replaced by IRT. Both CTT and IRT provide useful statistics to help us analyse test data. Generally, CTT and IRT provide complementary results. For many item analyses, CTT may be sufficient to provide the information we need. There are, however, theoretical differences between CTT and IRT, and many researchers prefer IRT because of enhanced measurement properties under IRT. IRT also provides a framework that facilitates test equating, computer adaptive testing and test score interpretation. While this book devotes a large part to IRT, we stress that CTT is an important part of the methodologies for educational and psychological measurement. In particular, the exposition of the concept of reliability in CTT sets the basis for evaluating measuring instruments. A good understanding of CTT lays the foundations for measurement principles. There are other approaches to measurement such as generalizability theory and structural equation modelling, but these are not the focus of attention in this book.

Assessing health-related quality of life (HRQoL) via Computerized Adaptive Tests (CAT) provides greater measurement precision coupled with a lower test burden compared to conventional tests. Currently, there are no European pediatric HRQoL CATs available. This manuscript aims at describing the development of a HRQoL CAT for children and adolescents: the Kids-CAT, which was developed based on the established KIDSCREEN-27 HRQoL domain structure. The Kids-CAT was developed combining classical test theory and item response theory methods and using large archival data of European KIDSCREEN norm studies (n = 10,577-19,580). Methods were applied in line with the US PROMIS project. Item bank development included the investigation of unidimensionality, local independence, exploration of Differential Item Functioning (DIF), evaluation of Item Response Curves (IRCs), estimation and norming of item parameters as well as first CAT simulations. The Kids-CAT was successfully built covering five item banks (with 26-46 items each) to measure physical well-being, psychological well-being, parent relations, social support and peers, and school well-being. The Kids-CAT item banks proved excellent psychometric properties: high content validity, unidimensionality, local independence, low DIF, and model conform IRCs. In CAT simulations, seven items were needed to achieve a measurement precision between .8 and .9 (reliability). It has a child-friendly design, is easy accessible online and gives immediate feedback reports of scores. The Kids-CAT has the potential to advance pediatric HRQoL measurement by making it less burdensome and enhancing the patient-doctor communication.

We used item response theory (IRT) methods for the assessment of the psychometric properties of a quality of life (QOL) questionnaire used to measure the effect of two nitrate treament regimens in patients with stable angina pectoris. IRT methods have been used little so far according to Medline 1990-1996 in this area, but these methods can be valuable. QOL is a highly dimensional construct and is often assessed in homogeneous populations, both of which are direct threats to reliability which is mostly used to evaluate the psychometric properties of QOL questionnaires. IRT methods do not use reliability as a measure of their applicability, but instead use formal goodness of fit tests. An additional advantage is that the scale used does not need an interval nature. In the present application IRT methods were applied successfully and we conclude that IRT methods are valuable tools for in QOL research.

African American youth, especially those who reside in low resourced communities, are exposed to higher levels of exposure to community violence relative to their counterparts from other race/ethnic groups. However, appropriate measures for assessing psychological stress related to such exposures are underresearched in the extant literature for this population. The aim of the current study was to examine the reliability and validity of the Brief Symptom Inventory (BSI-18) scale among African American youth exposed to community violence through classical test theory and item response theory methods. Internal reliability and construct validity were examined. Results indicated good internal reliability (α = .93). Construct validity of the BSI-18 was established through confirmatory factor analysis with the three-factor somatic, depression, and anxiety model. Construct validity was also determined with all items indicating adequate fit. Our study indicates good reliability and validity of the BSI-18 to assess psychological distress among African American youth exposed to community violence.

This article explores how traditional scores obtained from different forced-choice (FC) formats relate to their true scores and item response theory (IRT) estimates. Three FC formats are considered from a block of items, and respondents are asked to (a) pick the item that describes them most (PICK), (b) choose the two items that describe them the most and the least (MOLE), or (c) rank all the items in the order of their descriptiveness of the respondents (RANK). The multi-unidimensional pairwise-preference (MUPP) model, which is extended to more than two items per block and different FC formats, is applied to obtain the responses to each item block. Traditional and IRT (i.e., expected a posteriori) scores are computed from each data set and compared. The aim is to clarify the conditions under which simpler traditional scoring procedures for FC formats may be used in place of the more appropriate IRT estimates for the purpose of inter-individual comparisons. Six independent variables are considered: response format, number of items per block, correlation between the dimensions, item discrimination level, and sign-heterogeneity and variability of item difficulty parameters. Results show that the RANK response format outperforms the other formats for both the IRT estimates and traditional scores, although it is only slightly better than the MOLE format. The highest correlations between true and traditional scores are found when the test has a large number of blocks, dimensions assessed are independent, items have high discrimination and highly dispersed location parameters, and the test contains blocks formed by positive and negative items.

Test equating is a statistical procedure to ensure that scores from differenttest forms can be used interchangeably. There are several methodologiesavailable to perform equating, some of which are based on the Classical TestTheory (CTT) framework and others are based on the Item Response Theory (IRT)framework. This article compares equating transformations originated from threedifferent frameworks, namely IRT Observed-Score Equating (IRTOSE), KernelEquating (KE), and IRT Kernel Equating (IRTKE). The comparisons were made underdifferent data-generating scenarios, which include the development of a noveldata-generation procedure that allows the simulation of test data withoutrelying on IRT parameters while still providing control over some test scoreproperties such as distribution skewness and item difficulty. Our resultssuggest that IRT methods tend to provide better results than KE even when thedata are not generated from IRT processes. KE might be able to providesatisfactory results if a proper pre-smoothing solution can be found, while alsobeing much faster than IRT methods. For daily applications, we recommendobserving the sensibility of the results to the equating method, minding theimportance of good model fit and meeting the assumptions of the framework.

Both the 17-item Hamilton Rating Scale for Depression (HRSD17) and 30-item Inventory of Depressive Symptomatology – Clinician-rated (IDS-C30) contain a subscale that assesses anxious symptoms. We used classical test theory and item response theory methods to assess and compare the psychometric properties of the two anxiety subscales (HRSDANX and IDS-CANX) in a large sample (N = 3453) of outpatients with non-psychotic major depressive disorder in the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study. Approximately 48% of evaluable participants had at least one concurrent anxiety disorder by the self-report Psychiatric Diagnostic Screening Questionnaire (PDSQ). The HRSDANX and IDS-CANX were highly correlated (r = 0.75) and both had moderate internal consistency given their limited number of items (HRSDANX Cronbach’s alpha = 0.48; IDS-CANX Cronbach’s alpha = 0.58). The optimal threshold for ascribing the presence/absence of anxious features was found at a total score of eight or nine for the HRSDANX and seven or eight for the IDS-CANX. It would seem beneficial to delete item 17 (loss of insight) from the HRSDANX as it negatively correlated with the scale’s total score. Both the HRSDANX and IDS-CANX subscales have acceptable psychometric properties and can be used to identify anxious features for clinical or research purposes.

BackgroundWeb-based health behavior change interventions may be more effective if they offer autonomy-supportive communication facilitating the internalization of motivation for health behavior change. Yet, at this moment no validated tools exist to assess user-perceived autonomy-support of such interventions.ObjectiveThe aim of this study was to develop and validate the virtual climate care questionnaire (VCCQ), a measure of perceived autonomy-support in a virtual care setting.MethodsItems were developed based on existing questionnaires and expert consultation and were pretested among experts and target populations. The virtual climate care questionnaire was administered in relation to Web-based interventions aimed at reducing consumption of alcohol (Study 1; N=230) or cannabis (Study 2; N=228). Item properties, structural validity, and reliability were examined with item-response and classical test theory methods, and convergent and divergent validity via correlations with relevant concepts.ResultsIn Study 1, 20 of 23 items formed a one-dimensional scale (alpha=.97; omega=.97; H=.66; mean 4.9 [SD 1.0]; range 1-7) that met the assumptions of monotonicity and invariant item ordering. In Study 2, 16 items fitted these criteria (alpha=.92; H=.45; omega=.93; mean 4.2 [SD 1.1]; range 1-7). Only 15 items remained in the questionnaire in both studies, thus we proceeded to the analyses of the questionnaire’s reliability and construct validity with a 15-item version of the virtual climate care questionnaire. Convergent validity of the resulting 15-item virtual climate care questionnaire was confirmed by positive associations with autonomous motivation (Study 1: r=.66, P<.001; Study 2: r=.37, P<.001) and perceived competence for reducing alcohol intake (Study 1: r=.52, P<.001). Divergent validity could only be confirmed by the nonsignificant association with perceived competence for learning (Study 2: r=.05, P=.48).ConclusionsThe virtual climate care questionnaire accurately assessed participants’ perceived autonomy-support offered by two Web-based health behavior change interventions. Overall, the scale showed the expected properties and relationships with relevant concepts, and the studies presented suggest this first version of the virtual climate care questionnaire to be reasonably valid and reliable. As a result, the current version may cautiously be used in future research and practice to measure perceived support for autonomy within a virtual care climate. Future research efforts are required that focus on further investigating the virtual climate care questionnaire's divergent validity, on determining the virtual climate care questionnaire’s validity and reliability when used in the context of Web-based interventions aimed at improving nonaddictive or other health behaviors, and on developing and validating a short form virtual climate care questionnaire.