Using Classical Test Theory in Combination with Item Response Theory

Tests used for such purposes as determining educational quality, defining educational needs, hiring an employee, student selection and placement and performing guidance and clinic services have an important place in education and psychology. Of course, they should have certain psychometric features related to test scores' validity and reliability. Various test theories have helped to create more valid and reliable measurements and, as a result, to make better decisions regarding individuals. In education and psychology, Classical Test Theory (CTT) and Item Response Theory (IRT) are both widely used. CTT assumes that an individual's observed score is the total of the true score and the error score, while IRT estimates an individual's ability or latent trait from responses to test items (Embretson & Reise, 2000).When IRT assumptions and model-data fit are ensured, item and ability parameters' invariance occurs; this is known as the most important advantage IRT has over CTT. Item and ability parameters' invariance means estimating ability parameters independently of item sample and estimating item parameters independently of ability sample. IRT's invariance feature makes it very practicable in many applications, for instance, test development, computerized adaptive testing, bias studies, test equating and item mapping (Hambleton & Swaminathan, 1985). IRT is classified under two main categories as parametric IRT (PIRT) and nonparametric IRT (NIRT) (Olivares, 2005; Sijtsma & Molenaar, 2002).To analyse ordered items, such as Likert-type attitude items, partial credit cognitive items or not ordered graded items such as multiple-choice test items, item response models are developed towards polytomous items in IRT (Ostini & Nering, 2006). In these models developed for polytomous items, a non-linear relationship between an individual's latent trait and the possibility of choosing a certain category of item answer is explained (Embretson & Reise, 2000). Graded Response Model (GRM), part of IRT models developed for polytomous items, is often preferred by researchers for applications since it is more useful in presentations, portfolios, essays and Likert-type items with ordered item categories (DeMars, 2010; Ostini & Nering, 2006). To scale tests that consist of polytomous items by making true estimates according to GRM, evaluating PIRT's assumptions and model-data fit is necessary. And to provide these assumptions and model-data fit, large samples are needed. At this point, NIRT models draw attention because they provide a practical advantage in determining psychometric properties of tests with fewer items and respondents (Stout, 2001).NIRT models are defined as statistical scaling methods that require fewer assumptions than PIRT models for measuring persons and items (Stochl, 2007). With their wide application area, NIRT models are used in ordinal scales, applied research areas, sociology, marketing research and health research on quality of life (Sijtsma, 2005). The literature reveals that two models, namely, the Mokken model and nonparametric regression estimation models, are employed. These two models are themselves divided into sub-models. The Mokken model consists of the sub-models Monotone Homogeneity Model (MHM) and the Double Monotonicity Model (DMM). Nonparametric regression estimation models consist of such sub-models as the Kernel Smoothing Approach Model (KSAM), the Isotonic Regression Estimation and the Smoothed Isotonic Regression Estimation models (Lee, 2007; Sijtsma & Molenaar, 2002). Along with theoretical studies being conducted, new sub-models are being added to nonparametric regression estimation models.As a NIRT model, MHM requires unidimensionality, local independence and monotonicity assumptions, and it defines the relationship that latent variables and items with homogeneous (unidimensional) and monotone item characteristic curve (ICC) have (Meijer & Baneke, 2004; Sijtsma & Molenaar, 2002). …

Classical Test Theory (CTT), also known as the true score theory, refers to the analysis of test results based on test scores. The statistics produced under CTT include measures of item difficulty, item discrimination, measurement error and test reliability. The term “Classical” is used in contrast to “Modern” test theory which usually refers to item response theory (IRT). The fact that CTT was developed before IRT does not mean that CTT is outdated or replaced by IRT. Both CTT and IRT provide useful statistics to help us analyse test data. Generally, CTT and IRT provide complementary results. For many item analyses, CTT may be sufficient to provide the information we need. There are, however, theoretical differences between CTT and IRT, and many researchers prefer IRT because of enhanced measurement properties under IRT. IRT also provides a framework that facilitates test equating, computer adaptive testing and test score interpretation. While this book devotes a large part to IRT, we stress that CTT is an important part of the methodologies for educational and psychological measurement. In particular, the exposition of the concept of reliability in CTT sets the basis for evaluating measuring instruments. A good understanding of CTT lays the foundations for measurement principles. There are other approaches to measurement such as generalizability theory and structural equation modelling, but these are not the focus of attention in this book.

Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual change assessment. In this study, the authors compared the CTT and IRT methods with respect to their Type I error and detection rates. Preliminary results revealed that IRT is indeed superior to CTT in individual change detection, provided that the tests consist of at least 20 items. For shorter tests, however, CTT is generally better at correctly detecting change in individuals. The results and their implications are discussed.

PurposeTo validate the hepatitis B virus infection-related stigma scale (HBVISS) using Classical Test Theory and Item Response Theory in a sample of Chinese chronic HBV carriers.MethodsFeasibility, internal consistency reliability, split-half reliability and construct validity were evaluated using a cross-sectional validation study (n = 1,058) in Classical Test Theory. Content validity was assessed by COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) criteria. The Item Response Theory (IRT) model parameters were estimated using Samejima’s graded response model, after which item response category characteristic curves were drawn. Item information, test information, and IRT-based marginal reliability were calculated. Measurement invariance was assessed using differential item functioning (DIF). SPSS and R software were used for the analysis.ResultsThe response rate reached 96.4% and the scale was completed in an average time of 5 min. Content validity of HBVISS was sufficient (+) and the quality of the evidence was high according to COSMIN criteria. Confirmatory factor analysis showed acceptable goodness-of-fit (χ2/df = 5.40, standardized root mean square residual = 0.057, root mean square error of approximation = 0.064, goodness-of-fit index = 0.902, comparative fit index = 0.925, incremental fit index = 0.926, and Tucker-Lewis index = 0.912). Cronbach’s α fell in the range of 0.79–0.89 for each dimension and 0.93 for the total scale. Split-half reliability was 0.96. IRT discrimination parameters were estimated to range between 0.959 and 2.333, and the threshold parameters were in the range-3.767 to 3.894. The average score for test information was 12.75 (information >10) when the theta level reached between-4 and + 4. The IRT-based marginal reliability was 0.95 for the total scale and fell in the range of 0.83–0.91 for each dimension. No measurement invariance was detected (d-R2 < 0.02).ConclusionHBVISS exhibited good feasibility, reliability, validity, and item quality, making it suitable for assessing chronic Hepatitis B virus infection-related stigma.

BackgroundFamily support and patient satisfaction with treatment are crucial for aiding in the recovery from stroke. However, current validated stroke-specific questionnaires may not adequately capture the impact of these two variables on patients undergoing clinical trials of new drugs. Therefore, the aim of this study was to develop and evaluate a new stroke patient-reported outcome measure (Stroke-PROM) instrument for capturing more comprehensive effects of stroke on patients participating in clinical trials of new drugs.MethodsA conceptual framework and a pool of items for the preliminary Stroke-PROM were generated by consulting the relevant literature and other questionnaires created in China and other countries, and interviewing 20 patients and 4 experts to ensure that all germane parameters were included. During the first item-selection phase, classical test theory and item response theory were applied to an initial scale completed by 133 patients with stroke. During the item-revaluation phase, classical test theory and item response theory were used again, this time with 475 patients with stroke and 104 healthy participants. During the scale assessment phase, confirmatory factor analysis was applied to the final scale of the Stroke-PROM using the same study population as in the second item-selection phase. Reliability, validity, responsiveness and feasibility of the final scale were tested.ResultsThe final scale of Stroke-PROM contained 46 items describing four domains (physiology, psychology, society and treatment). These four domains were subdivided into 10 subdomains. Cronbach’s α coefficients for the four domains ranged from 0.861 to 0.908. Confirmatory factor analysis supported the validity of the final scale, and the model fit index satisfied the criterion. Differences in the Stroke-PROM mean scores were significant between patients with stroke and healthy participants in nine subdomains (P < 0.001), indicating that the scale showed good responsiveness.ConclusionsThe Stroke-PROM is a patient-reported outcome multidimensional questionnaire developed especially for clinical trials of new drugs and is focused on issues of family support and patient satisfaction with treatment. Extensive data analyses supported the validity, reliability and responsiveness of the Stroke-PROM.Electronic supplementary materialThe online version of this article (doi:10.1186/s12955-015-0246-0) contains supplementary material, which is available to authorized users.

In this study, the researchers assessed comparability of item statistics of 2017 basic education certificate mathematics examination of National Examinations Council (NECO) through Classical Test Theory (CTT) and Item Response Theory (IRT) measurement frameworks. The study adopted instrumentation design. A 60-item NECO basic education certificate education mathematics objective test paper I was administered to 978 basic nine examinees, randomly selected from Osogbo and Olorunda Local Government Area, Osun State, Nigeria. The responses of the examinees to the test data were analysed using Marginal Maximum Likelihood Estimation of JMETRIK software.&amp;nbsp; The result showed that the test data obey the assumption of unidimensionality of 3-parameter logistic model and Classical Test Theory measurement framework deleted more items 33 (55%) compare to IRT measurement framework 12 (20%). Also, it was observed that item statistics from the two contrasting frameworks (CTT and IRT) were not comparable. Moreover, further analysis showed that there was low correlation among the item statistics index. The implication of this is that NECO should jettison the use of Classical Test Theory and embrace utilization of Item Response Theory framework during their test development and item analysis.&amp;nbsp;

This study investigated the application of item response theory (IRT) to expand the range of ability estimates for gifted (hereinafter referred to as high-achieving) students’ performance on an above-level test. Using a sample of fourth- to sixth-grade high-achieving students ( N = 1,893), we conducted a study to compare estimates from two measurement theories, classical test theory (CTT) and IRT. CTT and IRT make different assumptions about the analysis that impact the reliability and validity of the scores obtained from the test. IRT can also differentiate students based on the student’s grade or within a grade by using the unique string of correct and incorrect answers the student makes while taking the test. This differentiation may have implications for identifying or classifying students who are ready for advanced coursework. An exploration of the differentiation for Math, Reading, and Science tests and the impact the different measurement frameworks can have on classification of students are explored. Implications for academic talent identification with the talent search model and development of academic talent are discussed.

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items.

Background Item response theory (IRT), particularly the graded response model, provides a latent trait (θ) that reflects balance ability and offers a complementary framework for psychometric evaluation. Although IRT is recommended for estimating clinical thresholds, its application to cutoff determination for stroke populations remains limited. We hypothesized that IRT-based approaches will yield cutoff values that differ from those derived using classical test theory and provide additional methodological insight for cutoff determination. Purpose To estimate and compare cutoff values for walking independence in inpatients with stroke using classical test theory and IRT. Methods Data from 165 inpatients with stroke were analyzed. Balance was assessed using the Brief Balance Evaluation Systems Test (Brief-BESTest). Cutoff scores were calculated using receiver operating characteristic (ROC), IRT combined with ROC (IRT+ROC), prevalence-based IRT method (old IRT), and embedded state-item IRT method (new IRT). Discriminative performance was evaluated using area under the ROC curve (AUC), sensitivity, specificity, likelihood ratios, and accuracy. Results The cutoff values for walking independence were 12.37 (ROC), 11.46 (IRT+ROC), 10.54 (old IRT), and 7.90 (new IRT). All methods demonstrated comparable performance, with AUC values ranging from 0.769 to 0.794, indicating good discriminative ability of the Brief-BESTest for identifying walking independence. The new IRT method achieved the highest AUC and sensitivity; ROC and IRT+ROC methods achieved similar AUCs. Although cutoff values and classification metrics differed slightly across methods, their confidence intervals overlapped. Conclusion To our knowledge, this is the first study to directly compare cutoff estimation methods incorporating IRT in stroke rehabilitation, suggesting that IRT-based approaches can yield cutoff values with stability comparable to that of classical test theory and providing methodological insight for future research and clinical assessment scale development.

The authors examined the in.uence of deviant scale items (i.e., items failing to adequately assess the construct of interest) on item parameter estimates of focal scale items and person parameter estimates through a comparison of item response theory (IRT) and classical test theory (CTT) methods. The authors used monte carlo methods to fully explore results found in a pilot investigation of jobattitude data. They found that graded response model item discrimination parameter estimates were robust when the number of focal items was larger than the number of deviant items, when the focal items were more discriminating than the deviant items, and when the correlation between the focal and deviant items was positive. IRT discrimination estimates were typically more robust than CTT discrimination estimates. Under extremely unfavorable conditions, however, CTT estimates were more robust than the IRT estimates. For person parameter estimates, the robustness of estimates depended primarily on the correlation of the deviant items with the focal construct. Implications are discussed for researchers employing IRT or CTT methods in scale development.

Despite theoretical differences between item response theory (IRT) and classical test theory (CTT), there is a lack of empirical knowledge about how, and to what extent, the IRT- and CTT-based item and person statistics behave differently. This study empirically examined the behaviors of the item and person statistics derived from these two measurement frameworks. The study focused on two issues: (a) What are the empirical relationships between IRT- and CTT-based item and person statistics? and (b) To what extent are the item statistics from IRT and those from CIT invariant across different participant samples? A large-scale statewide assessment database was used in the study. The findings indicate that the person and item statistics derived from the two measurement frameworks are quite comparable. The degree of invariance of item statistics across samples, usually considered as the theoretical superiority IRT models, also appeared to be similar for the two measurement fireworks.

Anxiety disorders in youth are frequently underdiagnosed and untreated, partly due to a lack of screening in primary care. The Generalized Anxiety Disorder 7-item (GAD-7) scale is a brief self-report measure designed to screen for anxiety in primary care settings. However, little is known about the psychometrics of this scale with adolescents. Participants included 579 youth age 11 to 17 years who received screening for depression in a primary care setting through a web-based application, VitalSign6, over a 4-year period. Psychometric analyses were completed based on classical test theory (CTT) and item response theory (IRT). Using CTT and IRT methods, the GAD-7 has a unidimensional structure with good psychometric properties. In addition, the IRT analysis demonstrates that items 1 and 2 are strongly associated with the total score, and thus are good choices as a 2-item screening tool. Convergent validity was demonstrated, with high correlations between the GAD-7 and other measures of anxiety, and discriminant validity was also demonstrated, with low correlations to measures of other psychological states. This psychometric evaluation of the GAD-7 provides support for the utility of this measure with adolescents. The GAD-2 is a good estimate of GAD-7 total score.

PurposeEngagement is a construct that varies according to the subject, object and context; this has been used to justify the coexistence of a variety of construct definitions and scales. Instead of proposing a new scale, this paper aims to create a procedure for comparing scales and to use it to evaluate brand engagement measures in social media.Design/methodology/approachThis study first defines a procedure for the selection, standardization and comparison of scales; this procedure considers both the classical test theory (CTT) and item response theory (IRT). The authors apply the procedure in a survey of 233 respondents to compare three scales for measuring consumer engagement with brands in social media.FindingsThe establishment of a procedure for scale comparison is useful in assisting researchers to choose specific measures. Results showed that the three scales have similar characteristics, but Vivek et al.’s (2014) scale is recommended when better discrimination between construct dimensions is required, Hollebeek et al.’s (2014) scale could be used as a one-dimensional scale and Dessart et al.’s (2016) reduced scale has better ability to capture information for the affective and cognitive dimensions. None of the scales were very efficient in discriminating weakly and strongly engaged individuals.Originality/valueThis study makes a substantive contribution by proposing a procedure for scale comparison that considers CTT and IRT and shows the advantages, limitations and recommendations for using three different scales of consumer engagement.

A new test index is defined as the probability of obtaining two randomly selected test scores (PDTS) as statistically different. After giving a concept definition of the test index, two simulation studies are presented. The first analyzes the influence of the distribution of test scores, test reliability, and sample size on PDTS within classical test theory. The second simulation manipulates the number of items and their locations over the latent continuum for a one-parameter logistic model. In addition, comparisons of both applications on the same data are made. The discussion focuses on advantages of applying PDTS for dimensional constructs in classical test theory and item response theory, as well as the disadvantages of making no assumptions about the test score distribution. The simulation studies provide the initial findings about the PDTS under several given empirical conditions. The discussion emphasizes why and when it is useful to consider PDTS in a test description.

BackgroundAppropriate dietary intake can have a significant influence on athletic performance. There is a growing consensus on sports nutrition and professionals working with athletes often provide dietary education. However, due to the limitations of existing sports nutrition knowledge questionnaires, previous reports of athletes’ nutrition knowledge may be inaccurate.MethodsAn updated questionnaire has been developed based on a recent review of sports nutrition guidelines. The tool has been validated using a robust methodology that incorporates relevant techniques from classical test theory (CTT) and Item response theory (IRT), namely, Rasch analysis.ResultsThe final questionnaire has 89 questions and six sub-sections (weight management, macronutrients, micronutrients, sports nutrition, supplements, and alcohol). The content and face validity of the tool have been confirmed based on feedback from expert sports dietitians and university sports students, respectively. The internal reliability of the questionnaire as a whole is high (KR = 0.88), and most sub-sections achieved an acceptable internal reliability. Construct validity has been confirmed, with an independent T-test revealing a significant (p < 0.001) difference in knowledge scores of nutrition (64 ± 16%) and non-nutrition students (51 ± 19%). Test-retest reliability has been assured, with a strong correlation (r = 0.92, p < 0.001) between individuals’ scores on two attempts of the test, 10 days to 2 weeks apart. Three of the sub-sections fit the Rasch Unidimensional Model.ConclusionsThe final version of the questionnaire represents a significant improvement over previous tools. Each nutrition sub-section is unidimensional, and therefore researchers and practitioners can use these individually, as required. Use of the questionnaire will allow researchers to draw conclusions about the effectiveness of nutrition education programs, and differences in knowledge across athletes of varying ages, genders, and athletic calibres.