Does the Length of a Questionnaire Matter?

Classical Test Theory (CTT), also known as the true score theory, refers to the analysis of test results based on test scores. The statistics produced under CTT include measures of item difficulty, item discrimination, measurement error and test reliability. The term “Classical” is used in contrast to “Modern” test theory which usually refers to item response theory (IRT). The fact that CTT was developed before IRT does not mean that CTT is outdated or replaced by IRT. Both CTT and IRT provide useful statistics to help us analyse test data. Generally, CTT and IRT provide complementary results. For many item analyses, CTT may be sufficient to provide the information we need. There are, however, theoretical differences between CTT and IRT, and many researchers prefer IRT because of enhanced measurement properties under IRT. IRT also provides a framework that facilitates test equating, computer adaptive testing and test score interpretation. While this book devotes a large part to IRT, we stress that CTT is an important part of the methodologies for educational and psychological measurement. In particular, the exposition of the concept of reliability in CTT sets the basis for evaluating measuring instruments. A good understanding of CTT lays the foundations for measurement principles. There are other approaches to measurement such as generalizability theory and structural equation modelling, but these are not the focus of attention in this book.

The reliability of event-related brain potential (ERP) scores depends on study context and how those scores will be used, and reliability must be routinely evaluated. Many factors can influence ERP score reliability; generalizability (G) theory provides a multifaceted approach to estimating the internal consistency and temporal stability of scores that is well suited for ERPs. G theory's approach possesses a number of advantages over classical test theory that make it ideal for pinpointing sources of error in observed scores. The current primer outlines the G-theory approach to estimating internal consistency (coefficients of equivalence) and test-retest reliability (coefficients of stability). This approach is used to evaluate the reliability of ERP measurements. The primer outlines how to estimate reliability coefficients that consider the impact of the number of trials, events, occasions, and groups. The uses of two different G-theory reliability coefficients (i.e., generalizability and dependability) in ERP research are elaborated, and a dataset from the companion manuscript, which examines N2 amplitudes to Go/NoGo stimuli, is used as an example of the application of these coefficients to ERPs. The developed algorithms are implemented in the ERP Reliability Analysis (ERA) Toolbox, which is open-source software designed for estimating score reliability using G theory. The toolbox facilitates the application of G theory in an effort to simplify the study-by-study evaluation of ERP score reliability. The formulas provided in this primer should enable researchers to pinpoint the sources of measurement error in ERP scores from multiple recording sessions and subsequently plan studies that optimize score reliability.

BackgroundHealth-related Quality of life (HRQOL) is now concerned worldwide in cancer clinical fields, with the specific scales for patients with hepatocellular carcinoma in China being seldom and no formal validation of Chinese Version of the FACT-Hep. This study was aimed to validate the Chinese FACT-Hep based on both Classical Test Theory and Generalizability Theory to rationalize its use in the Chinese population.MethodsOne hundred and fourteen in-patients with hepatocellular carcinoma were measured three times before and after treatments using the FACT-Hep. The FACT-Hep was evaluated by Cronbach α, Pearson correlation, intra-class correlation (ICC), and paired-t tests with standardized response mean. The Generalizability Theory was also applied to addresses the dependability of measurements deeply by estimation of multiple sources of variance.ResultsThe Cronbach’s α coefficients were greater than 0.70 for all domains, and test-retest reliability coefficients were all greater than 0.80 exception of EWB 0.74 with the range of 0.81–0.96. The domains of PWB, FWB and the overall scale had significant changes after treatments with SRM ranging from 0.40 to 0.69. G-coefficients and Ф-coefficients confirmed the reliability of the scale further with exact variance components.ConclusionThe Chinese version of FACT-Hep can be used to measure HRQOL for patients with hepatocellular carcinoma in China with being of good psychometric properties.

In this article, we illustrate ways in which generalizability theory (G-theory) can be used with continuous latent response variables (CLRVs) to address problems of scale coarseness resulting from categorization errors caused by representing ranges of continuous variables by discrete data points and transformation errors caused by unequal interval widths between those data points. The mechanism to address these problems is applying structural equation modeling (SEM) as a tool in deriving variance components needed to estimate indices of score consistency and validity. Illustrations include quantification of multiple sources of measurement error, use of non-nested and nested designs, derivation of indices of consistency for norm- and criterion-referenced interpretation of scores, estimation of effects when changing measurement procedures and designs, and disattenuation of correlation coefficients for measurement error. These illustrations underscore the effectiveness of G-theory with continuous latent response variables in providing stable indices of reliability and validity that are reasonably independent of the number of original scale points used, unevenness of scale intervals, and average degree of item skewness. We discuss general distinctions in reliability estimation within G-theory, SEM, and classical test theory; make specific recommendations for using G-theory on raw score and CLRV metrics; and provide computer code in an online supplement for doing all key analyses demonstrated in the article using R and Mplus. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

During the last decade researchers in the area of second language testing have adopted increasingly more sophisticated evaluation techniques in an attempt to better understand the process of learning a second language. Analysis of variance, regression analysis, and factor analysis are among the statistical procedures that have allowed for insightful evaluation of language testing data. This paper explores the feasibility of applying another procedure, generalizability theory analysis, to second language testing research. An earlier paper (Hinofotis, Bailey, and Stern 1981) reported on the development of a rating scale for assessing the oral proficiency of nonnative‐speaking teaching assistants and the actual use of the scale by a panel of raters in evaluating pilot data. The data from that study are reanalyzed here using generalizability theory, which provides simultaneous estimates of consistency along several dimensions of the research, e.g., raters, subjects, and occasions. The paper includes a comparison of generalizability theory and classical test theory, an analysis of the oral proficiency data, and a brief discussion involving the interpretation of generalizability study results.

Relative and absolute decisions in physical activity research are commonly based on a single assessment period that may not represent habitual activity behavior. This study used the generalizability theory to 1) quantify multiple sources of measurement error and 2) estimate the number of days and seasons needed to characterize long-term levels of activity. Between 2005 and 2006, youth participating in an intervention program were asked to wear a pedometer across seven consecutive days during three separate months (September, January, and May). Total variance in activity was partitioned and quantified according to differences among participants, inconsistency across days, relative differences among seasons, and the interactions among variables. Two coefficients (generalizability and phi) were calculated from multiple decision studies using a random and mixed design to determine the study protocol needed to achieve a reliability of 0.80 for relative and absolute decisions, respectively. Data were analyzed in 2009. Complete data were available for 42 boys and 38 girls. Residual variance accounted for the largest source of measurement error (55.64%), whereas smaller amounts were attributed to the participant (18.74%), season (6.59%), and day (2.67%) terms. Using a random design, both coefficients failed to reach an acceptable level of reliability using a single season. In contrast, using a mixed design, an acceptable level of reliability could be reached using 7-8 d from a single, fixed season. This study demonstrates the potential for using the generalizability theory to make decisions regarding the rank order of activity among individuals (relative decision) and compliance rates for physical activity recommendations (absolute decision).

Reliability: Measurement

With increasing interest in studies evaluating treatment outcome in children with attention deficit hyperactivity disorder (ADHD), there is a need for treatment-sensitive instruments that are feasible, yield valid and reliable scores, and measure outcome in a “time-locked” and “situation- and symptom-specific” manner. These instruments are needed to evaluate the outcome for which the treatment is targeted at specific settings (e.g., school), specific times of day (e.g., the late afternoon or early evening medication dose), and specific symptoms (e.g., hyperactivity). The Telephone Interview Probe (TIP) was developed to meet this need for outcome research with children with ADHD. The present study evaluated the feasibility, reliability, validity, and treatment sensitivity of scores on the TIP scales (Inattention, Hyperactivity-Impulsivity, Oppositional Behavior, and Problem Situation). Psychometric properties of the TIP were evaluated using classical test theory, as well as generalizability theory. The TIP proved to possess all the attributes required for use in large-scale treatment studies.

The efficacy of collaborative learning depends on the student interaction taking place. Monitoring student interaction can help teachers ensure their lesson goals are met. Experienced teachers describe, explain, and predict student interactions by drawing on their professional knowledge about types of student interaction that are associated with learning. This paper reports an assessment tool that measures teachers’ monitoring competence, operationalized as a form of professional vision: the ability to describe student interaction. Participants rate student interactions shown in videos on standardized items. Ratings are compared to an expert solution to provide a measure of monitoring competence. The assessment tool was evaluated with N = 193 teacher students that were in the pre-service or in-service phase of German teacher education. Classical test theory and generalizability theory were applied to evaluate the reliability of the assessment tool. Scores showed medium internal consistency. Measurement error resulting from videos was small. Reliability of scores was sufficient to detect differences in competence levels, before and after completing a training program on monitoring competence. Participants found the assessment tool appropriate for measuring monitoring competence. The findings show the feasibility of measuring professional vision of student interaction in collaborative learning. The assessment tool may be useful for adapting teacher dashboards, such as by advising beginning teachers how to intervene or, for more experienced teachers, by alerting when student interaction is not optimal for learning.

The chapter presents previous and near term applications and innovations for the assessment of rate-based measures such as fluency. The historical and future developments are discussed within the context of an ideographic behavioral and nomothetic psychometric paradigms of assessment. These paradigms are described and contrasted with descriptions of classical test theory (CTT), generalizability theory (GT), and item response theory (IRT). The interpretation and use argument (IUA) is used to frame the contemporary view on unified validity. These theoretical models are combined with an applied perspective to contextualize and encourage future developments in the measurement of fluency.

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and precision of scores within the CTT and IRT frameworks. This study presented new results pertaining to the relative precision (i.e., the test score conditional standard error of measurement for a given trait value) of CTT and IRT, and the new results shed light on the conditions where total scores and IRT estimates are more or less precisely measured. The relative reliability of CTT and IRT scores is examined as a function of item characteristics (e.g., locations, category thresholds, and discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution). CTT total scores were more reliable when the latent distribution was mismatched with category thresholds, but the discrepancy between CTT and IRT declined as the number of scale categories increased. This article also considered the appropriateness of linear approximations of polytomous items and presented circumstances where linear approximations are viable. A linear approximation may be appropriate for items with two response options depending on the item discrimination and the match between the item location and latent distribution. However, linear approximations are biased whenever items are located in the tails of the latent distribution and the bias is larger for more discriminating items.

Comparison of two psychometric scaling methods for ratings of acute musculoskeletal pain

The study is aimed at describing: (1) characteristics of the items of the national examination try-out test of the accounting subject matter in the 2015/2016 academic year on classical test theory and modern test theory; and (2) classification of students’ masteries in the learning of accounting. The study is explorative research. Analyses are conducted using the classical and modern test theories for item characteristics and descriptive quantitative for students’ masteries in accounting using the test set for the national examination try-out in the 2015/2016 academic year. A total of 414 students do the Package A test. Results show that (1) based on the classical test analyses, a number of 11 items (27.5%) belong to the “easy” category, 22 items (55%) “medium” category, and 7 items (17.5%) “difficult” category allowing a total of 19 (47.5%) to be categorized as good items; meanwhile, on the modern-theory analyses, a total of 34 items (85%) belong to the “good” category. (2) Around 38% of the students have competencies of the medium and low categories. Most students have difficulty in answering questions of the higher-order thinking levels.

This article introduces the Klasika software developed to run item and test analysis using the Classical Test Theory approach. Classical Test Theory is one of the specialized competencies and skills that undergraduate students of psychology must possess. Classical Test Theory becomes a mandatory course for all schools or departments of psychology in Indonesia. This article also provides a theoretical foundation of Classical Test Theory's essential concepts and statistical methods, specifically related to items and test statistics. The item analysis and test reliability procedures using Klasika, starting from the data preparation untill data interpretation, are explained with an empirical illustration. Finally, the analysis results using Klasika are compared with the results from Quest software to test the accuracy of the estimation results.

This study is about relations between classical test theory (CTT) and item response theory (IRT). It is shown that CTT is based on the assumption that measures are exchangeable, whereas IRT is based on conditional independence. Thus, IRT is presented as an extension of CTT, and concepts from both theories are related to one another. Furthermore, it is demonstrated that IRT can be used to provide CTT statistics in situations where CTT fails. Reliability, for instance, can be determined even though a test was not administered to the intended population.