For passage-based tests, items that belong to a common passage often violate the local independence assumption of unidimensional item response theory (UIRT). In this case, ignoring local item dependence (LID) and estimating item parameters using a UIRT model could be problematic because doing so might result in inaccurate parameter estimates, which, in turn, could impact the results of equating. Under the random groups design, the main purpose of this article was to compare the relative performance of the three-parameter logistic (3PL), graded response (GR), bifactor, and testlet models on equating passage-based tests when various degrees of LID were present due to passage. Simulation results showed that the testlet model produced the most accurate equating results, followed by the bifactor model. The 3PL model worked as well as the bifactor and testlet models when the degree of LID was low but returned less accurate equating results than the two multidimensional models as the degree of LID increased. Among the four models, the polytomous GR model provided the least accurate equating results.

The relationship of expected and actual difficulty of items on six mathematics question papers designed for 16-year olds in England was investigated through paired comparison using experts and testing with students. A variant of the Rasch model was applied to the comparison data to establish a scale of expected difficulty. In testing, the papers were taken by 2933 students using an equivalent-groups design, allowing the actual difficulty of the items to be placed on the same measurement scale. It was found that the expected difficulty derived using the comparative judgement approach and the actual difficulty derived from the test data was reasonably strongly correlated. This suggests that comparative judgement may be an effective way to investigate the comparability of difficulty of examinations. The approach could potentially be used as a proxy for pretesting high-stakes tests in situations where pretesting is not feasible due to reasons of security or other risks.

The aim of the present study was to examine Differential Distractor Functioning (DDF) as a means of improving the quality of a measure through understanding biased responses across groups. A DDF analysis could shed light on the potential sources of construct-irrelevant variance by examining whether the differential selection of incorrect choices (distractors), attracts various groups in different ways. To examine possible DDF effects, a method introduced by Penfield (2008, 2010a), based on odds ratio estimators, was utilized. Items from the Chemistry sub-scale of the Standard Achievement Admission Test (SAAT) in Saudi Arabia were used as an example. Statistical evidence for differential item functioning (DIF_ existed for five items, at either moderate or strong levels. Particularly three items (i.e., items 45, 54, and 61), reached category B levels (i.e., moderate DIF), and two items (items 51and 60) category C levels (strong DIF) based on Educational Testing Service guidelines. These items were then examined more closely for DDF in an attempt to potentially understand the causes of DIF and group biased responses. The manuscript concludes with a series of remedial actions, based on distractor-relevant information, with the goal of improving the psychometric properties of an instrument under study.

Contextualized items have been widely used in science testing. Despite common use of item contexts, how the influence of a chosen context on the reliability and validity of the score inferences remains unclear. We focused on sequential cues of contextual information, referring to the order of events or descriptions presented in item contexts. We hypothesized that the sequence of information presented should allow students to understand the logic of an item. We proposed an approach for capturing information on three dimensions of sequential cues: sequence of events, sequence of intention and action, and sequence of cause and effect. We found that the addition of contexts increased the item difficulty. There was limited evidence that items with intention did not differ in item difficulty from those without intention. Findings for the other two dimensions was inconclusive because of the low frequency of certain item categories.

Simulations were conducted to examine the effect of differential item functioning (DIF) on measurement consequences such as total scores, item response theory (IRT) ability estimates, and test reliability in terms of the ratio of true-score variance to observed-score variance and the standard error of estimation for the IRT ability parameter. The objective was to provide bounds of the likely DIF effects on these measurement consequences. Five factors were manipulated: test length, percentage of DIF items per form, item type, sample size, and level of group ability difference. Results indicate that the greatest DIF effect was less than 2 points on the 0 to 60 total score scale and about 0.15 on the IRT ability scale. DIF had a limited effect on the ratio of true-score variance to observed-score variance, but its influence on the standard error of estimation for the IRT ability parameter was evident for certain ability values.

We examined multimodality (the representation of information in multiple semiotic modes) in the context of international test comparisons. Using Program of International Student Assessment (PISA)-2009 data, we examined the correlation of the difficulty of science items and the complexity of their illustrations. We observed statistically significant mean correlation coefficient differences favoring higher-ranking over lower-ranking jurisdictions and non-Western over Western jurisdictions. We examined this correlation as a function of the variety of forms of representation of information used in illustrations. For the highest-ranking jurisdictions, the correlation tended to be positive and its magnitude tended to increase with the number of forms of representation. For the lowest-ranking jurisdictions, the correlation tended to be negative and its magnitude tended to increase with the number of forms of representation. Increased illustration complexity appears to be an affordance in making sense of science items for students from the highest-ranking jurisdictions and a challenge for students from the lowest-ranking jurisdictions. Our findings support the notion that integrating information provided in multiple semiotic modes is critical to science achievement and underscore the importance for science standards and international assessment frameworks to address this skill as intrinsic to (rather than an aspect of) science proficiency.

The use of computer-based assessments makes the collection of detailed data that capture examinees’ progress in the tests and time spent on individual actions possible. This article presents a study using process and timing data to aid understanding of an international language assessment and the examinees. Issues regarding test-taking strategies, test speededness, test design, and their relationship to examinees’ demographic backgrounds and performance are also discussed.

The purpose of this study was to enhance the meaning of the scores of an English-language test by developing performance levels and descriptors for reporting overall test performance. The levels and descriptors were intended to accompany the total scale scores of TOEFL Junior® Standard, an international test of English as a second/foreign language. The study addressed two issues: the number of performance levels that could be meaningfully reported and the information that should be included in the performance level descriptors. Data from 3,607 students who took an operational test form were used. Although our methodology built to some extent on earlier work (Papageorgiou et al., 2015), we demonstrate how content and construct differences between the tests of each study dictated use of different types of data in order to construct meaningful performance descriptors and select the cut-offs for the levels. We emphasize the importance of establishing the psychometric quality of the reported overall levels and the empirical basis for the content of their performance descriptors, and we discuss the implications of our work for similar efforts in the future.

Because schools worldwide use low-stakes tests to make important decisions, value-added indices computed from test scores must accurately reflect student learning, which requires equal test-taking effort across testing occasions. Evaluating change in effort assumes effort is measured equivalently across occasions. We evaluated the longitudinal measurement invariance of a globally used self-report measure of test-taking effort. Given measurement invariance, we assessed average change and variability in change in effort when examinees completed the same low-stakes tests twice during college. Average effort declined slightly, but there was large variability in change in effort. Conscientiousness, agreeableness, openness, and performance-approach orientation were uncorrelated with change in effort. Initial effort, change in mastery-approach orientation, change in work-avoidance orientation, and change in perceived test importance were correlated with change in effort. Because change in test importance had the strongest relationship with change in effort, globally applicable interventions to increase effort through increasing test importance are discussed.

Indicators of academic achievement for bilingual students can be inaccurate due to linguistic heterogeneity. For indigenous populations, language shift (the gradual replacement of one language by another) is a factor that can increase this heterogeneity and poses an additional challenge for valid testing. We investigated whether and how indigenous populations can be validly included in a large-scale assessment program. We gave Mexican preschool Mayan students aged 5 to 6 years the same set of mathematics items in three versions: (1) original in Spanish, (2) Mayan translation, and (3) content equivalent, developed from scratch in Mayan. Also, we collected information on the students’ and the teachers’ use of Spanish and Mayan, and on the communities’ support of the two languages. Students performed poorly on the three versions. Generalizability theory-based analyses revealed considerable performance inconsistency across items and language versions and low generalizability and dependability coefficients. This performance instability appears to stem from a dwindling support of the Mayan language in Mayan schools and communities. Fair, valid assessment of indigenous populations in either their languages or their countries’ dominant languages appears to be difficult to accomplish with current testing models and policies.