Human raters are normally involved in L2 performance assessment; as a result, rater behavior has been widely investigated to reduce rater effects on test scores and to provide validity arguments. Yet raters’ cognition and use of rubrics in their actual rating have rarely been explored qualitatively in L2 speaking assessments. In this study three rater groups (novice, developing, and expert) were first operationalized on the basis of four background variables (rating experience, teaching experience, rater training, and educational background) to predict different levels of expertise in rating. The three groups of raters then evaluated 18 ESL learners’ oral responses using an analytic scoring rubric across three occasions, separated by one-month intervals. Recorded verbal report data were analyzed (a) to compare rater behavior across the three groups and (b) to examine the development of rating performance within each group over time. The analysis revealed that the three groups of raters from different backgrounds presented varying levels of rating ability and different paces of improvement in their rating performance. The findings of the study suggest that a comprehensive consideration of rater characteristics contributes to a better understanding of raters’ different needs for training and rating.

Methods for detecting differential item functioning (DIF) and item bias are typically used in the process of item analysis when developing new measures; adapting existing measures for different populations, languages, or cultures; or more generally validating test score inferences. In 2007 in Language Assessment Quarterly, Zumbo introduced the concept of Third Generation DIF. In the current article we introduce a new methodology, latent class logistic regression, for Zumbo’s Third Generation DIF, whose foundation is a novel ecological model of item responding. The ecological model and the new statistical methodology are introduced, and a proof-of-concept is provided, in the context of an example of an international reading test focusing on DIF due to testing language. The new DIF framework is described and contrasted with other methods, Mplus code is provided, and the new method is shown to have potential for application in assessment.

In the late 1970s the language testing scene in North America was very lively, with the work, especially, of John Oller gaining prominence and the first meetings in Florida and Boston that led to t...

Looking back, in the inaugural issue in 2004, two invited articles by Alister Cumming and Constant Leung encouraged us at the helm of the journal to focus on two ideas. Cumming's (2004, p. 5) artic...

Test takers should have a voice in testing practices (Mathew, 2004). However, when incorporating their input, systematic processes to ensure the validity of testing practices must be followed. Such processes allow for test development to be a more democratic process (Shohamy, 2001), without sacrificing the value of the resulting inferences made from the test scores. This article describes a case study that incorporated the desires of test takers to change the procedures of a group oral discussion test in a university English as a Foreign Language program. A study was designed to determine the extent to which the proposed changes would threaten the validity of the testing process. Specifically, the procedures for the group oral were altered to investigate the effect of interlocutor familiarity. Students were randomly assigned to class-familiar (n = 146) and class-unfamiliar (n = 159) groups to identify to what extent group familiarity affected test takers' scores in the four assessed categories: pronunciation, fluency, lexis and grammar, and communication skills. For the two groups, no statistically significant difference in scores was found, and score reliability estimates were similar. The implications of the findings are addressed in terms of recommendations for using stakeholder input in the assessment design process.

The use of tests has been targeted as a critical point in validity, which suggests that contexts for test use warrant closer investigation. This article describes a study of decision making during test use in the context of a university intensive English language program. Over a period of 2½ years, data were collected by audio-recording placement sessions in which the program director and an instructor made decisions based on a three-part placement exam, TOEFL scores, course grades, instructor evaluations, and other information. Analysis of the sessions revealed four major areas impacting test use and the decision-making process: (a) test performance and score factors, (b) student factors, (c) test user factors, and (d) program factors (i.e., number and size of levels, curriculum, textbooks). These four areas represent factors in the placement process that are clearly about tests (test performance and score) and factors that have less relation to tests or scores such as test users and programmatic factors. The test and nontest factors interacted in the placement process to navigate borderlines between levels, confirm placements, and adhere to programmatic constraints.

This article reports on the development and piloting of a vocabulary recognition test designed for Indigenous Australian children. The research is both application oriented and development oriented. The aims of the article are to determine how well the test is used as a test instrument and the extent to which children recognize vocabulary items in Walmajarri (a local Indigenous language still spoken mainly by the older generation). The test was developed in collaboration with one community and extended to three neighboring communities where the same Indigenous language is spoken. Children's receptive knowledge of vocabulary items (nouns) in Walmajarri was tested, with 80 child participants across all communities. The usability of the test is discussed, as well as results of this language test, and recommendations are offered for future research in this area.

This study investigated the English writing skills developed by 42 children participating in an English as a Foreign Language (EFL) program for approximately 315 hr, spread over 18 months. The English writing abilities were measured 3 times in terms of grammar, content, coherence, spelling, and text length. A repeated measures multivariate analysis of variance demonstrated that the effect of time on the growth of writing skills was both large and significant: Grammar, coherence, and text length all showed substantial improvement in a linear progression; spelling and content developed in linear and quadratic trends. In addition, the improvement rate of grammar was similar to that of coherence, but text length showed a faster improvement. The vital common point was that students showed varying but remarkable improvements across all these components. This article makes idiosyncratic contributions to the understanding of young EFL writers and the longitudinal development of multiple writing components. This study may also serve as a reference for second-language educators as they notice (a) that these children achieved a high level of English writing skills although they did not live in an English-speaking society and (b) how they learned English in a curricular context where language acquisition principles were incorporated.

This research investigated the relation between rater cognition and rater behavior, focusing on differential severity/leniency regarding criteria. Participants comprised a sample of 18 raters examined in a previous rater cognition study (Eckes, 2008b). These raters, who were known to differ widely in their perceptions of criterion importance, provided ratings of live examinee writing performance. Based on these ratings, criterion-related bias measures were estimated using many-facet Rasch measurement. A cluster analysis of bias measures yielded four operational rater types. Each type was characterized by a distinct pattern of differentially severe or lenient ratings on particular criteria. The observed bias patterns were related to differential perceptions of criterion importance: Criteria perceived as highly important were more closely associated with severe ratings, and criteria perceived as less important were more closely associated with lenient ratings. Implications of the demonstrated link between rater cognition and rater behavior for future research into the nature of rater bias are discussed.

This commentary reviews recent literature on a number of problematic issues arising from the use of English proficiency tests by registration bodies as the sole assessment of the professional communication skills of overseas qualified health professionals from non-English-speaking backgrounds. It discusses differences between the assessment decisions of language specialists and those of medical communication specialists and the difficulties that many overseas qualified health professionals who have met the English requirements report experiencing as they endeavour to acquire professional communicative competence. It proposes that language specialists and medical communication specialists collaborate in the diagnostic assessment of overseas health professionals' language and communication needs and in the provision of appropriate courses of instruction.