Ethical Issues in Language Testing from a Validity Perspective

This article begins by exploring recent developments in the use of computers in language testing in four areas: (a) item banking, (b) computer-assisted language testing, (c) computerized-adaptive language testing, and (d) research on the effectiveness of computers in language testing.The article then examines the educational measurement literature in an attempt to forecast the directions future research on computers in language testing might take and suggests addressing the following issues: (a) piloting practices in computer adaptive language tests (CALTs), (b) standardizing or varying CALT lengths, (c) sampling CALT items, (d) changing the difficulty of CALT items, (e) dealing with CALT item sets, (f) scoring CALTS, (g) dealing with CALT item omissions, (h) making decisions about CALT cut-points, (i) avoiding CALT item exposure, (j) providing CALT item review opportunities, and (k) complying with legal disclosure laws when using CALTs.The literature on computer-assisted language learning indicates that language learners have generally positive attitudes toward using computers in the classroom (Reid, 1986;Neu & Scarcella, 1991;Phinney, 1991), and a fairly large literature has developed examining the effectiveness of computer-assisted language learning (for a review, see Dunkel, 1991).But less is known about the more specific area of computers in language testing.The purpose of this article is to examine recent developments in language testing that directly involve computer use including what we have learned in the process.The article will also examine the dominant issue of computer-adaptive testing in the educational measurement literature in an attempt to forecast some of the directions future research on computers in language testing might take. CURRENT STATE OF KNOWLEDGE ON COMPUTERS IN LANGUAGE TESTINGIn reviewing the literature on computers in language testing, I have found four recurring sets of issues: (a) item banking, (b) computer-assisted language testing, (c) computer-adaptive language testing, and (d) the effectiveness of computers in language testing.The discussion in this section will be organized under those four headings. Item BankingItem banking covers any procedures that are used to create, pilot, analyze, store, manage, and select test items so that multiple test forms can be created from subsets of the total "bank" of items.With a large item bank available, new forms of tests can be created whenever they are needed.Henning (1986) provides a description of how item banking was set up for the ESL Placement Examination at UCLA. (For further explanation and examples of item banking in educational testing, see Baker, 1989, pp.412-414, or Flaugher, 1990.)While the underlying aims of item banking can be accomplished by using traditional item analysis procedures (usually item facility and item discrimination indexes; for a detailed description of these

05–314Alderson, J. Charles (Lancaster U, UK) & Ari Huhta, The development of a suite of computer-based diagnostic tests based on the Common European Framework. Language Testing (London, UK) 22.3 (2005), 301–320.05–315Al-Hamly, Mashael & Christine Coombe (Kuwait U, Kuwait), To change or not to change: investigating the value of MCQ answer changing for Gulf Arab students. Language Testing (London, UK) 22.4 (2005), 509–531.05–316Broadfoot, Patricia M. (U of Bristol, UK), Dark alleys and blind bends: testing the language of learning. Language Testing (London, UK) 22.2 (2005), 123–141.05–317Cumming, Alister (U of Toronto, Canada; acumming@oise.utoronto.ca), Robert Kantor, Kyoko Baba, Usman Erdosy, Keanre Eouanzoui & Mark James, Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing (Amsterdam, the Netherlands) 10.1 (2005), 5–43.05–318Eckes, Thomas (TestDaF Institute, the Netherlands), Melanie Ellis, Vita Kalnberzina, Karmen Piorn, Claude Springer, Krisztina Szollás & Constance Tsagari, Progress and problems in reforming public language examinations in Europe: cameos from the Baltic States, Greece, Hungary, Poland, Slovenia, France and Germany. Language Testing (London,UK) 22.3 (2005), 355–377.05–319Figueras, Neus (Department of Education, Generalitat de Catalunya, Spain), Brian North, Sauli Takala, Norman Verhelst & Piet Van Avermaet, Relating examinations to the Common European Framework: a manual. Language Testing (London, UK) 22.3 (2005), 261–279.05–320Green, Anthony (Cambridge ESOL Examinations, Cambridge, UK), EAP study recommendations and score gains on the IELTS Academic Writing test. Assessing Writing (Amsterdam, the Netherlands) 10.1 (2005), 44–60.05–321Green, Rita & Dianne Wall (Lancaster U, UK), Language testing in the military: problems, politics and progress. Language Testing (London,UK) 22.3 (2005), 379–398.05–322Hasselgreen, Angela (The U of Bergen, Norway), Assessing the language of young learners. Language Testing (London,UK) 22.3 (2005), 337–354.05–323Klein, Joseph (kleinj@mail.biu.ac.il) & David Taub, The effect of variations in handwriting and print on evaluation of student essays. Assessing Writing (Amsterdam, the Netherlands) 10.2 (2005), 134–148.05–324Little, David (Trinity College, Dublin, Ireland), The Common European Framework and the European Language Portfolio: involving learners and their judgements in the assessment process. Language Testing (London, UK) 22.3 (2005), 321–336.05–325Lumley, Tom & Barry O'Sullivan (Australian Council for Educational Research, Australia), The effect of test-taker gender, audience and topic on task performance in tape-mediated assessment of speaking. Language Testing (London,UK) 22.4 (2005), 415–437.05–326Luxia, Qi (Guandong U of Foreign Studies, China), Stakeholders' conflicting aims undermine the washback function of a high-stakes test. Language Testing (London, UK) 22.2 (2005), 142–173.05–327Poehner, Matthew E. & James P. Lantolf (The Pennsylvania State U, USA), Dynamic assessment in the language classroom. Language Teaching Research (London, UK) 9.3 (2005), 233–265.05–328Stansfield, Charles W. & William E. Hewitt (Second Language Testing Inc., USA), Examining the predictive validity of a screening test for court interpreters. Language Testing (London, UK) 22.4 (2005), 438–462.05–329Trites, Latricia (Murray State U, USA) & Mary McGroarty, Reading to learn and reading to integrate: new tasks for reading comprehension tests?Language Testing (London, UK) 22.2 (2005), 174–210.05–330Uiterwijk, Henny (Citogroep, Arnem, the Netherlands) & Ton Vallen, Linguistic sources of item bias for second generation immigrants in Dutch tests. Language Testing (London, UK) 22.2 (2005), 211–234.05–331Weems, Gail H. (Arkansas Little Rock U, USA; ghweems@ualr.edu), Anthony J. Onwuegbuzie & Daniel Lustig, Profiles of respondents who respond inconsistently to positively- and negatively-worded items on rating scales. Evaluation & Research in Education (Clevedon, UK) 17.1 (2003), 45–60.05–332Weir, Cyril J. (Roehampton U, UK), Limitations of the Common European Framework for developing comparable examinations and tests. Language Testing (London, UK) 22.3 (2005), 281–300.05–333Xi, Xiaoming (U of California, USA), Do visual chunks and planning impact performance on the graph description task in the SPEAK exam?Language Testing (London, UK) 22.4 (2005), 463–508.

This entry defines high‐stakes language testing as the development and use of language tests that have major consequences for individuals, institutions, and educational or social systems. The entry delves into why high‐stakes language tests are developed and used, as well as the characteristics and research considerations of high‐stakes language testing. Following an introduction, the entry examines the rationale for high‐stakes language testing. The primary driving force is identified as the competition for resources and opportunities and the pursuit of fair competition. When decisions such as admissions, employment, and immigration are made, meritocracy is often practiced so that high‐performing candidates are rewarded with opportunities based on their achievements, rather than patronage. The entry then outlines the defining characteristics of high‐stakes language testing. First, tests with high stakes tend to be highly institutionalized and industrialized. Second, to avoid potential biases and minimize measurement errors, special attention is given to test standardization and objectivity. Third, high‐stakes testing has the capacity to shape teaching and learning through its washback effects. To demonstrate the features, two examples are provided: the Test of English as a Foreign Language (TOEFL) in the United States and the College English Test (CET) in China. In the final part, important considerations in research of high‐stakes language testing are discussed. Attention is drawn to the interdisciplinary and contextualized nature of high‐stakes language testing and the need for a broader validity agenda that goes beyond the measurement aspect of language testing and considers test consequences from a wider and more multifaceted social perspective.

Advances in language testing in the past decade have occurred in three areas: (a) the development of a theoretical view that considers language ability to be multicomponential and recognizes the influence of the test method and test taker characteristics on test performance, (b) applications of more sophisticated measurement and statistical tools, and (c) the development of communicative language tests that incorporate principles of communicative language teaching. After reviewing these advances, this paper describes an interactional model of language test performance that includes two components, language ability and test method. Language ability consists of language knowledge and metacognitive strategies, whereas test method includes characteristics of the environment, rubric, input, expected response, and relationship between input and expected response. Two aspects of authenticity are derived from this model. The situational authenticity of a given test task depends on the relationship between its test method characteristics and' the features of a specific language use situation, while its interactional authenticity pertains to the degree to which it invokes the test taker's language ability. The application of this definition of authenticity to test development is discussed.

This chapter considers how corpora can be used in the field of language testing. The general aim of language testing is to measure a latent trait in order to make inferences about an individual’s language ability. One useful way of thinking about testing language proficiency is to explore the essential principles, or test qualities, that underlie good assessment. The tasks used to assess language proficiency at any level should match the cognitive and psychological ability of the test taker as well as their linguistic knowledge; these are some of the test taker characteristics that influence how an individual performs on any test and which form part of a socio-cognitive framework for test development. Language testing occurs in a range of local, regional, national and international contexts, and language tests should take into account both the social context and the cognitive make-up of test takers and have various degrees of impact on them.

Despite the current need for reliable and valid test instruments to monitor the sign language acquisition of Deaf children in different countries, very few tests offering strong evidence for their psychometric properties are commercially available. A German Sign Language (DGS) test that focuses on linguistic structures acquired in preschool- and school-aged children (4–8 years old) is urgently needed. The present study uses as a template a test which has sound psychometric properties and has been standardized on another sign language as a starting point for tests of sign languages that are less documented, such as DGS. This book makes a novel contribution to the field by examining linguistic, cultural, methodological, and theoretical issues in the process of the adaptation from the source language test to the target language test, and by providing a model for future test adaptations. It also includes concrete steps for the test development and adaptation process. Adaptation and Evaluation of a German Sign Language Test addresses students and researchers alike who are involved in sign language test development and adaptation. It also provides a comprehensive summary in German.

High stakes English speaking tests have been used for evaluating applicants’ language proficiency in academic contexts or in the work place. In these situations, most Korean college students prepare for tests such as TOEIC Speaking or OPIc test in order to get better jobs. Although high-stakes tests pressure students and teachers to raise scores, there is little research about how test takers prepare for English speaking tests in cram schools. One aim of this study was to examine and compare TOEIC Speaking and OPIc test preparation activities in a cram school. Also, I aimed to investigate why and how test takers choose test preparation, the characteristics of test preparation, and how they are influenced by test preparation from the test-takers’ perspectives. Various types of data were collected through observation, interview and documentation. The results indicated that activities in the cram school mode were not significantly influenced by test type. Thus, test takers’ activities may be influenced by teacher’s instruction. The study raises awareness of the importance of instruction by a teacher for appropriate test preparation, and its impact on a test taker in the field of language teaching and testing.

Tests can give contributions if it is use naturally. It is needed to do a test to know students’ ability. Tests are also used to measure ability in language. Language testing is a field of study under the umbrella of applied linguistics. The purpose of testing is to collect objective information that may be used in conjunction with subjective information to make better. Language test developers and teachers should think of new types of intensive tests to measure examinees’ real language competence and performance to make the tests more reliable. Tes dapat memberikan kontribusi jika digunakan secara alami. Perlu dilakukan tes untuk mengetahui kemampuan siswa. Tes juga digunakan untuk mengukur kemampuan berbahasa. Pengujian bahasa adalah bidang studi di bawah payung linguistik terapan. Tujuan pengujian adalah untuk mengumpulkan informasi obyektif yang dapat digunakan bersama dengan informasi subyektif untuk menjadikannya lebih baik. Pengembang tes bahasa dan guru harus memikirkan jenis tes intensif baru untuk mengukur kompetensi dan kinerja bahasa peserta ujian yang sebenarnya agar tes tersebut lebih dapat diandalkan.

Test preparation has garnered considerable attention in second language (L2) education due to the significant implications that successful performance on a language test may have for academic advancement, future career opportunities, and immigration prospects. Meanwhile, an overemphasis on test preparation has been criticized for encouraging the cultivation of construct-irrelevant test-taking strategies at the expense of developing general language proficiency. To systematically explore how test preparation has been investigated in the literature, we conducted a scoping review of 66 studies on L2 test preparation. Specifically, this study examined the key characteristics of publications on test preparation, the main themes explored, the study and participant characteristics, as well as the essential aspects of their research methodologies. The results of this review revealed various trends in the literature on L2 test preparation, such as the exclusive focus on English as the target language, the lack of diversity in stakeholders as participants, the dominance of international language tests, and the paucity of experimental studies that utilize advanced statistical techniques. In addition to interpreting the results of our analysis, we discuss the implications of this scoping review and outline several directions for future research on test preparation.

The thought of Samuel Messick has influenced language testing in 2 main ways: in proposing a new understanding of how inferences made based on tests must be challenged, and in drawing attention to the consequences of test use. The former has had a powerful impact on language-testing research, most notably in Bachman's work on validity and the design of language tests. Messick's writing on test consequences has informed debate on ethics, impact, accountability, and washback in language testing in the work of several researchers. But the character of Messick's work challenges us in many additional ways. Messick located validity theory in the area of values. This article explores the implications of this position, highlighting the social construction of language test constructs. Language test constructs are increasingly the target of policy, a development that threatens to render traditionally conceived validation work of only marginal relevance. The less obvious, covert social construction of language test constructs is explored in the light of Butler's theory of performativity. The article concludes with a consideration of recent adaptations of Messick's work in the influential validation models of Mislevy and Kane, and considers their failure to address questions of values and the social context of assessment properly. Tackling these questions is the ongoing challenge of Messick's legacy.

This paper is primarily a methodological consideration, which critically reviews the literature on using protocol analysis to understand cognitive processes that students engage in while writing language tests. For the purpose of synthesizing the applicability and significance of protocol analysis in validating language tests, the author first discusses the potential of the concepts including validity, language test validation, and protocol analysis. Terminological classifications of different approaches of test validation are elaborated. The author then critically examines the state of evidence by interpreting studies that employed protocol analysis in language tests. This paper is intended to stimulate discussion on the merits and inadequacies of protocol analysis in different ESL/EFL tests, especially when the tests are designed for test-takers with culturally different backgrounds. Possible future research trends are suggested.

This study investigated the impact of different lengths of pre-task planning time on performance in a test of second language speaking ability for university admission. In the study, 47 Turkish-speaking learners of English took a test of English language speaking ability. The participants were divided into two groups according to their language proficiency, which was estimated through a paper-based English placement test. They each completed four monologue tasks: two picture-based narrative tasks and two description tasks. In a balanced design, each test taker was allowed a different length of planning time before responding to each of the four tasks. The four planning conditions were 30 seconds, 1 minute, 5 minutes, and 10 minutes. Trained raters awarded scores to the test takers using an analytic rating scale and a context-specific, binary-choice rating scale, designed specifically for the study. The results of the rater scores were analysed by using a multifaceted Rasch measurement. The impact of pre-task planning on test scores was found to be influenced by four variables: the rating scale; the task type that test takers completed; the length of planning time provided; and the test takers’ levels of proficiency in the second language. Increases in scores were larger on the picture-based narrative tasks than on the two description tasks. The results also revealed a relationship between proficiency and pre-task planning, whereby statistical significance was only reached for the increases in the scores of the lowest-level test takers. Regarding the amount of planning time, the 5-minute planning condition led to the largest overall increases in scores. The research findings offer contributions to the study of pre-task planning and will be of particular interest to institutions seeking to assess the speaking ability of prospective students in English-medium educational environments.

The search for fairness in language testing is distinct from other areas of educational measurement as the object of measurement, that is, language, is part of the identity of the test takers. So, a host of issues enter the scene when one starts to reflect on how to assess people's language abilities. As the quest for fairness in language testing is still in its infancy, even the need for such a research has been controversial, with some (e.g., Davies, 2010) arguing that such research is entirely in vain. This paper will provide an overview of some of the issues involved. Special attention will be given to critical language testing (CLT) as it has had a large impact on language testing research. It will be argued that although CLT has been very effective in revealing the ideological and value implications of the constructs of focus in language testing, extremism in this direction is not justified.

Validity in language testing and assessment has its long fundamental role in research along with reliability (Bachman & Palmer, 1996). This paper analyses basic theories and empirical research on language test validity in order to provide the notion, the classification of language test validity, the validation working frames and the trends of empirical research. Four key findings come out from the analysis. Firstly, language test validity refers to an evaluative judgment of the language test quality on the ground of evidence of the integrated components of test content, criterion and consequences through the interpretation of the meaning and utility of test scores. Secondly, construct validity is a dominating term in modern validity classification. The chronic division of construct validity into prior and post ones can help researchers have a clearer validation option. Plus, test validation can be grounded in light of Messick (1989), Bachman (1996) and Weir (2005). Finally, almost all empirical research on test validity the researcher has addressed concerns international and national high-stakes proficiency tests. The research results open gaps in test validation research for the future.

Measuring language ability is an inseparable part of any language education system or any institution that involves the use of a foreign/second language. This measurement serves many different purposes, including but not limited to evaluating learning and predicting ability in future contexts where a language is the primary channel of communication. Traditionally, tests of language ability treated language as comprising four separate skills, i.e., reading, listening, speaking, and writing. However, this view was replaced by a more integrated view of skills and resulted in tests that included tasks that involved two or more of these skills. For instance, to test speaking ability, some tasks ask test takers to read a passage, listen to a lecture, and synthesize information from these two sources in summary speech. This approach makes designing tasks more involved, especially because they must reflect the underlying construct they aim to measure at the same time. A further complication in designing these tests stems from the fact that there is not a single universal language ability construct that test developers can use. Each construct in the literature is born out of contextual factors, and test developers need to be aware of many constructs or be able to create or synthesize one from the existing models or constructs based on their needs. Then, they will have to operationalize this construct into test tasks that can elicit language samples and produce scores that are reflective of the construct. Apart from the test design, there is also the involved step of administering the test. While this step might be as simple as giving a multiple-choice test, sometimes it requires the coordination of a host of complex components and different people such as raters, proctors, etc. Many institutions cannot afford the time and resources required to design and administer such tests, and they might resort to using externally designed ones that might not reflect their needs. With tests like oral communication, examiners and raters must be trained to deliver and score the tests even when an external test is used. This might push some institutions to forego tests of oral communication completely or relegate this test to a low-stake one by minimizing its score impact on the overall language ability test they are implementing. The present work explored the possibility of addressing some of these challenges in the context of oral communication. More specifically, it focused on measuring interactional competence, an important aspect of oral communication (OC) construct, through a spoken dialogue system to address the challenges of choosing or synthesizing a construct, operationalizing that construct, and administering the test. This spoken dialogue system, called Shahryar, elicited 10 two-and-a-half-minute interactive oral communication samples related to topics common to university students. Shahryar also included an interactive avatar to study the effects of simulating human presence. Ninety-six speakers of English as a foreign/second language completed the discussions with Shahryar. Then, 10 human raters evaluated these samples using a four-level interactional competence rating scale developed and evaluated by the researcher, distributing the samples in a way that each sample received ratings from four to five raters. The rating scale was originally developed and evaluated in the context of a human-to-human paired oral communication task by Neiriz (2023). This rating scale has four subsections, i.e., topic management, turn management, breakdown repair, and interactive listening, and is developed based on the metaphorical conceptualization proposed by Galaczi and Taylor (2018). The scores from human raters were analyzed through a many-facet Rasch measurement, and the results indicated that the test takers could be separated into four levels of IC ability with three raters and three tasks. The qualitative analysis of the interviews with test takers and raters suggested not all aspects of IC manifested through conversations with Shahryar stemming from the nature of communication with a computer, and further modifications to the program and test taker preparation might be needed to reliability elicit all aspects of IC included in the rating scale. However, both test takers and raters found many useful aspects of Shahryar, making it a promising alternative to human-to-human OC test tasks in some contexts. The results from this study shed light on the possibility of implementing automated oral communication test delivery with a focus on IC. The findings have implications for both test developers with limited resources and language learners who lack access to affordable oral communication tests designed based on our recent understanding of oral communication ability.