Commentaries on Computer-based Language Assessment

The purpose of this preliminary study is to examine whether there is a relationship between the achievement scores and two different test formats used in speaking assessment – paired or individual. The study also attempts to explore the test takers’ preferences pertaining to the test formats employed and the reasons behind. The participants of this study are 25 freshman students studying in the English Language Teaching department of a well-known state university in Turkey. The data used in the quantitative analysis comes from the grades scored by the same cohort of students in two oral proficiency exams conducted within the scope of course assessment during the regular exam period of the program. Secondly, the qualitative data was obtained from the semi structured interviews carried out with 14 of the 25 students immediately after the paired speaking test. The interview questions specifically sought to reveal how the participants felt about the paired format of speaking test, how they would compare this format to the individual format, and if they were given the chance which format they would prefer and why. We have come to see that students got better scores in the paired speaking test format that we designed in the form of paired discussion in comparison to the individual format where the test takers’ interlocutor is the course lecturer. The difference in the mean scores obtained in the two different formatted exams was found statistically significant. Our qualitative findings showed that the majority of the students liked and supported the practice of being paired. Article visualizations:

Advances in natural language processing (NLP) and automatic speech recognition and processing technologies offer new opportunities for language testing. Despite their potential uses on a range of language test item types, relatively little work has been done in this area, and it is therefore not well understood by test developers, researchers or users in language assessment. This paper introduces NLP for language assessment as an area of inquiry and practice by describing the historical roots coming from computational linguistics, statistical NLP, speech recognition and processing technologies, language assessment, and computer-assisted language learning. It outlines uses of NLP and speech recognition and processing technologies in language assessment through illustrations of current testing projects, and identifies areas in need of further development.

This comprehensive review explores the utilization of Artificial Intelligence (AI) in language assessment and its transformative potential. Language assessment plays a vital role in education, employment, and societal integration. Language assessment has traditionally relied on human evaluators, who assess and score language performance based on standardized criteria. However, this manual assessment approach has limitations, including subjectivity, inter-rater variability, and scalability issues. With the rapid advancements in AI technologies, language assessment has experienced significant changes, offering more accurate, efficient, and innovative evaluation methods. The method used in this study is systematic review which critically examines the application of AI in language assessment, highlighting its benefits, challenges, and future prospects. The review covers various aspects, including automated scoring and evaluation, intelligent tutoring systems, natural language processing, advantages and benefits, challenges and considerations, and future directions. By harnessing the use of AI, language assessment can achieve unprecedented levels of objectivity, scalability, and personalization, while addressing ethical considerations. This review contributes to the understanding of AI's impact on language

Contents: Introduction. Part I: Important Social Contexts in Research on Second Language Teaching and Learning. Introduction. M.E. Brisk, Bilingual Education. M.R. Hawkins, ESL in Elementary Education. P.A. Duff, ESL in Secondary Schools: Programs, Problematics, and Possibilities. D.E. Murray, ESL in Adult Education. S. Carkin, English for Academic Purposes. P. Master, English for Specific Purposes. C. Roberts, English in the Workplace. B. Tomlinson, English as a Foreign Language: Matching Procedures to the Context of Learning. Y. Kachru, Teaching and Learning of World Englishes. Part II: Methods in Second Language Research. Introduction. L. Harklau, Ethnography and Ethnographic Research on Second Language Teaching and Learning. L. van Lier, Case Study. A. Lazaraton, Quantitative Research Methods. D. Nunan, Classroom Research. A. Burns, Action Research. Part III: Applied Linguistics and Second Language Research. Introduction. T. Pica, Second Language Acquisition Research and Applied Linguistics. S.L. McKay, Sociolinguistics and Second Language Learning. J. Zuengler, K.M. Cole, Language Socialization and Second Language Learning. G. Kasper, C. Roever, Pragmatics in Second Language Learning. J.P. Lantolf, Sociocultural and Second Language Learning Research: An Exegesis. N. Markee, Conversation Analysis for Second Language Acquisition. R.B. Kaplan, Contrastive Rhetoric. S. Conrad, Corpus Linguistics and L2 Teaching. Part IV: Second Language Processes and Development. Introduction. G. Ioup, Age in Second Language Development. R. DeKeyser, A. Juffs, Cognitive Considerations in L2 Learning. Z. Han, L. Selinker, Fossilization in L2 Learners. M. Swain, The Output Hypothesis: Theory and Research. E. Tarone, Speaking in a Second Language. M. Rost, L2 Listening. T.G. Wiley, Second Language Literacy and Biliteracy. P. Byrd, Instructed Grammar. D.E. Eskey, Reading in a Second Language. I.S.P. Nation, Teaching and Learning Vocabulary. J.S. Hedgcock, Taking Stock of Research and Pedagogy in L2 Writing. E. Hinkel, Analyses of Second Language Text and What Can Be Learned From Them. Part V: Methods and Curricula in Second Language Teaching. Introduction. S.J. Savignon, Communicative Language Teaching: Strategies and Goals. S. Fotos, Traditional and Grammar Translation Methods for Second Language Teaching. J. Williams, Form-Focused Instruction. M.A. Snow, A Model of Academic Literacy for Integrated Language and Content Instruction. R. Ellis, Instructed Language Learning and Task-Based Teaching. M. Celce-Murcia, E. Olshtain, Discourse-Based Approaches: A New Framework for Second Language Teaching and Learning. C.A. Chapelle, Computer-Assisted Language Learning. N.J. Anderson, L2 Learning Strategies. Part VI: Second Language Testing and Assessment. T. McNamara, Introduction. A.J. Kunnan, Language Assessment From a Wider Context. A. Davies, C. Elder, Validity and Validation in Language Testing. M. Chalhoub-Deville, C. Deville, A Look Back at and Forward to What Language Testers Measure. T. Lumley, A. Brown, Research Methods in Language Testing. D. Douglas, Testing Languages for Specific Purposes. C. Leung, Classroom Teacher Assessment of Second Language Development: Construct as Practice. Part VII: Identity, Culture, and Critical Pedagogy in Second Language Teaching and Learning. Introduction. T. Ricento, Considerations of Identity in L2 Learning. M. Byram, A. Feng, Teaching and Researching Intercultural Competence. S. Canagarajah, Critical Pedagogy in L2 Learning and Teaching. Part VIII: Language Planning and Policy and Language Rights. R.B. Baldauf, Jr., Introduction. R.B. Baldauf, Jr., Language Planning and Policy Research: An Overview. T. van Els, Status Planning for Learning and Teaching. A.J. Liddicoat, Corpus Planning: Syllabus and Materials Development. R.B. Baldauf, Jr., R.B. Kaplan, Language-in-Education Planning. D.E. Ager, Prestige and Image Planning. S. May, Language Planning and Minority Language Rights.

High stakes English speaking tests have been used for evaluating applicants’ language proficiency in academic contexts or in the work place. In these situations, most Korean college students prepare for tests such as TOEIC Speaking or OPIc test in order to get better jobs. Although high-stakes tests pressure students and teachers to raise scores, there is little research about how test takers prepare for English speaking tests in cram schools. One aim of this study was to examine and compare TOEIC Speaking and OPIc test preparation activities in a cram school. Also, I aimed to investigate why and how test takers choose test preparation, the characteristics of test preparation, and how they are influenced by test preparation from the test-takers’ perspectives. Various types of data were collected through observation, interview and documentation. The results indicated that activities in the cram school mode were not significantly influenced by test type. Thus, test takers’ activities may be influenced by teacher’s instruction. The study raises awareness of the importance of instruction by a teacher for appropriate test preparation, and its impact on a test taker in the field of language teaching and testing.

Commissioned as the lead paper in the Volume, this chapter shows how the terms assessment, testing, examining and evaluation have become increasingly intertwined, and multiple new terms have emerged. The chapter takes a somewhat historical view as it lays out a picture of the ways that the field of language ‘assessment’ has expanded its knowledge and skills but also its socio-political effects. Taking the notion of ‘purposes’ as a broad organizing principle, the chapter is intended to introduce new readers in this field to the wide range of issues current in the field, and prepare them for the more specific chapters that follow.

Investigations of the validity of a number of high-stakes language assessments are conducted using an argument-based approach, which requires evidence for inferences that are critical to score interpretation (Chapelle, Enright, & Jamieson, 2008b; Kane, 2013). The current study investigates the extrapolation inference for a high-stakes test of spoken English, the Michigan English Language Assessment Battery (MELAB) speaking task. This inference requires evidence that supports the inferential step from observations of what test takers can do on an assessment to what they can do in the target domain (Chapelle et al., 2008b; Kane, 2013). Typically, the extrapolation inference has been supported by evidence from a criterion measure of language ability. This study proposes an additional empirical method, namely corpus-based register analysis (Biber & Conrad, 2009), which provides a quantitative framework for examining the linguistic relationship between performance assessments and the domains to which their scores are extrapolated. This approach extends Bachman and Palmer’s (2010) focus on the target language use (TLU) domain analysis in their study of assessment use arguments by providing a quantitative approach for the study of language. We first explain the connections between corpus-based register analysis and TLU analysis. Second, an investigation of the MELAB speaking task compares the language of test-taker responses to the language of academic, professional, and conversational spoken registers, or TLU domains. Additionally, the language features at different performance levels within the MELAB speaking task are investigated to determine the relationship between test takers’ scores and their language use in the task. Following previous studies using corpus-based register analysis, we conduct a multi-dimensional (MD) analysis for our investigation. The comparison of the language features from the MELAB with the language of TLU domains revealed that support for the extrapolation inference varies across dimensions of language use.

Computer-based testing has been widely used across many testing environments, serving both high-stakes and low-stakes contexts. Its global use calls for the language assessment community’s attention to reevaluate some fundamental issues, in particular the extent to which these test scores are reliable, consistent, and meaningful. In light of these needs, Sadeghi and Douglas’s recently published edited volume Fundamental Considerations in Technology-Mediated Language Assessment provides a timely treatment on this topic, exploring not only critical issues in computer-based testing but also topics related to the implementation of technology in other aspects of language assessment. This book will appeal to a broader audience beyond just professionals in the field of language assessment because it offers research-based and theoretical-based knowledge for professionals, teachers, and policymakers interested in exploring and developing computer-based tests. [First Paragraph of Review]

The assessment of language in early childhood is essential for the early identification of children with special needs. However, administering traditional language assessments to large preschool populations can be prohibitively time-consuming and complicated. Thus, there is clear value in developing a standardized, norm-referenced, computer-administered language assessment battery that is both time-efficient and fun for children, that yields a meaningful profile of children's specific language competencies, and that can be administered by testers in a consistent manner without extensive training. Here we discuss research undertaken as part of an evaluation of the feasibility of developing such a language assessment tool for use with preschool children. Preschoolers (M = 3.60 years) were tested using a traditional, standardized language assessment (PLS-4) and a computer-administered task assessing (via the use of a touch-screen computer) verb vocabulary and comprehension of plural morphology, negation, and noun-verb agreement. All participants completed the entire test without difficulty. Moreover, analyses revealed significant correlations between performance on the computer-based language assessment, age, and performance on the PLS-4. These data support the notion that a computer-administered language assessment is methodologically feasible and can provide a practical and valid means by which to assess early language abilities.

For this issue of Teachers College, Columbia University Working Papers in TESOL & Applied Linguists, we invited three contributors to extensively comment on validity issues in second or foreign language assessment. Jorge Beltran looks at the English Language Arts Regents Exam, a test widely used in the K-12 system in New York. He takes an analytical approach in evaluating the exam in light of current validity frameworks in order to discuss the possible threats to validity and its impact on English language learners taking the exam. Next, Andrea Durkis looks at language assessments used as part of the citizenship process in France. While reviewing its history over the past twenty years, she examines validity issues through the lens of Kane’s framework. Finally, Heidi Liu Banerjee further explores fairness, an important test quality essential to the discussion of validity and its conceptualization in relation to Kane’s framework.

This article examines the relationship between test administration tools and item types. It firstly examines the difference in the test administration tools such as paper-based test, computer-based test, and web-based test, in particular, dealing with speaking test. The difference is relevant to item or task types. Based on the considerations of Bachman and Palmer (1996) and Chapelle and Douglas (2006) on test development, and Weir's (2005) definition of context validity, the application of computer technologies to test administration needs to be considered an important construct affecting test takers' performance. To investigate the item type difference among current speaking tests and its relationship with computer technology in administrating the tests, twelve test takers, five proctors, and two computer lab administrator participated in the phone interview and e-mail survey. Six popular speaking tests (TSE, PELT, IELTS, MATE, iBT TOEFL, ESPT) were selected to be analyzed. The data were analyzed in terms of item types, computer technology use, and test-takers perceptions. The implication is suggested for students, teachers, and language testers.

Coreference resolution is the problem of determining which mention in a text refer to the same entities, and is a crucial and difficult step in every natural language processing task. Despite the efforts that have been made in the past to solve this problem, its performance still does not meet today’s applications requirements. Given the importance of the verbs in sentences, in this work we tried to incorporate three types of their information on coreference resolution problem, namely, selectional restriction of verbs on their arguments, semantic relation between verb pairs, and the truth that arguments of a verb cannot be coreferent of each other. As a needed resource for supporting our model, we generate a repository of semantic relations between verb pairs automatically using Distributional Memory (DM), a state-of-the-art framework for distributional semantics. This resource consists of pairs of verbs associated with their probable arguments, their role mapping, and significance scores based on our measures. Our proposed model for coreference resolution encodes verbs’ knowledge with Markov logic network rules on top of deterministic Stanford coreference resolution system. Experiment results show that this semantic layer can improve the recall of the Stanford system while preserves its precision and improves it slightly.

Studies from various disciplines have reported that spatial location of options in relation to processing order impacts the ultimate choice of the option. A large number of studies have found a primacy effect, that is, the tendency to prefer the first option. In this paper we report on evidence that position of the key in four-option multiple-choice (MC) listening test items may affect item difficulty and thereby potentially introduce construct-irrelevant variance. Two sets of analyses were undertaken. With Study 1 we explored 30 test takers’ processing via eye-tracking on listening items from the Aptis Test. An unexpected finding concerned the amount of processing undertaken on different response options on the MC questions, given their order. Based on this, in Study 2 we looked at the direct effect of key position on item difficulty in a sample of 200 live Aptis items and around 6000 test takers per item. The results suggest that the spatial location of the key in MC listening tests affects the amount of processing it receives and the item’s difficulty. Given the widespread use of MC tasks in language assessments, these findings seem crucial, particularly for tests that randomize response order. Candidates who by chance have many keys in last position might be significantly disadvantaged.

Large-scale academic English language tests are often criticized for not reflecting the rich variation in the English language (Davidson 2009). While this issue has been raised in the context of world Englishes, it is relevant to the consideration of English as a lingua franca (ELF) as well.1 With the growing interest in how the notion of ELF affects English language instruction, learning and assessment, a relevant question that deserves close investigation is how we might go about assessing the English ability required for communication in a globalizing society. McNamara (2014) argues that adequately addressing this issue poses a major challenge to the current conceptualization of communicative language testing. Heated debates about this issue have continued between proponents of ELF and language assessment researchers for about a decade now (e.g. Canagarajah 2009; Elder and Davies 2006; Elder and Harding 2008; Jenkins 2006; Jenkins and Leung 2013; Taylor 2006). While the viewpoints expressed by these two groups have not often resonated with each other, collaboration between them to explore how to assess ELF is essential. This is because the demand for an English language assessment that can provide an accurate estimate of what a test taker would be able to do in the English language in real-life ELF communication is expected to grow with further increase of ELF speakers in the future.

Language assessment literacy has emerged as an important area of research within the field of language testing and assessment, garnering increasing scholarly attention. However, the existing literature on language assessment literacy primarily focuses on teachers and administrators, while students, who sit at the heart of any assessment, are somewhat neglected. Consequently, our understanding of student language assessment literacy and its impact on learning remains limited. Moreover, previous language assessment literacy research has been predominantly situated in classroom assessment contexts, and relatively little scholarly attention has been directed to large-scale testing contexts. To address these gaps, this study investigated the role of language assessment literacy in students’ learning for the writing section of the Duolingo English Test (DET) through a netnographic approach. Twenty-three online videos posted by test takers were analyzed with reference to the existing conceptualizations to examine learners’ language assessment literacy of the DET writing section. Highlighting learners’ voices, we propose a new model relating writing assessment literacy to learning which has the potential to develop the learner-centered approach in language assessment literacy research. It elucidates the internal relationships among different dimensions of students’ language assessment literacy and their impacts on the learning of writing. We, therefore, discussed the findings of this study to argue for the importance of the transparency of assessment and the opportunities to learn provided by large-scale assessments, and to call for teachers' attention to students' language assessment literacy and understanding of the writing construct.