Interactional competence has been variously defined as turn-taking ability, paralinguistic features of communication such as eye contact, gesture, and gesticulation, and listener responses. In existing assessment systems such as the oral proficiency interview (OPI), interactional competence is only rarely explicitly factored into the holistic task-based rating system. The present article explores the potential relevance of a facet of interactional competence, listener response, in contrastive interviewers conducted structurally distinct languages, Japanese and English. The analytic focus through micro-analysis of the interview interaction aims to consider evidence of how the candidate’s listener responses audible through backchannels might be consistently identified as distinct from existing rating criteria such as fluency, accuracy, and coherence, and whether listener responses as interactional competence might be distinct from, or subsumable under, these facets of speaker proficiency.

All educational testing is intended to have consequences, which are assumed to be beneficial, but tests may also have unintended, negative consequences (Messick, 1989). The issue is particularly important in the case of large-scale standardized tests, such as Australia’s National Assessment Program - Literacy and Numeracy (NAPLAN), the intended benefits of which are increased accountability and improved educational outcomes. The NAPLAN purpose is comparable to that of other state and national ‘core skills’ testing programs, which evaluate cross-sections of populations in order to compare results between population sub-groupings. Such comparisons underpin ‘accountability’ in the era of population-level testing. This study investigates the impact of NAPLAN testing on one population grouping that is prominent in the NAPLAN results’ comparisons and public reporting: children in remote Indigenous communities. A series of interviews with principals and teachers documents informants’ first-hand experiences of the use and effects of NAPLAN in schools. In the views of most participants, the language and content of the test instruments, the nature of the test engagement, and the test washback have negative impacts on students and staff, with little benefit in terms of the usefulness of the test data. The primary issue is the fact that meaningful participation in the tests depends critically on proficiency in Standard Australian English (SAE) as a first language. This study contributes to the broader discussion of how reform-targeted standardized testing for national populations affects sub-groups who are not treated equitably by the test instrument or reporting for accountability purposes. It highlights a conflict between consequential validity and the notion of accountability that drives reform-targeted testing.

Research investigating corpora of English learners’ language raises new questions about how syntactic complexity is defined theoretically and operationally for second language (L2) writing assessment. I show that syntactic complexity is important in construct definitions and L2 writing rating scales as well as in L2 writing research. I describe the operationalizations of syntactic complexity measurement in corpus-based L2 writing research, focusing on the Biber Tagger (Biber, Johansson, Leech, Conrad, & Finegan, 1999), Coh-Metrix (McNamara, Graesser, McCarthy, & Cai, 2014), and L2 Syntactic Complexity Analyzer (Lu, 2010), which are three tools commonly used to automate syntactic complexity analysis. A review of findings from recent corpus-based L2 writing studies on the relationship of syntactic complexity to L2 writing quality follows. I conclude with a discussion of the implications of these multiple perspectives on the definition of syntactic complexity in L2 studies.

Investigations of the validity of a number of high-stakes language assessments are conducted using an argument-based approach, which requires evidence for inferences that are critical to score interpretation (Chapelle, Enright, & Jamieson, 2008b; Kane, 2013). The current study investigates the extrapolation inference for a high-stakes test of spoken English, the Michigan English Language Assessment Battery (MELAB) speaking task. This inference requires evidence that supports the inferential step from observations of what test takers can do on an assessment to what they can do in the target domain (Chapelle et al., 2008b; Kane, 2013). Typically, the extrapolation inference has been supported by evidence from a criterion measure of language ability. This study proposes an additional empirical method, namely corpus-based register analysis (Biber & Conrad, 2009), which provides a quantitative framework for examining the linguistic relationship between performance assessments and the domains to which their scores are extrapolated. This approach extends Bachman and Palmer’s (2010) focus on the target language use (TLU) domain analysis in their study of assessment use arguments by providing a quantitative approach for the study of language. We first explain the connections between corpus-based register analysis and TLU analysis. Second, an investigation of the MELAB speaking task compares the language of test-taker responses to the language of academic, professional, and conversational spoken registers, or TLU domains. Additionally, the language features at different performance levels within the MELAB speaking task are investigated to determine the relationship between test takers’ scores and their language use in the task. Following previous studies using corpus-based register analysis, we conduct a multi-dimensional (MD) analysis for our investigation. The comparison of the language features from the MELAB with the language of TLU domains revealed that support for the extrapolation inference varies across dimensions of language use.

The present study discusses the relevance of measures of lexical diversity (LD) to the assessment of learner corpora. It also argues that existing measures of LD, many of which have become specialized for use with language corpora, are fundamentally measures of lexical repetition, are based on an etic perspective of language, and lack construct validity. The proposed solution draws from Zipf’s (1935) emic perspective of language, which views LD as a matter of perception, but which also assumes that competent speakers of a common language share similar perceptions. The present study tests whether this is true and specifically whether untrained human raters will show high levels of inter-rater reliability in their judgments of the levels of LD found in 60 texts extracted from a corpus of narratives written in English by a mix of language learners and native speakers. The results confirm Zipf’s assertion, but also indicate that a relatively large number of motivated raters are needed to demonstrate this tendency. The remainder of the study discusses the implications these results have for the development of an automated measure of LD to be used with learner corpora. The proposed method begins with human judgments of a representative subsample of a corpus, proceeds to a statistical model of objective measures that accurately predicts the human judgments, and ends with a multidimensional, corpus-specific automated measure that outputs reliable estimates of how a reliable group of human judges would rate the levels of LD in the texts of that corpus.

Over the past 45 years, the construct of syntactic sophistication has been assessed in L2 writing using what Bulté and Housen (2012) refer to as absolute complexity (Lu, 2011; Ortega, 2003; Wolfe-Quintero, Inagaki, & Kim, 1998). However, it has been argued that making inferences about learners based on absolute complexity indices (e.g., mean length of t-unit and mean length of clause) may be difficult, both from practical and theoretical perspectives (Norris & Ortega, 2009). Furthermore, indices of absolute complexity may not align with some prominent theories of language learning such as usage-based theories (e.g., Ellis, 2002a,b). This study introduces a corpus-based approach for measuring syntactic sophistication in L2 writing using a usage-based, frequency-driven perspective. Specifically, novel computational indices related to the frequency of verb argument constructions (VACs) and the strength of association between VACs and the verbs that fill them (i.e., verb–VAC combinations) are developed. These indices are then compared against traditional indices of syntactic complexity (e.g., mean length of T-unit and mean length of clause) with regard to their ability to model one aspect of holistic scores of writing quality in Test of English as a Foreign Language (TOEFL) independent essays. Indices related to usage-based theories of syntactic development explained greater variance (R2 = .142) in holistic scores of writing quality than traditional methods of assessing syntactic complexity (R2 = .058). The results have important implications for modeling syntactic sophistication, L2 writing assessment, and AES systems.

Research was conducted during the delivery of a series of workshops on language assessment with Haitian teachers in the spring of 2013. The final products of these workshops were several revised national English examinations presented to the Haitian Ministry of Education and Professional Training (MENFP). The research goal was to examine the language assessment literacy (LAL) development of both teachers and language assessment specialists during this collaboration. Data included the compiled feedback from Haitian teachers on draft examinations during the workshops, as well as survey and interview responses immediately following the workshops. Results reveal the complementary expertise of teachers and specialists, which facilitated LAL development by both parties. Results also identified challenges in collaborative decision making and consensus building to be addressed in future projects.

It is general practice to use rater judgments in speaking proficiency testing. However, it has been shown that raters’ knowledge and experience may influence their ratings, both in terms of leniency and varied focus on different aspects of speech. The purpose of this study is to identify raters’ relative responsiveness to fluency and linguistic accuracy in an occupational context, and to investigate whether professional and non-professional raters with a broad exposure to L2 speech demonstrate similar responsiveness to these two aspects. To this end, an experimental approach was applied. Fluency and accuracy were separated and systematically manipulated. As it is known that foreign accentedness of speech influences raters’ judgments, this factor was accounted for. Seventeen responses to a Dutch L2 exam in a vocational context were converted into four different versions manipulated for morpho-syntactical accuracy and/or fluency, and read by a Dutch L2 actor, resulting in 68 stimuli. Fifty-five professional raters and 41 non-trained, potential stakeholders holistically rated all stimuli. All raters had extensive prior exposure to spoken L2 Dutch. Linear mixed modeling showed that improvement of either fluency or accuracy led to significantly higher ratings by both linguistically trained and non-trained raters. This finding confirms that both groups perceive these aspects to be important features of speaking proficiency. Raters seemed to be more responsive to improvement of accuracy than of fluency. The linguistically non-trained raters seemed to appreciate the fluency improvement more than linguistically trained raters. The linguistically trained raters rewarded morpho-syntactical improvement relatively higher than the non-trained raters. This latter effect was explained by the finding that the linguistically trained raters seemed to be more preoccupied with accuracy, according to their responses to a questionnaire. This result suggests that raters with linguistic expertise were more attentive to accuracy whereas non-trained raters were relatively more attentive to fluency.

The present paper looks at the issue of standardization in L2 oral testing. Whereas external examiners are frequently used globally, some countries opt for test-takers’ own teachers as examiners instead. In the present study, Sweden is used as a case in point, with a focus on the mandatory, high-stakes, summative, ninth-grade national test in English (speaking part). The national test has the typical characteristics of standardized tests and its main objective is to contribute to equity in assessment and grading on a national level. However, using teachers as examiners raises problems for standardization. The aim of this study is to examine teachers’/examiners’ practices and views regarding four aspects of the speaking test – test-taker grouping, recording practices, the actual test occasion, and examiner participation in students’ test interactions – and to discuss findings in relation to issues concerning the normativity and practical feasibility of standardization, taking the perspectives of test-takers, teachers/examiners, and test constructors into account. In order to answer research questions linked to these four aspects of L2 oral testing, self-report survey data from a random sample of teachers ( N = 204) and teacher interviews ( N = 11) were collected and quantitative data were analyzed using inferential statistics. Survey findings revealed that despite thorough instructions, teacher practices and views vary greatly across all aspects, which was further confirmed by interview data. Three background variables – teacher certification, work experience, gender – were investigated to see whether they could provide explanations. Whereas certification and gender did not contribute significantly to explaining the findings, work experience bore some relevance, but effect sizes were generally small. The study concludes that using teachers as examiners is a well-functioning procedure in terms of assessment for learning, but raises doubts regarding assessment of learning and standardization; a solution for test authorities could be to frame the test as non-standardized.