ABSTRACT It is inevitable that second language (L2) learners often resort to test-wiseness (TW) strategies in reading tests to obtain higher scores. Previous research has reported mixed findings on whether TW strategies enhance L2 reading performance, but such strategies are often treated as a single, undifferentiated category. The aim of this study was to categorize TW strategies into distinct types and investigate their effects on L2 reading performance. Participants were 531 Taiwanese university students who completed a Test of English for International Communication (TOEIC) reading test and a TW strategy questionnaire to self-evaluate strategy use frequency. In the questionnaire, we divided TW strategies into four categories: time-using, error avoidance, deductive reasoning, and cue-using strategies. We then used Item Factor Analysis and structural equation modeling to examine the relationship between different types of TW strategies and students’ TOEIC reading scores. Results showed that error avoidance strategies were the only type with a positive effect on reading performance. While deductive reasoning strategies could provide shortcuts to potentially determine the correct answer, our results indicated that over-reliance on such strategies may negatively impact performance. The implications for L2 teaching and future research are also discussed.

ABSTRACT In response to the increasing importance of multimodal communication in the academic target language use (TLU) domain and the expanding construct of English for academic purposes (EAP) skills, the current study proposes a multimodal EAP test. The study consists of three stages: (i) construct definition, (ii) design of a digital multimodal test, and (iii) empirical validation. The research questions address the extent to which the new test design reflects the construct of multimodal EAP skills in relation to two aspects: language functions (as evidenced in test takers’ performance) and the perspectives of test takers and examiners. The findings showed that 79% of the target multimodal EAP language functions were observed in the dataset. This indicates that multimodality can be successfully operationalised in EAP assessment with a cumulative, thematically linked multimodal test design. The new design goes beyond what is typically operationalised by traditional independent skill-based tests and/or linear integrated language tests. The observation checklist developed can be used to help indicate the construct elicited by multimodal EAP assessments. Implications of the findings for the operationalisation of multimodal EAP constructs in large-scale assessment are discussed.

ABSTRACT The present article employs eye tracking methodology to investigate the influence of item type (explicit and implicit) on test-taker attention to visual cues in video L2 listening tests. The findings reveal that implicit items direct test takers’ attention toward nonverbal communicative cues related to affect more than explicit items, and that explicit items may direct more attention to illustrative gestures and objects. These findings raise questions about the construct addressed by implicit questions in multimodal video listening tests and call for careful consideration of test purpose when designing L2 listening tests.

ABSTRACT In the Editorial of this special issue – Multimodality in second language assessment, we will introduce the concept of multimodality in language education, in order to set the scene for the special issue. We will then summarise the key findings of the nine papers included in the special issue and highlight the key findings and the research methods and features of multimodal tasks used in these studies, to identify gaps in research and practice. We will also reflect on what we have achieved through this special issue and propose future directions for implementing and researching multimodality in second language (L2) assessment.

ABSTRACT Linguistic accuracy (LA) has been shown to be indicative of second language (L2) essay quality through the use of various measurements, such as error counts and error-free T-units. However, concerns persist regarding the precise definition of LA, as findings in recent literature often depend on the specific measures of LA employed. This study addresses this gap by using factor analysis and interviews to explore the dimensionality of LA in L2 writing assessments. The results revealed that current accuracy measures loaded onto two distinct factors, namely form accuracy and meaning accuracy, with cross-loadings observed for total errors per 100 words and the weighted clause ratio. Additionally, it was found that the quantity and type of errors differently impeded meaning comprehensibility. These findings support the bidimensionality of LA. This paper advances our understanding of the construct of LA and offers implications for L2 writing assessments and teaching.

ABSTRACT This quasi-experimental study examines the impact of a diagnosis-based pedagogical intervention on the development of micro English as a foreign language (EFL) reading abilities over time. Specifically, 105 first-year graduate students in a Chinese university completed three diagnostic reading assessments at the beginning, middle, and end of a semester. The reading assessments were developed based on a diagnostic language assessment (DLA) approach. While all students received standardized feedback generated from DLA, they were divided into three groups (i.e. self-learning, peer-collaborative learning, and teacher-guided learning) to receive different types of pedagogical intervention to help them incorporate the feedback. Results from linear mixed-effects models reveal that while all three groups’ reading ability improved on the micro abilities of summarizing and inferring, students from the peer-collaborative and teacher-guided groups outperformed those from the self-learning group on the other micro abilities (i.e. analyzing, differentiating, and locating target information across the three assessment occasions). The findings of this study have both theoretical and pedagogical implications regarding the validity of DLA, as well as the training and development of EFL reading comprehension abilities in the classroom.

ABSTRACT This article compares two methods for detecting local item dependence (LID): residual correlation examination and Rasch testlet modeling (RTM), in a commonly used 3:6 matching format and an extended matching test (EMT) format. The two formats are hypothesized to facilitate different levels of item dependency due to differences in the number of options and instructions regarding option recycling. The findings indicate that (1) RTM allows deeper LID inspection compared to residual correlation examination in matching tests, and (2) the EMT format has good resistance to LID while the traditional 3:6 format does not.

ABSTRACT Machines have a long-demonstrated ability to find statistical relationships between qualities of texts and surface-level linguistic indicators of writing. More recently, unlocked by artificial intelligence, the potential of using machines to identify content-related writing trait criteria has been uncovered. This development is significant, especially in formative assessment contexts where feedback is key. Yet the extent to which writing traits can be validly scored by machines remains under-researched, especially in the K-12 context. The present study investigated the validity of machine learning (ML) models designed for students in grades 3–6 to score three writing traits: task fulfillment, organization and coherence, and vocabulary and expression. The study utilized an argument-based approach, focusing on two primary inferences: evaluation and explanation. The evaluation inference investigated human-machine score alignment, the ability for the models to detect off-topic and gibberish responses, and the consistency of human-machine score alignment across grades and language backgrounds. The explanation inference investigated the relevance of features used in the models. Results indicated that human-machine score alignment was sufficient for all writing traits; however, validity concerns were raised regarding the models’ performances detecting off-topic and gibberish responses and the consistency across sub-groups. Implications for language assessment professionals and other educators were discussed.