Korean Test Takers' TOEIC-Speaking and OPIc Test Preparation

This paper investigates the comparability of test scores recorded for high-stakes English language Speaking Tests administered face-to-face in either a traditional centre-based mode (TM) or in an online proctored mode (OLP). The data comprise a large sample of test takers taking English language Speaking Tests at four CEFR (the 'Common European Framework of Reference for Languages') levels – B1 to C2 – via TM or OLP. The data were analysed using descriptive statistics, effect size differences and equivalence tests. While a degree of difference in scores obtained between modes was apparent at C2 level, the differences were not found to be statistically significant. The paper concludes that whether Speaking Tests are delivered in online proctored mode or in traditional face-to-face mode, test takers receive similar scores. The study confirms that mode of test delivery does not significantly affect test taker scores.

Language test preparation has often been studied within the consequential validity framework in relation to ethics, equity, fairness, and washback of assessment. The use of independent and integrated speaking tasks in the TOEFL iBT® test represents a significant development and innovation in assessing speaking ability in academic contexts. Integrated tasks that involve synthesizing and summarizing information presented in reading and listening materials have the potential to generate new test preparation strategies. This study investigated the experiences of over 1,500 Chinese test takers and 23 teachers who were preparing for the TOEFL iBT speaking tasks. It examined the frequency of use of a number of different test preparation activities and materials, reasons, and expectations for taking preparation courses and the features of preparation courses. In addition, we examined the usefulness of test preparation from two perspectives: students' and teachers' perceptions as well as the relationship between test preparation and performance. Data were collected via questionnaires, focus group discussions, interviews with test takers and teachers, and classroom observations. The data showed that (a) test preparation was a hugely complex, multiple‐components construct, and teaching and learning test‐taking strategies compose the most prominent feature of intensive preparation courses; (b) there were significant age‐related differences in students' preparation activities and focuses, although with small effect sizes; (c) there was a high agreement between teachers and students in their views on the usefulness of test preparation activities; and (d) there existed only a weak relationship between test preparation and performance. The only significant predictor of students' test performance was the frequency of their use of the TOEFL Practice Online TPO® practice tests. The findings of the study can enhance our understanding of the pedagogical practices that characterize test preparation programs and contribute to the ongoing validity argument for the TOEFL iBT Speaking test. The implications of the findings for test publishers, test takers, teachers, and test preparation schools are discussed with reference to the instructional, learning, and affective aspects of the multifaceted construct of test preparation.

This study investigated the impact of different lengths of pre-task planning time on performance in a test of second language speaking ability for university admission. In the study, 47 Turkish-speaking learners of English took a test of English language speaking ability. The participants were divided into two groups according to their language proficiency, which was estimated through a paper-based English placement test. They each completed four monologue tasks: two picture-based narrative tasks and two description tasks. In a balanced design, each test taker was allowed a different length of planning time before responding to each of the four tasks. The four planning conditions were 30 seconds, 1 minute, 5 minutes, and 10 minutes. Trained raters awarded scores to the test takers using an analytic rating scale and a context-specific, binary-choice rating scale, designed specifically for the study. The results of the rater scores were analysed by using a multifaceted Rasch measurement. The impact of pre-task planning on test scores was found to be influenced by four variables: the rating scale; the task type that test takers completed; the length of planning time provided; and the test takers’ levels of proficiency in the second language. Increases in scores were larger on the picture-based narrative tasks than on the two description tasks. The results also revealed a relationship between proficiency and pre-task planning, whereby statistical significance was only reached for the increases in the scores of the lowest-level test takers. Regarding the amount of planning time, the 5-minute planning condition led to the largest overall increases in scores. The research findings offer contributions to the study of pre-task planning and will be of particular interest to institutions seeking to assess the speaking ability of prospective students in English-medium educational environments.

Of the many possible institutional and individual factors bearing on test preparation, one is how individuals go about choosing their achievement goals. Yet, the literature on the relationship between the two phenomena remains slim. The objectives of this study are twofold. First, it explores the range of test preparation practices exercised by test takers in preparing for the English module of the Higher Education Admission Test in Iran. Secondly, it investigates how individual goal orientations mediate test preparation. A goal orientations scale was translated, validated and administered to the participants, who were 357 test candidates, a convenience sample. The participants also completed a test preparation questionnaire with two underlying factors including desired and undesired test preparation practices. Descriptive statistics and paired samples t-tests revealed that preparation for the Higher Education Admission Test entailed a mix of both detrimental and beneficial practices, with the frequency of the former being significantly higher. It was also revealed that mastery goal orientations are associated with educationally defensible test preparation practices. Findings carry implications for testers, test preparation instructors and educational policy makers.

Ethical issues in language testing have become one of the major research topics and challenges in the field of language testing in the 21st century. Ethics refers to the normative rules governing relationships and behaviors among individuals in human society, as well as the behavior order among people, society, and nations. Any group or professional behavior that affects society as a whole carries inherent ethical requirements. The exploration of ethical issues in language testing began to gain prominence in the field after Messick expanded the concept of validity in language testing. With the development of language testing, researchers have increasingly focused on the study of testing ethics. In 2000, the International Test Commission (ITC) issued a set of “Ethical Guidelines”, where ethical issues were reflected to varying degrees in various principles. In 2007, based on the specific implementation of the “Ethical Guidelines”, the ITC issued corresponding “Code of Conduct” to define the responsibilities and obligations of language testers and test takers. In essence, ethical issues in language testing fall within the scope of validity in language testing. This paper, starting from the current status of research on ethical issues in language testing in China, which is still in its infancy in both theory and practice, uses literature review, descriptive research, and qualitative analysis. Within the framework of validity, it discusses the factors involved in ethical issues in language testing: development of teaching and examination syllabus, test item production, exam preparation and administration, organization of scoring and score interpretation, result statistical analysis, feedback, and summarization. Based on the “Ethical Guidelines” and the corresponding “Code of Conduct” issued by the International Test Commission, it summarizes the causes of ethical issues and proposes corresponding solutions for each phase. Finally, some suggestions are made for the development of ethical issues in language testing.

Several analyses are presented on the relationships among the Test of Spoken English (TSE), the Test of Written English (TWE), and the Test of English as a Foreign Language (TOEFL). The multivariate prediction of each of these tests from the scores on the others is very accurate. Factor analysis reveals two prominent factors. One accounts mainly for oral proficiencies measured by the TSE, and the other accounts for the abilities measured by the TOEFL and the TWE. Scores on the TOEFL listening comprehension, TSE Grammar, TSE overall comprehensibility and the TWE share considerable variance with both factors. The factors may be related to different cognitive skills required to answer different types of questions.

This article examines the relationship between test administration tools and item types. It firstly examines the difference in the test administration tools such as paper-based test, computer-based test, and web-based test, in particular, dealing with speaking test. The difference is relevant to item or task types. Based on the considerations of Bachman and Palmer (1996) and Chapelle and Douglas (2006) on test development, and Weir's (2005) definition of context validity, the application of computer technologies to test administration needs to be considered an important construct affecting test takers' performance. To investigate the item type difference among current speaking tests and its relationship with computer technology in administrating the tests, twelve test takers, five proctors, and two computer lab administrator participated in the phone interview and e-mail survey. Six popular speaking tests (TSE, PELT, IELTS, MATE, iBT TOEFL, ESPT) were selected to be analyzed. The data were analyzed in terms of item types, computer technology use, and test-takers perceptions. The implication is suggested for students, teachers, and language testers.

ABSTRACTThis study investigated the effect of the interaction between test takers' background knowledge and language proficiency on their performance on the TOEFL iBT® reading section. Test takers with the target content background knowledge (the focal groups) and those without (the reference groups) were identified for each of the 5 selected passages based on their self‐identified academic and cultural backgrounds. The test takers were further classified into high and low proficiency groups based on their TOEFL iBT scores. Differential functioning was investigated at the item, item bundle, and passage levels. The results suggested that background knowledge interacted with language proficiency on certain items, which could be attributed to idiosyncratic passage and item characteristics (i.e., characteristics that were specific to a particular passage or item). Only 1 of the 5 passages investigated showed intermediate differential bundle functioning, favoring the focal group for both the high and low proficiency groups. There was no differential functioning at the passage level. This research sheds new light on our understanding of the effects of background knowledge and its interaction with language proficiency in the context of second language reading comprehension. It also has significant practical implications for test developers in advancing fair assessments.

ABSTRACTThe current study explores the nature and relationships among test takers’ performance appraisals, appraisal calibration, and reported cognitive and metacognitive strategy use in a language test situation. Performance appraisals are executive processes of strategic competence for judging test performance (e.g., evaluating the correctness or appropriateness of responses to given test tasks). Appraisal calibration denotes a perfect match between appraisal confidence and actual test performance. 294 English as a foreign language (EFL) students took an English test, which was designed to measure four language areas (listening, grammar, vocabulary, and reading). The students reported their level of appraisal confidence immediately after answering each test question. At the end of the test, they were asked to report their overall appraisal confidence and perceived cognitive and metacognitive strategy use in the test. First, it was found that test takers were not well calibrated in all test sections. Second, test takers’ appraisal confidence could predict just above one third of the test performance variance. Third, test takers tended to be underconfident in easy questions but overconfident in difficult questions. Fourth, appraisal calibration was not strongly related to reported metacognitive strategy use. The implications of the study in light of strategic competence research and recommendations for further research are discussed.

A proper speaking test may pose challenges to teachers due to suboptimal assessment literacy and the laborious administration of a speaking test. Technology can help alleviate the labor while enabling teachers to ascertain the accuracy and validity of the speaking test. In this direction, this study aims to develop a videoconferenced English-speaking test (henceforth VEST) with the aid of videoconferencing technology. Aligned with the empirics on the use of technology for speaking assessment, this study addresses the gapping void in how videoconferencing technology can be harnessed to escalate the practicality and positive impacts of a classroom-based speaking test in an Indonesian EFL setting. The development adhered to the ADDIE framework involving Analysis, Design, Development, Implementation, and Evaluation. The test prototype was evaluated using a test usefulness analysis framework. The evaluation results were coupled with reflection results of test takers’ experiences to guide test revision. Given the small sample size, this study has unveiled the potential of VEST to improve the praxis of speaking assessment and the resultant test takers’ experience. Implications and recommendations for future studies are also discussed.

The purpose of this preliminary study is to examine whether there is a relationship between the achievement scores and two different test formats used in speaking assessment – paired or individual. The study also attempts to explore the test takers’ preferences pertaining to the test formats employed and the reasons behind. The participants of this study are 25 freshman students studying in the English Language Teaching department of a well-known state university in Turkey. The data used in the quantitative analysis comes from the grades scored by the same cohort of students in two oral proficiency exams conducted within the scope of course assessment during the regular exam period of the program. Secondly, the qualitative data was obtained from the semi structured interviews carried out with 14 of the 25 students immediately after the paired speaking test. The interview questions specifically sought to reveal how the participants felt about the paired format of speaking test, how they would compare this format to the individual format, and if they were given the chance which format they would prefer and why. We have come to see that students got better scores in the paired speaking test format that we designed in the form of paired discussion in comparison to the individual format where the test takers’ interlocutor is the course lecturer. The difference in the mean scores obtained in the two different formatted exams was found statistically significant. Our qualitative findings showed that the majority of the students liked and supported the practice of being paired. Article visualizations:

In this study we examined the extent to which TOEIC® Speaking test scores relate to evaluations by professionals in the international workplace, the target language use domain of TOEIC tests. Linguistic laypersons in 10 countries were invited to participate in an online research survey. The survey incorporated a stratified sample of test-taker ( N = 99) responses to three representative tasks from the TOEIC Speaking test (reading a text aloud, responding to questions, expressing an opinion) that were cast as workplace role-play tasks. After completing each role-play task, participants used brief, descriptive six-point rating scales to rate the communicative effectiveness (comprehensibility, task fulfillment, elaboration, and coherence) of each of several speakers. Communicative effectiveness ratings from linguistic laypersons were strongly correlated with TOEIC Speaking test scaled scores ( r = 0.84). In addition, regression analysis was used to plot the relationship between layperson and test-based evaluations of speaking proficiency. Results suggested that test takers’ performances can be expected to be perceived as effective at score ranges typically associated with important decisions. The results are discussed in terms of their implications for claims about the generalizability of TOEIC Speaking test score interpretations in relation to the evaluations of linguistic laypersons in the international workplace.

This paper adds an organizational dimension to the body of literature on international student mobility. Existing studies examine push/pull factors and student motivations, neglecting that students’ motivations and demands are not necessarily spontaneous, but can be shaped by external forces. Drawing on interview, archival and observation data collected on four leading cram schools that prepare students for the TOEFL/GRE, IELTS and SAT in China, I argue that cram schools not only coach students on test preparation and “how to study abroad,” but they also adopt organizational framing to instill in students “why to study abroad.” Leading cram schools have played an integral role in promoting a certain organizational framing as the dominant approach of a niche market in a given era. During the 1990s, when the TOEFL/GRE niche market was rapidly expanding, the market leader in this niche promoted self-help and nationalism as dominant discourses. Self-help discourse frames overseas study and test preparation not as means, but as the ends of students’ lives: going beyond one’s limit and making one’s life complete. Nationalist discourse depicts overseas study as a detour to build a stronger China after learning from the West. After 2000, however, new organizational framing picked up momentum in the new niche markets of IELTS and SAT. Targeting urban middle-class consumers, market leaders in these new niches increasingly framed studying abroad as a springboard for immigration, a channel for becoming global elites and an opportunity for status improvement for the entire family. My article bridges literature on transnational higher education with studies on supplementary education.

Validity and fairness implications of varying time conditions on a diagnostic test of academic English writing proficiency

The rapid change and use of computer technology has had a profound influence in the field of language teaching and assessment, and the changes that computer and technology has brought to language assessment in particular are not only in the delivery method, but also in the scoring systems (Chapelle & Douglas, 2006). For instance, a wide variety of tasks be given as a test online worldwide, and scores can be available immediately to the test takers. However, along with such advances come potential problems and concerns.