The relationship between speaking test administration tools and item types

In 2004 national educational standards for chemistry were implemented in Germany. While the standards describe different competencies to be reached after grade 10, no compulsory contents are defined. The contents are defined by the different German states individually. This means that there are no defined common topics taught in all states that can be used for test items for the evaluation of the standards. As a result, two different types of test items are developed based on a model of competence. One type of items contains chemical content information that can be used to solve the task. The other type of items does not contain this information. The two types of items have different advantages and disadvantages; the first type are more dependent on the reading ability of the students, but their use leads to a fairer test across the different states, whereas the second type are more related to chemical content knowledge but are less fair. The two types of items are compared according to the item difficulty, correlations to reading skills, intelligence and chemical content knowledge. The results of this study can be generalized to other assessments using a similar design.

Computers have had a tremendous impact on assessment practices over the past half century. Advances in computer technology have substantially influenced the ways in which tests are made, administered, scored, and reported to examinees. These changes are particularly evident in computer-based testing, where the use of computers has allowed test developers to re-envision what test items look like and how they are scored. By integrating technology into assessments, it is increasingly possible to create test items that can sample as broad or as narrow a range of behaviors as needed while preserving a great deal of fidelity to the construct of interest. In this article we review and illustrate some of the current technological developments in computer-based testing, focusing on novel item formats and automated scoring methodologies. Our review indicates that a number of technological innovations in performance assessment are increasingly being researched and implemented by testing programs. In some cases, complex psychometric and operational issues have successfully been dealt with, but a variety of substantial measurement concerns associated with novel item types and other technological aspects impede more widespread use. Given emerging research, however, there appears to be vast potential for expanding the use of more computerized constructed-response type items in a variety of testing contexts.

High stakes English speaking tests have been used for evaluating applicants’ language proficiency in academic contexts or in the work place. In these situations, most Korean college students prepare for tests such as TOEIC Speaking or OPIc test in order to get better jobs. Although high-stakes tests pressure students and teachers to raise scores, there is little research about how test takers prepare for English speaking tests in cram schools. One aim of this study was to examine and compare TOEIC Speaking and OPIc test preparation activities in a cram school. Also, I aimed to investigate why and how test takers choose test preparation, the characteristics of test preparation, and how they are influenced by test preparation from the test-takers’ perspectives. Various types of data were collected through observation, interview and documentation. The results indicated that activities in the cram school mode were not significantly influenced by test type. Thus, test takers’ activities may be influenced by teacher’s instruction. The study raises awareness of the importance of instruction by a teacher for appropriate test preparation, and its impact on a test taker in the field of language teaching and testing.

The article substantiates the directions of informatization of the modern educational process in the institution of general secondary education. The analysis of normative maintenance is carried out and the needs of science, education and culture in informatization of educational process are defined. The purpose of the article: to substantiate the theoretical foundations of informatization of educational institutions. Research methods are used: analysis of scientific developments on the problem and normative sources; generalization and systematization − to identify promising areas of informatization; generalization of semantic characteristics of directions − for formulation of conclusions. The main directions of use of computer and information technologies in the institution of general secondary education are characterized, among which Computer and information technologies as an object of study, computer as a means of teaching various disciplines, as a tool to support subject lessons and other activities, as a means of development and upbringing of the child, the use of computers to perform educational and real tasks and to implement various activities, Information Technology as a means of diagnosing various functional systems of the child's body. The following arguments regarding the use of computer technology in the educational process of the school are emphasized: computer technology contributes to the improvement of teaching and increase the effectiveness of teaching; Computer literacy should be developed as early as possible so that students get used to and can quickly adapt to the constant updating of computer technology; thanks to computer technology, including the Internet, the mobility of the educational process increases, namely: students can communicate with teachers and students of other educational institutions, as well as professionals from both domestic and other countries, without leaving their class; computer technologies promote the development of self-regulation and self-control, teach students to manage their intellectual activity. It is concluded that the advantages of informatization include the following: increased activation of students' learning activities and motivation to learn; the emphasis is shifted to the development of practical skills, the development of creative thinking, memory, etc., the creation of conditions for active cooperation, the development of creativity, to develop communication skills.

This study investigated the impact of different lengths of pre-task planning time on performance in a test of second language speaking ability for university admission. In the study, 47 Turkish-speaking learners of English took a test of English language speaking ability. The participants were divided into two groups according to their language proficiency, which was estimated through a paper-based English placement test. They each completed four monologue tasks: two picture-based narrative tasks and two description tasks. In a balanced design, each test taker was allowed a different length of planning time before responding to each of the four tasks. The four planning conditions were 30 seconds, 1 minute, 5 minutes, and 10 minutes. Trained raters awarded scores to the test takers using an analytic rating scale and a context-specific, binary-choice rating scale, designed specifically for the study. The results of the rater scores were analysed by using a multifaceted Rasch measurement. The impact of pre-task planning on test scores was found to be influenced by four variables: the rating scale; the task type that test takers completed; the length of planning time provided; and the test takers’ levels of proficiency in the second language. Increases in scores were larger on the picture-based narrative tasks than on the two description tasks. The results also revealed a relationship between proficiency and pre-task planning, whereby statistical significance was only reached for the increases in the scores of the lowest-level test takers. Regarding the amount of planning time, the 5-minute planning condition led to the largest overall increases in scores. The research findings offer contributions to the study of pre-task planning and will be of particular interest to institutions seeking to assess the speaking ability of prospective students in English-medium educational environments.

The rapid change and use of computer technology has had a profound influence in the field of language teaching and assessment, and the changes that computer and technology has brought to language assessment in particular are not only in the delivery method, but also in the scoring systems (Chapelle & Douglas, 2006). For instance, a wide variety of tasks be given as a test online worldwide, and scores can be available immediately to the test takers. However, along with such advances come potential problems and concerns.

Ethical issues in language testing have become one of the major research topics and challenges in the field of language testing in the 21st century. Ethics refers to the normative rules governing relationships and behaviors among individuals in human society, as well as the behavior order among people, society, and nations. Any group or professional behavior that affects society as a whole carries inherent ethical requirements. The exploration of ethical issues in language testing began to gain prominence in the field after Messick expanded the concept of validity in language testing. With the development of language testing, researchers have increasingly focused on the study of testing ethics. In 2000, the International Test Commission (ITC) issued a set of “Ethical Guidelines”, where ethical issues were reflected to varying degrees in various principles. In 2007, based on the specific implementation of the “Ethical Guidelines”, the ITC issued corresponding “Code of Conduct” to define the responsibilities and obligations of language testers and test takers. In essence, ethical issues in language testing fall within the scope of validity in language testing. This paper, starting from the current status of research on ethical issues in language testing in China, which is still in its infancy in both theory and practice, uses literature review, descriptive research, and qualitative analysis. Within the framework of validity, it discusses the factors involved in ethical issues in language testing: development of teaching and examination syllabus, test item production, exam preparation and administration, organization of scoring and score interpretation, result statistical analysis, feedback, and summarization. Based on the “Ethical Guidelines” and the corresponding “Code of Conduct” issued by the International Test Commission, it summarizes the causes of ethical issues and proposes corresponding solutions for each phase. Finally, some suggestions are made for the development of ethical issues in language testing.

ABSTRACTThe controversy over the effectiveness of coaching for standardized admission tests has been fueled by studies that have sought dichotomous answers to questions that are more appropriately posed as ones of degree. Instead of asking categorically “Does coaching work?“, researchers would seem better advised to ask “How much time and effort, devoted to what kinds of coaching experiences, produce how much improvement in test performance?”To explore some relational answers, information on test preparation activities was collected from a large representative sample of candidates who took the GRE Aptitude Test in June 1980. About 3 percent of these candidates indicated that they had attended formal coaching programs for one or more sections of the test and provided information on the length, cost, and offerer of the courses.After adjusting for differences in the background characteristics of coached and uncoached students, effects on test scores were related to the length, the cost, and the type of programs offered. The effects on GRE verbal ability scores were not significantly related to the amount of coaching that examinees received. Quantitative coaching effects appeared to increase only slightly with time, but the relationship was not statistically significant. Effects on analytical ability scores, on the other hand, were related significantly to the length of coaching programs, through improved performance on two analytical item types, analysis of explanations and logical diagrams. These item types were shown previously to be susceptible to improvement through formal instructional intervention and have since been deleted from the test.Test performance was related to the kinds of coaching programs examinees attended only for the quantitative section of the test. With respect to the possibility that some kinds of test takers might profit more than others from coaching, exploratory analyses suggested that examinees intending to pursue higher‐level degrees may have benefited more than lower‐level degree seekers. This finding is consistent with the conjecture that highly motivated test takers may achieve greater effects from coaching than less motivated examinees.Overall, the data suggest that, when compared with the two highly susceptible item types that have been removed from the GRE Aptitude Test, the test item types in the current version of the test (now called the GRE General Test) appear to show relatively little susceptibility to formal coaching experiences of the kinds considered here.

The purpose of this preliminary study is to examine whether there is a relationship between the achievement scores and two different test formats used in speaking assessment – paired or individual. The study also attempts to explore the test takers’ preferences pertaining to the test formats employed and the reasons behind. The participants of this study are 25 freshman students studying in the English Language Teaching department of a well-known state university in Turkey. The data used in the quantitative analysis comes from the grades scored by the same cohort of students in two oral proficiency exams conducted within the scope of course assessment during the regular exam period of the program. Secondly, the qualitative data was obtained from the semi structured interviews carried out with 14 of the 25 students immediately after the paired speaking test. The interview questions specifically sought to reveal how the participants felt about the paired format of speaking test, how they would compare this format to the individual format, and if they were given the chance which format they would prefer and why. We have come to see that students got better scores in the paired speaking test format that we designed in the form of paired discussion in comparison to the individual format where the test takers’ interlocutor is the course lecturer. The difference in the mean scores obtained in the two different formatted exams was found statistically significant. Our qualitative findings showed that the majority of the students liked and supported the practice of being paired. Article visualizations:

Test items created for dentistry examinations are often individually written by content experts. This approach to item development is expensive because it requires the time and effort of many content experts but yields relatively few items. The aim of this study was to describe and illustrate how items can be generated using a systematic approach. Automatic item generation (AIG) is an alternative method that allows a small number of content experts to produce large numbers of items by integrating their domain expertise with computer technology. This article describes and illustrates how three modeling approaches to item content-item cloning, cognitive modeling, and image-anchored modeling-can be used to generate large numbers of multiple-choice test items for examinations in dentistry. Test items can be generated by combining the expertise of two content specialists with technology supported by AIG. A total of 5,467 new items were created during this study. From substitution of item content, to modeling appropriate responses based upon a cognitive model of correct responses, to generating items linked to specific graphical findings, AIG has the potential for meeting increasing demands for test items. Further, the methods described in this study can be generalized and applied to many other item types. Future research applications for AIG in dental education are discussed.

To collect pieces of evidence for the construct validity of the reading section of a high-stakes test, test taking strategies of 40 test takers were analyzed via a checklist of strategies. The checklist consisted of 28 strategies tapping test takers’ behaviors while taking some reading comprehension items. The goal was to see whether there was concordance between the type of strategies and the item types in the reading comprehension passages. For example, if the strategy of guessing is used on inference items, this jeopardizes the validity of the item because there is a mismatch between the intentions of the test makers and those of test takers (Cohen, 1984). Hopefully, it was found mostly the right strategies were used on the right item types. This speaks to the construct validity of the reading section of University of Tehran English Proficiency Test (the UTEPT) which was exposed to investigation.

The study examines the typicality effect in Spanish-English bilingual children and adults in their 2 languages. Two studies were conducted using a category-generation task to compare the typical items generated by children with those generated by adults. Children in the 1st study differed orthogonally with respect to age (older, younger) and language use (higher Spanish use, higher English use). In the 2nd study, the older and younger children were matched with adults on their current Spanish use to delineate the influence of test language and age. Children with higher English use generated more typical items, and these occurred earlier in their word lists in English than in Spanish. Participants at all levels of Spanish experience generated fewer typical items in Spanish than in English. Thus, there was less convergence of items considered typical among participants in Spanish. Older and younger children did not differ in the number of typical items generated. However, when participants were matched for language use, older children produced typical items earlier in their word lists than did younger children. This study demonstrates the influence of language use and test language in generation of typical items in bilingual children.

online testing was introduced for the Physics 1 course at the Higher Colleges of Technology; Fujairah Men's College in the Fall 2016 semester. The students wrote an online test 1 in Blackboard Learn and a paper-based test 2. The validity of the online testing was assessed by comparing the performance of students, relative to the other students in their section, for the paper-based and online tests. Analysis of the results did not show any measurable difference in relative student performance for the online test compared to the paper-based version. An online system-wide Physics 1 final exam for all the seventeen colleges in the Higher Colleges of Technology system was introduced in the Fall semester of 2018. The results for this final exam and those from the subsequent two semesters are shown and briefly discussed. These results indicate a general improvement in grades subsequent to the Fall 2018 semester. Current and future work is described.

ABSTRACTThis research provided a comprehensive evaluation and validation of the listening section of a newly introduced computerised test, Pearson Test of English Academic (PTE Academic). PTE Academic contains 11 item types assessing academic listening skills either alone or in combination with other skills. First, task analysis helped identify skills important for listening comprehension in academic settings. Aspects analysed included the purpose of assessment tasks, skills/constructs assessed, and task stimuli employed in PTE Academic. The findings indicated that modern technologies enabled PTE Academic, a computer-based test, to assess students’ academic listening abilities in real time using the integration of multi-modal sources. The statistical validation consisted of two stages. Exploratory factor analysis was performed first with a sample of over 5000 students who took PTE Academic, to examine the underlying listening constructs as measured by the scores on the 11 item types item on different listening skills; these scores were subjected to Rasch analysis using CONQUEST. Second, the difficulties of the item types were estimated and the effectiveness of these item types was evaluated by calculating the information function by item type. The study has implications for test developers and test users regarding the interpretation of student performance on listening assessments.

Praise for the 3rd Edition: Without question, this book should be on every nurse educator's bookshelf. --Nursing Education Perspectives An invaluable resource for theoretical and practical application of evaluation and testing of clinical nursing students. Graduate students and veteran nurses preparing for their roles as nurse educators will want to add this book to their library. Score: 93, 4 stars --Doody's has again given us philosophical, theoretical and social/ethical frameworks for understanding assessment and measurement, as well as fundamental knowledge to develop evaluation tools for individual students and academic programs. -Nancy F. Langston, PhD, RN, FAAN Dean and Professor,Virginia Commonwealth University School of Nursing The fourth edition of the for evaluation and testing in nursing education helps educators measure and evaluate the level of learning that their students have achieved and presents fundamental concepts of what constitutes reliable tools for measuring students' classroom and clinical performance. The text presents the concepts of assessment, measurement, evaluation, and testing and explains how to prepare all types of tests including the alternate-format NCLEX test items. It describes how to assemble, administer, and analyze tests and test performance for both classroom and clinical evaluation. The book also provides strategies for evaluating higher cognitive levels of learning and for evaluating written assignments with sample scoring rubrics. New to the 4th edition: Offers expanded coverage of essential concepts of assessment, evaluation, and testing in nursing education Presents a new chapter on online testing, with and without an online teaching platform such as BlackBoard, standardized online testing packages, establishing valid 'online based testing, and related issues. Updates each chapter with current research and references and new examples Presents stronger focus on evaluating higher-level cognitive functions Updates test preparation for NCLEX and other certification tests Explains how to develop test items that prepare students for licensure and certification exams, including discussion of innovative items being piloted for possible use on the NCLEX Provides expanded coverage on establishing rubrics for assessing written assignments Includes new information on using simulation for High Stakes Testing Offers online teaching guide that includes PowerPoint slides, sample core syllabus, and learner activities This text is essential for all educators and students who are studying to become nurse educators as well as for nurse faculty and mentors who wish to improve their ability to test and increase the validity of their test instruments. Key Features: Presents the gold standard for evaluation and testing in nursing education Firmly grounded in concepts of assessment, measurement, evaluation, and testing across all levels of nursing education Provides guidelines for writing all types of test items (including NCLEX alternate format test items), scoring, and analyzing tests Provides valid evaluation methods for both classroom and clinical evaluation Meets NLN Certification Competency #3: Use Assessment & Evaluation Strategies (15% of CNE Test BluePrint)