Using quizzes to enhance summative-assessment performance in a web-based class: An experimental study

Laboratory studies show that taking a test on studied material promotes subsequent learning and retention of that material on a final test (termed the testing effect). Educational research has virtually ignored testing as a technique to improve classroom learning. We investigated the testing effect in a college course. Students took weekly quizzes followed by multiple choice criterial tests (unit tests and a cumulative final). Weekly quizzes included multiple choice or short answer questions, after which feedback was provided. As an exposure control, in some weeks students were presented target material for additional reading. Quizzing, but not additional reading, improved performance on the criterial tests relative to material not targeted by quizzes. Further, short answer quizzes produced more robust benefits than multiple choice quizzes. This pattern converges with laboratory findings showing that recall tests are more beneficial than recognition tests for subsequent memory performance. We conclude that in the classroom testing can be used to promote learning, not just to evaluate learning.

Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

This contribution will report on, and evaluate, the use of multiple-choice questions, in both continuous assessment and terminal examination modes, on a first year module in electrical engineering at Dublin Institute of Technology. The author’s experiences are that multiple-choice questions tend to be student friendly (e.g. given a choice in a terminal examination, students will opt to attempt the questions with multiple choice parts). The author has used the assessment method over three academic years, and some assessment data is reported and analysed in the contribution. The pedagogical approach is explicitly pragmatic.

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one's ability in a particular domain) and self-monitoring processes (i.e., an "in the moment" awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general "trivia". The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one's initial answer. Differences in these measures as a function of the accuracy of the candidate's response were treated as indices of each candidate's ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

Very short answer questions (VSAQs) have gained attention for their superior psychometric properties compared to multiple-choice questions (MCQs). While VSAQs require knowledge recall, MCQs primarily involve knowledge recognition. This difference in cognitive processes may lead to varying cognitive workloads, defined as the amount of mental processing in working memory. Previous studies have not demonstrated consistent differences, likely due to reliance on self-reported measures. Eye tracking provides objective, process-level indicators of cognitive workload. This study investigated whether answering VSAQs requires a higher cognitive workload than answering MCQs. In a within-subject randomized crossover experiment, sixth-year medical students answered both VSAQs and MCQs. Cognitive workload was measured using screen-based eye tracking, focusing on the number of fixations and revisitations as objective indicators of mental effort. Data were analyzed using mixed-effects models. Thirty-four medical students participated, yielding 1,326 observations, which is the multiplication of the number of students by the number of questions (39 questions). Mixed-effects models showed a significant effect of question type on both workload indicators: VSAQs elicited more fixations and revisitations than MCQs (β_std = 0.30–0.39, p < .001). This effect remained after controlling for accuracy. Incorrect answers were associated with higher workload (β_std = −0.15–−0.16, p < .01). Heatmaps confirmed these findings, showing denser fixations on key diagnostic features for VSAQs and on answer options for MCQs. Answering VSAQs imposed a higher cognitive workload than MCQs. The presence of answer options in MCQs may reduce workload by providing unintentional cues, while VSAQs require active retrieval. Eye tracking proved valuable for distinguishing cognitive workload across assessment formats.

The COVID-19 pandemic forced educational institutions around the world to suddenly shift most of their traditional offline classes to the online in 2020. Korean universities were no exception to the trend, and it provided an opportunity to investigate student performance and satisfaction in an online versus offline university English course. In this study, an introductory English linguistics course conducted online in 2020 was compared to the same course conducted offline in 2019. The exams of the online English class probed in this study were the same versions used for the offline class consisting of multiple choice and short answer questions, but were administered in an open-book online format without a proctor. Then, students’ course evaluation results were analyzed to compare their satisfaction in different class delivery formats. Results of online and offline classes revealed that students’ exam results and course evaluation were almost identical, which suggest that the online format of university courses can be as effective as the traditional offline courses for the recent younger generation of online-friendly university students.

The implementation of 4C (Critical Thinking, Communication, Collaboration, Creativity) in the learning process is vital in order to prepare students for the 21st century (Supriano, 2019). In this era which the learning process conducts online, the teachers may still emphasize those values through kinds of questions served especially through e-learning madrasah. Thus, the objectives of this study are to be a reference for the other teachers by exploring the kinds of questions in learning Arabic, also give suggestions or even convey to certain parts so that a revision of e-learning madrasah will be conducted by analyzing the obstacles. Applying descriptive qualitative method, the researchers assume that through the futures of e-learning madrasah the teachers provide PPT/PDF, multiple choices, short answer and essay questions for maharah qiro’ah (reading skill), listening to a video/audio for maharah istima’ wa kalam (listening and speaking skills), fill-in-the-blank, short answer questions as well as asking students to create a simple essay for maharah kitabah (writing skill). Nevertheless, the Arabic font in e-learning madrasah and the learning schedule which is not specific and fixed become the obstacles.

Background: It is important to learn how to study for different examinations. The objective of the current study is to explore whether the assessment method selection would significantly affect the studying and learning approaches of students. Methods: This descriptive-analytical research consisted of 191 first-year undergraduate nursing students from three nursing schools and was conducted during two interval semesters. All of students were given examinations with half of multiple-choice questions (MCQs) and half of short-answer questions (SAQs) in mid and final term examinations. A structured 12-item questionnaire was designed based on a modified 5-point Likert scale in outline of visual analogue scale. The questionnaire with Cronbach’s alpha of at 0.814 sought information on student’s studying and learning approaches. Results: The students significantly rated agreement with queries of short-answer assessment part of questionnaire higher than in multiple-choice assessment part. Female students performed significantly better in the learning outcomes in SAQ and MCQ examinations grades and total grade of both examinations than male students who participated in the study (p= 0.001, 0.001 and 0.030, respectively). Neither lower-quartile nor higher-quartile of education promotion exposed significance difference on total score of studying and learning approaches questionnaires. Responses frequency to studying and learning approaches questionnaire revealed that most students selected short-answer assessment method. Conclusion: Therefore, the findings revealed that assessment method may shape and improve students’ studying and learning approaches. Short answer question is hypothesized to enhance the development of deep learning.

Two-stage testing (TST) involves individual testing followed by taking the same test in teams. Previously, Vogler and Robinson (The Journal of Experimental Education, 84(4), 787–803, 2016) found that TST facilitated individual performance. The present study addressed methodological limitations in the Vogler and Robinson study in two replications along with isolating the components of TST in a laboratory study. In Experiments 1 and 2, where undergraduates remained in testing conditions (individual or TST) for the entire semester, no TST advantages were found. Experiment 3 controlled for the effects of initial test format [short answer (SA) versus multiple choice (MC)], mode (individual vs. group), and repetition (once vs. twice) on new and old (appeared on previous tests) MC and SA items. Two weeks after watching a lecture and being tested once or twice individually or in teams, students were tested once more individually. Results revealed effects for repetition (taking an initial test twice was better than once—confirming test-enhanced learning) and format (taking an initial SA test was better than MC—confirming the superiority of supply over recognition tests). However, taking an initial team test was no better than taking an individual test. TST alone may not lead to individual learning gains.

The manual correction for short answer questions is a tedious and time consuming task. This issue leads to the importance of having automatic correction systems for short answer questions. Thus this paper proposes three different approaches for correcting short answer questions. One of them is NLP-based while the others are machine learning based approaches. The three approaches are tested and compared across another two short answer correction systems that are evaluated on the same dataset. Results show that the proposed approaches outperform others that are evaluated on the same dataset.

Introduction: The medical laboratory professional workforce shortage in the United States has been documented through several national survey reports that also describe the use of temporary staffing to fill vacancies....

Multiple-choice questions are a common means of assessment thought to be an objective way to evaluate students’ knowledge recall. However, there are several limitations when utilizing multiple-choice questions for assessment. Depending upon the way the question is formulated, multiple-choice questions often do not allow for assessment of higher order cognitive skills. Well-designed short-answer questions are generally considered superior to multiple-choice questions at requiring students to analyse, synthesize, and apply their knowledge. Additionally, when answering multiple-choice questions, students may be able to guess the correct answers, thus nullifying the ability of the question to validly assess student knowledge. This paper introduces a type of expanded multiple-choice question requiring students to explain why they chose the answer they did. Such examination questions potentially capitalize on the advantages, and minimize the disadvantages, of traditional multiple-choice questions. This paper discusses the use of expanded multiple-choice questions for assessment and compares student responses to expanded multiple-choice questions vs. short-answer questions to determine the effectiveness of such questions in an undergraduate class in speech acoustics.

abstractCareful communication between members of the obstetric team about intrapartum foetal heart rate is critical for clinical management and patient safety. This study evaluated the benefits of two testing modalities in assessing resident physician knowledge of the 2008 NICHD nomenclature. Multiple-choice (MC) and short-answer (SA) examinations were administered to Obstetrics and Gynecology resident physicians before an educational intervention and then immediately after the training, at 6 months and at 12 months. Test scores on both the MC and the SA examinations improved after the training session. The improvement was sustained over the course of the study. Residents performed higher on the MC examination than on the SA test. This study suggests that formalised teaching in foetal heart rate monitoring improves resident physician knowledge of the NICHD nomenclature and that SA examinations may better discriminate between residents who are and are not able to accurately articulate foetal heart rate monitoring terminology.

Formative assessments are primarily used as a tool to gauge learning throughout an anatomy course. They have also been demonstrated to improve student mastery and exam performance, although the precise nature of this relationship is poorly understood. In this study, it is hypothesized that formative assessment questions targeting higher cognitive levels, integrating topics from multiple lessons, and including visuospatial elements will increase student exam performance. Formative and summative questions provided to students during the Clinical Anatomy block at the University of Arizona College of Medicine-Phoenix between 2015 and 2018 were assessed for cognitive level, integration of targeted learning objectives, and presence or absence of visuospatial elements. These variables were entered into a hierarchical linear model along with demographic variables for each cohort to assess the relationships between these variables and cohort performance on exam questions. The best predictor of exam performance was the inclusion of constituent learning objectives within the formative assessment. Additionally, students performed better on exam questions with visuospatial elements when the targeted learning objectives were also associated with visuospatial elements on the formative assessment. Surprisingly, the cognitive level of formative questions and the integration of learning objectives within them were not correlated with student exam performance. This study demonstrates the importance of including a broad range of topics in formative assessments and highlights a potential benefit of adopting consistent question formats for formative assessments and exams.

Operation ARA is a serious game that teaches scientific inquiry using natural language conversations. Within the context of the game, students completed up to two distinct training modules that teach either didactic or applied conceptual information about research methodology (e.g., validity of dependent variables, need for control groups). An experiment using a 4-condition between-subjects pretest-interaction-posttest design was conducted in which 81 undergraduate college students interacted with varying modules of Operation ARA. The four conditions were designed to test the impact of the two distinct modules on different types of learning measured by multiple-choice, short answer, and case-based assessment questions. Results revealed significant differences on training condition and learning gains on two of the three types of questions.