Didactic Galactic: Types of Knowledge Learned in a Serious Game

Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

Very short answer questions (VSAQs) have gained attention for their superior psychometric properties compared to multiple-choice questions (MCQs). While VSAQs require knowledge recall, MCQs primarily involve knowledge recognition. This difference in cognitive processes may lead to varying cognitive workloads, defined as the amount of mental processing in working memory. Previous studies have not demonstrated consistent differences, likely due to reliance on self-reported measures. Eye tracking provides objective, process-level indicators of cognitive workload. This study investigated whether answering VSAQs requires a higher cognitive workload than answering MCQs. In a within-subject randomized crossover experiment, sixth-year medical students answered both VSAQs and MCQs. Cognitive workload was measured using screen-based eye tracking, focusing on the number of fixations and revisitations as objective indicators of mental effort. Data were analyzed using mixed-effects models. Thirty-four medical students participated, yielding 1,326 observations, which is the multiplication of the number of students by the number of questions (39 questions). Mixed-effects models showed a significant effect of question type on both workload indicators: VSAQs elicited more fixations and revisitations than MCQs (β_std = 0.30–0.39, p < .001). This effect remained after controlling for accuracy. Incorrect answers were associated with higher workload (β_std = −0.15–−0.16, p < .01). Heatmaps confirmed these findings, showing denser fixations on key diagnostic features for VSAQs and on answer options for MCQs. Answering VSAQs imposed a higher cognitive workload than MCQs. The presence of answer options in MCQs may reduce workload by providing unintentional cues, while VSAQs require active retrieval. Eye tracking proved valuable for distinguishing cognitive workload across assessment formats.

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one's ability in a particular domain) and self-monitoring processes (i.e., an "in the moment" awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general "trivia". The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one's initial answer. Differences in these measures as a function of the accuracy of the candidate's response were treated as indices of each candidate's ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

Tasks that involve retrieving information from memory, such as answering short answer questions, are more effective at improving learning than restudying, concept mapping, and other study techniques. However, little is known about how often teachers naturally provide these retrieval practice opportunities during lectures and classroom discussions. To identify how often teachers ask questions that require retrieval, what types of retrieval questions they ask, and whether teachers in high-growth classrooms differ in their use of retrieval questions compared to teachers in low-growth classrooms. The sample included twenty middle school mathematics classrooms that showed high growth on a test of mathematics achievement and twenty with low growth. For each classroom, we examined a videotape of one class period. We coded the number of teacher questions in each lesson, and the number and type of questions that provided an opportunity for retrieval. We found wide variability in the frequency and type of questions asked across classrooms. On average, almost half of the non-classroom management questions provided an opportunity for retrieval. However, teachers in high- and low-growth classrooms asked similar numbers and types of retrieval questions. Teachers naturally use a wide variety of retrieval questions in their mathematics classrooms. As such, improving their use of retrieval opportunities will require only small changes to their natural practice, rather than large changes to their instructional style.

The current study investigated predictors of shallow versus deep learning within a serious game known as Operation ARA. This game uses a myriad of pedagogical features including multiple-choice tests, adaptive natural language tutorial conversations, case-based reasoning, and an E-text to engage students. The game teaches 11 topics in research methodology across three distinct modules that target factual information, application of reasoning to specific cases, and question generation. The goal of this investigation is to discover predictors of deep and shallow learning by blending Evidence-Centered Design (ECD) with educational data mining. In line with ECD, time-honored cognitive processes or behaviors of time-on-task, discrimination, generation, and scaffolding were selected because there is a large research history supporting their importance to learning. The study included 192 college students who participated in a pretest-interaction-posttest design. These data were used to discover the best predictors of learning across the training experiences. Results revealed distinctly different patterns of predictors of deep versus shallow learning for students across the training environments of the game. Specifically, more interactivity is important for environments contributing to shallow learning whereas generation and discrimination is more important in training environments supporting deeper learning. However, in some training environments the positive impact of generation may be at the price of decreased discrimination.

Laboratory studies show that taking a test on studied material promotes subsequent learning and retention of that material on a final test (termed the testing effect). Educational research has virtually ignored testing as a technique to improve classroom learning. We investigated the testing effect in a college course. Students took weekly quizzes followed by multiple choice criterial tests (unit tests and a cumulative final). Weekly quizzes included multiple choice or short answer questions, after which feedback was provided. As an exposure control, in some weeks students were presented target material for additional reading. Quizzing, but not additional reading, improved performance on the criterial tests relative to material not targeted by quizzes. Further, short answer quizzes produced more robust benefits than multiple choice quizzes. This pattern converges with laboratory findings showing that recall tests are more beneficial than recognition tests for subsequent memory performance. We conclude that in the classroom testing can be used to promote learning, not just to evaluate learning.

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

This study evaluated the performance of four large language model (LLM)-based chatbots by comparing their test results with those of dental students on an oral and maxillofacial radiology examination. ChatGPT, ChatGPT Plus, Bard, and Bing Chat were tested on 52 questions from regular dental college examinations. These questions were categorized into three educational content areas: basic knowledge, imaging and equipment, and image interpretation. They were also classified as multiple-choice questions (MCQs) and short-answer questions (SAQs). The accuracy rates of the chatbots were compared with the performance of students, and further analysis was conducted based on the educational content and question type. The students' overall accuracy rate was 81.2%, while that of the chatbots varied: 50.0% for ChatGPT, 65.4% for ChatGPT Plus, 50.0% for Bard, and 63.5% for Bing Chat. ChatGPT Plus achieved a higher accuracy rate for basic knowledge than the students (93.8% vs. 78.7%). However, all chatbots performed poorly in image interpretation, with accuracy rates below 35.0%. All chatbots scored less than 60.0% on MCQs, but performed better on SAQs. The performance of chatbots in oral and maxillofacial radiology was unsatisfactory. Further training using specific, relevant data derived solely from reliable sources is required. Additionally, the validity of these chatbots' responses must be meticulously verified.

The Anatomy Learning Experiences Questionnaire (ALEQ) was designed by Smith and Mathias to explore students' perceptions and experiences of learning anatomy. In this study, the psychometric properties of a slightly altered 34-item ALEQ (ALEQ-34) were evaluated, and correlations with learning outcomes investigated, by surveying first- and second-year undergraduate medical students; 181 usable responses were obtained (75% response rate). Psychometric analysis demonstrated overall good reliability (Cronbach's alpha of 0.85). Exploratory factor analysis yielded a 27-item, three-factor solution (ALEQ-27, Cronbach's alpha of 0.86), described as: (Factor 1) (Reversed) challenges in learning anatomy, (Factor 2) Applications and importance of anatomy, and (Factor 3) Learning in the dissection laboratory. Second-year students had somewhat greater challenges and less positive attitudes in learning anatomy than first-year students. Females reported slightly greater challenges and less confidence in learning anatomy than males. Total scores on summative gross anatomy examination questions correlated with ALEQ-27, Pearson's r = 0.222 and 0.271, in years 1 and 2, respectively, and with Factor 1, r = 0.479 and 0.317 (all statistically significant). Factor 1 also had similar correlations across different question types (multiple choice; short answer or essay; cadaveric; and anatomical models, bones, or radiological images). In a retrospective analysis, Factor 1 predicted poor end-of-semester anatomy examination results in year 1 with a sensitivity of 88% and positive predictive value of 33%. Further development of ALEQ-27 may enable deeper understanding of students' learning of anatomy, and its ten-item Factor 1 may be a useful screening tool to identify at-risk students. Anat Sci Educ 10: 514-527. ©2017 American Association of Anatomists.

Using quizzes to enhance summative-assessment performance in a web-based class: An experimental study

The implementation of 4C (Critical Thinking, Communication, Collaboration, Creativity) in the learning process is vital in order to prepare students for the 21st century (Supriano, 2019). In this era which the learning process conducts online, the teachers may still emphasize those values through kinds of questions served especially through e-learning madrasah. Thus, the objectives of this study are to be a reference for the other teachers by exploring the kinds of questions in learning Arabic, also give suggestions or even convey to certain parts so that a revision of e-learning madrasah will be conducted by analyzing the obstacles. Applying descriptive qualitative method, the researchers assume that through the futures of e-learning madrasah the teachers provide PPT/PDF, multiple choices, short answer and essay questions for maharah qiro’ah (reading skill), listening to a video/audio for maharah istima’ wa kalam (listening and speaking skills), fill-in-the-blank, short answer questions as well as asking students to create a simple essay for maharah kitabah (writing skill). Nevertheless, the Arabic font in e-learning madrasah and the learning schedule which is not specific and fixed become the obstacles.

Introduction: The medical laboratory professional workforce shortage in the United States has been documented through several national survey reports that also describe the use of temporary staffing to fill vacancies....

Assessing student performance and understanding is very important in education generally and in Mathematics education in particular. DMAS is a Web-based Distributed Mathematics Assessment system that can be of great value to teachers and students of Mathematics. Features of DMAS include: DMAD (the core Distributed Mathematics Assessment database), test authoring tool for teachers, online taking of tests, grading and results administration, comprehension diagnoses and linking to remedial materials. DMAS supports many problem types and allows mathematical formulas/expressions as well as geometry drawings/graphs to be part of questions and answers. Furthermore, DMAS supports the use of test taking as a teaching method. A teacher can monitor all students from the teacher terminal in real-time and either verbally guide the whole class on certain points or interact privately with one or more students via the Teacher-Student Interaction Mechanism (TSIM), an instant text-messaging provided by DMAS. A new Assessment Markup Language, MAML, is designed to make tests and test questions easy to share widely on the Web. Although developed as a sub-system of WME (Web-based Mathematics Education), DMAS is an independent Web system easily interfaced to any Web page through a well-defined API. The DMAS system consists of local databases at individual school websites. Teachers may publish new or improved questions to their local DMAS database to be shared with other teachers in the same school and in different schools. Teachers can use the powerful search engine (DMASEngine) to search for assessment questions. Search can be narrowed by subject, topic, grade-level, question type, keyword, and author. DMAD is designed to work as one distributed database while providing power, unity, and convenience at each participating school. The system helps create, revise, administer, and grade exams that can contain various types of questions: multiple choices, true-or-false, extended (essay) questions, short answers, two-columns matching, and fill-the-blank. Questions use a well-designed representation (encoding) allowing correct answers, rubrics, formulas, images, geometrical objects (SVG), and multimedia as well as interactive contents. DMAS system provides different views (modes): teacher and student modes. It also supports two types of testing: online and paper testing. The teacher can set Assessment test display options: display questions by a given order, by random questions none-random options, or by all questions/options are in random order. Tests taken online can be graded automatically and stored for teacher review, analysis, and printing. DMAS Assessment system has been piloted into seventh grade students at Kimpton Middle School, Stow, Ohio on 10/05/2007 for real-life testing and apparently teacher and student's feedback gave good impression and encouraging feedback about the DMAS system. However, our goal is to put DMAS system under extensive trial in schools and collecting feedback and suggestions from teachers, students, school administrators and education experts to help evolve DMAS. As more schools adopt DMAS, the distributed nature of DMAS will be demonstrated in realistic situations. Because DMAS has been designed in consultation with middle school teachers and education experts, the underline DMAS system design and framework is easy and flexible for future changes, requirements, expansions, and customizations. DMAS provides features to support real-life test giving such as loss of power to a laptop and retaking of tests.

The two cognitive styles, ‘field dependent/field independent’ and ‘convergent/divergent’, were explored in relation to three formats of assessment (multiple choice, short answer and structural communication grid) in five classroom chemistry tests. The study was conducted in Greece with the participation of Grade-10 (upper secondary school) pupils (age 15-16). The field dependent/independent characteristic correlated with pupils’ performance in all the tests, and in almost all the formats of assessment. The convergent/divergent characteristic correlated with pupils’ performance in assessment where language was an important factor, but not in algorithmic types of questions or in questions where there is a greater use of symbols and less use of words. It seems that, in relation to the convergent/divergent characteristic, the chemistry content and presentation of the test is a factor affecting the type of questions being asked. This study suggests that some of the factors that affect pupils’ performance might be: (a) the content and presentation of the test, (b) the format of the test, (c) the psychology of the individual. [Chem. Educ. Res. Pract., 2006, 7 (2), 64-83]

Automatic assessment of short answers is one of the Computer Assisted Test works that can assess answers in natural language. Several methods have been used to create a system capable of assessing short answers that are close to human markings. In Indonesian, it might be easy to use string-based similarity methods by matching keywords, as has been done in previous studies. However, short answers have characteristics that focus on content, question type, and answer length, which cannot be accommodated only by string-based methods. This study aims to implement a hybrid method using corpus and string-based similarities. The Semantic Text Similarity (STS) method was used in this study to assess short answers in Indonesian. The STS method consists of three combinations of similarity methods, namely Normalized and Modified Longest Common Subsequence, Second Order Co-occurrence Pointwise Mutual Information, and Common Word Order Similarity. We also use a dynamic corpus with the advantage of being relatively small and adaptable to the learning domain. The Gensim Module is used to generate a dynamic corpus. The dynamic corpus uses the top five answers from students obtained from the Gensim module. The STS method is compared with the Cosine Similarity method since Cosine Similarity is the most commonly used method to assess answers in Indonesian. The results show that the STS method can outperform the Cosine Similarity method based on the Mean Absolute Error value, but still not outperformed in terms of correlation.