Toward Evidence-Based Guidelines on Answer Keys’ Placement in Multiple-Choice Tests: A Simulation Study

Assessment is essential in the education system to monitor educational development and make appropriate instructional decisions. This study aims to analyze the quality of multiple choice questions based on high-level thinking skills (HOTS) in the Geography subject using Anates Windows Version 4.0.9 software. This evaluative research involves analysis of final semester exam questions for class X Geography at SMA Islam Al Azhar 14 Semarang. Data consisting of questions, answer keys and exam results were collected from 32 students. The analysis was carried out descriptively and quantitatively to assess the level of difficulty, discriminating power, reliability, effectiveness of distractors, and validity of the questions. The analysis results show that the difficulty level of most of the questions is in the medium category (50%). The discriminating power is mostly good, although several questions are wrong or bad. Most of the distractors worked well, but some questions had ineffective distractors. Regarding validity, 17 questions were valid, while 11 questions were invalid. The reliability of the test is in the high category, with a reliability coefficient of 0.71. Anates Windows Version 4.0.9 software effectively analysed the quality of HOTS questions. These findings indicate that although most of the questions are of good quality, some require revision to increase their validity and effectiveness. This research emphasizes the importance of using technology in educational assessment to ensure the quality and fairness of exam questions. It contributes to developing assessment tools to improve students' higher-order thinking skills.Keywords: Item Analysis; Multiple Choice; HOTS; SMA Islam Al-Azhar

The manual preparation of question papers is a time-consuming and difficult task for the teachers, as it involves lots of efforts on the part of the teachers to ensure that proper coverage of the syllabus, proper level of cognitive difficulty, and proper preparation of answer keys. The paper will briefly describe, implement and evaluate AIScribe, an intelligent web based product that automatically generates the question papers and key answers using Artificial Intelligence (AI).The system provides a structured workflow where authorized staff can upload a syllabus document, configure examination parameters (type and difficulty distribution and receive a complete, formatted question paper alongside an answer key containing essential evaluation keywords. The core intelligence is driven by the integration of Large Language Model(LLM) API, specifically the google Gemini API , which is guided by meticulously engineered prompts to generate pedagogically relevant content. The system shows the significant reduction in the time to create a set of question papers from several hours (manual) to less than two minutes automatically. While the need for expert review to ensure factual accuracy does exist, AIScribe appears to be a very helpful aid for standardizing assessment creation, minimizing faculty workload, and increasing efficiency within the academy. The paper ends with a discussion of limitations and suggestions for future improvements, which include having multiple choice questions and working with Bloom's Taxonomy for better difficulty calibration.

The 2nd edition of has been prepared as a result of the enthusiastic reaction from students, educators, and laboratory professionals. General Microbiology, immunology, Bacteriology, Virology, Mycology, Parasitology, and Applied Microbiology are among the subjects covered in this updated edition in accordance with the new BMLT stream syllabus. New material has been added on molecular diagnostic tools, automated culture and identification systems, updated biosafety guidelines, advanced antimicrobial susceptibility testing, and recent developments in hospital-acquired infection surveillance. Each chapter concludes with expected questions and multiple-choice questions with an answer key. The goal of these updates is to improve the connection between theoretical understanding and practical laboratory experience.

This study aims to develop critical thinking instruments in high school students with static fluid material. The development model used in this study is the Analysis-Design-Develop-Implement-Evaluate (ADDIE) learning design model. The subjects of this study were 90 students of XI MIA class in MAN 1 of Medan. The research methods used were interviews, tests, questionnaires, and documentation. The developed test instrument consisted of a grid of test questions, questions, answer sheets, answer keys and scoring guidelines, and guidelines for interpretation of results. Validity test used is content validity test. The results of the validity test by three expert validators showed that the critical thinking test instrument developed was valid. The reliability of the questions developed was 0.82, which means a reliable test item. There are 15 questions, and 12 questions are used for limited scale tests, 10 questions are used for broad scale tests. Keywords: critical thinking skill, multiple choice, static fluid.

This research was aimed at developing autonomous listening learning materials for TOEFL preparation for learners at independent language learning center (ILLC) of the language development center at YSU to help them improve their listening skills. This research was a Research and Development study which refers to development model proposed by Tomlinson (1998). The subjects of this research were 30 learners who are members of ILLC. Data were collected using questionnaires and open-ended interviews. The data were analyzed qualitatively and quantitatively. The results of the research reveal that the appropriate listening learning materials for TOEFL preparation consist of printed materials and audio recording containing the listening skills to find the main topic, to find the main idea, to find the implied meaning, to make prediction, to determine the meaning of idiomatic expressions, to find details information, to draw conclusions, to find suggestions, and to make inferences. The learners’ role is as active listeners and to answer multiple choice questions, the learning activities include many listening practices, clear instructions, reflection and answer keys. Units are designed of Unit number/title, learning objectives, hints/tips, main activities and reflection. The appropriate tasks consist of warm-up, matching expressions to the pictures, matching words to their meanings, guessing, anticipating the topics and or questions, listening practices, note-taking.

This research was a quantitative descriptive study that aimed to find out the quality of questions made by teachers of SMA Negeri 10 Gowa in terms of the level of difficulty, discrimination item, and to determine the ability of these questions to measure the high-level thinking students in SMA Negeri 10 Gowa. The sample in this study was 36 students. Data collection technique was documentation, by collecting all daily questions, answer keys, and the marks of student answers of Biology class XI of SMA Negeri 10 Gowa. The instrument used in this study was a review of the Higher Order Thinking (HOTS) adapted from a module of Making HOTS Questions published by the Education and Culture Office. The data of this study were analyzed quantitatively and qualitatively. Based on the results of the quantitative analysis, the level of difficulty of the question description used in daily tests was quite good, while questions for multiple-choice categorized as good. The quality of discrimination items for both essays and multiple-choice questions categorized as poor. The results of a qualitative analysis found that there were no questions categorized as HOTS, but all of these questions had very good in use and construction of language.

The progress of science and technology in various countries has accelerated toward the 21 century. The measurement of scientific literacy is very important to find out the extent to which students have been literated in science as an effort to improve the quality of education in Indonesia. The preparation of test instruments based on scientific literacy is an effort to measure the literacy skills of students in the field of science. The purpose of this study was to develop a valid, practical and effective science literacy test instrument to measure the scientific literacy skills of students on the theme of global warming. The research method used in this study was the method of Research and Development (R & D) with the research design of ADDIE (Analysis, Design, Development, Implementation, and Evaluation). The final product in this research development is a scientific literacy-based test instrument to measure the scientific literacy skill of students that have been declared valid and feasible by experts and is valid in the content, in the form of 20 reasonable multiple-choice questions that are tailored to the indicators of the scientific literacy, complete with a grid of questions, answer keys, and bibliography. Based on the results, it is concluded that the validity of the test instruments received an average percentage of 79.99% and was included in the category of valid and feasible to use. The practicality of the test instrument is in the very practical category seen based on the results of student responses by 87.5% and teacher responses by 94.23%. The effectiveness of the test instrument is in the effective category, Since this studycan reveal or position the scientific literacy skills of students into three criteria items, namely high, medium, and low, it can be used as input and thought contribution in an effort to improve the quality of making the test instruments to measure the skill of scientific literacy on the theme of global warming.

Objective of this research was to analysis and to describe the level of validity of the test madeby Mathematics teacher according to the Classical Test Theory at the Junior High School Level in Baubau City. This Research was a descriptive quantitative research. The subjects of this research were all multiple-choice tests made by the teacher for the final even-semester test for class VIII students in the 2020/2021 academic year in the field of mathematics subject at 7 state Junior High School in Baubau City. The Instruments in this research were student answer sheets, answer keys and recapitulation of student criteria at grade VIII. Data collection techiques used documentation techiques. Analysis of the data used was using the SPSS Statistic application version 26. Based on the results of the research, it could be concluded that, the results of the analysis of the validity of the test made by the teacher at SMP Negeri 1 Baubau of 0.684 which meant the validity was in the fairly good category, SMP Negeri 3 Baubau was 0.770 which meant the validity was in the good category, SMP Negeri 5 Baubau was 0.491 which meant the validity was in the good enough category, SMP Negeri 7 Baubau of 0.404 which meant the validity was included in the category of good enough, SMP Negeri 9 Baubau of 0.439 which meant the validity was in the category of good enough, SMP Negeri 11 Baubau of 0.434 which meant the validity was in the category of good enough, SMP Negeri 17 Baubau was 0.611 which meant that the validity was categorized as good enough. So that the validity of the test made by the teacher in the field of Mathematics Subject according to the Classical Test Theory at the Junior High level in Baubau City was 0.547 with the category of good enough.

During the COVID-19 pandemic, teaching and learning activities switch of technology such as e-learning. In the Department of Electrical Engineering Education, Universitas Negeri Medan, 80% of electrical lecturers provide multiple choice exams and short answers. The scoring system cannot be performed automatically by a computer. The algorithm used in this study is a Mixed Heuristic algorithm. This algorithm is used because it is an effective way of working for sentences of more than 1 word. And give the results in the form of ranking in the form of similarity proportions. From the process of using a mixed heuristic algorithm, it can be proven that this algorithm can perform keywords on the answer keys and answers of users or students. The algorithm is also able to provide value to a user or student's answers

The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments

Tests of educational achievement typically present items in the multiple-choice format. Some achievement test items may be so saturated with aptitude (Willingham, 1980) as to be insensitive to skills acquired through education. Multiple-choice tests are ill-suited for assessing productive thinking and problem-solving skills, skills that often constitute important objectives of education. Viewed as incentives for learning, multiple-choice tests may impede student progress toward these objectives. There is need for accelerated research to develop alternatives to multiple-choice achievement tests, with content selected to match the specified educational objectives.

This study analyzed the quality of multiple-choice test used as middle test made by the English teachers in one school in Singaraja. This study was essential to be conducted since the items of the multiple-choice test must have good quality to be used to assess the students’ achievement levels. This study used content analysis method in analyzing100 items from 3 different instruments. In collecting the data, the checklist analysis form was used to compare the items of teacher-made multiple-choice test with the norms as one of the standards in making a good multiple-choice test, then clarified through interview. From the data that have been obtained, there are 72% of students who got bad scores in the middle test. There is 1% of the items has sufficient quality, 8% of items have good quality, and the rest of the items have very good quality. The most common mistake found is in the punctuation and capitalization. It is supported by the results of the interview, which show that the teachers did not know precisely the norm of punctuation and capitalization. It can be concluded that the teachers already follow the norms of making a multiple-choice test, and it is indicated that the quality of the multiple-choice test is not the only factor that affects students’ achievement levels.Keywords: instrument quality, norms, teacher-made multiple-choice test

Multiple-choice questions (MCQs) are widely used for assessment in medical education. While human-generated MCQs benefit from pedagogical insight, creating high-quality items is time intensive. With the advent of artificial intelligence (AI), tools like DeepSeek R1 offer potential for automated MCQ generation, though their educational validity remains uncertain. With this background, this study compared the psychometric quality of Physiology MCQs generated by faculty and an AI chatbot. A total of 200 MCQs were developed following the standard syllabus and question design guidelines: 100 by the Physiology faculty and 100 by the AI chatbot DeepSeek R1. Fifty questions from each group were randomly selected and administered to undergraduate medical students in 2 hours. Item analysis was conducted postassessment using difficulty index (DIFI), discrimination index (DI), and nonfunctional distractors (NFDs). Statistical comparisons were made using t tests or nonparametric equivalents, with significance at P < 0.05. Chatbot-generated MCQs had a significantly higher DIFI (0.64 ± 0.22) than faculty MCQs (0.47 ± 0.19; P < 0.0001). No significant difference in DI was found between the groups (P = 0.17). Faculty MCQs had significantly fewer NFDs (median 0) compared to chatbot MCQs (median 1; P = 0.0063). AI-generated MCQs demonstrated comparable discrimination ability but were generally easier and contained more ineffective distractors. While chatbots show promise in MCQ generation, further refinement is needed to improve distractor quality and item difficulty. AI can complement but not yet replace human expertise in assessment design.NEW & NOTEWORTHY This study contributes to the growing research on artificial intelligence (AI)- versus faculty-generated multiple-choice questions in Physiology. Psychometric analysis showed that AI-generated items were generally easier but had comparable discrimination ability to faculty-authored questions, while containing more nonfunctional distractors. By focusing on Physiology, this work offers discipline-specific insights and underscores both the potential and current limitations of AI in assessment development.

BackgroundThis study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education.MethodMultiple-choice questions with too high or too low difficulty (difficulty index < 0.4 or > 0.8) and insufficient discriminatory ability (point-biserial correlation < 0.2) on previous administration were identified. Items in Experimental Subgroup A underwent removal of item writing flaws along with enhancement of tested cognitive level (21 multiple-choice questions), while Experimental Subgroup B underwent replacement or removal of non-functioning distractors (11 multiple-choice questions). A control group of items (Group C) did not undergo any intervention (23 multiple-choice questions).ResultPost-intervention, the average number of functioning distractors (≥ 5 % selection frequency) per multiple-choice question increased from 0.67 to 0.81 in Subgroup A and from 0.91 to 1.09 in Subgroup B; a statistically significant increase in the number of multiple-choice questions with sufficient point-biserial correlation was also noted. No significant changes were noted in psychometric characteristics of the control group of items.ConclusionCorrection of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions. This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions.Electronic supplementary materialThe online version of this article (doi:10.1007/s40037-015-0212-x) contains supplementary material, which is available to authorized users.

The practice of assessments in third level education is extremely important and a rarely disputed part of the university curriculum as a method to demonstrate a student’s learning. However, assessments to test a student’s knowledge and level of understanding are challenging to apply given recent trends which are showing that student numbers are increasing, student demographics are wide ranging and resources are being stretched. As a result of these emerging challenges, lecturers are required to develop a comprehensive assessment to effectively demonstrate student learning, whilst efficiently managing large class sizes. One form of assessment which has been used for efficient assessment is multiple choice questions (MCQs); however this method has been criticised for encouraging surface learning, in comparison to other methods such as essays or case studies. This research explores the impact of blended assessment methods on student learning. This study adopts a rigorous three-staged qualitative methodology to capture third level lecturers’ and students’ perception to (1) the level of learning when using MCQs; (2) the level of learning when blended assessment in the form of case based MCQs are used. The findings illuminate the positive impact of cased based MCQs as students and lecturers suggest that it leads to a higher level of learning and deeper information processing over that of MCQs without case studies. The implications of this research is that this type of assessment contributes to the current thinking within literature on the use of assessments methods, as well as the blending of assessment methods to reach a higher level of learning. It further serves to reinforce the belief that assessments are the greatest influence on students’ learning, and the requirement for both universities and lecturers to reflect on the best form of assessment to test students’ level of understanding, whilst also balancing the real challenges of large class size teaching.