Affix-based Distractor Generation for Tamil Multiple Choice Questions using Neural Word Embedding

Multiple Choice Questions (MCQs) feature more and more in modern-day assessments due to more straightforward exams and faster checking. However, with this comes the challenge of creating a diverse pool of MCQs specific to a given subject matter. This work proposes an MCQ generation system that uses linguistic features and Deep Learning techniques to create MCQs from a given text. The entire process has three steps: i) The pre-trained DL state-of-the-art model summarizes a text paragraph to get relevant information. ii) Linguistic features generate question (stem) and answer (key) pairs. iii) Distractor generation using the key or correct answer. DL-based paraphrasing models are used to augment the MCQs dataset with questions of similar type and difficulty level.

Background: Multiple-choice questions (MCQs) are the most frequently accepted tool for the evaluation of comprehension, knowledge, and application among medical students. In single best response MCQs (items), a high order of cognition of students can be assessed. It is essential to develop valid and reliable MCQs, as flawed items will interfere with the unbiased assessment. The present paper gives an attempt to discuss the art of framing well-structured items taking kind help from the provided references. This article puts forth a practice for committed medical educators to uplift the skill of forming quality MCQs by enhanced Faculty Development programs (FDPs). Objectives: The objective of the study is also to test the quality of MCQs by item analysis. Methods: In this study, 100 MCQs of set I or set II were distributed to 200 MBBS students of Late Shri Lakhiram Agrawal Memorial Govt. Medical College Raigarh (CG) for item analysis for quality MCQs. Set I and Set II were MCQs which were formed by 60 medical faculty before and after FDP, respectively. All MCQs had a single stem with three wrong and one correct answers. The data were entered in Microsoft excel 2016 software to analyze. The difficulty index (Dif I), discrimination index (DI), and distractor efficiency (DE) were the item analysis parameters used to evaluate the impact on adhering to the guidelines for framing MCQs. Results: The mean calculated difficulty index, discrimination index, and distractor efficiency were 56.54%, 0.26, and 89.93%, respectively. Among 100 items, 14 items were of higher difficulty level (DIF I < 30%), 70 were of moderate category, and 16 items were of easy level (DIF I > 60%). A total of 10 items had very good DI (0.40), 32 had recommended values (0.30 - 0.39), and 25 were acceptable with changes (0.20 - 0.29). Of the 100 MCQs, there were 27 MCQs with DE of 66.66% and 11 MCQs with DE of 33.33%. Conclusions: In this study, higher cognitive-domain MCQs increased after training, recurrent-type MCQ decreased, and MCQ with item writing flaws reduced, therefore making our results much more statistically significant. We had nine MCQs that satisfied all the criteria of item analysis.

A comparison of clinical-scenario (case cluster) versus stand-alone multiple choice questions in a problem-based learning environment in undergraduate medicine.

Introduction: Multiple choice questions (MCQs) are widely used in medical education. Difficulty Index (DIF I), Discrimination Index (DI), and Distractor Efficiency (DE) are the most commonly used item analyses. MCQ authors should anticipate the possible difficulty level in writing their questions and carefully arrange distractors. This study aims to evaluate the quality of MCQ and the predictions of MCQ authors about the difficulty levels of their questions. Methods: In this study, the DIF I, DI and DE values of 688 MCQs in the exams held in the first year of the 2021-2022 academic year of Dokuz Eylül University Faculty of Medicine were investigated. The effect of the number of Non-Functional Distractors (NFD) on DIF I and DI, and the compliance between the DIF I predictions of the question authors and the actual values was evaluated. Results: DIF I mean was 0.57±0.21, and 47.5% was at the ideal difficulty level. There was a significant difference between the estimated and the actual DIF I (p=0.002). The DI average was 0.31 ± 0.17, and the discrimination level of 43.6% was excellent. 36.8% of distractors were NFD. MCQ's difficulty and discriminatory ability were significantly different according to the number of NFDs (p=0.000 for all). Discussion: It was determined that the number of NFDs significantly affected difficulty and discriminatory ability. There was a difference between the estimates of the difficulty and the actual values. Reducing the number of options in MCQs and being more careful when crafting questions can improve the quality of the questions.

We performed 4 experiments assessing the learning that occurs when taking a test. Our experiments used multiple-choice tests because the processes deployed during testing can be manipulated by varying the nature of the choice alternatives. Previous research revealed that a multiple-choice test that includes “none of the above” (NOTA) produces better performance on a subsequent test only when the correct answer is something other than NOTA (Odegard & Koen, 2007). However, when NOTA was an incorrect choice alternative, the correct answer was the only familiar alternative. Thus, familiarity may have allowed participants to identify the answer, which was then restudied. In other words, the testing benefit might have reflected a familiarity-guided restudy process rather than retrieval practice. In the current study, we examined the role of familiarity in the multiple-choice testing effect, manipulating the familiarity of alternatives. If NOTA was the correct answer, there was no testing benefit when the alternatives were all novel (Experiment 1) or all familiar (Experiment 3). Familiarity-guided restudy predicts memory impairment when there is a single familiar alternative for a NOTA-correct question. In contradiction to this hypothesis, there was a testing benefit for this condition (Experiments 2 and 4). Experiment 4 further collected metacognitive confidence ratings during the multiple-choice test, providing evidence of a recall-to-reject strategy for this condition. These results suggest that learning from multiple-choice tests is mainly due to retrieval practice rather than the use of familiarity.

Comparison of examination methods based on multiple-choice questions and constructed-response questions using personal computers

Background: Multiple-choice questions (MCQs) provide useful information about correct and incorrect answers, but they do not offer information about students’ confidence.Methods: Ninety and another 81 medical students participated each in a curricular neurology multiple-choice exam and indicated their confidence for every single MCQ. Each MCQ had a defined level of potential clinical impact on patient safety (uncritical, risky, harmful). Our first objective was to detect informed (IF), guessed (GU), misinformed (MI), and uninformed (UI) answers. Further, we evaluated whether there were significant differences for confidence at correct and incorrect answers. Then, we explored if clinical impact had a significant influence on students’ confidence.Results: There were 1818 IF, 635 GU, 71 MI, and 176 UI answers in exam I and 1453 IF, 613 GU, 92 MI, and 191 UI answers in exam II. Students’ confidence was significantly higher for correct than for incorrect answers at both exams (p < 0.001). For exam I, students’ confidence was significantly higher for incorrect harmful than for incorrect risky classified MCQs (p = 0.01). At exam II, students’ confidence was significantly higher for incorrect harmful than for incorrect benign (p < 0.01) and significantly higher for correct benign than for correct harmful categorized MCQs (p = 0.01).Conclusions: We were pleased to see that there were more informed than guessed, more uninformed than misinformed answers and higher students’ confidence for correct than for incorrect answers. Our expectation that students state higher confidence in correct and harmful and lower confidence in incorrect and harmful MCQs could not be confirmed.

ABSTRACT. Evaluating learning results is essential in all learning processes. Multiple choice tests are frequently used to objectively evaluate and recognize these results. Solving multiple choice tests by well constructed items requires not only recalling existing knowledge but needs comprehension, interpretation as well as analysis and synthesis. Multiple choice tests are widely used, being easy to administer and to check, especially in big groups. In this article, we further suggest how to improve the definition and the options for each item (correct answer and distractors) and propose how to analyze test-formats already used such as to create a pool of high quality items (for future use). This analysis is based on the percentage of correct answers, the discrimination index and the degree of difficulty of each item in different student groups. Analyzing items of multiple choice tests already used may obviously be useful for improving the ability to construct such tests in the future but also may serve in the classroom (teachers and students) to discuss the results of a test as a metacognitive exercise, reflecting the mental strategies by which students intend to solve the items recognizing difficulties to do so. The analysis often shows weak points in the teaching and learning process, to be considered when reviewing the curriculum.

INTRODUCTION Towards exams, students want to get the most out of revising, but some study methods may be ineffective and boring. Using quizzes for retrieval practice is a valuable technique, and peer learning can help with understanding topics in a new way. We adapted aspects of the team‐based learning method into an engaging revision session; e.g. students undertaking a test individually then as a team, thereby incorporating retrieval practice and peer learning. Including many questions in a short timeframe was designed to reveal knowledge gaps and keep the session enjoyable. AIM This study sought to explore students’ experience of teamwork during a broad‐ranging, fastpaced anatomy revision session. METHODS The revision session formed the final anatomy “laboratory” session before end‐of‐year exams for the 2019 cohort of Stage 1 postgraduate students (n=274) in the Sydney Medical Program (University of Sydney). Students were allocated to one of three classes, and were asked to form teams of 6 – 7 students. Multiple choice questions (MCQs) were presented in three rounds of 10 questions, with 30 seconds per question. Students attempted each round of 10 MCQs first by themselves, then immediately completed the same MCQs as a team. Correct answers were then explained to all students. Four extended matching questions (EMQs) providing anatomically‐themed clinical scenarios were also presented for team discussion only. All teams revealed their answers simultaneously, before solutions were explained. Paper‐based surveys were distributed at the end of the session, asking students to rate the usefulness of the MCQs and teamwork on a five‐point Likert‐type scale; freeform comments were also requested. Data were analysed with descriptive statistics. Ethics approval was obtained via the University of Sydney Human Research Ethics Committee . RESULTS From the 191 students who attended (70% of cohort), 518 individual sets of MCQ responses could be compared with team responses across all rounds. Most (83.2%) responses were higher in their team scores than individual scores. Only 3.7% of responses were lower in team rather than individual scores (13.1% achieved the same score). The mean rate of improvement from individual to team score was 157.7%. Surveys were returned by 157 students (82.2%). Of these, 72.6% “strongly agreed” (a further 21.0% “agreed”) that the MCQs helped identify knowledge gaps. Regarding working as a team following individual work, almost all strongly agreed (79.0%) or agreed (15.9%) that it had been beneficial to their learning process. Written feedback reflected these feelings, with students also complimenting the range of questions, though a few felt they were too difficult, especially for the short time allowed. More EMQs were requested. DISCUSSION AND CONCLUSION Students responded positively to this team‐based revision. Although fast‐paced, most students coped with the short timeframes, and appreciated being exposed to multiple high‐quality questions (and, importantly, answers). Through collaboration in their small teams, students benefitted from peer learning, and at the same time enjoyed the teamwork and camaraderie in the otherwise tense build‐up to exams.

Objectives&#x0D; One-best MCQs (Multiple Choice Questions) is a popular form of assessment where the student selects the best possible answer from the options provided. The objective of this study was to analyze the MCQs used in preliminary examinations in terms of Difficulty Index, Differentiation Index and Distracter efficiency.&#x0D; Method&#x0D; Total 100 MCQ items with one correct option and three distracters were taken from past 5 years preuniversity examinations and analyzed for level of difficulty and power of discrimination. A total of 149, 156, 154, 156 and 148 students appeared for the exam in each year. The Difficulty index was calculated by the formula P = R/T. High and low groups consisting of upper and lower 27% students, respectively, were taken after arranging the scores in descending order for calculating the Discrimination Index as follows: DI = 2 x [(H-L)/N]. Distracter efficiency was determined on the basis of the number of Nonfunctioning Distracters (NF-Ds). An ideal item should have a DFI between 0.30–70.0, a DI of &gt;0.24 and a DE of 100%.&#x0D; Results&#x0D; The Difficulty indices for the ten papers of 10 MCQ`s each were (Mean±S. D) 0.50±0.24, 0.49±0.25, 0.52±0.23, 0.55±0.19, 0.73±0.22, 0.68±0.18, 0.53±0.24, 0.65±0.18, 0.54±0.3, 0.47±0.26 respectively. Differentiation indices were (Mean±S. D)0.37±0.12, 0.34±0.17, 0.35±0.14, 0.43±0.15, 0.40±0.15, 0.43±0.17, 0.40±0.20, 0.46±0.11, 0.40±0.16, 0.40±0.20. Thus as a whole, the papers were of average difficulty and were able to differentiate between the students. Thirtyfive MCQs were ideal MCQs. In individual analysis, total 37 MCQs were very easy (Difficulty Index&gt;0.70) and 18 were found to be very tough (Difficulty Index&lt;0.30). In Distracter analysis, 20% of the distracters were found to be nonfunctional (NFDs). 30% of the MCQ`s had One NFD, while 9% and 4% had two and three NFD`s respectively.&#x0D; Conclusion&#x0D; Overall, the papers had very good difficulty and differentiation indices which were quite consistent over the five years but there are a few MCQs and distracters which need to be modified to further enhance accurate assessment of the students.

Assessment is a prominent topic including everything from large scale tests to district benchmark or interim tests to everyday classroom tests.In order to discover what seems to be an overuse of testing, educators should frame their views of testing as assessment and that assessment is getting information rather than making learners stressed.The more we know about the students, the clearer the picture we have about achievement or where gaps may occur.This study attempts to find the difference between using multiple choice type of quizzes and true/false type of quizzes as means of formative assessment in teaching intermediate adult students.What we exactly tried to do was checking that whether learners can improve their knowledge of grammar and have better results in their achievement test based on different exam types they take during the term and in the class.The findings showed that there was no significant difference between the learners exposed to multiple-choice exams and those who were given true/false tests.The type of exam which was used for final exam was even a multiple choice one but this did not lead to a significantly better performance for the ones who had the same exam type during the instructional sessions.

The paper deals with two-tier multiple choice tests working as a reliable tool for students'evaluation. The test was applied within the engineering education in the foreign language instruction at the Faculty of Informatics and Management, University of Hradec Kralove, Czech Republic. The multiple choice tests are often considered easier to pass even if the level of respondent's knowledge is low, and the reached result thus does not reflect the knowledge adequately. Two hypotheses were set to verify whether learners reach higher test scores in translations in comparison to multiple choice tests and to multiple choice tests with feedback. Under some conditions of "good luck in guessing" the correct answer might influence the test result to some extent.

The lack of informed knowledge about listening subskills and their relationships has hindered the development of the diagnostic English language track assessment (DELTA) in three participating Hong Kong universities. This study investigates English as a foreign language (EFL) learners' listening proficiency development in understanding different spoken genres in the Hong Kong Chinese tertiary contexts. It aims to: i) identify the subskills and/or cognitive processes that underlie student performance on the DELTA listening component; ii) examine the difficulty levels of the DELTA listening subskills, and, consequentially, their hierarchical order; iii) investigate the impact of text type on difficulty level and the hierarchical order of the subskills; and iv) infer principles underlying the development of listening proficiency in the Hong Kong tertiary education contexts. A multi-method approach was employed for data collection and analysis. The primary quantitative data were derived from the DELTA listening component items answered by 2830 Chinese ELF learners who studied in their first or second year in the DELTA participating universities in the 2013-14 academic year. The item pool included 207 multiple-choice questions (MCQ) from 33 texts of three text types – conversation, interview and lecture. Each MCQ is intended to measure a particular listening subskill, including: 1) identifying specific information (SSK1); 2) understanding main idea and supporting ideas (SSK2); 3) understanding information and making an inference (SSK3); 4) interpreting a word or phrase as used by the speaker (SSK4); 5) inferring the attitude or intention of the speaker (SSK5); and 6) inferring the speaker's reasoning (SSK6). By adopting inter-related Rasch analyses using Winsteps and Facets, all test items were calibrated and analysed to determine their difficulty measures and their respective difficulties across the three text types. Qualitative Stimulated Recall Protocol (SRP) discussions were then conducted with 62 examinees of varying estimated listening abilities one month later, in a simulated test situation, where the test-taking process was video-recorded and the participants were asked to recall and to verbalise their thought processes and strategies they used to answer each question. The SRP results reveal an array of both cognitive processes and test-taking strategies in the listening comprehension and test-answering process. Firstly, various combinations of cognitive processes were utilised by both the high and low ability examinees to answer questions targeting the same listening sub-skill; however, the dominant cognitive process that was reported to have been used to answer each question corresponded with the particular listening subskill intended by DELTA item writers. Secondly, an array of test-taking strategies best identified as elimination, and guessing, were reported as used by examinees during the test. While this finding might not be surprising given the exam-oriented atmosphere prevailing in Hong Kong secondary school education, it alerted the researcher to scrutinise the validity of the DELTA listening component. The most striking observation from the listening test analysis is that, the DELTA listening subskills are measurably separable from each other, and a hierarchical pattern is established. In terms of their interaction with text type, the results showed that SSK1 and SSK6 were, respectively, the easiest and the most difficult subskills, whereas the hierarchical orders of the other four subskills varied across the three text types. More generally, these findings provide empirical evidence for the proposition that EFL listening comprehension is composed of multiple listening subskills, which operate interactively and interdependently in the listening process. The results regarding the difficulty level and the hierarchy of listening subskills corroborate the findings of prior research that low-level processing, such as identifying specific information, poses less challenge than high-level processing, such as summarising and inferencing. Because of the complexity in the interaction between text type and listening subskills, it is difficult to identify an overarching hierarchical order of the six listening subskills across the three text types. A general pattern, however, is that the difficulty increased from SSK1, SSK2 to SSK6 irrespective of the text type, and this corresponds to the general subskill hierarchy. The study will benefit teachers and students with diagnostic profiling and bridge the gap in diagnostic test design with targeted items of appropriate difficulty for predicting learners' listening development. It will extend second language acquisition theory with a hierarchical trajectory of listening proficiency growth. Limitations and future research recommendations are discussed.

본 논문에서는 난이도를 고려하여 선택형 문항을 자동으로 생성하는 방법을 고안하였으며, 학습자 수준에 적합하도록 동적인 형태로 다양한 문항 제시를 할 수 있는 시스템을 구현하였다. 선택형 문제를 통한 평가에서는 적절한 규모의 문제 은행이 필요하다. 이와 같은 요구를 만족시키기 위해서는 보다 쉽고 빠른 방식으로 다양하고 많은 문제 및 문항을 생성할 수 있는 시스템이 필요한데, 본 논문에서는 문제 및 문항의 생성을 위하여 워드넷이라는 언어 자원을 이용한 자동 생성 방법을 고안하였다. 자동 생성을 위해서는 주어진 문장에서 형태소 분석을 통해 키워드를 추출하고, 각 키워드마다 워드넷의 계층적 특성에 따라 유사한 의미를 가진 후보 단어를 제시한다. 의미 유사 후보 단어를 제시할 때, 기존의 한국어 워드넷의 스키마를 개념간 의미 유사도 행렬을 구할 수 있는 형태의 스키마로 변경한다. 단어의 의미 유사도는 동의어를 의미하는 수준 0에서 거의 유사도가 없다고 볼 수 있는 수준 9까지 다양하게 제시될 수 있으며, 생성될 문항에 어느 정도의 유사도를 가진 어휘를 포함시키느냐에 따라서 출제자의 의도에 따른 난이도의 조정이 가능하다. 후보 어휘들의 의미 유사도 측정을 위해서, 본 논문에서는 두 가지 방법을 사용하여 구현하였다. 첫째는 단순히 두 어휘의 워드넷 상에서의 거리만을 고려한 것이고 둘째는 두 어휘가 포함되어 있는 트리 구조의 크기까지 추가적으로 고려한 것이다. 이러한 방법을 통하여 실제 출제자가 기존에 출제된 문제를 토대로 더 다양한 내용과 난이도를 가진 문제 또는 문항을 더 쉽게 출제할 수 있는 시스템을 개발할 수 있었다. We implemented a system being able to suggest example sentences for multiple choice tests, considering the level of students. To build the system, we designed an automatic method for sentence generation, which made it possible to control the difficulty degree of questions. For the proper evaluation in the multiple choice tests, proper size of question pools is required. To satisfy this requirement, a system which can generate various and numerous questions and their example sentences in a fast way should be used. In this paper, we designed an automatic generation method using a linguistic resource called WordNet. For the automatic generation, firstly, we extracted keywords from the existing sentences with the morphological analysis and candidate terms with similar meaning to the keywords in Korean WordNet space are suggested. When suggesting candidate terms, we transformed the existing Korean WordNet scheme into a new scheme to construct the concept similarity matrix. The similarity degree between concepts can be ranged from 0, representing synonyms relationships, to 9, representing non-connected relationships. By using the degree, we can control the difficulty degree of newly generated questions. We used two methods for evaluating semantic similarity between two concepts. The first one is considering only the distance between two concepts and the second one additionally considers positions of two concepts in the Korean Wordnet space. With these methods, we can build a system which can help the instructors generate new questions and their example sentences with various contents and difficulty degree from existing sentences more easily.

A college English test generation model was constructed based on a corpus in this study. By combining common linguistic datasets, the automatic item generation method was adopted for large-scale testing. The corpus-based approach was applied for English language instruction. Corpus construction, preprocessing, vocabulary analysis, and other relevant components were integrated for effective test item generation. A methodology using word lists with word ratios and other new metrics was derived from preference words and levels of difficulty to calculate sentence difficulty and its text complexity index. To address the challenges of previous systems, challenges in multiple-choice tests were addressed. The developed model uses corpus processing and machine learning algorithms to generate test questions at all levels of difficulty. The developed system solves problems of the current college English systems.