Deep Learning and Linguistic Feature Based Automatic Multiple Choice Question Generation from Text

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are not useful in real educational applications. This is because of the inability to produce the diverse and human alike distractors. Distractors are the wrong choices given along with the correct answer (key) to confuse the examinee. Especially, in educational domain (grammar learning) the MCQs deal with affix-based or morphologically transformed distractors. In this paper, we present a method for automatic generation of affix-based distractors for fill-in-the-blanks for learning Tamil Vocabulary. Affix-based distractor generation relies on certain regularities manifest in high dimensional spaces. We investigate the quality of distractors generated by a number of criteria, including Part-Of-Speech, difficulty level, spelling, word co-occurrence, semantic similarity and affixation. We evaluated our proposed method in grammar based Multiple Choice Questions (MCQs) dataset. The result shows that affix-based distractors, yield significantly more plausible outcomes in certain grammar based questions.

Background: Multiple-choice questions (MCQs) are the most frequently accepted tool for the evaluation of comprehension, knowledge, and application among medical students. In single best response MCQs (items), a high order of cognition of students can be assessed. It is essential to develop valid and reliable MCQs, as flawed items will interfere with the unbiased assessment. The present paper gives an attempt to discuss the art of framing well-structured items taking kind help from the provided references. This article puts forth a practice for committed medical educators to uplift the skill of forming quality MCQs by enhanced Faculty Development programs (FDPs). Objectives: The objective of the study is also to test the quality of MCQs by item analysis. Methods: In this study, 100 MCQs of set I or set II were distributed to 200 MBBS students of Late Shri Lakhiram Agrawal Memorial Govt. Medical College Raigarh (CG) for item analysis for quality MCQs. Set I and Set II were MCQs which were formed by 60 medical faculty before and after FDP, respectively. All MCQs had a single stem with three wrong and one correct answers. The data were entered in Microsoft excel 2016 software to analyze. The difficulty index (Dif I), discrimination index (DI), and distractor efficiency (DE) were the item analysis parameters used to evaluate the impact on adhering to the guidelines for framing MCQs. Results: The mean calculated difficulty index, discrimination index, and distractor efficiency were 56.54%, 0.26, and 89.93%, respectively. Among 100 items, 14 items were of higher difficulty level (DIF I < 30%), 70 were of moderate category, and 16 items were of easy level (DIF I > 60%). A total of 10 items had very good DI (0.40), 32 had recommended values (0.30 - 0.39), and 25 were acceptable with changes (0.20 - 0.29). Of the 100 MCQs, there were 27 MCQs with DE of 66.66% and 11 MCQs with DE of 33.33%. Conclusions: In this study, higher cognitive-domain MCQs increased after training, recurrent-type MCQ decreased, and MCQ with item writing flaws reduced, therefore making our results much more statistically significant. We had nine MCQs that satisfied all the criteria of item analysis.

A comparison of clinical-scenario (case cluster) versus stand-alone multiple choice questions in a problem-based learning environment in undergraduate medicine.

PurposeTo differentiate complicated appendicitis (CA) from uncomplicated appendicitis (UA) in children, we developed and validated an artificial intelligence (AI) model using a multimodal approach integrating ultrasound images and clinical data.MethodsA retrospective analysis was performed on 372 pathologically confirmed pediatric appendicitis cases (230 male, 142 female) from three centers, all with preoperative abdominal ultrasound. Deep learning (DL) features and radiomic features were extracted from appendiceal ultrasound images using deep transfer learning (DTL) and conventional radiomic methods, respectively. We selected 12 radiomic features, 9 DL features, and 3 clinical features—namely, white blood cell count (WBC), neutrophil count (NEU), and C-reactive protein (CRP)—for building the machine learning classification model. Based on these features, four distinct models were constructed: the Rad model (radiomic features only), the DL model (DL features only), the DTL model (combined radiomic and DL features), and the Combine model (integrating all three feature types: radiomic, DL, and clinical features). Model performance was evaluated using receiver operating characteristic (ROC) curves, decision curve analysis (DCA), and the DeLong test. Finally, the combined model was compared to the performance of clinicians with varying levels of experience.ResultsThe combined model demonstrated consistently favorable performance across all cohorts (AUC: 0.940, 0.895, 0.866, and 0.783 for training and validation sets, respectively). The model's accuracy (0.862) and positive predictive value (0.896) were comparable to senior surgeons (0.741, 0.970) and superior to junior surgeons (0.672, 0.865) in the internal validation cohort. DCA confirmed the clinical utility of the combined model over conventional strategies.ConclusionOur ultrasound-based AI model provides reliable differentiation between CA and UA in children, offering potential value as a diagnostic support tool for clinical decision making.

Introduction: Multiple choice questions (MCQs) are widely used in medical education. Difficulty Index (DIF I), Discrimination Index (DI), and Distractor Efficiency (DE) are the most commonly used item analyses. MCQ authors should anticipate the possible difficulty level in writing their questions and carefully arrange distractors. This study aims to evaluate the quality of MCQ and the predictions of MCQ authors about the difficulty levels of their questions. Methods: In this study, the DIF I, DI and DE values of 688 MCQs in the exams held in the first year of the 2021-2022 academic year of Dokuz Eylül University Faculty of Medicine were investigated. The effect of the number of Non-Functional Distractors (NFD) on DIF I and DI, and the compliance between the DIF I predictions of the question authors and the actual values was evaluated. Results: DIF I mean was 0.57±0.21, and 47.5% was at the ideal difficulty level. There was a significant difference between the estimated and the actual DIF I (p=0.002). The DI average was 0.31 ± 0.17, and the discrimination level of 43.6% was excellent. 36.8% of distractors were NFD. MCQ's difficulty and discriminatory ability were significantly different according to the number of NFDs (p=0.000 for all). Discussion: It was determined that the number of NFDs significantly affected difficulty and discriminatory ability. There was a difference between the estimates of the difficulty and the actual values. Reducing the number of options in MCQs and being more careful when crafting questions can improve the quality of the questions.

We performed 4 experiments assessing the learning that occurs when taking a test. Our experiments used multiple-choice tests because the processes deployed during testing can be manipulated by varying the nature of the choice alternatives. Previous research revealed that a multiple-choice test that includes “none of the above” (NOTA) produces better performance on a subsequent test only when the correct answer is something other than NOTA (Odegard & Koen, 2007). However, when NOTA was an incorrect choice alternative, the correct answer was the only familiar alternative. Thus, familiarity may have allowed participants to identify the answer, which was then restudied. In other words, the testing benefit might have reflected a familiarity-guided restudy process rather than retrieval practice. In the current study, we examined the role of familiarity in the multiple-choice testing effect, manipulating the familiarity of alternatives. If NOTA was the correct answer, there was no testing benefit when the alternatives were all novel (Experiment 1) or all familiar (Experiment 3). Familiarity-guided restudy predicts memory impairment when there is a single familiar alternative for a NOTA-correct question. In contradiction to this hypothesis, there was a testing benefit for this condition (Experiments 2 and 4). Experiment 4 further collected metacognitive confidence ratings during the multiple-choice test, providing evidence of a recall-to-reject strategy for this condition. These results suggest that learning from multiple-choice tests is mainly due to retrieval practice rather than the use of familiarity.

Automatic question generation from text or paragraph is a great challenging task which attracts broad attention in natural language processing. Because of the verbose texts and fragile ranking methods, the quality of top generated questions is poor. In this paper, we present a novel framework Automatic Chinese Question Generation (ACQG) to generate questions from text or paragraph. In ACQG, we use an adopted TextRank to extract key sentences and a template-based method to construct questions from key sentences. Then a multi-feature neural network model is built for ranking to obtain the top questions. The automatic evaluation result reveals that the proposed framework outperforms the state-of-the-art systems in terms of perplexity. In human evaluation, questions generated by ACQG rate a higher score.

Comparison of examination methods based on multiple-choice questions and constructed-response questions using personal computers

Background: Multiple-choice questions (MCQs) provide useful information about correct and incorrect answers, but they do not offer information about students’ confidence.Methods: Ninety and another 81 medical students participated each in a curricular neurology multiple-choice exam and indicated their confidence for every single MCQ. Each MCQ had a defined level of potential clinical impact on patient safety (uncritical, risky, harmful). Our first objective was to detect informed (IF), guessed (GU), misinformed (MI), and uninformed (UI) answers. Further, we evaluated whether there were significant differences for confidence at correct and incorrect answers. Then, we explored if clinical impact had a significant influence on students’ confidence.Results: There were 1818 IF, 635 GU, 71 MI, and 176 UI answers in exam I and 1453 IF, 613 GU, 92 MI, and 191 UI answers in exam II. Students’ confidence was significantly higher for correct than for incorrect answers at both exams (p < 0.001). For exam I, students’ confidence was significantly higher for incorrect harmful than for incorrect risky classified MCQs (p = 0.01). At exam II, students’ confidence was significantly higher for incorrect harmful than for incorrect benign (p < 0.01) and significantly higher for correct benign than for correct harmful categorized MCQs (p = 0.01).Conclusions: We were pleased to see that there were more informed than guessed, more uninformed than misinformed answers and higher students’ confidence for correct than for incorrect answers. Our expectation that students state higher confidence in correct and harmful and lower confidence in incorrect and harmful MCQs could not be confirmed.

ABSTRACT. Evaluating learning results is essential in all learning processes. Multiple choice tests are frequently used to objectively evaluate and recognize these results. Solving multiple choice tests by well constructed items requires not only recalling existing knowledge but needs comprehension, interpretation as well as analysis and synthesis. Multiple choice tests are widely used, being easy to administer and to check, especially in big groups. In this article, we further suggest how to improve the definition and the options for each item (correct answer and distractors) and propose how to analyze test-formats already used such as to create a pool of high quality items (for future use). This analysis is based on the percentage of correct answers, the discrimination index and the degree of difficulty of each item in different student groups. Analyzing items of multiple choice tests already used may obviously be useful for improving the ability to construct such tests in the future but also may serve in the classroom (teachers and students) to discuss the results of a test as a metacognitive exercise, reflecting the mental strategies by which students intend to solve the items recognizing difficulties to do so. The analysis often shows weak points in the teaching and learning process, to be considered when reviewing the curriculum.

INTRODUCTION Towards exams, students want to get the most out of revising, but some study methods may be ineffective and boring. Using quizzes for retrieval practice is a valuable technique, and peer learning can help with understanding topics in a new way. We adapted aspects of the team‐based learning method into an engaging revision session; e.g. students undertaking a test individually then as a team, thereby incorporating retrieval practice and peer learning. Including many questions in a short timeframe was designed to reveal knowledge gaps and keep the session enjoyable. AIM This study sought to explore students’ experience of teamwork during a broad‐ranging, fastpaced anatomy revision session. METHODS The revision session formed the final anatomy “laboratory” session before end‐of‐year exams for the 2019 cohort of Stage 1 postgraduate students (n=274) in the Sydney Medical Program (University of Sydney). Students were allocated to one of three classes, and were asked to form teams of 6 – 7 students. Multiple choice questions (MCQs) were presented in three rounds of 10 questions, with 30 seconds per question. Students attempted each round of 10 MCQs first by themselves, then immediately completed the same MCQs as a team. Correct answers were then explained to all students. Four extended matching questions (EMQs) providing anatomically‐themed clinical scenarios were also presented for team discussion only. All teams revealed their answers simultaneously, before solutions were explained. Paper‐based surveys were distributed at the end of the session, asking students to rate the usefulness of the MCQs and teamwork on a five‐point Likert‐type scale; freeform comments were also requested. Data were analysed with descriptive statistics. Ethics approval was obtained via the University of Sydney Human Research Ethics Committee . RESULTS From the 191 students who attended (70% of cohort), 518 individual sets of MCQ responses could be compared with team responses across all rounds. Most (83.2%) responses were higher in their team scores than individual scores. Only 3.7% of responses were lower in team rather than individual scores (13.1% achieved the same score). The mean rate of improvement from individual to team score was 157.7%. Surveys were returned by 157 students (82.2%). Of these, 72.6% “strongly agreed” (a further 21.0% “agreed”) that the MCQs helped identify knowledge gaps. Regarding working as a team following individual work, almost all strongly agreed (79.0%) or agreed (15.9%) that it had been beneficial to their learning process. Written feedback reflected these feelings, with students also complimenting the range of questions, though a few felt they were too difficult, especially for the short time allowed. More EMQs were requested. DISCUSSION AND CONCLUSION Students responded positively to this team‐based revision. Although fast‐paced, most students coped with the short timeframes, and appreciated being exposed to multiple high‐quality questions (and, importantly, answers). Through collaboration in their small teams, students benefitted from peer learning, and at the same time enjoyed the teamwork and camaraderie in the otherwise tense build‐up to exams.

Objectives&#x0D; One-best MCQs (Multiple Choice Questions) is a popular form of assessment where the student selects the best possible answer from the options provided. The objective of this study was to analyze the MCQs used in preliminary examinations in terms of Difficulty Index, Differentiation Index and Distracter efficiency.&#x0D; Method&#x0D; Total 100 MCQ items with one correct option and three distracters were taken from past 5 years preuniversity examinations and analyzed for level of difficulty and power of discrimination. A total of 149, 156, 154, 156 and 148 students appeared for the exam in each year. The Difficulty index was calculated by the formula P = R/T. High and low groups consisting of upper and lower 27% students, respectively, were taken after arranging the scores in descending order for calculating the Discrimination Index as follows: DI = 2 x [(H-L)/N]. Distracter efficiency was determined on the basis of the number of Nonfunctioning Distracters (NF-Ds). An ideal item should have a DFI between 0.30–70.0, a DI of &gt;0.24 and a DE of 100%.&#x0D; Results&#x0D; The Difficulty indices for the ten papers of 10 MCQ`s each were (Mean±S. D) 0.50±0.24, 0.49±0.25, 0.52±0.23, 0.55±0.19, 0.73±0.22, 0.68±0.18, 0.53±0.24, 0.65±0.18, 0.54±0.3, 0.47±0.26 respectively. Differentiation indices were (Mean±S. D)0.37±0.12, 0.34±0.17, 0.35±0.14, 0.43±0.15, 0.40±0.15, 0.43±0.17, 0.40±0.20, 0.46±0.11, 0.40±0.16, 0.40±0.20. Thus as a whole, the papers were of average difficulty and were able to differentiate between the students. Thirtyfive MCQs were ideal MCQs. In individual analysis, total 37 MCQs were very easy (Difficulty Index&gt;0.70) and 18 were found to be very tough (Difficulty Index&lt;0.30). In Distracter analysis, 20% of the distracters were found to be nonfunctional (NFDs). 30% of the MCQ`s had One NFD, while 9% and 4% had two and three NFD`s respectively.&#x0D; Conclusion&#x0D; Overall, the papers had very good difficulty and differentiation indices which were quite consistent over the five years but there are a few MCQs and distracters which need to be modified to further enhance accurate assessment of the students.

The paper deals with two-tier multiple choice tests working as a reliable tool for students'evaluation. The test was applied within the engineering education in the foreign language instruction at the Faculty of Informatics and Management, University of Hradec Kralove, Czech Republic. The multiple choice tests are often considered easier to pass even if the level of respondent's knowledge is low, and the reached result thus does not reflect the knowledge adequately. Two hypotheses were set to verify whether learners reach higher test scores in translations in comparison to multiple choice tests and to multiple choice tests with feedback. Under some conditions of "good luck in guessing" the correct answer might influence the test result to some extent.

The lack of informed knowledge about listening subskills and their relationships has hindered the development of the diagnostic English language track assessment (DELTA) in three participating Hong Kong universities. This study investigates English as a foreign language (EFL) learners' listening proficiency development in understanding different spoken genres in the Hong Kong Chinese tertiary contexts. It aims to: i) identify the subskills and/or cognitive processes that underlie student performance on the DELTA listening component; ii) examine the difficulty levels of the DELTA listening subskills, and, consequentially, their hierarchical order; iii) investigate the impact of text type on difficulty level and the hierarchical order of the subskills; and iv) infer principles underlying the development of listening proficiency in the Hong Kong tertiary education contexts. A multi-method approach was employed for data collection and analysis. The primary quantitative data were derived from the DELTA listening component items answered by 2830 Chinese ELF learners who studied in their first or second year in the DELTA participating universities in the 2013-14 academic year. The item pool included 207 multiple-choice questions (MCQ) from 33 texts of three text types – conversation, interview and lecture. Each MCQ is intended to measure a particular listening subskill, including: 1) identifying specific information (SSK1); 2) understanding main idea and supporting ideas (SSK2); 3) understanding information and making an inference (SSK3); 4) interpreting a word or phrase as used by the speaker (SSK4); 5) inferring the attitude or intention of the speaker (SSK5); and 6) inferring the speaker's reasoning (SSK6). By adopting inter-related Rasch analyses using Winsteps and Facets, all test items were calibrated and analysed to determine their difficulty measures and their respective difficulties across the three text types. Qualitative Stimulated Recall Protocol (SRP) discussions were then conducted with 62 examinees of varying estimated listening abilities one month later, in a simulated test situation, where the test-taking process was video-recorded and the participants were asked to recall and to verbalise their thought processes and strategies they used to answer each question. The SRP results reveal an array of both cognitive processes and test-taking strategies in the listening comprehension and test-answering process. Firstly, various combinations of cognitive processes were utilised by both the high and low ability examinees to answer questions targeting the same listening sub-skill; however, the dominant cognitive process that was reported to have been used to answer each question corresponded with the particular listening subskill intended by DELTA item writers. Secondly, an array of test-taking strategies best identified as elimination, and guessing, were reported as used by examinees during the test. While this finding might not be surprising given the exam-oriented atmosphere prevailing in Hong Kong secondary school education, it alerted the researcher to scrutinise the validity of the DELTA listening component. The most striking observation from the listening test analysis is that, the DELTA listening subskills are measurably separable from each other, and a hierarchical pattern is established. In terms of their interaction with text type, the results showed that SSK1 and SSK6 were, respectively, the easiest and the most difficult subskills, whereas the hierarchical orders of the other four subskills varied across the three text types. More generally, these findings provide empirical evidence for the proposition that EFL listening comprehension is composed of multiple listening subskills, which operate interactively and interdependently in the listening process. The results regarding the difficulty level and the hierarchy of listening subskills corroborate the findings of prior research that low-level processing, such as identifying specific information, poses less challenge than high-level processing, such as summarising and inferencing. Because of the complexity in the interaction between text type and listening subskills, it is difficult to identify an overarching hierarchical order of the six listening subskills across the three text types. A general pattern, however, is that the difficulty increased from SSK1, SSK2 to SSK6 irrespective of the text type, and this corresponds to the general subskill hierarchy. The study will benefit teachers and students with diagnostic profiling and bridge the gap in diagnostic test design with targeted items of appropriate difficulty for predicting learners' listening development. It will extend second language acquisition theory with a hierarchical trajectory of listening proficiency growth. Limitations and future research recommendations are discussed.

Multiple Choice Questions (MCQs) are a very well known, traditional and accepted method of assessment. The use of MCQs for testing students has produced numerous debates amongst academics concerning their effectiveness as they are viewed as practical and efficient but also perceived as possibly „too easy‟ and potentially unable to appropriately test the higher order cognitive skills that essay questions can assess.The use of MCQs in a forensic science context is currently being investigated, not only for use within forensic science education, but also for the testing of competency of qualified forensic practitioners. This paper describes a Higher Education Academy funded project that is investigating the design and the implementation of MCQs for testing forensic practitioners and the lessons that have been learnt so far, that will assist academics in the development of robust MCQ assessments within forensic science degrees to promote and assess deep learning.