TreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice Questions

PurposeTechnological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy. Methods and MaterialsA set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers could be clearly assigned to one of the answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers. ResultsFor 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as “acceptable,” “good,” or “very good” regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered “very good” in 29.3% and 28%, “good” in 28% and 29.3%, “acceptable” in 19.3% and 19.3%, “bad” in 9.3% and 9.3%, and “very bad” in 14% and 14% regarding correctness/helpfulness. ConclusionsModern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.

Background/Objectives: Otolaryngologists are increasingly using head and neck ultrasound (HNUS). Determining whether a practitioner of HNUS has achieved adequate theoretical knowledge remains a challenge. This study assesses the performance of two large language models (LLMs) in generating multiple-choice questions (MCQs) for head and neck ultrasound, compared with MCQs generated by physicians. Methods: Physicians and LLMs (ChatGPT, GPT4o, and Google Gemini, Gemini Advanced) created a total of 90 MCQs that covered the topics of lymph nodes, thyroid, and salivary glands. Experts in HNUS additionally evaluated all physician-drafted MCQs using a Delphi-like process. The MCQs were assessed by an international panel of experts in HNUS, who were blinded to the source of the questions. Using a Likert scale, the evaluation was based on an overall assessment including six assessment criteria: clarity, relevance, suitability, quality of distractors, adequate rationale of the answer, and an assessment of the level of difficulty. Results: Four experts in the clinical field of HNUS assessed the 90 MCQs. No significant differences were observed between the two LLMs. Physician-drafted questions (n = 30) had significant differences with Google Gemini in terms of relevance, suitability, and adequate rationale of the answer, but only significant differences in terms of suitability compared with ChatGPT. Compared to MCQ items (n = 16) validated by medical experts, LLM-constructed MCQ items scored significantly lower across all criteria. The difficulty level of the MCQs was the same. Conclusions: Our study demonstrates that both LLMs could be used to generate MCQ items with a quality comparable to drafts from physicians. However, the quality of LLM-generated MCQ items was still significantly lower than MCQs validated by ultrasound experts. LLMs are therefore cost-effective to generate a quick draft for MCQ items that afterward should be validated by experts before being used for assessment purposes. In this way, the value of LLM is not the elimination of humans, but rather vastly superior time management.

Reinforcement learning (RL) for large language models (LLMs) typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling.We investigate whether LLMs benefit from training on mixed data with varying reward clarity.Our approach combines Multiplechoice questions (MCQs), which offer clear binary rewards, with OE questions, for which we use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators.We hypothesize that MCQs can stabilize training when mixed with OE questions.Our experiments show this mixed-data approach consistently improves medical question-answering performance across model scales.

Aim/Purpose: The study investigates the factors influencing the acceptance and utilisation of large language models (LLMs) (predictor variables of LLM usage), such as ChatGPT, in Learning design by instructional designers and university-teaching academics from various countries. Background: Large language models (LLMs) have exploded onto the scene, transforming the landscape of learning design. Instructional designers and university teaching academics have been overburdened with content creation for their teaching programmes, and the arrival of LLM models will help in this regard by developing more interactive content that drives student engagement and, in turn, contributes to student success. Since LLMs are a relatively new phenomenon, little is known about the factors influencing their acceptance in learning design; therefore, this research is needed, as learning design principles are the bedrock of student engagement and success. Methodology: A cross-sectional correlational quantitative study was employed. Data was collected using an online questionnaire posted on social media, including LinkedIn, from 203 instructional designers and university teaching academics. Purposive and snowball sampling methods were used to target instructional designers and university teaching academics at colleges and universities worldwide. Participants were asked to share the survey link with fellow instructional designers and university-teaching academics in their communities. The factor structure of the data was determined using exploratory factor analysis. Nonetheless, the factor structure derived from the LLMs did not entirely reflect the original configuration of the Unified Theory of Acceptance and Use of Technology (UTAUT3), as certain predictors appeared to coalesce, indicating LLMs’ unique nature in learning design. Confirmatory factor analysis was used to verify the fit of the data on the measurement model. First-order and second-order structural modelling were used to identify the structural relationships among the variables. Contribution: The study determines significant factors for the acceptance of LLMs by instructional designers and academic teaching staff in learning design, enabling possible opportunities for best practices in the field through interventions to optimize LLM usage. The study applies the technology acceptance model to the emerging LLM technology and extends the technology acceptance model by adding the trust construct as a predictor variable. Findings: The structural analysis results indicated that the ingrained LLM practices, LLM peer-driven expectations, innovative propensity towards LLM adoption, reliability and provider trust in LLMs, and ease of use and support influenced perceived LLM benefits and usage, but community standards and infrastructure had no influence. The second-order structural equation modelling indicated that perceived LLM benefits and usage and ingrained LLM habits contributed most to the learning design. Recommendations for Practitioners: Teaching academics and instructional designers must use LLMs in designing content, assessments, and interactive learning activities, and attend LLM training workshops on prompting and best practices in integrating LLMs into learning and teaching to see their benefits; hence, regular use of LLMs will then lead to trust and innovation in LLMs usage, enhancing learning design and improving student learning outcomes. Recommendation for Researchers: Researchers must use mixed methods approaches to have a deeper understanding of the factors influencing LLMs. Since habit and perceived LLM benefits and usage contributed the most variance to learning design, researchers must investigate strategies that optimise these factors in learning design, such as effective intervention strategies that can help form positive LLM habits. In addition, the findings provide researchers with a starting point for future research. Further researchers must investigate interventions that optimise the influence of personal innovativeness and trust that contributed the least variance to learning design, hence unlocking the potential of LLMs in learning design through innovation, responsible, and ethical use. Impact on Society: The use of LLMs in learning design has a high possibility of transforming education, specifically the learning design landscape. Using LLMs will free up more time for teaching academics and instructional designers so that they spend more time on higher-order thinking skill demands. Consequently, the students will be exposed to more engaging and interactive content, resulting in improved learning outcomes. Future Research: Future research must include context-derived external variables in technology acceptance models, such as levels of prompting competencies, to provide a deeper understanding of LLMs. In addition, future research must be based on the application and impact of LLMs on student engagement and success, and their attainment of 21st-century skills.

This contribution will report on, and evaluate, the use of multiple-choice questions, in both continuous assessment and terminal examination modes, on a first year module in electrical engineering at Dublin Institute of Technology. The author’s experiences are that multiple-choice questions tend to be student friendly (e.g. given a choice in a terminal examination, students will opt to attempt the questions with multiple choice parts). The author has used the assessment method over three academic years, and some assessment data is reported and analysed in the contribution. The pedagogical approach is explicitly pragmatic.

Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform.We first reveal flaws in MCQA's format, as it struggles to: 1) test generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.We instead advocate for generative formats based on human testing-where LLMs construct and explain answers-better capturing user needs and knowledge while remaining easy to score.We then show even when MCQA is a useful format, its datasets suffer from: leakage; unanswerability; shortcuts; and saturation.In each issue, we give fixes from education, like rubrics to guide MCQ writing; scoring methods to bridle guessing; and Item Response Theory to build harder MCQs.Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful explanations-showing how our prior solutions better measure or address these issues.While we do not need to desert MCQA, we encourage more efforts in refining the task based on educational testing, advancing evaluations. Questioning Multiple Choice QuestionsMultiple choice question answering (MCQA) is the standard for large language model (LLM) evaluations, prized for simplicity and similarity to human testing (Robinson and Wingate, 2023).When designing new benchmarks, MCQA seems easy to implement (Guo et al., 2023), and when selecting new LLMs to use, MCQA leaderboards inform our decisions (Fourrier et al., 2024).If you want to build a popular dataset, prove your LLM is smart, or even publish a position paper, it is hard to avoid MCQA.Standardized testing groups have long explored ways to better use MCQA for student testing (Angoff, 1971).But despite years of use in NLP (Turney et al., 2003), few have asked: 1) should MCQA be a standard model evaluation format; and 2) are its datasets well-designed?This position paper argues: Evaluating LLMs with MCQA has flaws Q1.What's wrong with MCQA's format?A) It doesn't apply to many tasks ( 3.1) B) It's misaligned with LLM use cases ( 3.2) C) It doesn't fully test knowledge ( 3.3) Q2.What's wrong with MCQA datasets?A) Test sets are contaminated ( 5.1) B) They have unanswerable questions ( 5.2) C) They contain shortcuts ( 5.3) D) They're too easy for LLMs ( 5.4) Q3.How do LLMs struggle with MCQA?A) They lack robustness ( 6.1) B) They exhibit biases ( 6.2) C) They give unfaithful explanations ( 6.3) Q4.How can insights from education improve MCQA?A) Improve knowledge testing via generative formats ( 4) B) Combat test set leakage with fresh questions ( 5.1) C) Write MCQs informed by educational rubrics ( 5.2) D) Use calibration scoring to curb guessing ( 5.3.1)E) Find harder MCQs with item response theory ( 5.4.1)7 gpt-3 has seen 45% of RACE's test set (Sainz et al., 2023).

Background & Objective: In dental education, understanding optimal assessment methods and factors like stress and confidence is essential.This research assessed second-year dental students' performance in fixed prosthodontics using multiple-choice questions (MCQs) and preclinical practical exams, examining impacts of gender, stress, and confidence.Materials & Methods: Using a quasi-experimental design, 495 students from a single faculty underwent assessment.Selected via convenience sampling, they were exposed to MCQs and practical exams in fixed prosthodontics.An expert-reviewed questionnaire gauged their stress and confidence.Data was analyzed using descriptive statistics, t-tests, and Pearson's correlation.Additionally, a balanced sub-set of 176 students (88 males and 88 females) was chosen for gender-based analysis.Results: Findings indicate statistical parity between MCQ and practical exam performances (p>0.001).Females slightly outperformed in MCQs, while males excelled in practicals, without reaching statistical significance (p>0.05).Stress correlated with practical exam outcomes (r=0.34,p=0.001), and confidence with MCQ scores (r=0.41,p<0.0001). Conclusion:The research underscores near-equivalence of MCQs and practical exams for student assessments in fixed prosthodontics.Recognizing the roles of stress and confidence in assessments offers insights for balanced evaluations.Dental faculties should integrate these findings, and future work should pivot towards tool validations for enriched learning.

While large language models (LLMs) perform well in medical education, their ability to accurately interpret and answer English and Arabic dental multiple-choice questions (MCQs) remains underexplored. This study evaluated the performance of advanced LLMs in answering dental MCQs in both languages, identifying language-specific challenges and assessing their applicability in multilingual dental education. A total of 300 MCQs from ten dental specialties were sourced from question banks. The MCQs were translated into Arabic and reviewed for linguistic and technical accuracy. Four LLMs, ChatGPT-4o, ChatGPT-4, Gemini, and Claude, were tested separately on Arabic and English datasets. Accuracy was the primary metric, along with specialty-specific performance, question type differentiation, and cross-language consistency. Claude achieved the highest accuracy in English (89%), while Gemini performed best in Arabic (80%). Most models showed better performance in English, with notable translation inconsistencies, particularly for ChatGPT models. Specialty-wise, Claude and Gemini excelled in endodontics and operative dentistry. No significant differences were observed between knowledge-based and clinical questions, but Arabic interpretation posed challenges. Statistical analysis confirmed significant differences between models and across languages. Gemini demonstrated robust performance in Arabic, while Claude excelled in English. ChatGPT models exhibited limitations, particularly in Arabic datasets. Performance varied across specialties, highlighting the need for improved multilingual adaptability and specialty-specific training. Expanding specialised and culturally relevant datasets is essential for optimising LLMs' educational utility. This study provides key insights into LLM performance in bilingual dental education, supporting future advancements in AI-driven learning tools.

UNSTRUCTURED This study aims to assess the potential of large language models (LLMs) to enhance reporting efficiency and accuracy in oncological imaging, specifically evaluating their knowledge of RECIST 1.1 guidelines. While the capabilities of LLMs have been explored across various domains, their specific applications in radiology are of significant interest due to the intricate and time-consuming nature of image evaluation in oncology. We conducted a comparative analysis involving seven different LLMs and a general radiologist (GR) to determine their proficiency in responding to RECIST 1.1-based multiple-choice questions. Our methodology involved the creation of 25 multiple-choice questions by a board-certified radiologist, ensuring alignment with RECIST 1.1 guidelines. These questions were presented to seven LLMs—Claude 3 Opus, ChatGPT 4, ChatGPT 4o, Gemini 1.5 Pro, Mistral Large, Meta Llama 3 70B, and Perplexity Pro—as well as to a GR with six years of experience. The LLMs were prompted to answer as an experienced radiologist, and their responses were compared to those of the GR. The results demonstrated that Claude 3 Opus achieved a perfect accuracy of 100% (25/25), followed closely by ChatGPT 4o with 96% (24/25). ChatGPT 4 and Mistral Large both scored 92% (23/25), while Meta Llama 3 70B, Perplexity Pro, and Gemini 1.5 each scored 88% (21/25). The GR also achieved a score of 92% (23/25). These findings highlight the impressive proficiency of current LLMs in understanding and applying RECIST 1.1 guidelines, suggesting their potential as valuable tools in radiology. The outstanding performance of Claude 3 Opus raises the prospect of LLMs becoming integral to oncology practices, potentially enhancing the accuracy and efficiency of radiology reporting. However, the variations in performance among different models underscore the need for further refinement and evaluation. Additionally, while this study focused on text-based responses, the visual assessment capabilities of multimodal LLMs remain unexplored. Given the visual nature of radiology, future research should investigate the integration of visual analysis in LLMs to fully harness their potential in clinical settings. In conclusion, our study underscores the high potential of LLMs to assist radiologists in oncological reporting, providing a consistent and reliable approach to interpreting RECIST 1.1 guidelines. These findings advocate for the continued development and integration of LLMs in radiology to enhance diagnostic accuracy and reporting efficiency.

The purpose of this case study is to examine Korean university students’ perceptions about the academic English writing class they took during the Spring semester of 2022. More specifically, this study aims to understand how satisfied students were with their learning experiences in the academic English writing class, and how they perceived its appropriateness as a mandatory English general education offered in the Korean higher education context.A survey questionnaire consisting of Likert-scale, multiple choice, and open-ended questions was the main source of the data, and a total of 65 freshmen taking the course responded. To triangulate, in-depth interviews were also conducted with three focal participants via email or Zoom upon their choice. Descriptive statistics were calculated to analyze the responses to the Likert-scale and the multiple-choice questions, and the content analysis method was applied for open-ended survey questions and interviews.Analysis of data showed that students viewed their academic English writing experiences positively due to the content knowledge and sense of achievement gained through the class as well as the instructor’s teaching style. Some participants were dissatisfied with their learning experiences because of content difficulty, relatively heavy workload, and their insufficient English proficiency which hindered them from following the class. Most students self-evaluated that their writing skills after the class somewhat improved. Students chose writing an essay, learning about academic writing conventions, such as APA styles and in-text citations, and instructor’s feedback as class activities that helped them improve their academic English writing.To the question about the appropriateness of academic English writing as a mandatory general education subject, a vast majority of students hold a positive view. Their reasons include their belief of academic English writing as essential knowledge that they need to possess as a university student belonging to an academic discourse community as well as transferable knowledge needed not only in an English class but also in various disciplines. They also recognize its practicality for their academic activities and its rarity to learn anywhere else except university.The findings suggest that school administrators should allocate more credit hours for an academic English writing course and offer it for students in their 2nd or 3rd year of university. The findings also emphasize the importance of instructors’ effort to tailor their instruction according to their students and to provide supplementary learning materials, such as pre-recorded videos so as to effectively guide students to improve their skills for English writing for academic purposes.

This study aimed to evaluate the performance of large language models (LLMs) and multimodal LLMs in interpreting the Breast Imaging Reporting and Data System (BI-RADS) categories and providing clinical management recommendations for breast radiology in text-based and visual questions. This cross-sectional observational study involved two steps. In the first step, we compared ten LLMs (namely ChatGPT 4o, ChatGPT 4, ChatGPT 3.5, Google Gemini 1.5 Pro, Google Gemini 1.0, Microsoft Copilot, Perplexity, Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Opus 200K), general radiologists, and a breast radiologist using 100 text-based multiple-choice questions (MCQs) related to the BI-RADS Atlas 5th edition. In the second step, we assessed the performance of five multimodal LLMs (ChatGPT 4o, ChatGPT 4V, Claude 3.5 Sonnet, Claude 3 Opus, and Google Gemini 1.5 Pro) in assigning BI-RADS categories and providing clinical management recommendations on 100 breast ultrasound images. The comparison of correct answers and accuracy by question types was analyzed using McNemar's and chi-squared tests. Management scores were analyzed using the Kruskal- Wallis and Wilcoxon tests. Claude 3.5 Sonnet achieved the highest accuracy in text-based MCQs (90%), followed by ChatGPT 4o (89%), outperforming all other LLMs and general radiologists (78% and 76%) (P < 0.05), except for the Claude 3 Opus models and the breast radiologist (82%) (P > 0.05). Lower-performing LLMs included Google Gemini 1.0 (61%) and ChatGPT 3.5 (60%). Performance across different categories of showed no significant variation among LLMs or radiologists (P > 0.05). For breast ultrasound images, Claude 3.5 Sonnet achieved 59% accuracy, significantly higher than other multimodal LLMs (P < 0.05). Management recommendations were evaluated using a 3-point Likert scale, with Claude 3.5 Sonnet scoring the highest (mean: 2.12 ± 0.97) (P < 0.05). Accuracy varied significantly across BI-RADS categories, except Claude 3 Opus (P < 0.05). Gemini 1.5 Pro failed to answer any BI-RADS 5 questions correctly. Similarly, ChatGPT 4V failed to answer any BI-RADS 1 questions correctly, making them the least accurate in these categories (P < 0.05). Although LLMs such as Claude 3.5 Sonnet and ChatGPT 4o show promise in text-based BI-RADS assessments, their limitations in visual diagnostics suggest they should be used cautiously and under radiologists' supervision to avoid misdiagnoses. This study demonstrates that while LLMs exhibit strong capabilities in text-based BI-RADS assessments, their visual diagnostic abilities are currently limited, necessitating further development and cautious application in clinical practice.

BackgroundThe creation of high-quality multiple-choice questions (MCQs) is essential for medical education assessments but is resource-intensive and time-consuming when done by human experts. Large language models (LLMs) like ChatGPT-4o offer a promising alternative, but their efficacy remains unclear, particularly in high-stakes exams.ObjectiveThis study aimed to evaluate the quality and psychometric properties of ChatGPT-4o-generated MCQs compared to human-created MCQs in a high-stakes medical licensing exam.MethodsA prospective cohort study was conducted among medical doctors preparing for the Primary Examination on Emergency Medicine (PEEM) organised by the Hong Kong College of Emergency Medicine in August 2024. Participants attempted two sets of 100 MCQs—one AI-generated and one human-generated. Expert reviewers assessed MCQs for factual correctness, relevance, difficulty, alignment with Bloom’s taxonomy (remember, understand, apply and analyse), and item writing flaws. Psychometric analyses were performed, including difficulty and discrimination indices and KR-20 reliability. Candidate performance and time efficiency were also evaluated.ResultsAmong 24 participants, AI-generated MCQs were easier (mean difficulty index = 0.78 ± 0.22 vs. 0.69 ± 0.23, p < 0.01) but showed similar discrimination indices to human MCQs (mean = 0.22 ± 0.23 vs. 0.26 ± 0.26). Agreement was moderate (ICC = 0.62, p = 0.01, 95% CI: 0.12–0.84). Expert reviews identified more factual inaccuracies (6% vs. 4%), irrelevance (6% vs. 0%), and inappropriate difficulty levels (14% vs. 1%) in AI MCQs. AI questions primarily tested lower-order cognitive skills, while human MCQs better assessed higher-order skills (χ² = 14.27, p = 0.003). AI significantly reduced time spent on question generation (24.5 vs. 96 person-hours).ConclusionChatGPT-4o demonstrates the potential for efficiently generating MCQs but lacks the depth needed for complex assessments. Human review remains essential to ensure quality. Combining AI efficiency with expert oversight could optimise question creation for high-stakes exams, offering a scalable model for medical education that balances time efficiency and content quality.

Assessment plays a vital role in evaluating students’ academic performance, particularly in higher education contexts where language proficiency is essential for professional development. This study investigates the written performance of English for Medical Purposes (EMP) students at the University of Tlemcen, with a specific focus on assessment practices that contribute to enhancing students’ writing skills. The study aims to achieve two primary objectives: first, to explore the effectiveness of different assessment techniques in measuring writing proficiency; and second, to assess how writing-focused evaluations contribute to the development of academic language skills among medical students. A total of 108 students from three academic departments—Medicine, Pharmacy, and Dental Medicine—participated in the research. Data collection was carried out using a combination of student questionnaires and both formative and summative assessments. The evaluation included three types of assessment tools: multiple choice questions (MCQs), short open responses (QROC), and extended writing exams. The findings revealed strong student performance in MCQs, where 83.33% of participants provided correct answers. However, the writing exams exposed notable difficulties in grammar, spelling, and idea generation. In particular, 83.33% of the total written errors were related to spelling, and 55.56% of students experienced difficulties in organizing and expressing ideas effectively. These results underscore the need for more targeted writing assessments that can diagnose specific learner challenges and inform effective teaching strategies. The study concludes that the integration of varied assessment methods is essential for identifying linguistic weaknesses, supporting student learning, and ultimately improving writing proficiency in medical English contexts.

This paper investigates the performance of a diverse set of large language models (LLMs) including leading closed-source (GPT-4, GPT-4o mini, Claude 3.5 Haiku) and open-source (Llama 3.1 70B, Llama 3.1 8B) models, alongside the earlier GPT-3.5 within the context of U.S. tax resolutions. AI-driven solutions like these have made substantial inroads into legal-critical systems with significant socio-economic implications. However, their accuracy and reliability have not been assessed in some legal domains, such as tax. Using the Volunteer Income Tax Assistance (VITA) certification tests—endorsed by the US Internal Revenue Service (IRS) for tax volunteering—this study compares these LLMs to evaluate their potential utility in assisting both tax volunteers as well as taxpayers, particularly those with low and moderate income. Since the answers to these questions are not publicly available, we first analyze 130 questions with the tax domain experts and develop the ground truths for each question. We then benchmarked these diverse LLMs against the ground truths using both the original VITA questions and syntactically perturbed versions (a total of 390 questions) to assess genuine understanding versus memorization/hallucinations. Our comparative analysis reveals distinct performance differences: closed-source models (GPT-4, Claude 3.5 Haiku, GPT-4o mini) generally demonstrated higher accuracy and robustness compared to GPT-3.5 and the open-source Llama models. For instance, on basic multiple-choice questions, top models like GPT-4 and Claude 3.5 Haiku achieved 83.33% accuracy, surpassing GPT-3.5 (54.17%) and the open-source Llama 3.1 8B (50.00%). These findings generally hold across both original and perturbed questions. However, the paper acknowledges that these developments are initial indicators, and further research is necessary to fully understand the implications of deploying LLMs in this domain. A critical limitation observed across all evaluated models was significant difficulty with open-ended questions, which require accurate numerical calculation and application of tax rules. We hope that this paper provides a means and a standard to evaluate the efficacy of current and future LLMs in the tax domain.

BackgroundWritten assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity.DiscussionAn extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format.SummaryThe evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-014-0249-2) contains supplementary material, which is available to authorized users.