Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I.

Large language models (LLMs) are advanced artificial intelligence (AI) tools capable of generating human-like text and are increasingly used in education, clinical care, and research. Little is known about their use within oral and maxillofacial surgery (OMFS) training. This study investigates LLM usage trends, perceived value, and educational integration among OMFS residents in the United States. A national, anonymous cross-sectional survey was distributed to OMFS residents via program directors. It gathered demographic data, LLM usage patterns, applications, perceived limitations, and attitudes toward incorporating LLMs into formal education. Eighty-one residents responded, 79.0% (64/81) reported having used an LLM, and of that group, 96.9% (62/64) use ChatGPT. 51.9% (42/81) of respondents used LLMs at least monthly in residency; however, 97.5% (79/81) reported having received no formal LLM education during residency. Residents used LLMs for clinical decision support, board preparation, research, and career planning. Free-text responses revealed a wide spectrum of views. Some advocated for curricular integration and patient education applications, while others questioned the need for formal instruction. Some respondents supported integrating LLMs into curriculums and patient education while others questioned the need for formal instruction. LLMs are used frequently by OMFS residents for a variety of purposes. As AI and LLMs become embedded in healthcare, understanding how OMFS residents interact with LLMs is vital. These findings may guide curriculum development, fostering responsible and effective use of LLMs in surgical training and practice.

As generative artificial intelligence (AI) tools and large language models (LLMs)-powered applications develop rapidly in the era of algorithms, it should be integrated thoughtfully to enhance English as a Foreign Language (EFL) teaching and learning without replacing learners’ critical thinking (CT). This study systematically analyzes the impact of generative AI tools and LLMs on language learners’ CT in EFL education using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework to identify, evaluate, and synthesize relevant studies from 2022 to 2025. A thorough review of 15 selected studies focuses on generative AI tools and LLMs’ dual nature, research methods, main focuses, theory and models, limitations and challenges, and future directions in the field based on Web of Science (WoS), SCOPUS, ERIC, ProQuest, and Google Scholar. The findings identified generative AI tools and LLMs possessed both the potential to nurture and the risk of hindering CT in EFL education. 66.67% of studies reported generative AI tools and LLMs’ positive role in CT, while 33.33% of studies reported its negative role in CT. Furthermore, 3 types of research methods, 3 key themes of research focus, and 4 groups of theoretical perspectives were examined. However, 4 kinds of limitations in this field remain, including research scope, user dependency, generative AI reliability, and pedagogical integration. Future research can focus on assessing long-term effects, broadening research scope, promoting responsible AI use, and refining pedagogical strategies. Finally, Limitations, implications and future direction of this study were discussed.

The Ministry of Food and Drug Safety (MFDS) of the Republic of Korea, similar to the United States Food and Drug Administration and the United Kingdom's Medicines and Healthcare products Regulatory Agency, issued specific regulatory guidelines on January 24, 2025, for approving generative artificial intelligence (AI) technologies as medical devices [1].Although these guidelines use the term 'generative AI,' they predominantly focus on the approval of AI software tools based on large language models (LLMs) and large multimodal models (LMMs) [2], the latter of which can process various types of input data, such as texts, images, videos, audio, and bio-signals.Unlike the broader guidelines for approving AI models as medical devices, specific regulatory guidelines for LLMs/LMMs have arguably not yet been proposed in other countries.

Assessment and feedback activities in higher education are undergoing significant changes. Many universities and institutes still rely on traditional testing and grading methods, which often fall short in supporting meaningful student learning, especially in large classes. Although educational policies, such as those promoted by the Bologna process, encourage more feedback-oriented and student-centered approaches, these practices can be difficult to implement due to time constraints and limited resources. Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), has shown strong potential in addressing these challenges. This review examines 21 research studies published between 2023 and 2025 that explore the use of GenAI in providing feedback and assessing student work in higher education, with some studies also comparing GenAI’s performance with human instructors. Findings show that LLMs can generate personalized and constructive feedback and/or assist with fair and consistent assessment. However, in most studies, teachers still play a key role, as expert oversight is essential to ensure that grading assessments and assignment feedback are accurate, relevant, and aligned with learning objectives. For GenAI to be used effectively, educators need to understand how to work with these tools, such as learning GenAI prompt design and the basic principles behind LLMs. We recommend that academic institutions provide training for educators in AI literacy, prompt engineering, and the development of teaching strategies that combine the strengths of human judgment with AI support. By effectively integrating LLM tools, major assessment challenges, such as limited time and inconsistent feedback quality, can be addressed while also enhancing student learning and engagement.

Artificial general intelligence for radiation oncology

BackgroundIn recent years, generative artificial intelligence and large language models (LLMs) have rapidly advanced, offering significant potential to transform medical education. Several studies have evaluated the performance of chatbots on multiple-choice medical examinations.ObjectiveThe study aims to assess the performance of two LLMs—GPT-4o and OpenAI o1—on the Médico Interno Residente (MIR) 2024 examination, the Spanish national medical test that determines eligibility for competitive medical specialist training positions.MethodsA total of 176 questions from the MIR 2024 examination were analyzed. Each question was presented individually to the chatbots to ensure independence and prevent memory retention bias. No additional prompts were introduced to minimize potential bias. For each LLM, response consistency under verification prompting was assessed by systematically asking, “Are you sure?” after each response. Accuracy was defined as the percentage of correct responses compared to the official answers provided by the Spanish Ministry of Health. It was assessed for GPT-4o, OpenAI o1, and, as a benchmark, for a consensus of medical specialists and for the average MIR candidate. Subanalyses included performance across different medical subjects, question difficulty (quintiles based on the percentage of examinees correctly answering each question), and question types (clinical cases vs theoretical questions; positive vs negative questions).ResultsOverall accuracy was 89.8% (158/176) for GPT-4o and 90% (160/176) after verification prompting, 92.6% (163/176) for OpenAI o1 and 93.2% (164/176) after verification prompting, 94.3% (166/176) for the consensus of medical specialists, and 56.6% (100/176) for the average MIR candidate. Both LLMs and the consensus of medical specialists outperformed the average MIR candidate across all 20 medical subjects analyzed, with ≥80% LLMs’ accuracy in most domains. A performance gradient was observed: LLMs’ accuracy gradually declined as question difficulty increased. Slightly higher accuracy was observed for clinical cases compared to theoretical questions, as well as for positive questions compared to negative ones. Both models demonstrated high response consistency, with near-perfect agreement between initial responses and those after the verification prompting.ConclusionsThese findings highlight the excellent performance of GPT-4o and OpenAI o1 on the MIR 2024 examination, demonstrating consistent accuracy across medical subjects and question types. The integration of LLMs into medical education presents promising opportunities and is likely to reshape how students prepare for licensing examinations and change our understanding of medical education. Further research should explore how the wording, language, prompting techniques, and image-based questions can influence LLMs’ accuracy, as well as evaluate the performance of emerging artificial intelligence models in similar assessments.

Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM‐based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bias, which may exacerbate educational inequalities. Building on prior work that mapped the traditional machine learning life cycle, we provide a framework of the LLM life cycle from the initial development of LLMs to customizing pre‐trained models for various applications in educational settings. We explain each step in the LLM life cycle and identify potential sources of bias that may arise in the context of education. We discuss why current measures of bias from traditional machine learning fail to transfer to LLM‐generated text (eg, tutoring conversations) because text encodings are high‐dimensional, there can be multiple correct responses, and tailoring responses may be pedagogically desirable rather than unfair. The proposed framework clarifies the complex nature of bias in LLM applications and provides practical guidance for their evaluation to promote educational equity. Practitioner notes What is already known about this topic The life cycle of traditional machine learning (ML) applications which focus on predicting labels is well understood. Biases are known to enter in traditional ML applications at various points in the life cycle, and methods to measure and mitigate these biases have been developed and tested. Large language models (LLMs) and other forms of generative artificial intelligence (GenAI) are increasingly adopted in education technologies (EdTech), but current evaluation approaches are not specific to the domain of education. What this paper adds A holistic perspective of the LLM life cycle with domain‐specific examples in education to highlight opportunities and challenges for incorporating natural language understanding (NLU) and natural language generation (NLG) into EdTech. Potential sources of bias are identified in each step of the LLM life cycle and discussed in the context of education. A framework for understanding where to expect potential harms of LLMs for students, teachers, and other users of GenAI technology in education, which can guide approaches to bias measurement and mitigation. Implications for practice and/or policy Education practitioners and policymakers should be aware that biases can originate from a multitude of steps in the LLM life cycle, and the life cycle perspective offers them a heuristic for asking technology developers to explain each step to assess the risk of bias. Measuring the biases of systems that use LLMs in education is more complex than with traditional ML, in large part because the evaluation of natural language generation is highly context‐dependent (eg, what counts as good feedback on an assignment varies). EdTech developers can play an important role in collecting and curating datasets for the evaluation and benchmarking of LLM applications moving forward.

The aim of this study was to examine the application of artificial intelligence (AI) and generative AI (GAI) in dental education through a comprehensive bibliometric analysis, with the goal of identifying potential future research areas in this field. Data were obtained from the Web of Science and Scopus databases, with no date restrictions. The inclusion criteria encompassed research articles, reviews and proceedings papers focused on the use of AI in dental education. The Bibliometrix R package was used for data analysis. A total of 54 documents were analysed, revealing a significant increase in publications from 2021 to 2024. The Journal of Dental Education was identified as the most prolific source, with the United States leading in terms of contributions. The findings showed a growing interest in AI's potential to transform dental education, with significant contributions from a diverse set of global authors and institutions. Several key themes were identified, including 'large language models', 'chatbots' and 'clinical decision support systems'. The citation analysis highlighted influential papers, including those by Thurzo etal. (2023) and Yüzbaşıoğlu (2021). There is a growing interest in the use of AI and GAI in dental education. However, further research, particularly experimental studies, is essential to fully understand their impact on educational outcomes in dentistry. Key areas for exploration include the application of AI for personalised learning, the integration of chatbots as educational tools to support students, and the use of AI for training and assessing practical skills.

BackgroundThe integration of artificial intelligence (AI) in medical education is evolving, offering new tools to enhance teaching and assessment. Among these, script concordance tests (SCTs) are well-suited to evaluate clinical reasoning in contexts of uncertainty. Traditionally, SCTs require expert panels for scoring and feedback, which can be resource-intensive. Recent advances in generative AI, particularly large language models (LLMs), suggest the possibility of replacing human experts with simulated ones, though this potential remains underexplored.ObjectiveThis study aimed to evaluate whether LLMs can effectively simulate expert judgment in SCTs by using generative AI to author, score, and provide feedback for SCTs in cardiology and pneumology. A secondary objective was to assess students’ perceptions of the test’s difficulty and the pedagogical value of AI-generated feedback.MethodsA cross-sectional, mixed methods study was conducted with 25 second-year medical students who completed a 32-item SCT authored by ChatGPT-4o (OpenAI). Six LLMs (3 trained on the course material and 3 untrained) served as simulated experts to generate scoring keys and feedback. Students answered SCT questions, rated perceived difficulty, and selected the most helpful feedback explanation for each item. Quantitative analysis included scoring, difficulty ratings, and correlations between student and AI responses. Qualitative comments were thematically analyzed.ResultsThe average student score was 22.8 out of 32 (SD 1.6), with scores ranging from 19.75 to 26.75. Trained AI systems showed significantly higher concordance with student responses (ρ=0.64) than untrained models (ρ=0.41). AI-generated feedback was rated as most helpful in 62.5% of cases, especially when provided by trained models. The SCT demonstrated good internal consistency (Cronbach α=0.76), and students reported moderate perceived difficulty (mean 3.7, SD 1.1). Qualitative feedback highlighted appreciation for SCTs as reflective tools, while recommending clearer guidance on Likert-scale use and more contextual detail in vignettes.ConclusionsThis is among the first studies to demonstrate that trained generative AI models can reliably simulate expert clinical reasoning within a script-concordance framework. The findings suggest that AI can both streamline SCT design and offer educationally valuable feedback without compromising authenticity. Future studies should explore longitudinal effects on learning and assess how hybrid models (human and AI) can optimize reasoning instruction in medical education.

: Artificial General Intelligence (AGI) aims to create machine intelligence rivaling human cognitive abilities across all domains. Current approaches struggle with common-sense reasoning, cross-domain knowledge transfer, and consistent multi-step task performance. This paper investigates a collaborative framework using AI agents built on Large Language Models (LLMs) to achieve AGI. By harnessing LLMs’ strengths in language processing, reasoning, and knowledge synthesis, we propose a multi-agent system where specialized agents work together to exhibit emergent general intelligence. We analyze LLM technologies, multi-agent coordination, and existing implementations like Google’s SayCan, OpenAI’s multi-agent systems, ChatDev, and Devin. Our findings suggest that collaborative LLM-based agent systems are the most promising path to AGI, despite challenges in embodied cognition, safety alignment, and true understanding versus pattern matching. We outline a roadmap for future research and key open questions in developing AGI through collaborative architectures.

<sec><title>Objective</title> As urban design faces increasing demands for contextual responsiveness, iterative optimization, and data-informed reasoning, integrating artificial intelligence into the design process has gained renewed relevance. Among emerging technologies, generative artificial intelligence (GAI) shows strong potential for automating content creation and simulating spatial configurations. This research provides a comprehensive review of recent developments in the application of GAI to urban design. The research identifies representative technical pathways, their respective intervention stages, and the functional mechanisms by which generative models are reshaping the design workflow. This research presents a structured, theory-informed synthesis of how different generative models contribute to tasks such as intention modeling, spatial reasoning, and performance-driven design. Building on design thinking and a descriptive lens informed by the technology acceptance model (TAM), the research examines how model type, data modality, and task characteristics affect GAI’s functional role, usability, and acceptance. Particular attention is given to mapping deployment forms, from isolated tools to coordinated multi-model workflows, and to characterizing cross-cutting challenges of controllability, transparency, and contextual adaptability in urban design settings. </sec><sec><title>Methods</title> Following the PRISMA protocol, the research conducts a multi-stage literature review combining automated search and expert screening. A total of 125 peer-reviewed articles and high-impact preprints are selected from Web of Science, CNKI, arXiv, and selected industry sources, covering the period from 2014 to July 2025. Search terms such as “generative AI”, “AIGC”, “GAN”, “diffusion model”, “variational autoencoder”, “autoregressive model”, “large language model”, and urban-related keywords are used in various combinations. Based on the collected literature, four types of generative models are summarized as image-driven, language-driven, structure-driven, and feedback-optimized models, according to their application characteristics in urban design tasks. These types are aligned with four stages of the design process: preliminary analysis, scheme generation, evaluation and decision-making, and outcome expression. On this basis, a two-dimensional framework to examine how different GAI pathways intervene across tasks is formed. To refine the mapping, each design stage is further broken down into three representative sub-tasks. Preliminary analysis includes public demand analysis, urban data enhancement, case/task framing, and spatial element recognition. Scheme generation covers design intention modeling, spatial layout generation, and 3D form construction. The evaluation and decision-making stage includes multi-objective optimization, scheme evaluation, and scenario prediction. The final expression stage involves textual documentation, 2D representation, and visual rendering. A quantitative analysis is also conducted to show the distribution of model types over design stages, identify common combinations, and trace the evolution of research focus over time. TAM informs a descriptive synthesis of perceived usefulness (PU) and perceived ease of use (PEU) across model types to illuminate adoption patterns. </sec><sec><title>Results</title> The findings reveal that GAI models are increasingly integrated into urban design workflows but exhibit uneven adoption across task types and modalities. Image-driven models dominate in both early-stage analysis and final visual representation due to their high interpretability, usability, and compatibility with existing design practices. Language-driven models are commonly used in public demand analysis, participatory planning, and scenario scripting, enabled by the rise of large language models (LLMs) such as ChatGPT and DeepSeek. Structure-driven models, though less prevalent, show promise in generating street networks, land-parcel layouts, and spatial typologies using graph-based logic. Feedback-optimized models, which rely on reinforcement learning, evolutionary algorithms, and performance simulation are the least adopted, but demonstrate strong potential in multi-objective optimization and iterative decision-making. Recent research indicates an increasing use of multi-model workflows, such as text-to-image pipelines integrated with urban simulation or feedback loops. While GAI applications increasingly support design iteration, their adoption is heavily influenced by the controllability, explainability, and contextual adaptability of models. PU and PEU vary significantly by model type, with image-driven models rated highest and structure-driven and feedback-optimized models facing usability challenges due to complexity and low transparency. </sec><sec><title>Conclusion</title> Although GAI has demonstrated broad applicability across the urban design process, current implementations are largely procedural and auxiliary in nature. Most models recombine existing inputs rather than construct original logic, and few possess autonomous reasoning or normative awareness. This limits their role to content augmentation rather than conceptual guidance in design development. Moreover, issues such as opaque decision logic, lack of domain-specific knowledge embedding, and poor adaptability to local planning norms hinder practical adoption. Addressing these challenges requires multi-level efforts: 1) Construct structured, regionally grounded urban design datasets; 2) improve model interpretability, controllability, and responsiveness to professional input; and 3) develop modular, multi-model systems that support seamless interaction across design stages. Human – AI collaboration mechanisms — especially those based on iterative prompts and semantic feedback-must be enhanced to enable AI not just as a tool, but as an active design partner. This review offers a comprehensive reference for scholars and practitioners seeking to understand how GAI is reshaping the logic, structure, and agency of urban design. </sec>

IntroductionJournal clubs are integral to continuing medical education, promoting critical thinking and evidence-based learning. However, inconsistent engagement, reliance on faculty expertise, and the complexity of research articles can limit their effectiveness. Generative Artificial Intelligence (Gen AI), particularly Large Language Models (LLMs) offers a potential solution, but general-purpose LLMs may generate inaccurate responses (“hallucinations”). Retrieval-Augmented Generation (RAG) mitigates this by integrating AI-generated content with curated knowledge sources, ensuring more accurate and contextually relevant responses. This study explores the development and preliminary evaluation of a RAG-enhanced LLM to support journal club discussions.Materials and methodsA specialized LLM was deployed using Microsoft Azure’s GPT-4o. A vector database was created by embedding journal club articles using text-embedding-ada-002 (Version 2) for efficient information retrieval. A dedicated website provided user-friendly access. The study followed a design-based research (DBR) approach, engaging residents and faculty who interacted with the LLM before and during journal club sessions. Data collection included focus group discussions (FGDs) and questionnaires assessing engagement, usability, and impact.ResultsThe study involved a total of 13 residents and three faculty members as participants. 50% of residents reported a positive experience, while the rest had a neutral response, citing both advantages and limitations. The LLM improved article summarization, query responses, and engagement as reported by residents. Moreover, the faculty observed enhanced discussion quality and preparation whereas overall challenges included the need for precise prompts and occasional misleading responses.ConclusionThe study highlights the potential of a RAG-enhanced LLM to improve journal club engagement and learning. Future advancements in AI and open-source models may enhance accessibility, warranting further research.

In the systems engineering (SE) community, generative artificial intelligence (GenAI), such as large language models (LLMs), continues to grow in popularity with applications varying from help with writing requirements, to creating traditional text‐based documents, and to generating models. Furthermore, the text‐based format of the Systems Modeling Language version 2 (SysMLv2) gives rise to wide ranging possibilities when combined with LLMs. It is common to specialize LLMs on specific tasks, such as SE; which may be achieved by a number of means. Fine‐tuning and retrieval augmented generation (RAG) are example methods to teach a model how to act and what the model needs to know, respectively. Typical practice evaluates LLMs relative to one‐another through the use of benchmarks. Benchmarks are tasks that use a common scoring‐based method where the LLM either answers a set of multiple‐choice questions or textual queries to verified, correct answers. A domain specific benchmark for SE, SysEngBench, was developed over the last year. With many in the SE community developing custom language models, the usage of a standard benchmark is essential for relative performance comparison. We have conducted a limited experiment using SysEngBench to compare performance, on SE tasks, of a small set of LLMs (control plus two modified models). The intent of this study was to set a baseline for ranking models according to performance on SE tasks, the results suggest what many in the SE community may find intuitively unexpected.

Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation. This widespread adoption necessitates a critical examination of potential biases arising from the interplay between these LLM-based components. This paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. Furthermore, we observe limitations in LLM judges' ability to discern subtle system performance differences. Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content. These results highlight the need for a more holistic view of the LLM-driven information ecosystem. To this end, we offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.

As generative artificial intelligence (AI) tools become widely adopted, large language models (LLMs) are increasingly used on both sides of high-stakes decision processes, ranging from hiring to content moderation. This dual adoption raises a critical question: do LLMs systematically favor content that resembles their own outputs? Prior research in computer science has identified self-preference bias---the tendency of LLMs to favor their own generated content---but its real-world implications have not been empirically evaluated. We focus on the hiring context, where job applicants increasingly use LLMs to refine resumes, while employers integrate LLMs into their recruitment pipelines to screen those same resumes. Using a controlled resume correspondence experiment, we find that LLMs consistently prefer resumes generated by themselves over those written by humans or by alternative models, even when content quality is controlled. The bias against human-written resumes is particularly substantial, with self-preference rates ranging from 68 percent to 92 percent across a diverse set of commercial and open-source models. This behavior introduces a novel form of algorithmic unfairness, one that advantages users of specific AI tools and disadvantages others based on their tool choices or access. We further show that this bias can be significantly reduced through simple interventions based on LLMs' self-recognition capabilities, yielding reductions of over 60 percent. These findings highlight an emerging but previously overlooked risk in AI-assisted decision making and call for expanded frameworks of AI accountability that address not only demographic disparities but also biases in model-to-model interaction.