Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis

In today’s rapidly evolving business landscape, Artificial Intelligence (AI), and specifically Large Language Models (LLMs), are redefining how organizations operate, make decisions, and engage with customers. AI-driven technologies have become indispensable, providing businesses with powerful tools to streamline operations, derive actionable insights from vast data, and foster more meaningful customer interactions. For business leaders, scholars, and practitioners alike, understanding the transformative potential of AI isn’t just advantageous—it’s essential to staying competitive in an increasingly data-driven world.This editorial delves into recent scholarly advancements in LLM applications within business contexts, analyzing studies that explore AI’s potential across various domains, from decision support to creative industries. By introducing a structured framework, this editorial highlights key insights and contributions from recent studies, assessing their value to academia and industry. The following comparative analysis sheds light on how these innovations shape our understanding of AI’s role in business while pointing to future research directions.Puyt and Madsen's (2024) study stands out as a foundational exploration of LLM accuracy, assessing ChatGPT-4's ability to recount the history of the SWOT analysis-a vital business strategy tool. Their findings reveal that, while ChatGPT-4 effectively conveys general concepts, it struggles with detailed historical information, often producing inaccuracies or "hallucinations." This gap underscores the need for LLMs to be trained with verified academic data, particularly for strategic business applications that demand precision. This study not only contributes to the literature by proposing methods to evaluate AI accuracy in historical contexts but also highlights the importance of rigorous information vetting in industry settings where reliability is crucial.In contrast, Raikov et al. (2024) explore a hybrid intelligence model that combines LLM capabilities with explainable AI (XAI) principles to enhance human-machine collaboration. Their approach emphasizes cognitive semantics, improving transparency and decision-making efficiency. The hybrid model's real-time adaptability addresses the needs of complex, regulated industries such as finance and healthcare, where trust in AI decisions is paramount. Academically, this study provides a valuable addition to XAI literature by demonstrating how LLMs can bridge the gap between AI autonomy and human oversight, making it a model for future human-AI interactions in complex business environments.Another significant study by Mariotti and colleagues (2024) examines the integration of LLMs with enterprise knowledge graphs to enhance data-driven decision-making. By enabling organizations to leverage knowledge graphs for more accurate and scalable data retrieval, this research provides a robust framework for businesses seeking efficient knowledge management systems. The academic contribution here lies in advancing the dialogue between LLMs and knowledge graphs, emphasizing ethical data handling and quality standards essential for industry applications. For enterprises, the study offers practical solutions to achieve streamlined data management, balancing automation with privacy and security. 2024) take a different approach, investigating LLMs' role in creative industries, specifically within fashion design. They introduce a hybrid intelligence model that supports creative processes, allowing AI to complement rather than replace human ingenuity. While LLMs in this field demonstrate potential in automating repetitive design tasks and enhancing customer personalization, the study reveals limitations in AI's ability to handle spatial and stylistic nuances. This study's academic contribution lies in promoting human-AI co-creation, inspiring further research into AI applications across diverse creative sectors, including media and marketing.Collectively, these studies not only illuminate LLMs' transformative potential in business but also highlight critical ethical and operational considerations. Ensuring accuracy, transparency, and data privacy are vital to responsibly integrating AI into business workflows. Future research should focus on enhancing LLM accuracy, refining hybrid intelligence models, and exploring creative AI applications, all while maintaining ethical standards. As LLMs evolve, interdisciplinary collaborations will be essential to harness their full potential, making AI an ethical, effective, and innovative force in the business world.

Background Syphilis, caused by Treponema pallidum, is a significant global health concern with potentially severe complications if untreated. Advances in artificial intelligence (AI), particularly large language models (LLMs), offer opportunities to enhance medical diagnosis and public health education. This study aims to assess LLMs' ability to provide readable, accurate, and comprehensive syphilis information by comparing it with WHO datasheets and validating through specialist evaluation for clinical relevance.Material and Methods Ten AI-based LLMs were evaluated. Ten questions addressing symptoms, transmission, diagnosis, treatment, and prevention were crafted by researchers. Responses from the LLMs were compared to World Health Organization (WHO) syphilis fact sheets, and a panel of specialists assessed the accuracy, clinical relevance, and readability of the AI-generated information.Results Among the evaluated LLMs, ChatGPT 4.0 and Claude demonstrated the highest accuracy, scoring 92% and 89% alignment with WHO standards, respectively. Perplexity and Llama3 performed less reliably, with scores between 60-70%, especially in areas like tertiary syphilis and neurosyphilis. Specialists identified common errors, such as outdated treatment protocols and incorrect descriptions of transmission pathways. Expert reviews further revealed that while LLMs provided adequate information on early syphilis symptoms, they struggled with complex clinical nuances. The specialists' evaluation showed that only 60% of the AI-generated content was deemed clinically reliable without further edits, with ChatGPT 4.0 rated highest by experts in terms of readability and clinical accuracy.Conclusions LLMs hold promise for disseminating syphilis information, but human oversight is crucial. AI models need refinement to improve their accuracy, especially in complex medical scenarios. Key words:Artificial intelligence, large language models, pathology, syphilis.

Objective To evaluate and synthesise current applications of large language models (LLMs) in systematic reviews and meta-analyses (SRMAs), identify key limitations and propose an enhanced theoretical framework to improve the efficiency, scalability and reliability of evidence synthesis. Methods and analysis We conducted a narrative review of recent studies applying LLMs across key SRMA stages. A total of 21 publications were analysed for model type, task application, accuracy metrics and workflow impact. Building on this evidence base, we designed a comprehensive LLM-enhanced SRMA framework that categorises LLM roles as consultants and assistants, integrates human-in-the-loop strategies and uses retrieval-augmented generation (RAG) and agent-based architectures to address critical challenges including hallucinations, bias and workflow inefficiency. Results The reviewed literature demonstrated that LLMs can support various SRMA tasks with reported accuracy ranging from 61% to 99%, showing particular promise in literature screening and data extraction. Our proposed framework conceptualises modular integration of LLMs across all six SRMA stages, with LLMs serving as consultants for research question formulation and search strategy development and as assistants for task automation including abstract screening and structured data extraction. The framework incorporates RAG technology to reduce hallucinations by grounding outputs in retrieved literature and employs agent-based orchestration for complex analytical workflows. Theoretical analysis suggests potential for significant efficiency gains while maintaining methodological rigour through strategic human oversight. Conclusion LLMs offer substantial theoretical potential to transform evidence synthesis by improving efficiency, scalability and consistency across SRMA workflows. The proposed LLM-enhanced framework provides a systematic, theoretically grounded approach for integrating advanced artificial intelligence capabilities into existing SRMA methodologies while preserving essential human oversight and analytical integrity. Future empirical studies are needed to validate the framework’s practical effectiveness, establish implementation protocols and demonstrate real-world benefits in evidence-based medicine.

This paper explores the application of large language models (LLMs) to automate the evaluation of programming assignments in an undergraduate “Introduction to Programming” course. This study addresses the challenges of manual grading, including time constraints and potential inconsistencies, by proposing a system that integrates several LLMs to streamline the assessment process. The system utilizes a graphic interface to process student submissions, allowing instructors to select an LLM and customize the grading rubric. A comparative analysis, using LLMs from OpenAI, Google, DeepSeek and ALIBABA to evaluate student code submissions, revealed a strong correlation between LLM-generated grades and those assigned by human instructors. Specifically, the reduced model using statistically significant variables demonstrates a high explanatory power, with an adjusted R2 of 0.9156 and a Mean Absolute Error of 0.4579, indicating that LLMs can effectively replicate human grading. The findings suggest that LLMs can automate grading when paired with human oversight, drastically reducing the instructor workload, transforming a task estimated to take more than 300 h of manual work into less than 15 min of automated processing and improving the efficiency and consistency of assessment in computer science education.

<sec> <title>BACKGROUND</title> Patient recruitment remains a major bottleneck in clinical trials. Large language models (LLMs) show promise for patient–trial matching and eligibility screening, but existing work emphasizes algorithmic performance in controlled settings. Less is known about how LLMs should be embedded in multi-stakeholder recruitment workflows where human oversight, process design, and economic incentives jointly determine outcomes. </sec> <sec> <title>OBJECTIVE</title> To propose a theory-grounded conceptual framework—the LLM-Embedded Clinical Recruitment Architecture (LECRA)—that explains how LLM integration in clinical trial recruitment may influence operational and economic outcomes through intermediate mechanisms involving data complexity, LLM processing, and human–AI collaboration. </sec> <sec> <title>METHODS</title> We synthesize evidence from recent empirical and review studies on LLM-based patient–trial matching, human–AI teaming in clinical settings, and AI governance in healthcare. Socio-technical systems theory is used to justify joint optimization of technical and social subsystems, and transaction cost economics is used to interpret information-processing costs, uncertainty, and scale effects. The framework is specified as four interdependent layers, and six propositions are derived for future empirical testing. </sec> <sec> <title>RESULTS</title> LECRA distinguishes (1) data complexity, (2) LLM processing, (3) human–AI collaboration, and (4) economic impact. The framework posits that LLM value is not solely a property of model accuracy; collaboration design (e.g., teaming mode, oversight, explainability) mediates how technical capability translates into recruitment performance, and economic returns may increase with scale as fixed deployment costs are spread across cases. The propositions link complexity, teaming, and scale to expected differences in processing efficiency and cost-related outcomes. </sec> <sec> <title>CONCLUSIONS</title> This conceptual work reframes LLM-enabled recruitment as a socio-technical and economic problem, not only an algorithmic one. It offers a structured basis for empirical studies—especially quasi-experimental evaluations in organizational settings—and for practice by highlighting workflow integration and governance choices that determine whether LLM adoption realizes its potential benefits. </sec> <sec> <title>CLINICALTRIAL</title> Not applicable. This manuscript presents a conceptual framework and literature synthesis; it does not report a prospectively registered interventional clinical trial. </sec>

Advances in artificial intelligence, particularly in natural language processing, offer promising tools for addressing mental health challenges in online contexts, potentially identifying at-risk individuals and informing timely interventions. This study investigates the potential of Large Language Models (LLMs) for automatically triaging social media posts expressing psychological distress. Using a dataset of 425 Italian-language Reddit posts, we compared the triage performance of three state-of-the-art LLMs - ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro - with trained clinician assessment using an adapted version of the Mental Health Triage Scale (MHTS), a validated instrument used in psychiatric screening services. A zero-shot prompting approach, with and without role assignment (simulating a clinician's perspective), evaluated the models' capability to assess intervention urgency. Results revealed that LLMs consistently overestimated urgency compared to human raters, although correlations with human judgments were moderate to strong, with GPT-4o and Claude 3.5 Sonnet demonstrating higher agreement. GPT-4o achieved the best classification performance, highlighting its potential for this task. Claude 3.5 Sonnet showed high sensitivity but lower precision, indicating a tendency toward false positives, while Gemini 1.5 Pro exhibited more balanced but generally lower performance. These findings suggest that while LLMs show promise for mental health triage in social media, their tendency to overestimate urgency and model-specific variations in performance underscore the need for careful interpretation, and human oversight when applying LLMs in mental health contexts.

As autonomous agents and assistants, large language models (LLMs) often struggle with "hallucinations."Fundamentally, the problem is one of prioritization and balance: the LLM needs to understand or infer when it needs to be creative and balance that with its need to be accurate.Most efforts focus on either updating intrinsic knowledge via targeted post-training or by adding external knowledge sources which the LLM can reference neurosymbolically (e.g., via retrieval-augmented generation).However, these all eventually rely on the LLM's implicit reasoning ability during generation, still allowing for these random hallucinations despite high-quality training examples and references.Using aspect-oriented summarization as a case study, we propose LOgit REwriting (LORE), a new controlled generation paradigm which can simultaneously be faithful to external knowledge and to the LLM's intentions.LORE works by adding a rewriting module at left-to-right inference time, continuously reflecting on the newest prediction and trying to find a replacement that is more faithful to the source document.Then, it merges the logits of the replacement with those of the original prediction to generate the next token.We proposed a new long-context aspect-oriented summarization dataset, SLPAspect, and find that LORE generates 5.8% better summaries compared to the LLM without LORE-rewriting. 1

The growing interest in advanced large language models (LLMs) like ChatGPT has sparked debate about how best to use them in various human activities. However, a neglected issue in the debate concerning the applications of LLMs is whether they can reason logically and follow rules in novel contexts, which are critical for our understanding and applications of LLMs. To address this knowledge gap, this study investigates five LLMs (ChatGPT-4o, Claude, Gemini, Meta AI, and Mistral) using word ladder puzzles to assess their logical reasoning and rule-adherence capabilities. Our two-phase methodology involves (1) explicit instructions about word ladder puzzles and rules regarding how to solve the puzzles and then evaluate rule understanding, followed by (2) assessing LLMs’ ability to create and solve word ladder puzzles while adhering to rules. Additionally, we test their ability to implicitly recognize and avoid HIPAA privacy rule violations as an example of a real-world scenario. Our findings reveal that LLMs show a persistent lack of logical reasoning and systematically fail to follow puzzle rules. Furthermore, all LLMs except Claude prioritized task completion (text writing) over ethical considerations in the HIPAA test. Our findings expose critical flaws in LLMs’ reasoning and rule-following capabilities, raising concerns about their reliability in critical tasks requiring strict rule-following and logical reasoning. Therefore, we urge caution when integrating LLMs into critical fields and highlight the need for further research into their capabilities and limitations to ensure responsible AI development.

621 Background: Effective communication between clinical researchers and participants is vital for successful clinical trials. However, informed consent documents often contain complex language, making it difficult for many patients to fully understand key information and make informed decisions. This comprehension gap impacts patient experience and impedes trial enrollment. Large Language Models (LLMs) show promise for translating specialized medical content into accessible language; however, their effectiveness in clinical trial communication remains underexplored. Multimodal approaches combining LLM-generated content with visual aids may significantly enhance participant understanding and trust in the consent process. This study aims to design and test the feasibility of creating a lay clinical trial summary using LLM technology. Methods: This feasibility pilot leveraged an LLM to extract and customize key clinical trial information from research protocols for prospective participants. An initial literature review leveraged FDA guidance and prior work by Hill et al (2024) to establish a template and delineate content requirements. We utilized Amazon Bedrock with Anthropic's Claude 3.7 to engineer and test prompts. We extracted each information topic separately into phrases or short sentences in JavaScript Object Notation (JSON) and used placeholders to insert content into the template. Clinical experts, communication specialists, and patient advocates reviewed content for accuracy and clarity. Results: Feedback received from patient advocates were incorporated to optimize content relevance, literacy level, and acceptability before producing 2 final outputs for IRB review. Iterative prompt optimization using one-to-two shot examples in conversational, second-person language achieved optimal LLM outputs at 4-6 grade reading level. The template was converted from PowerPoint shapes to a Word table for improved text wrapping, visual presentation, and Section 508 accessibility compliance. Human oversight remained essential for content validation and managing text constraints within fixed cell dimensions. Patient reviewers provided highly positive feedback, endorsing bullet points, simplified study titles, and enhanced typography with wider spacing. Graphics and layout enhancements significantly improved engagement compared to traditional consent materials. Final outputs, after human review, were submitted to IRB for approval. Conclusions: Utilizing an LLM increased efficiency in creating summaries, provided consistency in language and structure, and customized trial content. Implementation required balancing risks of oversimplification against benefits, maintaining human oversight for critical evaluation, and ensuring regulatory compliance. A future study will evaluate participant preference for layperson abstract plus consent form versus consent form alone.

Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant

Purpose: Large language models (LLMs) improve medical writing efficiency but introduce methodological, ethical, and legal risks. This review examines current evidence on the limitations of LLM-assisted medical writing and proposes principles for its responsible integration into biomedical research.Current concepts: LLMs are commonly used for draft generation, language editing, literature summarization, reference handling, statistical code generation, and manuscript structuring. Studies consistently report improved readability and reduced writing time, particularly among non-native English-speaking authors. However, recurrent challenges include factual hallucinations; fabricated or inaccurate citations; incomplete retrieval of recent literature due to training cutoffs; prompt-sensitive statistical errors; ambiguity regarding authorship and accountability; risks of unintended plagiarism; and concerns related to patient data privacy. These limitations arise from the probabilistic nature of LLMs and their lack of intrinsic fact-verification mechanisms or ethical reasoning.Discussion and conclusion: Risks associated with LLM use vary by manuscript stage and therefore require differentiated oversight. LLMs should be confined primarily to language refinement rather than fact generation, and literature-related outputs must be verified against primary sources, preferably using retrieval-augmented tools. Statistical analyses should remain under human control, with independent validation of all outputs. Ethical governance requires transparent disclosure of LLM use, clear assignment of human responsibility, and strict safeguards for sensitive data. A dual framework combining human-in-the-loop and human-on-the-loop oversight offers a pragmatic model for balancing efficiency with scientific rigor. When positioned as augmentative tools rather than autonomous agents, LLMs can be responsibly integrated into medical research without compromising integrity or reproducibility.

Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements. We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties. Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety. Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).

This study compares the design practices and performance of ChatGPT 4.0, a large language model (LLM), against graduate engineering students in a 48-h prototyping hackathon, based on a dataset comprising more than 100 prototypes. The LLM participated by instructing two participants who executed its instructions and provided objective feedback, generated ideas autonomously and made all design decisions without human intervention. The LLM exhibited similar prototyping practices to human participants and finished second among six teams, successfully designing and providing building instructions for functional prototypes. The LLM’s concept generation capabilities were particularly strong. However, the LLM prematurely abandoned promising concepts when facing minor difficulties, added unnecessary complexity to designs, and experienced design fixation. Communication between the LLM and participants was challenging due to vague or unclear descriptions, and the LLM had difficulty maintaining continuity and relevance in answers. Based on these findings, six recommendations for implementing an LLM like ChatGPT in the design process are proposed, including leveraging it for ideation, ensuring human oversight for key decisions, implementing iterative feedback loops, prompting it to consider alternatives, and assigning specific and manageable tasks at a subsystem level.

Large Language Models (LLMs) in Intelligent Computer-Assisted Language Learning enable highly personalized learning, yet raise significant challenges related to pedagogical grounding, data privacy, and instructional validity. Although Knowledge Graphs (KGs) and Federated Learning (FL) can mitigate these issues in isolation, evidence on systematic FL–KG–LLM integration for educational language learning remains limited. This scoping review maps the FL–KG–LLM convergence landscape. Following PRISMA-ScR guidelines, we searched six databases and screened 51 papers (2019–2025) using automated extraction. Our findings indicate limited convergence: no papers integrate all three domains, and 58.8% of approaches remain confined to isolated technological silos. Reporting is also uneven across the corpus, with an average “Not Reported” (NR) rate of 84.5%, most notably for privacy mechanisms (92.2%), validation metrics (90.2%), and Common European Framework of Reference for Languages (CEFR) alignment (88.2%). Domain-specific analysis reveals two distinct patterns: inter-domain gaps (disciplinary silos resulting in expected CEFR absence in single-domain papers) and intra-domain gaps (failure to report domain-critical variables, including 100% parameter NR in FL studies, 86.7% validation NR in KG studies, and 100% CEFR NR in convergence papers). Taken together, these gaps suggest that pedagogical grounding is treated as optional rather than structural. We therefore identify two pillars of pedagogical grounding: a Grounding Pillar, which constrains LLM outputs via Knowledge Graph rules, and a Validation Pillar, which concerns how authoritative frameworks (e.g., CEFR) are mapped onto Knowledge Graph schemas and evaluated. The near-universal absence of CEFR alignment and validation reporting suggests that this second pillar is currently missing, which we term the Integrity Gap—a systematic disconnection between technological innovation and pedagogical grounding inin Intelligent Computer-Assisted Language Learning. By reframing the problem as upstream control and validation, this review informs the design of user-facing automated systems where trust, transparency, and human oversight are critical.

Background and Purpose: We present a pilot benchmark dataset of 378 preclinical histological samples for evaluating large language model (LLM) performance on multi-dimensional classification tasks. This dataset addresses the lack of standardized benchmarks for assessing LLMs in preclinical histopathology, encompassing species identification (mouse, rabbit, rat), organ recognition, staining methods, and preparation techniques. Methods: We evaluated the LLMs GPT-4.1, GPT-4o-mini, and Llama 3.2 on 378 histological samples across four classification dimensions: species identification (mouse, rabbit, rat), organ recognition (kidney, liver, prostate, spleen), staining method classification (H&E, Elastica van Gieson, collagen, iron, IHC-elastin, MOVAT's pentachrome), and preparation type determination (frozen vs. paraffin-embedded). Performance was assessed using sensitivity and specificity metrics with confusion matrix analysis. Results: Model performance varied substantially across tasks and exhibited strong sensitivity to class imbalance. For preparation type classification, GPT-4.1 achieved the most balanced performance (50% frozen sensitivity, 85.7% paraffin sensitivity), while Llama 3.2 failed to recognize paraffin samples (0% sensitivity). In species classification, Llama 3.2 was the only model capable of identifying all three species (rabbit: 75% sensitivity, rat: 85.7% sensitivity) despite poor mouse recognition (0.3% sensitivity). GPT-4.1 achieved higher mouse sensitivity within this dataset (70.4% sensitivity) but failed with minority species. For staining classification, Llama 3.2 demonstrated highest overall performance, achieving >88% sensitivity for most staining types, while GPT-4o-mini showed perfect H&E recognition (100% sensitivity). Conclusions: Current LLMs demonstrate variable performance for histological classification with substantial sensitivity to class imbalance. While not suitable for standalone diagnostic use, they may serve as useful screening tools in research settings with appropriate human oversight.