An Adaptive Prompt Optimization Framework for Domain-Specific Large Language Models

Purpose: Large language models (LLMs) improve medical writing efficiency but introduce methodological, ethical, and legal risks. This review examines current evidence on the limitations of LLM-assisted medical writing and proposes principles for its responsible integration into biomedical research.Current concepts: LLMs are commonly used for draft generation, language editing, literature summarization, reference handling, statistical code generation, and manuscript structuring. Studies consistently report improved readability and reduced writing time, particularly among non-native English-speaking authors. However, recurrent challenges include factual hallucinations; fabricated or inaccurate citations; incomplete retrieval of recent literature due to training cutoffs; prompt-sensitive statistical errors; ambiguity regarding authorship and accountability; risks of unintended plagiarism; and concerns related to patient data privacy. These limitations arise from the probabilistic nature of LLMs and their lack of intrinsic fact-verification mechanisms or ethical reasoning.Discussion and conclusion: Risks associated with LLM use vary by manuscript stage and therefore require differentiated oversight. LLMs should be confined primarily to language refinement rather than fact generation, and literature-related outputs must be verified against primary sources, preferably using retrieval-augmented tools. Statistical analyses should remain under human control, with independent validation of all outputs. Ethical governance requires transparent disclosure of LLM use, clear assignment of human responsibility, and strict safeguards for sensitive data. A dual framework combining human-in-the-loop and human-on-the-loop oversight offers a pragmatic model for balancing efficiency with scientific rigor. When positioned as augmentative tools rather than autonomous agents, LLMs can be responsibly integrated into medical research without compromising integrity or reproducibility.

Large Language Models (LLMs) demonstrate increasing capabilities in medical knowledge assessment, yet limitations remain in cross-population validation, direct human-AI comparisons, and evaluation of newer models in anesthesiology contexts. This study addresses these gaps by conducting a head-to-head comparison between newer LLMs and human examinees on official Israeli multiple-choice board examinations. We evaluated two LLMs (Claude 3.7 Sonnet and ChatGPT-4) against anonymized aggregate data from 381 examinees on three consecutive official Israeli anesthesiology board examinations (2023–2024), comprising 450 multiple-choice questions stratified by difficulty, discrimination ability, and topic. Each model was tested twice per exam. Claude 3.7 Sonnet achieved 73.67% accuracy, significantly outperforming both human examinees (62.77%, P < 0.001) and ChatGPT-4 (64.44%, P < 0.001). However, both LLMs performed below the upper quartile of human performance (78.05%). While LLMs excelled on easy questions and theoretical domains like cardiac physiology (Claude: 96.88%, ChatGPT-4: 81.25%), they showed lower performance in areas such as ambulatory (Claude: 30.00%, ChatGPT-4: 10.00%) and regional anesthesia (Claude: 44.44%, ChatGPT-4: 38.89%). Human examinees demonstrated consistent performance across all domains, whereas LLMs showed extreme variability. Self-consistency was substantial for both LLMs (κ = 0.66–0.68), but agreement with human responses was moderate (κ = 0.34–0.39). While advanced LLMs currently exceed average examinee performance on anesthesiology board examinations, they fall short of top-quartile examinees at present and demonstrate significant performance variability across different topic areas.

BackgroundGestational diabetes mellitus (GDM) is a prevalent chronic condition that affects maternal and fetal health outcomes worldwide, increasingly in underserved populations. While generative artificial intelligence (AI) and large language models (LLMs) have shown promise in health care, their application in GDM management remains underexplored.ObjectiveThis study aimed to investigate whether retrieval-augmented generation techniques, when combined with knowledge graphs (KGs), could improve the contextual relevance and accuracy of AI-driven clinical decision support. For this, we developed and validated a graph-based retrieval-augmented generation (GraphRAG)–enabled local LLM as a clinical support tool for GDM management, assessing its performance against open-source LLM tools.MethodsA prototype clinical AI assistant was developed using a GraphRAG constructed from 1212 peer-reviewed research articles on GDM interventions, retrieved from the Semantic Scholar API (2000‐2024). The GraphRAG prototype integrated entity extraction, KG construction using Neo4j, and retrieval-augmented response generation. The performance was evaluated in a simulated environment using clinical and layperson prompts, comparing the outputs of the systems against ChatGPT (OpenAI), Claude (Anthropic), and BioMistral models across 5 common natural language generation metrics.ResultsThe GraphRAG-enabled local LLM showed higher accuracy in generating clinically relevant responses. It achieved a bilingual evaluation understudy score of 0.99, Jaccard similarity of 0.98, and BERTScore of 0.98, outperforming the benchmark LLMs. The prototype also produced accurate, evidence-based recommendations for clinicians and patients, demonstrating its feasibility as a clinical support tool.ConclusionsGraphRAG-enabled local LLMs show much potential for improving personalized GDM care by integrating domain-specific evidence and contextual retrieval. Our prototype proof-of-concept serves two purposes: (1) the local LLM architecture gives practitioners from underserved locations access to state-of-the-art medical research in the treatment of chronic conditions and (2) the KG schema may be feasibly built on peer-reviewed, indexed publications, devoid of hallucinations and contextualized with patient data. We conclude that advanced AI techniques such as KGs, retrieval-augmented generation, and local LLMs improve GDM management decisions and other similar conditions and advance equitable health care delivery in resource-constrained health care environments.

The application of large language models in qualitative nursing research: A scoping review.

Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text. Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems. Motivated by the common collective and directionality of text generation and evolution, this paper first illustrates the conceptual parallels between LLMs and EAs at a micro level, which includes multiple one-to-one key characteristics: token representation and individual representation, position encoding and fitness shaping, position embedding and selection, Transformers block and reproduction, and model training and parameter adaptation. These parallels highlight potential opportunities for technical advancements in both LLMs and EAs. Subsequently, we analyze existing interdisciplinary research from a macro perspective to uncover critical challenges, with a particular focus on evolutionary fine-tuning and LLM-enhanced EAs. These analyses not only provide insights into the evolutionary mechanisms behind LLMs but also offer potential directions for enhancing the capabilities of artificial agents.

The evolution of communication technology has driven the demand for intelligent power grids and data analysis in power systems. However, obtaining and annotating electrical data from intelligent terminals is time-consuming and challenging. We propose Meta In-Context Learning (M-ICL), a new approach that harnesses large language models to classify time series electrical data, which largely alleviates the need for annotated data when adapting to new tasks. The proposed M-ICL consists of two stages: meta-training and meta-testing. In meta-training, the model is trained on various tasks that have an adequate amount of training data. The meta-training stage aims to learn the mapping between electrical data and the embedding space of large language models. In the meta-testing stage, the trained model makes predictions on new tasks. By utilizing the in-context learning ability of large language models, M-ICL adapts models to new tasks effectively with only a few annotated instances (e.g., 1–5 training instances per class). Our contributions lie in the new application of large language models to electrical data classification and the introduction of M-ICL to improve the classification performance with the strong in-context learning ability of large language models. Furthermore, we conduct extensive experiments on 13 real-world datasets, and the experimental results show that the proposed M-ICL improves the average accuracy over all datasets by 19.06%, 12.06%, and 6.63% when only one, two, and five training instances for each class are available, respectively. In summary, M-ICL offers a promising solution to the challenges of electrical data classification.

Purpose: The study analyses the potential of Generative Artificial Intelligence (GenAI) in the field of tax law by assessing the ability of Large Language Models (LLMs) to analyse legal texts. The aim is to determine whether LLMs can perform correct and practically useful analyses of tax law. Need for the study: The increasing complexity of legal regulations and the demand for effective legal research require the implementation of AI tools to support lawyers. Tax law, due to partial harmonisation in European Union countries, is an ideal case for evaluating the effectiveness of LLMs in legal analysis. Methodology: The study used an experimental method to test three LLMs: GPT o1-preview (OpenAI), Gemini 2.0 Flash Thinking Experimental 01-21 (GoogleAI Studio) and DeepSeek. The models analysed 54 rulings of Polish administrative courts. Three hypotheses were verified regarding the ability of LLMs to: (1) correctly identify legal provisions and rulings, (2) recognise key legislation and legal institutions, and (3) extract legal arguments from rulings. The results were compared with expert assessments. Findings: The study showed that LLMs can effectively support lawyers in the analysis of tax law. The models achieved high precision in the identification of legal provisions and rulings, with the Gemini 2.0 model achieving the best results. In the more complex analysis of legal theses, the GPT o1-preview model showed the best performance. Overall, the use of LLMs reduced the time needed for legal analysis by up to 90% compared to the work of experts. Practical Implications: The study emphasises the potential of LLMs in increasing the efficiency of tax law analysis and legal cases research. Lawyers can use these models to automate document review, improve legal argumentation and reduce costs. Future research should focus on integrating LLMs into the preparation of pleadings, court justifications and administrative decisions.

Large Language Models (LLMs) are increasingly used in cybersecurity, both as practical tools in real-world tasks like penetration testing and reverse engineering, and as educational aids for students learning complex analysis techniques. While recent research highlights their potential to automate code analysis, deobfuscation, and threat detection, less is known about how students actually use these models during malware analysis courses. To address this gap, we conducted a survey of 37 students enrolled in university-level malware analysis courses. Our findings show that all participants reported using LLMs, primarily for assignments and labs (70.2%) and to better understand course content (59.4%). Students primarily analyzed outputs from Interactive Disassembler Pro (IDA Pro) (83.7%), followed by OllyDbg and Wireshark (43.2%). They mainly used LLMs for advanced static analysis, especially for disassembled code interpretation (37.8%). While 59.4% of students reported no major issues when using LLMs, 27% encountered refusals to respond, primarily due to ethical safeguards built into the models, and others noted inaccurate or overly generic responses and token-size limits. In terms of satisfaction, 67.5% of students reported positive experiences with LLMs, and 81% indicated they were likely to continue using them for cybersecurity-related tasks in the future. These findings suggest the need for responsible integration of LLMs into cybersecurity education through lecturer guidance, ethical transparency, and effective assessment design. Overall, they highlight both the strengths and limitations of LLMs in supporting advanced technical learning.

The rapid evolution of AI has driven advancements across numerous sectors. In the domain of government affairs, large language models (LLMs) hold significant potential for applications such as policy analysis, data processing, and decision support. However, their adoption in government settings faces considerable challenges, including data accessibility issues, the absence of standardized evaluation criteria, and concerns regarding model accuracy, reliability, and security. To address these challenges, we propose a comprehensive evaluation framework specifically designed for LLMs in government affairs. Built on modular principles, this framework ensures adaptability across various industries. Additionally, we introduce the Multi-Scenario Government Affairs Benchmark (MSGABench 1 ) dataset, a Chinese-language dataset specifically crafted to meet the practical needs of government professionals. Employing the proposed framework and the MSGA dataset, we conducted an empirical evaluation of 15 prominent LLMs, revealing critical insights: (1) Performance: Many models demonstrated low accuracy and reliability, particularly under minor input variations, with some dropping below 35% accuracy, whereas GPT-4 achieved above 95% reliability; (2) Security and Compliance: Significant concerns were identified, including privacy vulnerabilities, legal compliance risks, and persistent biases, which may hinder secure deployments in government contexts; (3) Task Avoidance: Certain models exhibited excessive caution, often avoiding responses to basic tasks like document classification and government-related inquiries, which restricts their usability. These findings highlight essential limitations and opportunities for improvement, contributing to the safe and effective application of LLMs in the government sector.

ATLASky-AI: An Autonomous Framework for Physics-Based Trustworthy Verification of LLM-Generated Spatiotemporal Knowledge

This paper presents a novel approach to enhancing educational question-answering (Q&A) systems by combining Retrieval-Augmented Generation (RAG) with Large Language Model (LLM) Code Interpreters. Traditional educational Q&A systems face challenges in areas such as knowledge updates, reasoning accuracy, and the handling of complex computational tasks. These limitations are particularly evident in domains requiring multi-step reasoning or access to real-time, domain-specific knowledge. To address these issues, we propose a system that utilizes RAG to dynamically retrieve up-to-date, relevant information from external knowledge sources, thus mitigating the common "hallucination" problem in LLMs. Additionally, the integration of an LLM Code Interpreter enables the system to perform multi-step logical reasoning and execute Python code for precise calculations, significantly improving its ability to solve mathematical problems and handle complex queries. We evaluated our proposed system on five educational datasets-AI2_ARC, OpenBookQA, E-EVAL, TQA, and ScienceQA-which represent diverse question types and domains. Compared to vanilla Large Language Models (LLMs), our approach combining Retrieval-Augmented Generation (RAG) with Code Interpreters achieved an average accuracy improvement of 10-15 percentage points. Among tested models, GPT-4o and Gemini-pro-1.5 consistently showed the strongest performance, excelling particularly in scientific reasoning, multi-step computations. Despite these advancements, we identify several challenges that remain, including knowledge retrieval failures, code execution errors, difficulties in synthesizing cross-disciplinary information, and limitations in multi-modal reasoning, particularly when combining text and images. These challenges provide important directions for future research aimed at further optimizing educational Q&A systems. Our work shows that integrating RAG and Code Interpreters offers a promising path toward more accurate, transparent, and personalized educational Q&A systems, and can significantly improve the learning experience in various educational contexts.

The computational significance of consciousness is an important and potentially more tractable research theme than the hard problem of consciousness, as one could look at the correlation of consciousness and computational capacities through, e.g., algorithmic or complexity analyses. In the literature, consciousness is defined as what it is like to be an agent (i.e., a human or a bat), with phenomenal properties, such as qualia, intentionality, and self-awareness. The absence of these properties would be termed "unconscious." The recent success of large language models (LLMs), such as ChatGPT, has raised new questions about the computational significance of human conscious processing. Although instances from biological systems would typically suggest a robust correlation between intelligence and consciousness, certain states of consciousness seem to exist without manifest existence of intelligence. On the other hand, AI systems seem to exhibit intelligence without consciousness. These instances seem to suggest possible dissociations between consciousness and intelligence in natural and artificial systems. Here, I review some salient ideas about the computational significance of human conscious processes and identify several cognitive domains potentially unique to consciousness, such as flexible attention modulation, robust handling of new contexts, choice and decision making, cognition reflecting a wide spectrum of sensory information in an integrated manner, and finally embodied cognition, which might involve unconscious processes as well. Compared to such cognitive tasks, characterized by flexible and ad hoc judgments and choices, adequately acquired knowledge and skills are typically processed unconsciously in humans, consistent with the view that computation exhibited by LLMs, which are pretrained on a large dataset, could in principle be processed without consciousness, although conversations in humans are typically done consciously, with awareness of auditory qualia as well as the semantics of what are being said. I discuss the theoretically and practically important issue of separating computations, which need to be conducted consciously from those which could be done unconsciously, in areas, such as perception, language, and driving. I propose conscious supremacy as a concept analogous to quantum supremacy, which would help identify computations possibly unique to consciousness in biologically practical time and resource limits. I explore possible mechanisms supporting the hypothetical conscious supremacy. Finally, I discuss the relevance of issues covered here for AI alignment, where computations of AI and humans need to be aligned.

Understanding source code is a cognitively demanding task, yet existing complexity metrics primarily rely on structural features and overlook actual human effort. This paper introduces a time-based approach to measuring code comprehension using large language models (LLMs). We investigate whether LLMs can estimate the number of seconds a junior developer would need to understand Java source files. Using a dataset of 371 files from the Apache Hive project, we evaluated five LLMs under a uniform prompt design. Human comprehension times, collected through a controlled study involving junior developers, were used as ground truth. Among the tested models, Gemini 2.0 Flash most closely approximated human estimates, while others exhibited systematic bias or output instability. Our findings show that while LLMs can produce interpretable, numeric predictions of comprehension time, achieving consistent accuracy requires prompt refinement and potential model fine-tuning. This work contributes a behavioral perspective to software complexity analysis, offering time-based estimations as an alternative to static metrics.

Accurate prediction of molecular toxicity is vital for drug development. Most mainstream methods rely on fingerprints or graph-based feature extraction, the emergence of large language models (LLMs) offers new prospects for molecular representation learning in toxicity prediction. Although several studies attempt to leverage LLMs to integrate molecular sequence data for pretraining molecular representations, certain limitations remain. Current LLM-based approaches usually utilize solely on class embedding features, overlooking the rich information in sequence embedding. Moreover, integrating pre-trained molecular representations with multi-modal molecular data may further enhance performance in toxicity prediction. To address these challenges, we propose BFGTP, a BERT-guided two-stage molecular representation learning framework for toxicity prediction. Firstly, we design independent encoders for molecular descriptions of three modalities, where the fingerprint encoder with dual level attention mechanisms effectively integrates multi-category fingerprints. Then, the two-stage guide strategy is introduced to fully utilize the prior knowledge of LLMs, employing contrastive learning to align and fuse the tri-modal representations and knowledge distillation to align predicted value distributions. BFGTP ultimately combines fingerprint and graph representations to predict molecular toxicity. Experiments on seven toxicity datasets show that BFGTP outperforms baselines, achieving the highest AUC on five datasets and the best average performance across five evaluation metrics. Ablation studies, t-SNE visualization and case study confirm the effectiveness of BFGTP's components and its ability to capture meaningful molecular representations.

Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs).The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance.This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest.We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case.We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points.