Towards a Natural Language Interface for Flexible Multi-Agent Task Assignment

Evaluating gpt-4 for zero-shot classification of bleeding and clotting events: Can large language models serve as second reviewers?

Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

Interacting with Large Language Models (LLMs) via declarative queries is increasingly popular for tasks like question answering and data extraction, thanks to their ability to process vast unstructured data. However, LLMs often struggle with answering complex factual questions, exhibiting low precision and recall in the returned data. This challenge highlights that executing queries on LLMs remains a largely unexplored domain, where traditional data processing assumptions often fall short. Conventional query optimization, typically cost-driven, overlooks LLM-specific quality challenges such as contextual understanding. Just as new physical operators are designed to address the unique characteristics of LLMs, optimization must consider these quality challenges. Our results highlight that adhering strictly to conventional query optimization principles fails to generate the best plans in terms of result quality. To tackle this challenge, we present a novel approach to enhance SQL results by applying query optimization techniques specifically adapted for LLMs. We introduce a database system, GALOIS, that sits between the query and the LLM, effectively using the latter as a storage layer. We design alternative physical operators tailored for LLM-based query execution and adapt traditional optimization strategies to this novel context. For example, while pushing down operators in the query plan reduces execution cost (fewer calls to the model), it might complicate the call to the LLM and deteriorate result quality. Additionally, these models lack a traditional catalog for optimization, leading us to develop methods to dynamically gather such metadata during query execution. Our solution is compatible with any LLM and balances the trade-off between query result quality and execution cost. Experiments show up to 144% quality improvement over questions in Natural Language and 29% over direct SQL execution, highlighting the advantages of integrating database solutions with LLMs.

Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM‐based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bias, which may exacerbate educational inequalities. Building on prior work that mapped the traditional machine learning life cycle, we provide a framework of the LLM life cycle from the initial development of LLMs to customizing pre‐trained models for various applications in educational settings. We explain each step in the LLM life cycle and identify potential sources of bias that may arise in the context of education. We discuss why current measures of bias from traditional machine learning fail to transfer to LLM‐generated text (eg, tutoring conversations) because text encodings are high‐dimensional, there can be multiple correct responses, and tailoring responses may be pedagogically desirable rather than unfair. The proposed framework clarifies the complex nature of bias in LLM applications and provides practical guidance for their evaluation to promote educational equity. Practitioner notes What is already known about this topic The life cycle of traditional machine learning (ML) applications which focus on predicting labels is well understood. Biases are known to enter in traditional ML applications at various points in the life cycle, and methods to measure and mitigate these biases have been developed and tested. Large language models (LLMs) and other forms of generative artificial intelligence (GenAI) are increasingly adopted in education technologies (EdTech), but current evaluation approaches are not specific to the domain of education. What this paper adds A holistic perspective of the LLM life cycle with domain‐specific examples in education to highlight opportunities and challenges for incorporating natural language understanding (NLU) and natural language generation (NLG) into EdTech. Potential sources of bias are identified in each step of the LLM life cycle and discussed in the context of education. A framework for understanding where to expect potential harms of LLMs for students, teachers, and other users of GenAI technology in education, which can guide approaches to bias measurement and mitigation. Implications for practice and/or policy Education practitioners and policymakers should be aware that biases can originate from a multitude of steps in the LLM life cycle, and the life cycle perspective offers them a heuristic for asking technology developers to explain each step to assess the risk of bias. Measuring the biases of systems that use LLMs in education is more complex than with traditional ML, in large part because the evaluation of natural language generation is highly context‐dependent (eg, what counts as good feedback on an assignment varies). EdTech developers can play an important role in collecting and curating datasets for the evaluation and benchmarking of LLM applications moving forward.

Extracting information from unstructured and semi-structured geoscience literature is a crucial step in conducting geological research. The traditional machine learning extraction paradigm requires a substantial amount of high-quality manually annotated data for model training, which is time-consuming, labor-intensive, and not easily transferable to new fields. Recently, large language models (LLMs) (e.g., ChatGPT, GPT-4, and LLaMA), have shown great performance in various natural language processing (NLP) tasks, such as question answering, machine translation, and text generation. A substantial body of work has demonstrated that LLMs possess strong in-context learning (ICL) and even zero-shot learning capabilities to solve downstream tasks without specifically designed supervised fine-tuning. In this paper, we propose utilizing LLMs for geoscience literature information extraction. Specifically, we design a hierarchical PDF parsing pipeline and an automated knowledge extraction process, which can significantly reduce the need for manual data annotation, assisting geoscientists in literature data mining. For the hierarchical PDF parsing pipeline, firstly, a document layout detection model fine-tuned on geoscience literature is employed for layout detection, obtaining layout detection information for the document. Secondly, based on the document layout information, an optical character content parsing model is used for content parsing, obtaining the text structure and plain text corresponding to the content. Finally, the text structure and plain text are combined and reconstructed to ultimately obtain the parsed structured data. For the automated knowledge extraction process, firstly, the parsed long text is segmented into paragraphs to adapt to the input length limit of LLMs. Subsequently, a few-shot prompting method is employed for structured knowledge extraction, encompassing two tasks: attribute value extraction and triplet extraction. In attribute value extraction, prompts are generated automatically by the LLMs based on the subdomain and attribute names, facilitating the location and extraction of values related to subdomain attribute names in the text. For triplet extraction, the LLMs employ a procedural approach to entity extraction, entity type extraction, and relation extraction, following the knowledge graph structure pattern. Finally, the extracted structured knowledge is stored in the form of knowledge graphs, facilitating further analysis and integration of various types of knowledge from the literature. Our proposed approach turns out to be simple, flexible, and highly effective in geoscience literature information extraction. Demonstrations of information extraction in subdomains such as radiolarian fossils and fluvial facies have yielded satisfactory results. The extraction efficiency has significantly improved, and feedback from domain experts indicates a relatively high level of accuracy in the extraction process. The extracted results can be used to construct a foundational knowledge graph for geoscience literature, supporting the comprehensive construction and efficient application of a geoscience knowledge graph.

This paper presents Hyborg Agency, a computational ecosystem that explores how AI agents can meaningfully coexist with human society through the metaphor of a digital forest. By reimagining AI agents as mechanical deer, mutated from discarded electronics, the project defamiliarizes common perceptions of anthropomorphized AI agents, transforming them into non-human creatures with conversational abilities. The system employs Large Language Models (LLMs) to process community conversations, treating them as nutrients for AI growth, and features a dual-platform structure that connects an immersive 3D environment with Discord, a widely used communication platform. Based on community chats, Hyborgs generate daily summaries of their observations and share them with one another, fostering consistent memories and perceptions of the world. Through thematic analysis of public exhibitions and structured interviews with fourteen experts (seven pairs), we identified three key themes: social expansion in human relationships, language evolution through code-infused communication, and creative engagement through defamiliarized interaction. These findings highlight how AI agents can enrich human social relationships while maintaining transparency about their artificial nature. This work contributes to ongoing discussions on AI participation in society through a speculative scenario: human discourse serves as the foundation for constructing a digital forest that nurtures AI agents. In this virtual forest, humans and AI grow together in a symbiotic relationship, shaping one another and their shared environment, ultimately achieving harmonious coexistence.

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

Introduction Game theory has long served as a foundational tool in cybersecurity to test, predict, and design strategic interactions between attackers and defenders. The recent advent of Large Language Models (LLMs) offers new tools and challenges for the security of computer systems. In this work, we investigate whether classical game-theoretic frameworks can effectively capture the behaviors of LLM-driven actors and bots. Methods Using a reproducible framework for game-theoretic LLM agents, we investigate two canonical scenarios—the one-shot zero-sum game and the dynamic Prisoner's Dilemma—and we test whether LLMs converge to expected outcomes or exhibit deviations due to embedded biases. We experiments on four state-of-the-art LLMs and five natural languages (English, French, Arabic, Vietnamese, and Mandarin Chinese) to assess linguistic sensitivity. Results For both games, we observe that the final payoffs are influenced by agents characteristics such as personality traits or knowledge of repeated rounds. We also uncover an unexpected sensitivity of the final payoffs to the choice of languages, which should warn against indiscriminate application of LLMs in cybersecurity applications and call for in-depth studies, as LLMs may behave differently when deployed in different countries. We also employ quantitative metrics to evaluate the internal consistency and cross-language stability of LLM agents. Discussion In addition to uncovering unexpected behaviors requiring attention by scholars and practitioners, our work can help guide the selection of the most stable LLMs and optimizing models for secure applications.

This study explores the application of the Computational Theory of Mind (ToM) in artificial intelligence, using large language models (LLMs) enhanced with cognitive biases, meta-programs and an artificial endocrine system (AES). These agent-based models simulate complex human behavior and decision-making in a structured philosophical debate, where agents modeled after Socrates, Plato, Aristotle, and Kant displayed distinct reasoning styles influenced by their internal bias architecture, cognitive strategies and hormone-driven emotional modulation. The simulation demonstrated that integrating ToM elements into LLMs significantly enhances behavioral realism and predictive potential. It also introduced the concept of "artificial psychopathologies" - emergent maladaptive behaviors such as paranoia, depression or narcissism in AI agents - highlighting both the promise and the peril of psychologically-informed AI. As such, this work contributes not only to the advancement of cognitive AI modeling, but also to a broader ethical discourse. The findings support the potential of ToM-AI systems to revolutionize human-computer interaction, provided they are developed under rigorous epistemic and moral safeguards.

Designing preference elicitation (PE) methodologies that can quickly ascertain a user’s top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions – requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to guide LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline’s MRR@10 of 0.17, despite relying on earlier and smaller LLMs.1

As the Internet of Things (IoT) revolutionizes intelligent environments, energy efficiency remains a critical challenge. Traditional IoT-based energy management systems rely on predefined rule-based automation, which lacks adaptability to dynamic environmental conditions. This study proposes an Agentic AIoT system that integrates AI Agents, Large Language Models (LLMs), and Digital Twin technology to enhance IoT decision-making and optimize energy consumption in smart homes. Unlike conventional IoT frameworks, our system dynamically analyzes contextual factors—including real-time environmental conditions, user behaviors, and external data sources (e.g., weather forecasts and occupancy schedules)—to generate optimal energy-saving decisions autonomously. The proposed system consists of three key components: (1) Information Modules, which aggregate multi-source data beyond traditional IoT sensor inputs and construct a Digital Twin model for real-time environment simulation; (2) AI Agents & LLMs, which process contextual data and generate adaptive energy-saving strategies through intelligent reasoning; and (3) Automated Decision Execution, which seamlessly integrates optimized actions into IoT operations. By incorporating Digital Twin technology, the system can simulate various energy consumption scenarios, predict future power demands, and refine decision-making through AI-driven iterative learning. Experimental results demonstrate that our AIoT system achieves a 47.77% reduction in energy consumption compared to non-IoT environments while enhancing automation, adaptability, and user-centric control. Unlike conventional rule-based IoT solutions, integrating Digital Twin with LLM-powered AI Agents enables real-time context-aware decision-making and predictive energy optimization. This study contributes to intelligent IoT applications, Digital Twin-driven energy management, and AI-enhanced automation. Future research will focus on refining AI-driven autonomous decision-making, improving Digital Twin’s predictive accuracy, and further quantifying the impact of LLM-powered AIoT systems on energy efficiency. Our findings highlight the trans-formative potential of LLM-integrated AIoT with Digital Twin technology in revolutionizing energy management for intelligent environments while ensuring seamless interoperability with existing IoT infrastructures.

Foundation large language models (LLMs) face limitations in specialized auditing domains, including accuracy issues, contextual memory constraints, and manual document management requirements. This proposal introduces an AI agent framework specifically designed for auditing workflows, integrating three core components: retrieval augmented generation (RAG) for seamless access to private knowledge bases, customizable workflows with intelligent query classification and multiagent coordination, and orchestrated prompts that embed standardized audit methodologies. The proposed framework reduces workflow disruptions and token consumption while maintaining accuracy. The proposal demonstrates the agent's workflow and its capabilities in document retrieval and analytical calculations. The evaluation plans to compare foundation LLM applications with customized AI agents using both baseline RAG and graph RAG approaches across auditing tasks, measuring accuracy against manually generated ground truth and efficiency through time and token consumption metrics. JEL Classifications: M42; O33; C88; C63; D83.

Vehicular Clouds processing is a new field of research that aims to exploit the vehicles' onboard computational resources as a part of a cooperative distributed cloud computing environment. In this paper, we propose a vehicular cloud network architecture where a group of vehicles near a traffic light cluster and form a temporal vehicular cloud by aggregating their computational resources in that cluster. The goal of the proposed architecture is to minimize the processing and network power consumed in the data center of a cloud operator. To this end, arriving processing tasks are optimally assigned to the centralized cloud and/or the formed vehicular clouds to reduce the total power consumption of the centralized cloud by reducing its average processing workload and network traffic. Furthermore, task assignment among vehicular clouds is constrained by tasks completion time. Our proposed system is analyzed using a mixed integer linear programming (MILP) model where two task assignment approaches were considered: single task assignment and distributed task assignment. In the first approach, each task is not split among multiple clouds, while splitting is allowed in the second approach. It was found that the power consumption of the centralized cloud is reduced by 45% (in the first approach) and 60% (in the second approach) compared to the case where all tasks are assigned to the centralized cloud only. The higher power saving of the centralized cloud in the second approach comes from the ability of vehicular clouds to host more processing workload, an average of 37% more workload, compared to the single task assignment approach.

BackgroundThe launch of ChatGPT (OpenAI) in November 2022 attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including health care. Numerous studies have since been conducted regarding how to use state-of-the-art LLMs in health-related scenarios.ObjectiveThis review aims to summarize applications of and concerns regarding conversational LLMs in health care and provide an agenda for future research in this field.MethodsWe used PubMed, ACM, and the IEEE digital libraries as primary sources for this review. We followed the guidance of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to screen and select peer-reviewed research articles that (1) were related to health care applications and conversational LLMs and (2) were published before September 1, 2023, the date when we started paper collection. We investigated these papers and classified them according to their applications and concerns.ResultsOur search initially identified 820 papers according to targeted keywords, out of which 65 (7.9%) papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT (60/65, 92% of papers), followed by Bard (Google LLC; 1/65, 2% of papers), LLaMA (Meta; 1/65, 2% of papers), and other LLMs (6/65, 9% papers). These papers were classified into four categories of applications: (1) summarization, (2) medical knowledge inquiry, (3) prediction (eg, diagnosis, treatment recommendation, and drug synergy), and (4) administration (eg, documentation and information collection), and four categories of concerns: (1) reliability (eg, training data quality, accuracy, interpretability, and consistency in responses), (2) bias, (3) privacy, and (4) public acceptability. There were 49 (75%) papers using LLMs for either summarization or medical knowledge inquiry, or both, and there are 58 (89%) papers expressing concerns about either reliability or bias, or both. We found that conversational LLMs exhibited promising results in summarization and providing general medical knowledge to patients with a relatively high accuracy. However, conversational LLMs such as ChatGPT are not always able to provide reliable answers to complex health-related tasks (eg, diagnosis) that require specialized domain expertise. While bias or privacy issues are often noted as concerns, no experiments in our reviewed papers thoughtfully examined how conversational LLMs lead to these issues in health care research.ConclusionsFuture studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications bring bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in health care.

This landscape study aims to provide an understanding of Large Language Models (LLMs), especially with a view point of supporting scientific and community queries for climate resilience and preparedness. Typically, for queries related to climate resilience and preparedness, the data span multiple heterogeneous sources and are complex by nature. Scientists as well as communities seek answers to real-world questions from LLMs, yet the accuracy of the feedback is not always guaranteed to be consistent and free of hallucinations. Given the complex landscape and the number of LLM models available, the choice of a model becomes complex too. In addition, LLM ensemble models are also taking on an important role where high level of trustworthiness and accuracy is needed. To address this gap, we performed a review of the LLM research landscape, identifying common themes. This was followed by a deep dive into ensemble models. We evaluate the ensembles as well as a popular single LLM model (that is, ChatGPT) with two sample queries: (i) ‘Are there any linkages between sea ice retreat and ice shelf melt in the Antarctic?’, (ii) ‘What is the best crop for sandy soils?’. The responses were assessed for robustness and accuracy by the domain expert. Through our analysis we concluded that a single LLM provides a parameter free setup whereas the ensemble models needed fine tuning to arrive to robust outcomes. However, the paper does not address ensemble performance issues and neither does it diminish their importance. Instead, this indicates that a lot more research is needed to get ensembles to a more user-friendly and approachable state of use unlike single LLMs. We also outline criteria that an ensemble model selection will need to consider such as initial model selection, retraining, ensemble optimization, energy utilization.