Cracks in Collaboration: Threat Models and Attacks on Multi-LLM Collaborative Systems

The rapid advancements of large language models (LLMs) have raised public concerns about the privacy leakage of personally identifiable information (PII) within their extensive training datasets. Recent studies have demonstrated that an adversary could extract highly sensitive privacy data from the training data of LLMs with carefully designed prompts. However, these attacks suffer from the model's tendency to hallucinate and catastrophic forgetting (CF) in the pre-training stage, rendering the veracity of divulged PIIs negligible. In our research, we propose a novel attack, Janus, which exploits the fine-tuning interface to recover forgotten PIIs from the pre-training data in LLMs. We formalize the privacy leakage problem in LLMs and explain why forgotten PIIs can be recovered through empirical analysis on open-source language models. Based upon these insights, we evaluate the performance of Janus on both open-source language models and two latest LLMs, i.e., GPT-3.5-Turbo and LLaMA-2-7b. Our experiment results show that Janus amplifies the privacy risks by over 10 times in comparison with the baseline and significantly outperforms the state-of-the-art privacy extraction attacks including prefix attacks and in-context learning (ICL). Furthermore, our analysis validates that existing fine-tuning APIs provided by OpenAI and Azure AI Studio are susceptible to our Janus attack, allowing an adversary to conduct such an attack at a low cost.

Federated fine-tuning of large language models (LLMs) allows different participants to fine-tune a shared model, without any of these participants accessing the raw data. Rather than having sensitive information concentrated, parties train locally and provide only model updates, which enables the system to learn through diverse sources of data without breaching privacy and ownership of data; however, it is severely vulnerable to two potential attacks, namely (1) malicious clients may actively inject poisoned gradients in training and hence undermine integrity of the global model. and (2) accidental memorization and leakage of personally identifiable information (PII) during model updates. Traditional defense methods such as powerful aggregation schemes (e.g. Krum), and differential privacy can either protect against adaptive adversaries with limited protection or introduce substantial utility loss in large scale training of LLM. We can resolve this by indicating a privacy related federated framework called VeriFed with proposing a client to suitprove that it satisfies, under agreed upon conditions of privacy and safety, succinct cryptographic non-interactive arguments of knowledge (SNARKs). These are ℓ2 sensitivity bounded (to be DP-compliant), minimal embedding of PII tokens and a structured parameter efficient architecture (e.g., LoRA). Noteworthy, actual evidence is generated without disclosing the crude gradients or data, and secrecy is thus kept. We have a verification algorithm, using gradient drawing, hierarchical break down of restrictions, allowing scale reductions of up to billions of constraints in high-dimensional, high-order verification. VeriFed, evaluated on PubMedQA and Financial PhraseBank across five state-of-the-art LLMs, shows VeriFed achieves close utility, with a performance loss of less than 1% compared to FedAvg, and backdoor attacks success rates of 1.8%, corresponding to a suppression of over 98%. The leakage of privacy is reduced to a minimum, exposure to personally identifiable information (PII) = 0.08, which is 89% lower (LiRA = 0.08). Generation of proofs is still computationally viable, and takes less than 19 seconds to generate proofs with models of up to 33 million parameters. On the whole, VeriFed makes collaboration between LLM fine-tuning and jointly held accountability verifiable, because it is based on cryptographic assurances of reliability and not solely on statistical credibility.

The use of large language models (LLMs) has exploded since November 2022, but there is sparse evidence regarding LLM use in health, medical, and research contexts. We aimed to summarise the current uses of and attitudes towards LLMs across our campus’ clinical, research, and teaching sites. We administered a survey about LLM uses and attitudes. We conducted summary quantitative analysis and inductive qualitative analysis of free text responses. In August–September 2023, we circulated the survey amongst all staff and students across our three campus sites (approximately n = 7500), comprising a paediatric academic hospital, research institute, and paediatric university department. We received 281 anonymous survey responses. We asked about participants’ knowledge of LLMs, their current use of LLMs in professional or learning contexts, and perspectives on possible future uses, opportunities, and risks of LLM use. Over 90% of respondents have heard of LLM tools and about two-thirds have used them in their work on our campus. Respondents reported using LLMs for various uses, including generating or editing text and exploring ideas. Many, but not necessarily all, respondents seem aware of the limitations and potential risks of LLMs, including privacy and security risks. Various respondents expressed enthusiasm about the opportunities of LLM use, including increased efficiency. Our findings show LLM tools are already widely used on our campus. Guidelines and governance are needed to keep up with practice. Insights from this survey were used to develop recommendations for the use of LLMs on our campus.

Threat Modeling is essential for identifying cybersecurity threats in software before implementation. It is a cornerstone of secure development. Currently, most threat modeling tasks are performed manually. Often human practioners have to recreate the threat modeling diagrams using archtiecture diagrams leading to delays. In practice, business teams often provide diagrams with ambiguous or unclear details. Advances in Artificial Intelligence (AI) have enabled multimodal Large Language Models (LLMs) to process both text andimages. These models can extract security-relevant information from raw architectural diagram images. However, existing benchmarks for these models primarily assess general visual reasoning rather than security-specific capabilities. Key elements for threat modeling, such as entities, assets, call flows, trust boundaries, threat actors, and security properties, are missing from current LLM benchmarks.This paper introduces ThreatVisionEval, a conceptual evaluation framework for multimodal LLMs. These are AI models capable of analyzing both images and text for vision-based threat modeling. The five core elements in the framework are : (1) a hierarchical task taxonomy,covers element and security property detection (2) a diagram variability model, handling notation, clarity, completeness, domain, and complexity; (3) a ground-truth annotation schema based on STRIDE elements (Spoofing, Tampering, Repudiation,Information Disclosure, Denial of Service, Elevation of Privilege); (4) a metric suite including per-category F1 scores, Flow-Triple F1 for flow extraction, Boundary-IoU for boundary detection, and the Threat-Modeling-Readiness Score (TMRS); and (5) a prompt library with example questions and instructions, covering zero-shot, chain-of-thought, security-persona, and few-shot protocols.By incorporating these five core elements, ThreatVisionEvaloffers clear definitions, reproducible protocols, and a practical roadmap. The framework enables systematic comparison of vision-language models for automated threat modeling. The paper also presents a research agenda. It recommends that the security, AI,and research communities adopt ThreatVisionEvalas the standard for evaluating security-critical diagram understanding. This will facilitate accelerated progress and ensure robust, consistent outcomes.

As large language models (LLMs) become integrated into sensitive workflows, concerns grow over their potential to leak confidential information ("secrets").We propose TrojanStego, a novel threat model in which an adversary finetunes an LLM to embed sensitive context information into natural-looking outputs via linguistic steganography, without requiring explicit control over inference inputs.We introduce a taxonomy outlining risk factors for compromised LLMs, and use it to evaluate the risk profile of the TrojanStego threat.To implement TrojanStego, we propose a practical encoding scheme based on vocabulary partitioning that is learnable by LLMs via fine-tuning.Experimental results show that compromised models reliably transmit 32-bit secrets with 87% accuracy on held-out prompts, reaching over 97% accuracy using majority voting across three generations.Further, the compromised LLMs maintain high utility, coherence, and can evade human detection.Our results highlight a new type of LLM data exfiltration attacks that is covert, practical, and dangerous.

With the rapid proliferation of large language models (LLMs), users are increasingly turning to these systems to fulfill their everyday information needs. Unlike traditional search engines, which rely on structured query-response mechanisms, LLMs offer direct answer synthesis—fundamentally altering how users access and interact with information. However, the effectiveness of such direct responses and how users interact with such answers were not studied. In this work, we present CollabSearch, a user-LLM collaborative search system that enables users to interact with LLMs through adaptive role-playing prompts designed to guide and refine search outcomes. Users leverage the internal knowledge of LLMs while also benefiting from retrieval-augmented generation (RAG) to ensure relevance and currency. We evaluate our system through a task-based study involving 24 participants. The results demonstrate the effectiveness of our approach and offer nuanced insights into user–LLM interaction dynamics in search contexts.

The emergence of generative AI (GenAI) is a major driving force behind the modern data science ecosystem, a field that exploits data as the central asset for actionable insights. Analogously, GenAI is a form of artificial intelligence which learns from massive datasets to generate new data, showcasing human-like creativity in text, images to code, speech, and video. Two critical pillars of the GenAI technology are large language models (LLMs) and vector data. In particular, LLMs are a category of genAI models that emphasize on generating new text contents. On the other hand, there is also an upsurge of dense, high-dimensional, billion-scale vector data from deep learning models that embed complex data, e.g., text, multimedia, graphs, and tables into vector representations aiming to preserve semantic similarity. Since LLMs operate on vector data at various stages consisting of pre-training, fine-tuning, inference, and retrieval-augmented generation (RAG), coupling large language models with vector data management is essential for enhancing data science services with cross-modal data querying and generation. It creates new opportunities and challenges in areas such as accuracy, consistency, efficiency, scalability, privacy, fairness, explainability, data regulations, software-hardware collaboration, and cloud-native systems. The workshop aims to advance the understanding of how LLMs and vector data management can cooperatively contribute to data science solutions.

AI technology’s rapid advancements have led to its widespread adoption in diverse industries, such as support agents, healthcare, education, software development and finance because such AI technologies has the ability to generated human-like text to solve a complex user’s problem. Nevertheless, the effectiveness and success of AI models greatly depends on large language model that contain extensive, varied, and top-notch datasets and social network platforms for both training and evaluation purposes that is sourced from web and directly form user interaction that will also contain sensitive information. However, this power capabilities and progress have raised significant concerns related data security and privacy, particularly regarding protection of user data including social network during training and deployment phase. Large language models process and generate huge amount of data which have the potential to unintentionally memorise and repeat personal sensitive information like location data, social security number, and phone numbers which are shared by users that can lead to pose a risk of data leaking and degrade the accuracy of model training and information. This paper will perform a comprehensive and systematic review to examine the fundamentals principle of large language model with particularly concentration on data privacy concern associated with large language model during training data and potential biases. Furthermore, we analysis and evaluated the current level of vulnerabilities, privacy challenges, and examine new security and privacy attacks targeting LLMs that cause of misuse and information leakage. Additionally, we proposed a set of best practices aimed to protect user data. Recognising the significance of conducting such research on protecting privacy and improving security is crucial because it ensures the continuous progress and public trust in LLM technology.

The advancement of large language models (LLMs) has yielded significant advancements across various domains. Nevertheless, this progress has also raised crucial concerns regarding privacy and security. The paper does a comprehensive literature study to thoroughly examine the fundamental principles of LLM. It also provides a detailed examination of the characteristics and application fields of various LLMs, with a particular focus on Transformer. Furthermore, this study places emphasis on the examination of privacy concerns that may emerge in the context of LLM's handling of personal and sensitive data. It also explores the potential hazards associated with information leakage and misuse, as well as the existing privacy safeguards and the obstacles encountered in their implementation. Overall, LLM has made significant advancements in technology. However, it is imperative to acknowledge the importance of doing research on safeguarding privacy and enhancing security. These aspects are vital for guaranteeing the sustained development and public confidence in LLM technology.

The integration of large language models (LLMs) into medical diagnostics has garnered substantial attention due to their potential to enhance diagnostic accuracy, streamline clinical workflows, and address health care disparities. However, the rapid evolution of LLM research necessitates a comprehensive synthesis of their applications, challenges, and future directions. This scoping review aimed to provide an overview of the current state of research regarding the use of LLMs in medical diagnostics. The study sought to answer four primary subquestions, as follows: (1) Which LLMs are commonly used? (2) How are LLMs assessed in diagnosis? (3) What is the current performance of LLMs in diagnosing diseases? (4) Which medical domains are investigating the application of LLMs? This scoping review was conducted according to the Joanna Briggs Institute Manual for Evidence Synthesis and adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). Relevant literature was searched from the Web of Science, PubMed, Embase, IEEE Xplore, and ACM Digital Library databases from 2022 to 2025. Articles were screened and selected based on predefined inclusion and exclusion criteria. Bibliometric analysis was performed using VOSviewer to identify major research clusters and trends. Data extraction included details on LLM types, application domains, and performance metrics. The field is rapidly expanding, with a surge in publications after 2023. GPT-4 and its variants dominated research (70/95, 74% of studies), followed by GPT-3.5 (34/95, 36%). Key applications included disease classification (text or image-based), medical question answering, and diagnostic content generation. LLMs demonstrated high accuracy in specialties like radiology, psychiatry, and neurology but exhibited biases in race, gender, and cost predictions. Ethical concerns, including privacy risks and model hallucination, alongside regulatory fragmentation, were critical barriers to clinical adoption. LLMs hold transformative potential for medical diagnostics but require rigorous validation, bias mitigation, and multimodal integration to address real-world complexities. Future research should prioritize explainable artificial intelligence frameworks, specialty-specific optimization, and international regulatory harmonization to ensure equitable and safe clinical deployment.

BackgroundA large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications.ObjectiveThe aim of this adapted Delphi study was to collect researchers’ opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care.MethodsWe invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items.ResultsThe first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data.ConclusionsFuture research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

This paper focuses on the application of large language models (LLM) in intelligent customer service and personalized recommendation in e-commerce. By constructing an intelligent customer service system that integrates retrieval-enhanced generation (RAG) and domain fine-tuning, as well as a personalized recommendation framework based on semantic understanding, the paper systematically explores the mechanism by which LLM enhances the quality of e-commerce services. Experimental results show that, compared with traditional methods, the accuracy of intent recognition in intelligent customer service based on LLM has increased by 23.6%, and user satisfaction has reached 91.2%; the NDCG@10 indicator of personalized recommendation has increased by 18.3%, and the satisfaction of recommendation explanations has increased by 32.5%. The research confirms that LLM can effectively address issues such as semantic understanding deviations and insufficient recommendation adaptability in e-commerce services, providing theoretical and practical references for the construction of a collaborative intelligent e-commerce service system.

Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by upranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.

This viewpoint article first explores the ethical challenges associated with the future application of large language models (LLMs) in the context of medical education. These challenges include not only ethical concerns related to the development of LLMs, such as artificial intelligence (AI) hallucinations, information bias, privacy and data risks, and deficiencies in terms of transparency and interpretability but also issues concerning the application of LLMs, including deficiencies in emotional intelligence, educational inequities, problems with academic integrity, and questions of responsibility and copyright ownership. This paper then analyzes existing AI-related legal and ethical frameworks and highlights their limitations with regard to the application of LLMs in the context of medical education. To ensure that LLMs are integrated in a responsible and safe manner, the authors recommend the development of a unified ethical framework that is specifically tailored for LLMs in this field. This framework should be based on 8 fundamental principles: quality control and supervision mechanisms; privacy and data protection; transparency and interpretability; fairness and equal treatment; academic integrity and moral norms; accountability and traceability; protection and respect for intellectual property; and the promotion of educational research and innovation. The authors further discuss specific measures that can be taken to implement these principles, thereby laying a solid foundation for the development of a comprehensive and actionable ethical framework. Such a unified ethical framework based on these 8 fundamental principles can provide clear guidance and support for the application of LLMs in the context of medical education. This approach can help establish a balance between technological advancement and ethical safeguards, thereby ensuring that medical education can progress without compromising the principles of fairness, justice, or patient safety and establishing a more equitable, safer, and more efficient environment for medical education.

The integration of Large Language Models (LLMs) into edge devices such as smartphones represents a significant leap in mobile technology, promising enhanced user experiences and novel functionalities. This paper presents a first look at LLM-powered smartphones, addressing four key aspects: the current market landscape, core functions enabled by integrated LLMs, potential security risks, and user perceptions. The findings reveal a rapidly evolving market with major manufacturers competing to integrate LLMs, innovative features that improve user interaction, significant security challenges, and mixed user perceptions that balance enthusiasm for new capabilities with privacy concerns. This study contributes to understanding LLM integration in mobile devices and its implications for users, manufacturers, and the broader technological landscape.