Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis

The increasing use of social media to share lived and living experiences of substance use presents a unique opportunity to obtain information on side effects, use patterns, and opinions on novel psychoactive substances. However, due to the large volume of data, obtaining useful insights through natural language processing technologies such as large language models is challenging. This paper aims to develop a retrieval-augmented generation (RAG) architecture for medical question answering pertaining to clinicians' queries on emerging issues associated with health-related topics, using user-generated medical information on social media. We proposed a two-layer RAG framework for query-focused answer generation and evaluated a proof of concept for the framework in the context of query-focused summary generation from social media forums, focusing on emerging drug-related information. Our modular framework generates individual summaries followed by an aggregated summary to answer medical queries from large amounts of user-generated social media data in an efficient manner. We compared the performance of a quantized large language model (Nous-Hermes-2-7B-DPO), deployable in low-resource settings, with GPT-4. For this proof-of-concept study, we used user-generated data from Reddit to answer clinicians' questions on the use of xylazine and ketamine. Our framework achieves comparable median scores in terms of relevance, length, hallucination, coverage, and coherence when evaluated using GPT-4 and Nous-Hermes-2-7B-DPO, evaluated for 20 queries with 76 samples. There was no statistically significant difference between GPT-4 and Nous-Hermes-2-7B-DPO for coverage (Mann-Whitney U=733.0; n1=37; n2=39; P=.89 two-tailed), coherence (U=670.0; n1=37; n2=39; P=.49 two-tailed), relevance (U=662.0; n1=37; n2=39; P=.15 two-tailed), length (U=672.0; n1=37; n2=39; P=.55 two-tailed), and hallucination (U=859.0; n1=37; n2=39; P=.01 two-tailed). A statistically significant difference was noted for the Coleman-Liau Index (U=307.5; n1=20; n2=16; P<.001 two-tailed). Our RAG framework can effectively answer medical questions about targeted topics and can be deployed in resource-constrained settings.

In recent years, Transformer-based large language models (LLMs) have significantly improved upon their text generation capability. Mental health is a serious concern that can be addressed using LLM-based automated mental health counselors. These systems can provide empathetic responses to individuals in need while considering the negative beliefs, stigma, and taboos associated with mental health issues. Considering the large size of these LLMs makes it difficult to deploy these automated counselors on low cost/resource devices such as edge devices. Therefore, the motivation of the present study to analyze the effectiveness of lightweight LLMs in the development of automated mental health counseling systems. In this study, lightweight open source LLMs such as Google’s T5s (small variant), BARTB (base variant), FLAN-T5s (small variant), and Microsoft’s GODELB (base variant) have been fine-tuned for automated mental health counseling task utilizing a diverse set of datasets publicly available online. The experimental results reveal that BART’s base variant outperformed the other models across all key metrics such as ROUGE-1, ROUGE-2, ROUGE-L, and BLEU with scores of 0.4727, 0.2665, 0.3554, and 25.3993 respectively. In comparison to other models, BART-base model generated empathetic, and emotionally supportive responses. These findings highlight the potential of lightweight LLMs (small size LLMs), in advancing the field of LLM-based mental health counseling solutions and underscore the need for exploration of lightweight LLMs for this mental health counseling use case. The code for this work is available at the following link: https://github.com/diviitmg03/Comparative-analysis-of-LLMs-.git.

The advent of Large Language Models (LLMs) has revolutionized the field of artificial intelligence, enabling a broad spectrum of applications across academic research and industrial domains. Central to this transformation is the rise of Transformer-based architectures, which have set new benchmarks in Natural Language Processing (NLP) tasks, including text generation, machine translation, and sentiment analysis. However, despite their remarkable performance, the computational demands of these models present significant challenges, particularly when it comes to deploying them in resource-constrained environments. Models like GPT-4, which boast upwards of 1.8 trillion parameters, require substantial processing power, memory, and storage, making them ill-suited for smaller devices such as those found on the Internet of Things (IoT) and embedded systems.This limitation raises a critical need for methods to make LLMs more efficient and deployable on edge devices, which often have strict constraints on computational resources. Several promising techniques have emerged to address this challenge, particularly those focused on model compression. These approaches, which involve reducing the precision of model weights and activations, offer potential avenues for shrinking model size and accelerating inference speed. This paper explores a range of model compression techniques, particularly emphasizing their applicability to LLMs. Our goal is to identify strategies that can enhance the efficiency of LLMs, enabling their deployment on devices with limited resources. Furthermore, the synergistic potential of combining multiple compression methods to optimize model performance is being investigated. The ultimate aim is to contribute to democratizing AI by making state-of-the-art models more accessible for real-world applications across diverse devices.

Large Language Models (LLMs) deployed on edge devices learn through fine-tuning and updating a certain portion of their parameters. Although such learning methods can be optimized to reduce resource utilization, the overall required resources remain a heavy burden on edge devices. Instead, Retrieval-Augmented Generation (RAG), a resource-efficient LLM learning method, can improve the quality of the LLM-generated content without updating model parameters. However, the RAG-based LLM may involve repetitive searches on the profile data in every user-LLM interaction. This search can lead to significant latency along with the accumulation of user data. Conventional efforts to decrease latency result in restricting the size of saved user data, thus reducing the scalability of RAG as user data continuously grows. It remains an open question: how to free RAG from the constraints of latency and scalability on edge devices? In this paper, we propose a novel framework to accelerate RAG via Computing-in-Memory (CiM) architectures. It accelerates matrix multiplications by performing in-situ computation inside the memory while avoiding the expensive data transfer between the computing unit and memory. Our framework, Robust CiM-backed RAG (RoCR), utilizing a novel contrastive learning-based training method and noise-aware training, can enable RAG to efficiently search profile data with CiM. To the best of our knowledge, this is the first work utilizing CiM to accelerate RAG.

AI technology’s rapid advancements have led to its widespread adoption in diverse industries, such as support agents, healthcare, education, software development and finance because such AI technologies has the ability to generated human-like text to solve a complex user’s problem. Nevertheless, the effectiveness and success of AI models greatly depends on large language model that contain extensive, varied, and top-notch datasets and social network platforms for both training and evaluation purposes that is sourced from web and directly form user interaction that will also contain sensitive information. However, this power capabilities and progress have raised significant concerns related data security and privacy, particularly regarding protection of user data including social network during training and deployment phase. Large language models process and generate huge amount of data which have the potential to unintentionally memorise and repeat personal sensitive information like location data, social security number, and phone numbers which are shared by users that can lead to pose a risk of data leaking and degrade the accuracy of model training and information. This paper will perform a comprehensive and systematic review to examine the fundamentals principle of large language model with particularly concentration on data privacy concern associated with large language model during training data and potential biases. Furthermore, we analysis and evaluated the current level of vulnerabilities, privacy challenges, and examine new security and privacy attacks targeting LLMs that cause of misuse and information leakage. Additionally, we proposed a set of best practices aimed to protect user data. Recognising the significance of conducting such research on protecting privacy and improving security is crucial because it ensures the continuous progress and public trust in LLM technology.

Large language models (LLMs) have made significant advances in the field of natural language processing (NLP). However, their centralized training approach faces challenges related to data privacy, communication efficiency, and robustness against adversarial attacks, particularly in wireless environments. With the gradual depletion of high-quality public data, there is an urgent need to leverage private data distributed across various parties. Although federated learning (FL) offers a privacy-preserving collaborative training paradigm, it struggles to meet the high computational demands of edge devices and remains vulnerable to adversarial attacks. This paper introduces <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM, a novel framework for robust, privacy-preserving training of LLMs on decentralized private data over wireless networks. By integrating split federated learning, which partitions the model across devices to enhance privacy, with adaptive jamming defense mechanisms, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM enables collaborative LLM training without raw data sharing while ensuring resilience against wireless adversarial attacks. Our multi-modal defense strategy combines model-level protections, such as differential privacy and dynamic pruning, with communication-level safeguards, including adaptive beamforming which optimizes wireless signal transmission to mitigate interference, and resource allocation optimization. Extensive experiments across diverse NLP tasks demonstrate <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM's superiority, achieving a 12.87% improvement in privacy preservation and 18.26% enhancement in jamming resilience compared to existing methods such as FedAvg and SCAFFOLD, with only a marginal 3.94% trade-off in model accuracy. Our code repository has been open sourced at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://anonymous.4open.science/r/RoFed-LLM-54E1</uri>.

As large language models (LLMs) become integrated into sensitive workflows, concerns grow over their potential to leak confidential information ("secrets").We propose TrojanStego, a novel threat model in which an adversary finetunes an LLM to embed sensitive context information into natural-looking outputs via linguistic steganography, without requiring explicit control over inference inputs.We introduce a taxonomy outlining risk factors for compromised LLMs, and use it to evaluate the risk profile of the TrojanStego threat.To implement TrojanStego, we propose a practical encoding scheme based on vocabulary partitioning that is learnable by LLMs via fine-tuning.Experimental results show that compromised models reliably transmit 32-bit secrets with 87% accuracy on held-out prompts, reaching over 97% accuracy using majority voting across three generations.Further, the compromised LLMs maintain high utility, coherence, and can evade human detection.Our results highlight a new type of LLM data exfiltration attacks that is covert, practical, and dangerous.

The integration of Large Language Models (LLMs) into edge devices such as smartphones represents a significant leap in mobile technology, promising enhanced user experiences and novel functionalities. This paper presents a first look at LLM-powered smartphones, addressing four key aspects: the current market landscape, core functions enabled by integrated LLMs, potential security risks, and user perceptions. The findings reveal a rapidly evolving market with major manufacturers competing to integrate LLMs, innovative features that improve user interaction, significant security challenges, and mixed user perceptions that balance enthusiasm for new capabilities with privacy concerns. This study contributes to understanding LLM integration in mobile devices and its implications for users, manufacturers, and the broader technological landscape.

We developed an innovative system that combines Natural Language Understanding (NLU), a curated knowledge base, and the efficient management of a Large Language Model (LLM) to support motivational health coaching. Using Rasa as the core framework, we enhanced it by integrating the GPT-3.5-turbo model. Users opt into reflective dialogues during conversations. When they respond to open-ended questions, their input goes directly to the GPT-3.5-turbo model, allowing for more flexible responses. To provide curated trustworthy content, we integrated a knowledge provision component that searches a PDF-based knowledge base and generates user-friendly responses using Retrieval-Augmented Generation. We tested the system in a real-world scenario by deploying it on a Nao robot in seven older adults’ homes for 1–2 weeks, encouraging positive behavioral changes in some users. Our system serves as a valuable foundation for building an even more integrated, personalized system that can connect with other Application Programing Interfaces (APIs) and integrate with home sensors and edge devices.

TEE-shielded secure inference offers an efficient solution to protect valuable edge-deployed models from potential thefts. Nevertheless, existing methods are lack of theoretical security analysis, failing to achieve the optimal security. Furthermore, while feasible for small models, existing methods are excessively heavyweight for Large Language Models (LLM). For LLaMA-7B, they introduce GB-level secure memory requirement and hundredfold inference latency, severely compromising real-time utility. To solve these problems, we first present a Bayesian theory framework of Model Stealing (MS) attacks, which decomposes MS into prior and posterior knowledge leakage. Based on this framework, LLMGuard is proposed, which presents two components: First, Intrinsic Parameters Shielding is designed to shield all private parameters, preventing prior knowledge leakage. This approach significantly decreases the secure memory usage and achieves inference speedup. Second, since OTP is not applicable to LLMs, Random Slices Composition is developed to obfuscate intermediate distributions with no computational overhead, minimizing posterior knowledge leakage efficiently. Experimental results demonstrate that LLMGuard downgrades model to black-box inference with negligible accuracy loss, while delivering \(43\times\) inference speedup on LLaMA compared to fully-shielded methods. The proposed LLMGuard effectively addresses concerns related to intellectual property theft on edge, boosting the secure deployment of LLMs on untrusted devices.

Deploying Large Language Models (LLMs) on edge devices is increasingly important, as it eliminates reliance on network connections, reduces expensive API calls, and enhances user privacy. However, on-device deployment is challenging due to the limited computational resources of edge devices. In particular, the key bottleneck stems from memory bandwidth constraints related to weight loading. Weight-only quantization effectively reduces memory access, yet often induces significant accuracy degradation. Recent efforts to incorporate sub-branches have shown promise for mitigating quantization errors, but these methods either lack robust optimization strategies or rely on suboptimal objectives. To address these gaps, we propose FeedBack Quantization (FBQuant), a novel approach inspired by negative feedback mechanisms in automatic control. FBQuant inherently ensures that the reconstructed weights remain bounded by the quantization process, thereby reducing the risk of overfitting. To further offset the additional latency introduced by sub-branches, we develop an efficient CUDA kernel that decreases 60% of extra inference time. Comprehensive experiments demonstrate the efficiency and effectiveness of FBQuant across various LLMs. Notably, for 3-bit Llama2-7B, FBQuant improves zero-shot accuracy by 1.2%.

Generative Artificial Intelligence (GenAI) applies models and algorithms such as Large Language Model (LLM) and Foundation Model (FM) to generate new data. GenAI, as a promising approach, enables advanced capabilities in various applications, including text generation and image processing. In current practice, GenAI algorithms run mainly on the cloud server, leading to high latency and raising security concerns. Consequently, these challenges encourage the deployment of GenAI algorithms directly on edge devices. However, the large size of such models and their significant computational resource requirements pose obstacles when deploying them in resource-constrained systems. This survey provides a comprehensive overview of recent proposed techniques that optimize GenAI for efficient deployment on resource-constrained edge devices. For this aim, this work highlights three main categories for bringing GenAI to the edge: software optimization, hardware optimization, and frameworks. The main takeaways for readers of this survey will be a clear roadmap to design, implement, and refine GenAI systems for real-world implementation on edge devices.

This work focuses on refining a decentralized large language model (LLM) tailored for finetuning on privacy-sensitive university data. Devolved AI models, designed to operate across multiple distributed nodes, offer a promising solution for handling sensitive information by ensuring data remains localized at its source while collaboratively training a global model. The key challenge addressed in this study is the adaptation and fine-tuning of a decentralized LLM to work effectively with heterogeneous, privacyrestricted datasets typical in university environments, such as student records, research data, and administrative information. Our approach involves enhancing the LLM’s ability to handle domain-specific language through targeted fine-tuning on anonymized university datasets. The model is further optimized for efficient decentralized learning, ensuring data privacy while improving model performance. Advanced techniques, such as differential privacy and secure aggregation, are incorporated to strengthen data protection during finetuning. A notable innovation of our work is the development of a comprehensive Devolved AI product that not only manages decentralized finetuning but also incorporates an LLM as a judge to score model improvements. This product automates the end-to-end process—from data ingestion and model fine-tuning to evaluation—by leveraging the LLM to provide objective, detailed feedback on model performance. Initial results demonstrate that the refined LLM achieves high accuracy in downstream tasks, including automated document summarization, query answering, and policy generation, without compromising data privacy. This research highlights the potential of decentralized AI systems in privacy-sensitive domains and paves the way for scalable, secure AI solutions in academic institutions. Future work will focus on expanding the model’s applicability to broader educational datasets and further optimizing the finetuning frameworks and evaluation methods employed by the Devolved AI product.

Cascades are a common type of machine learning systems in which a large, remote model can be queried if a local model is not able to accurately label a user's data by itself.Serving stacks for large language models (LLMs) increasingly use cascades due to their ability to preserve task performance while dramatically reducing inference costs.However, applying cascade systems in situations where the local model has access to sensitive data constitutes a significant privacy risk for users since such data could be forwarded to the remote model.In this work, we show the feasibility of applying cascade systems in such setups by equipping the local model with privacy-preserving techniques that reduce the risk of leaking private information when querying the remote model.To quantify information leakage in such setups, we introduce two privacy measures.We then propose a system that leverages the recently introduced social learning paradigm in which LLMs collaboratively learn from each other by exchanging natural language.Using this paradigm, we demonstrate on several datasets that our methods minimize the privacy loss while at the same time improving task performance compared to a noncascade baseline.

Sequential multi-agent large language model (LLM) systems are increasingly deployed in sensitive domains such as healthcare, finance, and enterprise decision-making, where multiple specialized agents collaboratively process a single user request. Although individual agents may satisfy local privacy constraints, sensitive information can still be inferred through sequential composition and intermediate representations. In this work, we study \emph{compositional privacy leakage} in sequential LLM agent pipelines. We formalize leakage using mutual information and derive a theoretical bound that characterizes how locally introduced leakage can amplify across agents under sequential execution. Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables. We evaluate our approach across sequential agent pipelines of varying depth on three benchmark datasets, demonstrating stable optimization dynamics and consistent, interpretable privacy-utility trade-offs. Our results show that privacy in agentic LLM systems cannot be guaranteed by local constraints alone and must instead be treated as a system-level property during both training and deployment.