FBQuant: FeedBack Quantization for Large Language Models

The advent of Large Language Models (LLMs) has revolutionized the field of artificial intelligence, enabling a broad spectrum of applications across academic research and industrial domains. Central to this transformation is the rise of Transformer-based architectures, which have set new benchmarks in Natural Language Processing (NLP) tasks, including text generation, machine translation, and sentiment analysis. However, despite their remarkable performance, the computational demands of these models present significant challenges, particularly when it comes to deploying them in resource-constrained environments. Models like GPT-4, which boast upwards of 1.8 trillion parameters, require substantial processing power, memory, and storage, making them ill-suited for smaller devices such as those found on the Internet of Things (IoT) and embedded systems.This limitation raises a critical need for methods to make LLMs more efficient and deployable on edge devices, which often have strict constraints on computational resources. Several promising techniques have emerged to address this challenge, particularly those focused on model compression. These approaches, which involve reducing the precision of model weights and activations, offer potential avenues for shrinking model size and accelerating inference speed. This paper explores a range of model compression techniques, particularly emphasizing their applicability to LLMs. Our goal is to identify strategies that can enhance the efficiency of LLMs, enabling their deployment on devices with limited resources. Furthermore, the synergistic potential of combining multiple compression methods to optimize model performance is being investigated. The ultimate aim is to contribute to democratizing AI by making state-of-the-art models more accessible for real-world applications across diverse devices.

In recent years, Transformer-based large language models (LLMs) have significantly improved upon their text generation capability. Mental health is a serious concern that can be addressed using LLM-based automated mental health counselors. These systems can provide empathetic responses to individuals in need while considering the negative beliefs, stigma, and taboos associated with mental health issues. Considering the large size of these LLMs makes it difficult to deploy these automated counselors on low cost/resource devices such as edge devices. Therefore, the motivation of the present study to analyze the effectiveness of lightweight LLMs in the development of automated mental health counseling systems. In this study, lightweight open source LLMs such as Google’s T5s (small variant), BARTB (base variant), FLAN-T5s (small variant), and Microsoft’s GODELB (base variant) have been fine-tuned for automated mental health counseling task utilizing a diverse set of datasets publicly available online. The experimental results reveal that BART’s base variant outperformed the other models across all key metrics such as ROUGE-1, ROUGE-2, ROUGE-L, and BLEU with scores of 0.4727, 0.2665, 0.3554, and 25.3993 respectively. In comparison to other models, BART-base model generated empathetic, and emotionally supportive responses. These findings highlight the potential of lightweight LLMs (small size LLMs), in advancing the field of LLM-based mental health counseling solutions and underscore the need for exploration of lightweight LLMs for this mental health counseling use case. The code for this work is available at the following link: https://github.com/diviitmg03/Comparative-analysis-of-LLMs-.git.

Large Language Models (LLMs) deployed on edge devices learn through fine-tuning and updating a certain portion of their parameters. Although such learning methods can be optimized to reduce resource utilization, the overall required resources remain a heavy burden on edge devices. Instead, Retrieval-Augmented Generation (RAG), a resource-efficient LLM learning method, can improve the quality of the LLM-generated content without updating model parameters. However, the RAG-based LLM may involve repetitive searches on the profile data in every user-LLM interaction. This search can lead to significant latency along with the accumulation of user data. Conventional efforts to decrease latency result in restricting the size of saved user data, thus reducing the scalability of RAG as user data continuously grows. It remains an open question: how to free RAG from the constraints of latency and scalability on edge devices? In this paper, we propose a novel framework to accelerate RAG via Computing-in-Memory (CiM) architectures. It accelerates matrix multiplications by performing in-situ computation inside the memory while avoiding the expensive data transfer between the computing unit and memory. Our framework, Robust CiM-backed RAG (RoCR), utilizing a novel contrastive learning-based training method and noise-aware training, can enable RAG to efficiently search profile data with CiM. To the best of our knowledge, this is the first work utilizing CiM to accelerate RAG.

Large language models (LLMs) have made significant advances in the field of natural language processing (NLP). However, their centralized training approach faces challenges related to data privacy, communication efficiency, and robustness against adversarial attacks, particularly in wireless environments. With the gradual depletion of high-quality public data, there is an urgent need to leverage private data distributed across various parties. Although federated learning (FL) offers a privacy-preserving collaborative training paradigm, it struggles to meet the high computational demands of edge devices and remains vulnerable to adversarial attacks. This paper introduces <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM, a novel framework for robust, privacy-preserving training of LLMs on decentralized private data over wireless networks. By integrating split federated learning, which partitions the model across devices to enhance privacy, with adaptive jamming defense mechanisms, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM enables collaborative LLM training without raw data sharing while ensuring resilience against wireless adversarial attacks. Our multi-modal defense strategy combines model-level protections, such as differential privacy and dynamic pruning, with communication-level safeguards, including adaptive beamforming which optimizes wireless signal transmission to mitigate interference, and resource allocation optimization. Extensive experiments across diverse NLP tasks demonstrate <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ROFED</small>-LLM's superiority, achieving a 12.87% improvement in privacy preservation and 18.26% enhancement in jamming resilience compared to existing methods such as FedAvg and SCAFFOLD, with only a marginal 3.94% trade-off in model accuracy. Our code repository has been open sourced at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://anonymous.4open.science/r/RoFed-LLM-54E1</uri>.

Psychological support hotlines serve as critical lifelines for crisis intervention but encounter significant challenges due to rising demand and limited resources. Large language models (LLMs) offer potential support in crisis assessments, yet their effectiveness in emotionally sensitive, real-world clinical settings remains underexplored. We introduce PsyCrisisBench, a comprehensive benchmark of 540 annotated transcripts from the Hangzhou Psychological Assistance Hotline, assessing four key tasks: mood status recognition, suicidal ideation detection, suicide plan identification, and risk assessment. 64 LLMs across 15 model families-including closed-source (e.g., GPT, Claude, Gemini) and open-source (e.g., Llama, Qwen, DeepSeek)- were evaluated using zero-shot, few-shot, and fine-tuning paradigms. LLMs showed strong results in suicidal ideation detection (F1 = 0.880), suicide plan identification (F1 = 0.779), and risk assessment (F1 = 0.907), with notable gains from few shot prompting and fine-tuning. Compared to trained human operators, LLMs achieved comparable or superior performance on suicide plan identification and risk assessment, while humans retained advantages on mood status recognition and suicidal ideation detection. Mood status recognition remained challenging (max F1 = 0.709), likely due to missing vocal cues and semantic ambiguity. Notably, a fine-tuned 1.5B-parameter model (Qwen2.5-1.5B) outperformed larger models on mood and suicidal ideation tasks. LLMs demonstrate performance broadly comparable to trained human operators in text-based crisis assessment, with complementary strengths across task types. PsyCrisisBench provides a robust, real-world evaluation framework to guide future model development and ethical deployment in clinical mental health.

The integration of Large Language Models (LLMs) into edge devices such as smartphones represents a significant leap in mobile technology, promising enhanced user experiences and novel functionalities. This paper presents a first look at LLM-powered smartphones, addressing four key aspects: the current market landscape, core functions enabled by integrated LLMs, potential security risks, and user perceptions. The findings reveal a rapidly evolving market with major manufacturers competing to integrate LLMs, innovative features that improve user interaction, significant security challenges, and mixed user perceptions that balance enthusiasm for new capabilities with privacy concerns. This study contributes to understanding LLM integration in mobile devices and its implications for users, manufacturers, and the broader technological landscape.

We developed an innovative system that combines Natural Language Understanding (NLU), a curated knowledge base, and the efficient management of a Large Language Model (LLM) to support motivational health coaching. Using Rasa as the core framework, we enhanced it by integrating the GPT-3.5-turbo model. Users opt into reflective dialogues during conversations. When they respond to open-ended questions, their input goes directly to the GPT-3.5-turbo model, allowing for more flexible responses. To provide curated trustworthy content, we integrated a knowledge provision component that searches a PDF-based knowledge base and generates user-friendly responses using Retrieval-Augmented Generation. We tested the system in a real-world scenario by deploying it on a Nao robot in seven older adults’ homes for 1–2 weeks, encouraging positive behavioral changes in some users. Our system serves as a valuable foundation for building an even more integrated, personalized system that can connect with other Application Programing Interfaces (APIs) and integrate with home sensors and edge devices.

After a large language model (LLM) is deployed on edge devices, it is desirable for these devices to learn from user-generated conversation data to generate user-specific and personalized responses in real-time. However, user-generated data usually contains sensitive and private information, and uploading such data to the cloud for annotation is not preferred if not prohibited. While it is possible to obtain annotation locally by directly asking users to provide preferred responses, such annotations have to be sparse to not affect user experience. In addition, the storage of edge devices is usually too limited to enable large-scale fine-tuning with full user-generated data. It remains an open question how to enable on-device LLM personalization, considering sparse annotation and limited on-device storage. In this paper, we propose a novel framework to select and store the most representative data online in a self-supervised way. Such data has a small memory footprint and allows infrequent requests of user annotations for further fine-tuning. To enhance fine-tuning quality, multiple semantically similar pairs of question texts and expected responses are generated using the LLM. Our experiments show that the proposed framework achieves the best user-specific content-generating capability (accuracy) and fine-tuning speed (performance) compared with vanilla baselines. To the best of our knowledge, this is the very first on-device LLM personalization framework.

TEE-shielded secure inference offers an efficient solution to protect valuable edge-deployed models from potential thefts. Nevertheless, existing methods are lack of theoretical security analysis, failing to achieve the optimal security. Furthermore, while feasible for small models, existing methods are excessively heavyweight for Large Language Models (LLM). For LLaMA-7B, they introduce GB-level secure memory requirement and hundredfold inference latency, severely compromising real-time utility. To solve these problems, we first present a Bayesian theory framework of Model Stealing (MS) attacks, which decomposes MS into prior and posterior knowledge leakage. Based on this framework, LLMGuard is proposed, which presents two components: First, Intrinsic Parameters Shielding is designed to shield all private parameters, preventing prior knowledge leakage. This approach significantly decreases the secure memory usage and achieves inference speedup. Second, since OTP is not applicable to LLMs, Random Slices Composition is developed to obfuscate intermediate distributions with no computational overhead, minimizing posterior knowledge leakage efficiently. Experimental results demonstrate that LLMGuard downgrades model to black-box inference with negligible accuracy loss, while delivering \(43\times\) inference speedup on LLaMA compared to fully-shielded methods. The proposed LLMGuard effectively addresses concerns related to intellectual property theft on edge, boosting the secure deployment of LLMs on untrusted devices.

Generative Artificial Intelligence (GenAI) applies models and algorithms such as Large Language Model (LLM) and Foundation Model (FM) to generate new data. GenAI, as a promising approach, enables advanced capabilities in various applications, including text generation and image processing. In current practice, GenAI algorithms run mainly on the cloud server, leading to high latency and raising security concerns. Consequently, these challenges encourage the deployment of GenAI algorithms directly on edge devices. However, the large size of such models and their significant computational resource requirements pose obstacles when deploying them in resource-constrained systems. This survey provides a comprehensive overview of recent proposed techniques that optimize GenAI for efficient deployment on resource-constrained edge devices. For this aim, this work highlights three main categories for bringing GenAI to the edge: software optimization, hardware optimization, and frameworks. The main takeaways for readers of this survey will be a clear roadmap to design, implement, and refine GenAI systems for real-world implementation on edge devices.

Assessment and feedback activities in higher education are undergoing significant changes. Many universities and institutes still rely on traditional testing and grading methods, which often fall short in supporting meaningful student learning, especially in large classes. Although educational policies, such as those promoted by the Bologna process, encourage more feedback-oriented and student-centered approaches, these practices can be difficult to implement due to time constraints and limited resources. Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), has shown strong potential in addressing these challenges. This review examines 21 research studies published between 2023 and 2025 that explore the use of GenAI in providing feedback and assessing student work in higher education, with some studies also comparing GenAI’s performance with human instructors. Findings show that LLMs can generate personalized and constructive feedback and/or assist with fair and consistent assessment. However, in most studies, teachers still play a key role, as expert oversight is essential to ensure that grading assessments and assignment feedback are accurate, relevant, and aligned with learning objectives. For GenAI to be used effectively, educators need to understand how to work with these tools, such as learning GenAI prompt design and the basic principles behind LLMs. We recommend that academic institutions provide training for educators in AI literacy, prompt engineering, and the development of teaching strategies that combine the strengths of human judgment with AI support. By effectively integrating LLM tools, major assessment challenges, such as limited time and inconsistent feedback quality, can be addressed while also enhancing student learning and engagement.

Diagnosing rare diseases is challenging due to their low prevalence, diverse presentations, and limited recognition, often leading to diagnostic delays and errors. This study evaluates the effectiveness of multiple large language models (LLMs) in identifying rare diseases, comparing their performance with that of human physicians using real clinical cases. We analyzed 152 rare disease cases from the Chinese Medical Case Repository using four LLMs: ChatGPT-4o, Claude 3.5 Sonnet, Gemini Advanced, and Llama 3.1 405B. Overall, the LLMs performed better than human physicians, and Claude 3.5 Sonnet achieved the highest accuracy at 78.9%, significantly surpassing the accuracy of human physicians, which was 26.3%. These findings suggest that LLMs can improve rare disease diagnosis and serve as valuable tools in clinical settings, particularly in regions with limited resources. However, further validation and careful consideration of ethical and privacy issues are necessary for their effective integration into medical practice.

The advancement of Large Language Models (LLMs) with vision capabilities in recent years has elevated video analytics applications to new heights. To address the limited computing and bandwidth resources on edge devices, edge-cloud collaborative video analytics has emerged as a promising paradigm. However, most existing edge-cloud video analytics systems are designed for traditional deep learning models (e.g., image classification and object detection), where each model handles a specific task. In this paper, we introduce VaVLM, a novel edge-cloud collaborative video analytics system tailored for Vision-Language Models (VLMs), which can support multiple tasks using a single model. VaVLM aims to enhance the performance of VLM-powered video analytics systems in three key aspects. First, to reduce bandwidth consumption during video transmission, we propose a novel Region-of-Interest (RoI) generation mechanism based on the VLM’s understanding of the task and scene. Second, to lower inference costs, we design a task-oriented inference trigger that processes only a subset of video frames using an optimized inference logic. Third, to improve inference accuracy, the model is augmented with additional information from both the environment and auxiliary analytics models during the inference stage. Extensive experiments on real-world datasets demonstrate that VaVLM achieves an 80.3% reduction in bandwidth consumption and an 89.5% reduction in computational cost compared to baseline methods.

Energy consumption has emerged as a critical design constraint in deploying high-performance neural networks, especially on edge devices with limited power resources. In this paper, a comparative study is conducted for two prevalent deep learning paradigms—convolutional neural networks (CNNs), exemplified by ResNet18, and transformer-based large language models (LLMs), represented by GPT3-small, Llama-7B, and GPT3-175B. By analyzing how the scaling of memory energy versus computing energy affects the energy consumption of neural networks with different batch sizes (1, 4, 8, 16), it is shown that ResNet18 transitions from a memory energy-limited regime at low batch sizes to a computing energy-limited regime at higher batch sizes due to its extensive convolution operations. On the other hand, GPT-like models remain predominantly memory-bound, with large parameter tensors and frequent key–value (KV) cache lookups accounting for most of the total energy usage. Our results reveal that reducing weight-memory energy is particularly effective in transformer architectures, while improving multiply–accumulate (MAC) efficiency significantly benefits CNNs at higher workloads. We further highlight near-memory and in-memory computing approaches as promising strategies to lower data-transfer costs and enhance power efficiency in large-scale deployments. These findings offer actionable insights for architects and system designers aiming to optimize artificial intelligence (AI) performance under stringent energy budgets on battery-powered edge devices.

This viewpoint article first explores the ethical challenges associated with the future application of large language models (LLMs) in the context of medical education. These challenges include not only ethical concerns related to the development of LLMs, such as artificial intelligence (AI) hallucinations, information bias, privacy and data risks, and deficiencies in terms of transparency and interpretability but also issues concerning the application of LLMs, including deficiencies in emotional intelligence, educational inequities, problems with academic integrity, and questions of responsibility and copyright ownership. This paper then analyzes existing AI-related legal and ethical frameworks and highlights their limitations with regard to the application of LLMs in the context of medical education. To ensure that LLMs are integrated in a responsible and safe manner, the authors recommend the development of a unified ethical framework that is specifically tailored for LLMs in this field. This framework should be based on 8 fundamental principles: quality control and supervision mechanisms; privacy and data protection; transparency and interpretability; fairness and equal treatment; academic integrity and moral norms; accountability and traceability; protection and respect for intellectual property; and the promotion of educational research and innovation. The authors further discuss specific measures that can be taken to implement these principles, thereby laying a solid foundation for the development of a comprehensive and actionable ethical framework. Such a unified ethical framework based on these 8 fundamental principles can provide clear guidance and support for the application of LLMs in the context of medical education. This approach can help establish a balance between technological advancement and ethical safeguards, thereby ensuring that medical education can progress without compromising the principles of fairness, justice, or patient safety and establishing a more equitable, safer, and more efficient environment for medical education.