CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems

Large Language Models (LLMs) are being increasingly applied in various natural language processing tasks including safety-critical systems (e.g., medical diagnosis querying, and code generation for self-driving), where resilience to hardware transient faults is essential for guaranteed safety. Traditional Fault Injection (FI) approaches are time consuming due to a large number of repeated executions, which limits their scalability to fast evaluate large-scale resilience. To address these challenges, we propose LLM-IARE, a novel Input-Aware Resilience Estimation Model for LLMs under hardware transient faults. It takes advantage of twice static analysis and once dynamic execution to extract critical parameters to compute general resilience metrics such as Silent Data Corruption (SDC) rates. Fast static analysis can obtain the primary LLM parameters, while dynamic execution can provide input-sensitive attention profiling to characterize how input variations influence internal attention patterns dynamically. More importantly, our proposed LLM-IARE uses the obtained parameters for modeling at three levels (operation, module, and layer) so that the SDC rates of transient fault impacts on LLMs can be calculated quickly and accurately. Additionally, LLM-IARE is further extended to estimate the LLM application-level resilience metric, the cosine similarity reflecting the bit-upset induced semantic fault impacts on final output quality. Comprehensive experiments on six representative LLMs (for example, GPT-2, T5 and RoBERTa), and 30 BERT variants demonstrate that LLM-IARE achieves a fast and accurate LLMs resilience evaluation, with up to 7335 × (average 4500 ×) speedup and an average logarithmic relative error of 3.94% compared to advanced LLVM-based fault injection methods. We further extend the evaluation to four larger Qwen2.5 models (0.5B–7B), where LLM-IARE maintains stable accuracy with logarithmic relative errors between 1.96% and 4.21% (average 3.39%).

Now that we have so much information available, recommender systems are becoming more and more important in many different areas. Traditional methods are very popular because they work well, but they have problems with the cold-start problem and data sparsity. Large language models (LLMs) are a new breakthrough for recommender systems because they can understand and generate text in natural language. This study suggests a better recommendation method that uses both collaborative filtering and LLMs. This method combines the strengths of both methods, using collaborative filtering to model user preferences and LLMs to improve understanding of user and item information. This improves the accuracy and variety of recommendations. In this paper, we start by explaining the basic ideas behind collaborative filtering and LLMs. Then, we design a recommendation system that combines the two and test how well it works. The results show that the hybrid model based on collaborative filtering and LLM significantly improves accuracy, recall and user satisfaction, demonstrating its potential in complex recommendation scenarios.

Recommender systems (RecSys) have become critical tools for enhancing user engagement by delivering personalized content across diverse digital platforms. Recent advancements in large language models (LLMs) demonstrate significant potential for improving RecSys, primarily due to their exceptional generalization capabilities and sophisticated contextual understanding, which facilitate the generation of flexible and interpretable recommendations. However, the direct deployment of LLMs as primary recommendation policies presents notable challenges, including persistent latency issues stemming from frequent API calls and inherent model limitations such as hallucinations and biases. To address these issues, this paper proposes a novel offline reinforcement learning (RL) framework that leverages imitation learning from LLM-generated trajectories. Specifically, inverse reinforcement learning is employed to extract robust reward models from LLM demonstrations. This approach negates the need for LLM fine-tuning, thereby substantially reducing computational overhead. Simultaneously, the RL policy is guided by the cumulative rewards derived from these demonstrations, effectively transferring the semantic insights captured by the LLM. Comprehensive experiments conducted on two benchmark datasets validate the effectiveness of the proposed method, demonstrating superior performance when compared against state-of-the-art RL-based and in-context learning baselines. The code can be found at https://github.com/ArronDZhang/IL-Rec.

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1

Large Language Models (LLMs) have achieved remarkable success in recent years, owing to their impressive generalization capabilities and rich world knowledge. To capitalize on the potential of using LLMs as recommender systems, mainstream approaches typically focus on two paradigms. The first paradigm designs multi-domain or multi-task instruction data for generalizable recommendation, so as to align LLMs with general recommendation areas and deal with cold-start recommendation. The second paradigm focuses on enhancing domain-specific recommendation tasks, improving performance in warm recommendation scenarios. While most previous works treat these two paradigms separately, we argue that they have complementary advantages, and combining them can yield better results. In this paper, we propose a generalizable and efficient LLM-based recommendation framework RecCocktail. Our approach begins with fine-tuning a "base spirit" LoRA module using domain-general recommendation instruction data to align LLM with recommendation knowledge. Next, given users' behavior of a specific domain, we construct a domain-specific "ingredient" LoRA module. We then provide an entropy-guided adaptive merging method to mix the "base spirit" and the "ingredient" in the weight space. Please note that, RecCocktail combines the advantages of the existing two paradigms without introducing additional time or space overhead during the inference phase. Moreover, RecCocktail is efficient with plug and play, as the "base spirit" LoRA is trained only once, and any domain-specific "ingredient" can be efficiently mixed with only domain-specific fine-tuning. Extensive experiments on multiple datasets under both warm and cold-start recommendation scenarios validate the effectiveness and generality of the proposed RecCocktail.

Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support. This potential has been illustrated by the state-of-the-art performance obtained by LLMs in Medical Question Answering, with striking results such as passing marks in licensing medical exams. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations, written by medical doctors, of the correct and incorrect options in the exams. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs, with best results around 75 accuracy for English, still has large room for improvement, especially for languages other than English, for which accuracy drops 10 points. Therefore, despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. Data, code, and fine-tuned models will be made publicly available.11https://huggingface.co/datasets/HiTZ/MedExpQA.

Cross-category recommendation systems aim to suggest items to users from various categories, enhancing the user experience by broadening their exposure to diverse products or services. Large Language Models (LLMs) offer a promising avenue for improving the effectiveness of such systems. LLMs, trained on vast datasets, excel at understanding complex relationships between different categories of products by analysing patterns in user interactions, preferences, and item descriptions. By leveraging natural language understanding, these models can identify similarities and connections between disparate product categories, enabling more personalized and relevant cross-category recommendations. This approach goes beyond traditional recommendation algorithms that typically focus on a single category or rely heavily on collaborative filtering techniques. Instead, LLMs can process and integrate information from multiple domains, making it easier to suggest complementary products from different categories that users may not have initially considered. Additionally, LLMs can interpret contextual information, such as user reviews and sentiment analysis, to further refine recommendations based on the user's unique preferences and behavioural patterns. The potential of LLMs in cross-category recommendations opens new possibilities for e-commerce, entertainment, and other industries where personalized experiences are key to user engagement and satisfaction. As these models continue to evolve, they offer the promise of more intelligent and comprehensive recommendation systems that can better meet the needs of users by offering diverse and relevant suggestions across categories. These abstract outlines the key role of LLMs in advancing cross-category recommendation systems, highlighting their capacity to improve personalization and broaden the scope of recommendations.

Large Language Models (LLMs) have achieved significant successes in various research domains by learning the relationship between words. However, while these models are capable of making predictions and inferences based on the learned patterns, they lack logical reasoning abilities, which are crucial for solving problems in both theoretical and practical domains. In addition, traditional logic inference methods are effective in solving problems that are based on logic, but not suitable for general tasks such as recommendations. In response to these challenges, this paper introduces a Logical Large Language Model (L3M) that integrates the strengths of logical reasoning and large language models. The data in L3M is represented in logical expressions and the model uses logical constraints to learn the rules of basic logical operations such as And, Or, and Not. We conduct experiments on both theoretical tasks (solving logical equations) and practical tasks (recommender systems). The results of our theoretical experiments demonstrate that L3M is highly effective in solving logical expressions and variables. Additionally, L3M outperforms the state-of-the-art recommendation models in sequential recommendation tasks.

The rapid advancement of large language models (LLMs) has opened up promising opportunities for their downstream applications in question-answering (QA), such as ChatGPT, ChatGLM, etc. However, such LLMs do not perform very well in domain-specific QA tasks without fine-tuning. But directly fine-tuning LLMs on domain-specific corpus data may lead to catastrophic forgetting, causing the LLMs to lose their general language capability. To address this problem, we propose the Knowledge-Enhanced Fine-Tuning (KEFT) method, an unsupervised fine-tuning approach to enhance the knowledge capability of LLMs in domain-specific QA tasks while preserving their general language capability. KEFT leverages the inherent language comprehension of pre-trained LLMs to generate synthetic-QA datasets from domain-specific corpus data autonomously for fine-tuning, and adopts a Low-Rank Adaptation (LoRA) method to further alleviate over-fitting. Furthermore, to enhance the representation of domain-specific knowledge, we introduce a knowledge-enhanced fine-tuning loss function, which encourages the model to learn the knowledge-question connection, thereby generating natural and knowledgeable answers. Our evaluations across multiple domain-specific datasets demonstrate that KEFT surpasses state-of-the-art fine-tuning approaches, enhancing the performance of various LLMs in QA tasks in both English and Chinese languages.

Tabular data synthesis is crucial in machine learning, yet existing general methods-primarily based on statistical or deep learning models-are highly data-dependent and often fall short in recommender systems. This limitation arises from their difficulty in capturing complex distributions and understanding complicated feature relations from sparse and limited data, along with their inability to grasp semantic feature relations. Recently, Large Language Models (LLMs) have shown potential in generating synthetic data through few-shot learning and semantic understanding. However, they often suffer from inconsistent distribution and lack of diversity due to their inherent distribution disparity with the target dataset. To address these challenges and enhance tabular data synthesis for recommendation tasks, we propose a novel two-stage framework named SampleLLM to improve the quality of LLM-based tabular data synthesis for recommendations by ensuring better distribution alignment. In the first stage, SampleLLM employs LLMs with Chain-of-Thought prompts and diverse exemplars to generate data that closely aligns with the target dataset distribution, even when input samples are limited. The second stage uses an advanced feature attribution-based importance sampling method to refine feature relationships within the synthetic data, reducing any distribution biases introduced by the LLM. Experimental results on three recommendation datasets, two general datasets, and online deployment illustrate that SampleLLM significantly surpasses existing methods for recommendation tasks and holds promise for a broader range of tabular data scenarios.

Designing preference elicitation (PE) methodologies that can quickly ascertain a user’s top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions – requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to guide LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline’s MRR@10 of 0.17, despite relying on earlier and smaller LLMs.1

The increasing availability of Web APIs has amplified the complexity of mashup creation, where developers must identify compatible and functionally relevant APIs based on often ambiguous natural language descriptions. Traditional methods also fall short in capturing hierarchical semantic cues, modeling compatibility, and aligning with developer intent. Although large language models (LLMs) offer strong generalization capabilities, they remain unreliable in mashup recommendation due to hallucinated outputs, limited controllability, and token-length constraints when dealing with large-scale API repositories. To overcome these limitations, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LIMR</i>, an intent-aware mashup recommendation framework that combines LLM-augmented semantic reasoning with structured, multi-scale neural modeling. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LIMR</i> first prompts a LLM to extract high-level intent from user requirements, which serves as a global semantic signal. This intent is fused with low-level, multi-scale features extracted by a convolutional encoder, which are designed to capture fine-grained lexical/phrasal patterns at different granularities and provide precise semantic grounding for API matching. These heterogeneous representations are further contextually refined through a Transformer-based interaction module. To handle nonlinear semantic dependencies and compositional complexity, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LIMR</i> integrates a Kolmogorov-Arnold Network (KAN) with learnable activation functions, enhancing the model's capacity to capture intricate feature interactions. The entire framework is optimized via LLM, incorporating auxiliary objectives such as mashup category prediction and API quality estimation to guide generalization and reduce overfitting. Comprehensive experiments on the ProgrammableWeb and APIBench datasets show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LIMR</i> significantly outperforms state-of-the-art baselines, which the ranking-oriented metrics, including NDCG and mAP, achieves improvements of 17.1%–34.2% over the strongest competitors. These results confirm the effectiveness of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LIMR</i>'s hybrid design in delivering precise, robust, and intent-aware mashup API recommendations, especially in scenarios where LLMs alone fail to meet accuracy and scalability demands.

The mainstream approach of GNN-based recommendation aggregates high-order ID information associated with the node in the user-item graph. The aggregation pattern using ID as signal has two disadvantages: lack of textual semantics and the impact of interaction noise. These disadvantages pose a threat to effectively learn user preferences, especially in capturing intricate user-item semantic relationships. Although large language models (LLMs) allow the integration of rich textual information into recommenders and have had groundbreaking applications in recommender systems, current works need to bridge the gap between different representation spaces. This is because LLM-based methods align the representations of GNN-based models only by using text embedding of LLM, leading to unsatisfactory results. To address this challenge, we propose a denoising alignment framework with LLMs for GNN-based recommenders (DALR) , which aims to align structural representation with textual representation and mitigate the effects of noise. Specifically, we propose a modeling framework that integrates the representation of graph structure with textual information from LLMs to capture intricate user-item interactions. We also suggest an alignment paradigm to enhance representation performance by aligning semantic signals from LLMs and structural features from GNN models. Additionally, we introduce a contrastive learning scheme to relieve the impact of noise and improve model performance. Extensive experiments on public datasets demonstrate that our model consistently outperforms the state-of-the-art methods. DALR achieves improvements ranging from 2.82% to 12.20% in Recall@5 and from 1.04% to 3.48% in NDCG@5 compared to the strongest baseline model, using the Steam dataset as an example.

Large language models (LLMs) have been widely used in real-world applications due to their ability to decipher and make sense of perturbed information. However, the performance of LLMs that suffered from input perturbations has not been fully analyzed. In this paper, we propose a self-correction reflection method inspired by human cognition to improve robustness and mitigate the impact of perturbations on LLM performance. Firstly, we analyze the vulnerabilities of tokenization in LLMs through empirical demonstrations, comparative analysis with human cognition, and theoretical investigation of perturbation susceptibility. Secondly, we imitate the correction of input information in the human brain, which is enhanced by the consistency principle, aiming to enable the model to correct the perturbation and reduce the impact on the final response. Finally, we conduct experiments to validate the method’s efficacy, demonstrating improved performance across multiple models and datasets. We introduce a new evaluation metric, Model-Specific Cosine Similarity (MSCS), to quantify how well a specific model understands perturbation text, providing a more comprehensive evaluation of different LLM architectures. Particularly for different types of perturbations, after applying self-correction reflection, the average MSCS improvement across all models reaches 10.88%, while the average ACC improvement is 10.22%. The study also underscores the need for more innovative tokenization techniques and architectural design to achieve human-like cognitive robustness. This study could pave the way for more reliable, adaptable, and intelligent language models capable of thriving in practical applications.

Automated Program Repair (APR) aims to automatically generate patches for rectifying software bugs. Recent strides in Large Language Models (LLM), such as ChatGPT, have yielded encouraging outcomes in APR, especially within the conversation-driven APR framework. Nevertheless, the efficacy of conversation-driven APR is contingent on the quality of the feedback information. In this article, we propose ContrastRepair , a novel conversation-based APR approach that augments conversation-driven APR by providing LLMs with contrastive test pairs. A test pair consists of a failing test and a passing test, which offer contrastive feedback to the LLM. Our key insight is to minimize the difference between the generated passing test and the given failing test, which can better isolate the root causes of bugs. By providing such informative feedback, ContrastRepair enables the LLM to produce effective bug fixes. The implementation of ContrastRepair is based on the state-of-the-art LLM, ChatGPT, and it iteratively interacts with ChatGPT until plausible patches are generated. We evaluate ContrastRepair on multiple benchmark datasets, including Defects4J, QuixBugs, and HumanEval-Java. The results demonstrate that ContrastRepair significantly outperforms existing methods, achieving a new state-of-the-art in program repair. For instance, among Defects4J 1.2 and 2.0, ContrastRepair correctly repairs 143 out of all 337 bug cases, while the best-performing baseline fixes 124 bugs.