Cross Category Recommendations Using LLMs

Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

Designing preference elicitation (PE) methodologies that can quickly ascertain a user’s top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions – requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to guide LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline’s MRR@10 of 0.17, despite relying on earlier and smaller LLMs.1

Aim/Purpose: This paper focuses on app review analysis techniques, driven by the rapid advancement of the mobile app market and NLP techniques in optimizing mobile app user experiences. Background: Owing to technological advancements, app review analysis has rapidly evolved. This study examines both conventional and emerging techniques, including current advancements such as large language models (LLMs) in app review analysis. It provides an overview of the various methods used across different categories of app review analysis, comparing effective strategies for identifying user concerns and enhancing app functionality. Methodology: A systematic review was utilized based on two major standard guidelines, PRISMA and Kitchenham’s guidelines, for the period of 2014 to 2024. After defining the review protocol, papers were identified through keyword-based searches on six major online databases: Scopus, Web of Science, IEEE Xplore, ACM Digital Library, Science Direct, and Springer. Following screening and excluding papers based on defined quality criteria, 53 papers were considered for this study. The use of PRISMA ensures a transparent and reproducible review process, while Kitchenham’s guidelines provide a structured and rigorous approach for evaluating and synthesizing the literature. Contribution: This review study aims to evaluate the current state of knowledge on app review analysis techniques to improve mobile app user experiences. This study categorized the existing state-of-the-art papers into eight different categories, such as sentiment analysis, review classification, summarization, and prioritization, and examined challenges related to app review analysis. Furthermore, the study emphasizes the potential of LLMs for optimizing and automating app review analysis and provides future directions to address gaps in user-centric app development. Findings: Among the eight main categories defined in app review analysis, sentiment analysis is the most prevalent, followed by review classification and information extraction. Most studies use a combination of these categories to achieve a comprehensive goal. Prioritization techniques such as risk matrices, thumbs-up count-based approach, and anomaly detection are widely used to identify emerging issues. Extracting meaningful information and evaluating the proposed approach are the most common challenges identified. Novel LLMs, like Chat-GPT, significantly enhance review analysis by automating the process, improving feature extraction, and enabling context-aware review classification. Recommendations for Practitioners: The combination of conventional approaches and novel LLM-based methods can enhance both the efficiency and accuracy in identifying and addressing critical issues raised through mobile app user reviews. It effectively prioritizes user concerns by leveraging the strengths of both traditional preprocessing techniques and advanced LLMs. Recommendation for Researchers: Researchers are encouraged to explore the integration of emerging technologies like LLMs to enhance the of app review analysis, particularly in feature-specific sentiment analysis. Impact on Society: The results of this study contribute to enhancing the mobile app user experience through effective app review analysis, which improves user satisfaction and supports user-centered app development. This ultimately leads to a better mobile app ecosystem, benefiting both users and developers. Future Research: In the future, this research can be extended in multiple directions. Researchers can address the existing research gaps that LLMs have yet to address, particularly in prioritizing user concerns. Additionally, there is potential for further research on tool implementations focusing on identifying persistent issues through time series analysis by considering the app version and date of the app reviews. Moreover, there is a need to develop comprehensive frameworks that are more generalizable across different apps and categories, with a focus on identifying user concerns related to specific features.

This project focuses on leveraging generative AI to analyse user reviews from the Play Store for diverse applications. By utilizing advanced large language models (LLMs), the system processes extensive user feedback to identify key trends, sentiments, and actionable insights. The AI analyses reviews to categorize overall user sentiment (positive, negative, or neutral), highlight recurring issues, and identify popular feature requests. The generative AI’s capabilities enable it to provide nuanced suggestions to developers, such as improving app functionality, addressing common complaints, and implementing features aligned with user preferences. Additionally, the system can generate concise summaries of user reactions, offering developers a clear understanding of their app's strengths and areas needing improvement. This data-driven approach enhances app development by prioritizing updates based on real user needs, improving user satisfaction, and fostering higher ratings on the Play Store. The integration of generative AI streamlines the review analysis process, ensuring actionable recommendations for sustained app success. Key Words: Generative AI, Large Language Models (LLMs), sentiment analysis, App improvement, review analysis 1.INTRODUCTION This project utilizes generative AI to analyze Play Store reviews across various applications, providing developers with actionable insights. By leveraging large language models (LLMs), the system processes user feedback to identify sentiment trends, highlight common issues, and suggest improvements. This approach enables developers to understand user reactions effectively and prioritize updates, ultimately enhancing app quality, user satisfaction, and ratings. The integration of AI streamlines review analysis, ensuring data-driven decisions for better app performance.

Burnout has become a significant concern in high-stress industries, impacting mental health and productivity. Traditional burnout management solutions lack real-time adaptability and personalized intervention strategies. This research explores an AI-driven burnout management mechanism leveraging Generative AI, MongoDB Atlas, Python, and Large Language Models (LLMs) to provide real-time detection, personalized coping strategies, and predictive analytics. The system collects and stores user responses, behavioral patterns, and sentiment data in MongoDB Atlas, while LLMs analyze and generate tailored burnout mitigation recommendations. Experimental results demonstrate the effectiveness of the proposed approach in early burnout detection, personalized intervention, and user engagement, showing improvements in stress reduction and work-life balance. The proof of concept for AI-powered burnout detection and intervention demonstrates the technical feasibility of using Large Language Models (LLMs), Generative AI, MongoDB Atlas, and Python-based automation for real-time mental health support. The system successfully classifies burnout severity using LLMs trained on sentiment analysis, achieving an accuracy of 91.2%, which significantly outperforms traditional rule-based sentiment models. The use of MongoDB Atlas for scalable data storage ensures real-time burnout tracking, while FastAPI-based backend automation allows for seamless integration across platforms. Furthermore, Generative AI provides personalized stress interventions, dynamically adapting to user feedback and improving engagement. These technical validations highlight how AI-driven burnout management can be scalable, real-time, and enterprise-ready, paving the way for AI-powered diagnostics in mental health applications across various industries.

Now that we have so much information available, recommender systems are becoming more and more important in many different areas. Traditional methods are very popular because they work well, but they have problems with the cold-start problem and data sparsity. Large language models (LLMs) are a new breakthrough for recommender systems because they can understand and generate text in natural language. This study suggests a better recommendation method that uses both collaborative filtering and LLMs. This method combines the strengths of both methods, using collaborative filtering to model user preferences and LLMs to improve understanding of user and item information. This improves the accuracy and variety of recommendations. In this paper, we start by explaining the basic ideas behind collaborative filtering and LLMs. Then, we design a recommendation system that combines the two and test how well it works. The results show that the hybrid model based on collaborative filtering and LLM significantly improves accuracy, recall and user satisfaction, demonstrating its potential in complex recommendation scenarios.

Large language models (LLMs) offer significant promise as a knowledge source for task learning. Prompt engineering has been shown to be effective for eliciting knowledge from an LLM, but alone it is insufficient for acquiring relevant, situationally grounded knowledge for an embodied agent learning novel tasks. We describe a cognitive-agent approach, STARS, that extends and complements prompt engineering, mitigating its limitations and thus enabling an agent to acquire new task knowledge matched to its native language capabilities, embodiment, environment, and user preferences. The STARS approach is to increase the response space of LLMs and deploy general strategies, embedded within the autonomous agent, to evaluate, repair, and select among candidate responses produced by the LLM. We describe the approach and experiments that show how an agent, by retrieving and evaluating a breadth of responses from the LLM, can achieve 77-94% task completion in one-shot learning without user oversight. The approach achieves 100% task completion when human oversight (such as an indication of preference) is provided. Further, the type of oversight largely shifts from explicit, natural language instruction to simple confirmation/discomfirmation of high-quality responses that have been vetted by the agent before presentation to a user.

Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements. We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties. Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety. Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).

This paper proposes the utilization of large language models as recommendation systems for museum visitors. Since the aforementioned models lack the notion of context, they cannot work with temporal information that is often present in recommendations for cultural environments (e.g., special exhibitions or events). In this respect, the current work aims to enhance the capabilities of large language models through a fine-tuning process that incorporates contextual information and user instructions. The resulting models are expected to be capable of providing personalized recommendations that are aligned with user preferences and desires. More specifically, Generative Pre-trained Transformer 4, a knowledge-based large language model is fine-tuned and turned into a context-aware recommendation system, adapting its suggestions based on user input and specific contextual factors such as location, time of visit, and other relevant parameters. The effectiveness of the proposed approach is evaluated through certain user studies, which ensure an improved user experience and engagement within the museum environment.

Sentiment analysis is a computational technique employed to extract and interpret subjective information from textual data. It involves the identification and classification of sentiments, opinions, and emotions expressed within the text. By analyzing linguistic cues, such as word choice, syntax, and sentiment lexicons, sentiment analysis can discern a range of emotions, from positive to negative, as well as more nuanced sentiments, such as anger, joy, or surprise. This powerful tool has the potential to unlock valuable insights from vast amounts of unstructured text data, which enables informed decision-making and effective communication in various domains, including education. Recent advances in sentiment analysis have leveraged the power of deep neural networks, particularly general-purpose Large Language Models (LLMs) trained on extensive labeled datasets. However, real-world applications frequently encounter challenges related to the availability of large, high-quality labeled data and the computational resources necessary for training such models. This research addresses these challenges by investigating effective strategies for utilizing LLMs in scenarios with limited data and computational resources. Specifically, this study explores three techniques: zero-shot learning, <i>N</i>-shot learning and fine-tuning. By evaluating these methods, this research aims to demonstrate the feasibility of employing general-purpose LLMs for sentiment analysis within educational contexts even when access to computational resources and labeled data is limited. The findings of this study reveal that different adaptation methods lead to significantly different LLM performance outcomes.

Since interpretable feature-engineered pipelines, natural language processing (NLP) has developed into deep neural and transformer-based architectures, and finally, large language models (LLMs) that can generalize their results across tasks. This development has significantly enhanced the subjective language comprehension, such as sentiment, emotion, sarcasm, humor, stance, metaphor, intent and aesthetic judgments, and increased the requirements of explainability in high-stakes areas. Already trained transformers (e.g., encoder-only and decoder-only versions) and LLMs like BERT and GPT have established powerful baselines on text classification and zero/few-shot subjectivity modeling, but the opaqueness of these models drives language-based rationalization (extractive/abstractive natural-language justifications) and feature-attribution algorithms (e.g., LIME, Integrated Gradients, SHAP). The four strands brought together by this survey include: (i) sentiment-analysis methods and datasets (e.g., IMDb, Sentiment140, Twitter Airline, SemEval), (ii) transformer-era text classification, (iii) LLMs on subjective language, and (iv) rationalization of explainable NLP. We have provided a unified taxonomy of subjective tasks and have analyzed model families, starting with classical ML up to transformers/LLMs; have collected major data sets and benchmarks; and have systematized explainability methods, in particular rationalization, and their evaluations and classify open problems dataset bias and annotation ambiguity, constraints of faithfulness and explanatory comprehensibility, evaluation bias, compute cost, and ethical risks. The article will facilitate the formation of a coherent foundation of explainable, credible subjective NLP based on the functions of transformer/LLM and analyze feature extraction techniques methodically based on LLM. We have trained a Machine Learning classifier using 70% of the training data and 30% of the testing data. Based on our results, we find that the proposed ML based technique gives enhanced performance, with an improved accuracy of 99% in the UCI-ML reviews dataset and 96% in the Twitter Kaggle dataset. This study underscores the paramount significance of feature extraction in sentiment analysis, endowing pragmatic insights to elevate model performance and steer future research pursuits.

Large Language Models (LLMs) empower recommendation systems through their advanced reasoning and planning capabilities. However, the dynamic nature of user interests and content poses a significant challenge: While initial fine-tuning aligns LLMs with domain knowledge and user preferences, it fails to capture such real-time changes, necessitating robust update mechanisms. This paper investigates strategies for updating LLM-powered recommenders, focusing on the trade-offs between ongoing fine-tuning and Retrieval-Augmented Generation (RAG). Using an LLM-powered user interest exploration system as a case study, we perform a comparative analysis of these methods across dimensions like cost, agility, and knowledge incorporation. We propose a hybrid update strategy that leverages the long-term knowledge adaptation of periodic fine-tuning with the agility of low-cost RAG. We demonstrate through live A/B experiments on a billion-user platform that this hybrid approach yields statistically significant improvements in user satisfaction, offering a practical and cost-effective framework for maintaining high-quality LLM-powered recommender systems.

Recommender systems (RecSys) have become critical tools for enhancing user engagement by delivering personalized content across diverse digital platforms. Recent advancements in large language models (LLMs) demonstrate significant potential for improving RecSys, primarily due to their exceptional generalization capabilities and sophisticated contextual understanding, which facilitate the generation of flexible and interpretable recommendations. However, the direct deployment of LLMs as primary recommendation policies presents notable challenges, including persistent latency issues stemming from frequent API calls and inherent model limitations such as hallucinations and biases. To address these issues, this paper proposes a novel offline reinforcement learning (RL) framework that leverages imitation learning from LLM-generated trajectories. Specifically, inverse reinforcement learning is employed to extract robust reward models from LLM demonstrations. This approach negates the need for LLM fine-tuning, thereby substantially reducing computational overhead. Simultaneously, the RL policy is guided by the cumulative rewards derived from these demonstrations, effectively transferring the semantic insights captured by the LLM. Comprehensive experiments conducted on two benchmark datasets validate the effectiveness of the proposed method, demonstrating superior performance when compared against state-of-the-art RL-based and in-context learning baselines. The code can be found at https://github.com/ArronDZhang/IL-Rec.

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

As large language models (LLMs) achieve remarkable success in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and are being actively explored currently. In this article, we focus on adapting and enhancing large language models for recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation realms, i.e., LLMs fail to effectively extract useful information from a pure textual context of long user behavior sequence, even if the length of context is well below the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely, R etrieval- e nhanced L arge La nguage models Plus (ReLLaX), which provides full-stack optimization from three perspectives, i.e., data, prompt, and parameter. For data-level enhancement, we design semantic user behavior retrieval (SUBR) to reduce the heterogeneity of the behavior sequence, thus lowering the difficulty for LLMs to extract the essential information from user behavior sequences. Although SUBR can improve the data quality, further increase in the sequence length will still raise its heterogeneity to a level where LLMs can no longer comprehend it. Hence, we further propose to perform prompt-level and parameter-level enhancement, with the integration of conventional recommendation models (CRMs). As for prompt-level enhancement, we apply soft prompt augmentation (SPA) to explicitly inject collaborative knowledge from CRMs into the prompt. The item representations of LLMs are thus more aligned with recommendation, helping LLMs better explore the item relationships in the sequence and facilitating comprehension. Finally, for parameter-level enhancement, we propose component fully-interactive LoRA (CFLoRA). By enabling sufficient interaction between the LoRA atom components, the expressive ability of LoRA is extended, making the parameters effectively capture more sequence information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e., from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments are conducted on three real-world public datasets to demonstrate the superiority of ReLLaX compared with existing baseline models, as well as its capability to alleviate lifelong sequential behavior incomprehension. Our code is available. 1