Efficient Sequential Recommendation for Long Term User Interest Via Personalization

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

Recommending a limited number of Point-of-Interests (POIs) a user will visit next has become increasingly important to both users and POI holders for Location-Based Social Networks (LBSNs). However, POI recommendation is a challenging task since complex sequential patterns and rich contexts are contained in extremely sparse user check-in data. Recent studies show that embedding techniques effectively incorporate POI contextual information to alleviate the data sparsity issue, and Recurrent Neural Network (RNN) has been successfully employed for sequential prediction. Nevertheless, existing POI recommendation approaches are still limited in capturing user personalized preference due to separate embedding learning or network modeling. To this end, we propose a novel unified spatio-temporal neural network framework, named PPR, which leverages users’ check-in records and social ties to recommend personalized POIs for querying users by joint embedding and sequential modeling. Specifically, PPR first learns user and POI representations by joint modeling User-POI relation, sequential patterns, geographical influence, and social ties in a heterogeneous graph and then models user personalized sequential patterns using the designed spatio-temporal neural network based on LSTM model for the personalized POI recommendation. Furthermore, we extend PPR to an end-to-end recommendation model by jointly learning node representations and modeling user personalized sequential preference. Extensive experiments on three real-world datasets demonstrate that our model significantly outperforms state-of-the-art baselines for successive POI recommendation in terms of Accuracy, Precision, Recall and NDCG. The source code is available at: https://www.anonymous.4open.science/r/DSE-1BEC.

Sequential recommendation has been widely studied in the recommendation domain since it can capture users’ temporal preferences and provide more accurate and timely recommendations. To address user privacy concerns, the combination of federated learning and sequential recommender systems (FedSeqRec) has gained growing attention. Unfortunately, the performance of FedSeqRec is still unsatisfactory because the models used in FedSeqRec have to be lightweight to accommodate communication bandwidth and clients’ on-device computational resource constraints. Recently, large language models (LLMs) have exhibited strong transferable and generalized language understanding abilities and therefore, in the NLP area, many downstream tasks now utilize LLMs as a service to achieve superior performance without constructing complex models. Inspired by this successful practice, we propose a generic FedSeqRec framework, FELLAS, which aims to enhance FedSeqRec by utilizing LLMs as an external service. Specifically, FELLAS employs an LLM server to provide both item-level and sequence-level representation assistance. The item-level representation service is queried by the central server to enrich the original ID-based item embedding with textual information, while the sequence-level representation service is accessed by each client. However, invoking the sequence-level representation service requires clients to send sequences to the external LLM server. To safeguard privacy, we implement \(d_{\mathcal{X}}\) -privacy satisfied sequence perturbation, which protects clients’ sensitive data with guarantees. Additionally, a contrastive learning-based method is designed to transfer knowledge from the noisy sequence representation to clients’ sequential recommendation models. Furthermore, to empirically validate the privacy protection capability of FELLAS, we propose two interacted item inference attacks, considering the threats posed by the LLM server and the central server acting as curious-but-honest adversaries in cooperation. Extensive experiments conducted on three datasets with two widely used sequential recommendation models demonstrate the effectiveness and privacy-preserving capability of FELLAS.

Sequential recommendation models aim to predict the interested items to a user based on his historical behaviors. To train sequential recommenders, implicit feedback data is widely adopted since it is easier to obtain than explicit feedback data. In the setting of implicit feedback, a user’s historical behaviors can be characterized as a chronologically ordered sequence of interacted items. From a perspective of machine learning, the historical interaction sequence and the recommended items can be considered as context and label , respectively, which are usually in one-hot representations in the recommendation models. However, due to the discrete nature, one-hot representations are hard to sufficiently reflect the underlying user preference, and might also contain noise from implicit feedback that will mislead the model training. To solve these issues, we propose a general optimization framework, Multi-View Smoothness (MVS), to enhance the smoothness of sequential recommendation models in both data representations and model learning. Specifically, with the help of a complementary model, we smooth and enrich the one-hot representations of contexts and labels to better depict the underlying user preference (i.e., context smoothness and label smoothness), and devise a model regularization strategy to enforce the neighborhood smoothness of the model itself (i.e., model smoothness). Based on these strategies, we design three regularizers to constrain and improve the training of sequential recommendation models. Extensive experiments on five datasets show that our approach is able to improve the performance of various base models consistently and outperform other regularization training methods.

Sequential recommendation has become increasingly essential in various online services. It aims to model the dynamic preferences of users from their historical interactions and predict their next items. The accumulated user behavior records on real systems could be very long. This rich data brings opportunities to track actual interests of users. Prior efforts mainly focus on making recommendations based on relatively recent behaviors. However, the overall sequential data may not be effectively utilized, as early interactions might affect users' current choices. Also, it has become intolerable to scan the entire behavior sequence when performing inference for each user, since real-world system requires short response time. To bridge the gap, we propose a novel long sequential recommendation model, called Dynamic Memory-based Attention Network (DMAN). It segments the overall long behavior sequence into a series of sub-sequences, then trains the model and maintains a set of memory blocks to preserve long-term interests of users. To improve memory fidelity, DMAN dynamically abstracts each user's long-term interest into its own memory blocks by minimizing an auxiliary reconstruction loss. Based on the dynamic memory, the user's short-term and long-term interests can be explicitly extracted and combined for efficient joint recommendation. Empirical results over four benchmark datasets demonstrate the superiority of our model in capturing long-term dependency over various state-of-the-art sequential models.

Timeliness and contextual accuracy of recommendations are increasingly important when delivering contemporary digital marketing experiences. Conventional recommender systems (RS) suggest relevant but time-invariant items to users by accounting for their past purchases. These recommendations only map to customers’ general preferences rather than a customer’s specific needs immediately preceding a purchase. In contrast, RSs that consider the order of transactions, purchases, or experiences to measure evolving preferences can offer more salient and effective recommendations to customers: Sequential RSs not only benefit from a better behavioral understanding of a user’s current needs but also provide better predictive power. In this paper, we demonstrate and rank the effectiveness of a sequential recommendation system by utilizing a production dataset of over 2.7 million credit card transactions for 46K cardholders. The methodfirst employs an autoencoder on raw transaction data and submits observed transaction encodings to a GRU-based sequential model. The sequential model produces a MAP@1 metric of 47% on the out-of-sample test set, in line with existing research. We also discuss implications for embedding real-timepredictions using the sequential RS into Nexus, a scalable, lowlatency, event-based digital experience architecture.

The sequential recommendation system predicts user's future operations based on their historical interaction information and achieves good performance in recent work. However, when applying to the task of recommendation in software crowdsourcing platform, the accuracy of the previous recommendation models is significantly reduced because of the sparse interactive data and dynamic item list in the platform. The Generative Ranking based Sequential Recommendation Model (GRS) is proposed to solve the problems mentioned above. The generative layer is introduced into a translation-based recommendation model to prevent overfitting problem. By generating latent vector in feature space, the interpolation between encoded points is highly reduced and the model is adapted to achieve better performance by embedding auxiliary features into the model. The efficiency and feasibility of the model is validated by the experiment in different datasets extracted from crowdsourcing platforms.

IPSRM: An intent perceived sequential recommendation model

With the rapid development of recommendation models and device computing power, device-based recommendation has become an important research area due to its better real-time performance and privacy protection. Previously, Transformer-based sequential recommendation models have been widely applied in this field because they outperform Recurrent Neural Network (RNN)-based recommendation models in terms of performance. However, as the length of interaction sequences increases, Transformer-based models introduce significantly more space and computational overhead compared to RNN-based models, posing challenges for device-based recommendation. To balance real-time performance and high performance on devices, we propose Device-Cloud Collaborative Correction Framework for On-Device Recommendation (CoCorrRec). CoCorrRec uses a self-correction network (SCN) to correct parameters with extremely low time cost. By updating model parameters during testing based on the input token, it achieves performance comparable to current optimal but more complex Transformer-based models. Furthermore, to prevent SCN from overfitting, we design a global correction network (GCN) that processes hidden states uploaded from devices and provides a global correction solution. Extensive experiments on multiple datasets show that CoCorrRec outperforms existing Transformer-based and RNN-based device recommendation models in terms of performance, with fewer parameters and lower FLOPs, thereby achieving a balance between real-time performance and high efficiency. Code is available at https: //github.com/Yuzt-zju/CoCorrRec.

Top-$N$ sequential recommendation models each user as a sequence of items interacted in the past and aims to predict top-$N$ ranked items that a user will likely interact in a `near future'. The order of interaction implies that sequential patterns play an important role where more recent items in a sequence have a larger impact on the next item. In this paper, we propose a Convolutional Sequence Embedding Recommendation Model (\emph{Caser}) as a solution to address this requirement. The idea is to embed a sequence of recent items into an `image' in the time and latent spaces and learn sequential patterns as local features of the image using convolutional filters. This approach provides a unified and flexible network structure for capturing both general preferences and sequential patterns. The experiments on public datasets demonstrated that Caser consistently outperforms state-of-the-art sequential recommendation methods on a variety of common evaluation metrics.

ObjectiveAmid electronic health records, laboratory tests, and other technology, office-based patient and provider communication is still the heart of primary medical care. Patients typically present multiple complaints, requiring physicians to decide how to balance competing demands. How this time is allocated has implications for patient satisfaction, payments, and quality of care. We investigate the effectiveness of machine learning methods for automated annotation of medical topics in patient-provider dialog transcripts.Materials and MethodsWe used dialog transcripts from 279 primary care visits to predict talk-turn topic labels. Different machine learning models were trained to operate on single or multiple local talk-turns (logistic classifiers, support vector machines, gated recurrent units) as well as sequential models that integrate information across talk-turn sequences (conditional random fields, hidden Markov models, and hierarchical gated recurrent units).ResultsEvaluation was performed using cross-validation to measure 1) classification accuracy for talk-turns and 2) precision, recall, and F1 scores at the visit level. Experimental results showed that sequential models had higher classification accuracy at the talk-turn level and higher precision at the visit level. Independent models had higher recall scores at the visit level compared with sequential models.ConclusionsIncorporating sequential information across talk-turns improves the accuracy of topic prediction in patient-provider dialog by smoothing out noisy information from talk-turns. Although the results are promising, more advanced prediction techniques and larger labeled datasets will likely be required to achieve prediction performance appropriate for real-world clinical applications.

Analyzing sequences of interactions between users and items, sequential recommendation models can learn user intent and make predictions about the next item. Next to item interactions, most systems also have interactions with what we call non-item pages: these pages are not related to specific items but still can provide insights into the user’s interests, as, for example, navigation pages. We therefore propose a general way to include these non-item pages in sequential recommendation models to enhance next-item prediction. First, we demonstrate the influence of non-item pages on following interactions using the hypotheses testing framework HypTrails and propose methods for representing non-item pages in sequential recommendation models. Subsequently, we adapt popular sequential recommender models to integrate non-item pages and investigate their performance with different item representation strategies as well as their ability to handle noisy data. To show the general capabilities of the models to integrate non-item pages, we create a synthetic dataset for a controlled setting and then evaluate the improvements from including non-item pages on two real-world datasets. Our results show that non-item pages are a valuable source of information, and incorporating them in sequential recommendation models increases the performance of next-item prediction across all analyzed model architectures.

Federated learning (FL) is gradually gaining traction as the de facto standard for distributed recommendation model training that takes advantage of on-device user data while reducing server costs. However, the computation resources of user devices in FL are usually much more limited compared to servers in a datacenter, which hinders the application of some advanced recommendation models (e.g., Transformer-based models) in FL. In addition, models with better recommendation performance tend to have more parameters, which increases the cost of communication between servers and user devices. Therefore, it is difficult for existing federated recommendation methods to achieve a good trade-off between recommendation accuracy and computation and communication costs. As a response, we propose a novel federated recommendation framework for efficient recommendations. First, we propose an all-MLP model by replacing the self-attention sublayer in a Transformer encoder with a Fourier sublayer, in which the noise information in the user interaction data is effectively attenuated using Fast Fourier Transform and learnable filters. Second, we adopt an adaptive model pruning technique in the FL framework, which can significantly reduce the model size without affecting the recommendation performance. Extensive experiments on four real-world datasets demonstrate that our method outperforms existing federated recommendation methods and strikes a good trade-off between recommendation performance and model size.KeywordsRecommender systemSequential recommendationFederated recommendationFederated learningAll-MLP model

As technology continues to advance, it becomes increasingly integrated into daily life facilitating complex tasks across a range of environments. While some applications such as smartphones and smartwatches are less critical, others like healthcare devices and autonomous vehicles demand bug-free performance to prevent financial loss or harm. Traditionally, simulation-based testing and formal verification played a major role in ensuring a bug-free device. However, the simulation of bigger systems is limited to a definite number of scenarios on the Design under Verification (DUV). Hence, it is unable to explore all possible inputs that can occur. Formal verification, on the other hand, offers a higher level of assurance through mathematical proofs but is both time-consuming and suffers from scalability issues, especially as designs grow in complexity. Recently, Large Language Models (LLMs) have shown promise in tasks previously limited to human expertise. Their natural language processing capabilities can assist in handling extensive specifications and source code, particularly in debugging hardware descriptions and analyzing security and functionality. The utilization of Retrieval Augmented Generation (RAG) has further enhanced LLMs by incorporating large specification or source code bases, thereby improving their bug-identification and correction capabilities. While recent advancements in LLMs, particularly with RAG, have yielded promising results in bug identification and correction for a small class of hardware bugs, significant gaps remain in their full potential for systematically addressing a wide range of hardware bugs. For instance, existing LLM methodologies struggle to detect bugs involving incorrect constant values, i.e., the use of wrong constants in source code. This limitation underscores the need for further exploration in utilizing LLMs to fully optimize the verification process. To bridge this gap, we propose a 3-phased 4-stage LLM-assisted systematic bug closure methodology that focuses on functional bugs in Verilog HDL rather than structural or syntactic issues. Our approach extracts functional properties of the DUV and systematically breaks down complex expressions into smaller sub-expressions to facilitate bug detection and correction. By employing RAG, the LLM is guided using the functional specifications and source code to identify and correct bugs. If the initial guidance through RAG is insufficient, our methodology initiates an iterative bug closure process. This includes incorporating more extensive information from the specifications, fetching additional lines of code for bug localization, and breaking down complex Verilog HDL expressions. In our comprehensive evaluation, we assess the LLM’s capabilities using 9 different categories of bugs. As benchmarks, we use 5 OpenTitan Intellectual Property (IP) cores to demonstrate the scalability and effectiveness of our bug closure methodology where ≈ 60% of the bugs were corrected. Specifically, we evaluate OpenAI’s GPT-4 in its ability to identify and correct functional bugs in Verilog HDL code.

Geospatial data is ubiquitous in today’s world, playing a crucial role in various aspects of modern life. As businesses increasingly establish and expand their online presence, they leverage location-based insights to better understand customer behavior and optimize services. Simultaneously, the proliferation of GPS-enabled devices enables users to continuously share their locations and activities online, contributing to an ever-growing pool of geospatial information. Social networks and web services, in particular, heavily rely on this data, integrating it into their platforms to provide more personalized and relevant experiences. This surge in geospatial data generation necessitates the development of novel algorithms to mine meaningful patterns and enhancing user experiences by delivering accurate information at the right place and time. This dissertation presents a pipeline of novel algorithms, spanning from the integration of geospatial data, to its effective application in real-world use cases. Specifically it introduces (1) an algorithm for deduplicating and consolidating geospatial data from multiple sources, a key step in creating a comprehensive geospatial database; it studies (2) the problem of Geospatial Knowledge Graph (GeoKG) construction, and presents an algorithm to automatically mine fine-grained spatial relationships between entities from a database, within an open-world setting; it presents (3) a framework for self-supervised pre-training of geospatial foundation models, capable of generating task-agnostic representations that are readily applicable to a wide range of downstream tasks. Finally, (4) this thesis explores the fine-tuning of pre-trained large language models (LLMs) on city-specific data, to build urban virtual assistants, capable of delivering accurate geospatial recommendations in a conversational manner while minimizing hallucinations. The study on geospatial Entity Resolution (ER) proposes a state-of-the-art approach to link records from different sources, that refer to the same real-world geospatial entity. This is a crucial step for deduplicating merged data. The algorithm, called Geo-ER, leverages textual, spatial, and contextual information, making a step forward towards automatic geospatial data integration. A novel neighborhood embedding component is designed, using Graph Attention Networks, to produce a more contextualized representation of an entity. Extensive experiments, on eight real-world datasets, are performed to demonstrate the advantages of Geo-ER compared to existing state-of-the-art solution for geospatial and traditional ER. Cross-city validation experiments showcase the algorithm's robustness in scenarios where training data for a specific city is unavailable. Subsequently, this thesis presents GTMiner, a novel framework for automatically constructing a Geospatial Knowledge Graph (GeoKG) from tabular data, derived from the merging and consolidation of multiple data sources. The architecture for geospatial relation prediction is an open-world Knowledge Graph Completion solution, consisting of a pre-trained language model, a geospatial encoder, and a Geo-Textual interaction component, which jointly model the textual and geospatial characteristics of an entity. GTMiner further includes a refinement module, specifically designed to improve both coverage and correctness of a GeoKG. The study further introduces four real-world datasets, collected from publicly available sources, and carries out extensive experiments to evaluate the performance of each of the system’s components and perform ablation studies. A study on general-purpose representation learning in the geospatial domain introduces CityFM, a new framework that leverages mutual information maximization and incorporates Nodes, Ways, and Relations into its learning frame-work to train geospatial Pre-trained Foundation Models (PFMs) for a selected area of interest. CityFM-trained models are task-agnostic and multimodal, capable of integrating an entity’s textual, visual, and spatial attributes. The study conducts experiments on road, building, and region-level downstream tasks to demonstrate the effectiveness of the embeddings produced by the foundation models in real-world applications, compared to task-specific algorithms. The last study in this dissertation presents LAMP, a virtual assistant for urban applications. It introduces a novel strategy to generate synthetic conversational data about geospatial entities, to fine-tune a pre-trained LLM. The generated data enables the language model to learn and memorize thousands of places within a city, while also equipping it with spatial (proximity) awareness specific to the urban area of interest.