Sequential Recommendation Model for Next Purchase Prediction

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

In contrast to traditional recommender systems which usually pay attention to users' general and long-term preferences, sequential recommendation (SR) can model users' dynamic intents based on their behaviour sequences and suggest the next item(s) to them. However, most of existing sequential models learn the ranking score of an item only based on its relevance property, and the personalized user demands in terms of different learning objectives, such as diversity, tail novelty or recency, which have been played essential roles in multi-objective recommendation (MOR), are often neglected in SR. In this paper, we first discuss the importance of considering multiple different objectives within a learning model for recommender system. Next, to capture users' objective-level preferences by utilizing interactive information in the sequential context, we propose a novel Dynamic Multi-objective Recommendation (DMORec) framework with interactive evolution for SR. In particular, DMORec formulates a dynamic multi-objective optimization task to simultaneously optimize more than two varying objectives in an interactive recommendation process. Moreover, to resolve this optimization task in SR, we develop an evolutionary algorithm with supervised learning approach to obtain the Pareto-optimal solutions of the formulated problem. Comprehensive experiments on four real-world datasets demonstrate the effectiveness of the proposed DMORec for dynamic multi-objective recommendation in sequential recommender systems.

Sequential recommendation has been widely studied in the recommendation domain since it can capture users’ temporal preferences and provide more accurate and timely recommendations. To address user privacy concerns, the combination of federated learning and sequential recommender systems (FedSeqRec) has gained growing attention. Unfortunately, the performance of FedSeqRec is still unsatisfactory because the models used in FedSeqRec have to be lightweight to accommodate communication bandwidth and clients’ on-device computational resource constraints. Recently, large language models (LLMs) have exhibited strong transferable and generalized language understanding abilities and therefore, in the NLP area, many downstream tasks now utilize LLMs as a service to achieve superior performance without constructing complex models. Inspired by this successful practice, we propose a generic FedSeqRec framework, FELLAS, which aims to enhance FedSeqRec by utilizing LLMs as an external service. Specifically, FELLAS employs an LLM server to provide both item-level and sequence-level representation assistance. The item-level representation service is queried by the central server to enrich the original ID-based item embedding with textual information, while the sequence-level representation service is accessed by each client. However, invoking the sequence-level representation service requires clients to send sequences to the external LLM server. To safeguard privacy, we implement \(d_{\mathcal{X}}\) -privacy satisfied sequence perturbation, which protects clients’ sensitive data with guarantees. Additionally, a contrastive learning-based method is designed to transfer knowledge from the noisy sequence representation to clients’ sequential recommendation models. Furthermore, to empirically validate the privacy protection capability of FELLAS, we propose two interacted item inference attacks, considering the threats posed by the LLM server and the central server acting as curious-but-honest adversaries in cooperation. Extensive experiments conducted on three datasets with two widely used sequential recommendation models demonstrate the effectiveness and privacy-preserving capability of FELLAS.

Self-attention models have achieved the state-of-the-art performance in sequential recommender systems by capturing the sequential dependencies among user–item interactions. However, they rely on adding positional embeddings to the item sequence to retain the sequential information, which may break the semantics of item embeddings due to the heterogeneity between these two types of embeddings. In addition, most existing works assume that such dependencies exist solely in the item embeddings, but neglect their existence among the item features. In our previous study, we proposed a novel sequential recommendation model, i.e., MLP4Rec, based on the recent advances of MLP-Mixer architectures, which is naturally sensitive to the order of items in a sequence because matrix elements related to different positions of a sequence will be given different weights in training. We developed a tri-directional fusion scheme to coherently capture sequential, cross-channel, and cross-feature correlations with linear computational complexity as well as much fewer model parameters than existing self-attention methods. However, the cascading mixer structure, the large number of normalization layers between different mixer layers, and the noise generated by these operations limit the efficiency of information extraction and the effectiveness of MLP4Rec. In this extended version, we propose a novel framework – SMLP4Rec for sequential recommendation to address the aforementioned issues. The new framework changes the flawed cascading structure to a parallel mode, and integrates normalization layers to minimize their impact on the model’s efficiency while maximizing their effectiveness. As a result, the training speed and prediction accuracy of SMLP4Rec are vastly improved in comparison to MLP4Rec. Extensive experimental results demonstrate that the proposed method is significantly superior to the state-of-the-art approaches. The implementation code is available online to ease reproducibility.

STATISTICAL MODELS OF PILOT PERCEPTION AND CONTROL AS INDICATORS SENSITIVE TO STRESS

Sequential recommendation has become increasingly essential in various online services. It aims to model the dynamic preferences of users from their historical interactions and predict their next items. The accumulated user behavior records on real systems could be very long. This rich data brings opportunities to track actual interests of users. Prior efforts mainly focus on making recommendations based on relatively recent behaviors. However, the overall sequential data may not be effectively utilized, as early interactions might affect users' current choices. Also, it has become intolerable to scan the entire behavior sequence when performing inference for each user, since real-world system requires short response time. To bridge the gap, we propose a novel long sequential recommendation model, called Dynamic Memory-based Attention Network (DMAN). It segments the overall long behavior sequence into a series of sub-sequences, then trains the model and maintains a set of memory blocks to preserve long-term interests of users. To improve memory fidelity, DMAN dynamically abstracts each user's long-term interest into its own memory blocks by minimizing an auxiliary reconstruction loss. Based on the dynamic memory, the user's short-term and long-term interests can be explicitly extracted and combined for efficient joint recommendation. Empirical results over four benchmark datasets demonstrate the superiority of our model in capturing long-term dependency over various state-of-the-art sequential models.

Sequential recommendation system's goal is to predict users' next actions based on their historical behavior sequences. As we know, more recent items have a larger impact than the previous ones. Meanwhile, modeling users' current interests are challenging. Knowledge graph (KG) contains a vast of information, which can help us capture users' interests by propagating their interactions. In this paper, we propose an Attentive Dynamic Knowledge Graph Networks (ADKGN) for sequential recommendation, which includes an embedding module, an attentive dynamic knowledge graph module, a sequential process module and a prediction module. Specifically, embedding module learns initial item embedding vectors by combing latent features and sequential features. For users' recent interacted items, attentive dynamic knowledge graph module learns dynamic weights for each pretrained KG embedding, and then utilizes top-k layer and parallel-based aggregation layer to effectively aggregate useful information from multi-hop neighbors. Sequential process module combines a user's history interactions and processed current interactions, and employs sequential models over them to get final user representations. Prediction module predicts the clicking probability using final user representation and target item representation. We conduct experiments on a public dataset, finding that ADKGN significantly outperforms state-of-the-art solutions.

Sequential recommender systems (SRS) have gained increasing popularity due to their remarkable proficiency in capturing dynamic user preferences. In the current setup of SRS, a common configuration is to uniformly consider each historical behavior as a positive interaction. However, this setting has the potential to yield sub-optimal performance as each individual item often have a different impact on shaping the user's interests. Hence, in this paper, we propose a novel automatic sampling framework for sequential recommendation, named AutoSAM, to non-uniformly treat historical behaviors. Specifically, AutoSAM extends the conventional SRS framework by integrating an extra sampler to intelligently discern the skew distribution of the raw input, and then sample informative sub-sets to build more generalizable SRS. To tackle the challenges posed by non-differentiable sampling actions and to introduce multiple decision factors for sampling, we further design a novel reinforcement learning based method to guide the training of the sampler. Furthermore, we theoretically devise multi-objective sampling rewards including Future Prediction and Sequence Perplexity, and then optimize the whole framework in an end-to-end manner by combining the policy gradient. We conduct extensive experiments on benchmark recommendation models and four real-world datasets. The experimental results demonstrate the effectiveness of the proposed AutoSAM.

IPSRM: An intent perceived sequential recommendation model

Recommender systems help users to discover and filter new and interesting products based on their preferences. Session-Based Recommender systems are powerful tools for anonymous e-commerce visitors to understand their behaviours and recommend useful products. Diversity in the recommendations is an important parameter due to increasing the opportunity of recommending new and less similar items that users interacted. Effect of diversity has been investigated in many works for the collaborative filtering-based Recommender systems. However, for session-based Recommender systems, exploring the effect of diversity is still an open area. In this paper, we propose an approach to calculate the diversity level of the items in the session logs and analyse the effect of diversity level on the session-based recommendation. In order to test the impact of diversity awareness, we propose a sequential Item-KNN recommendation model. The final recommendation list is created as a contribution of the interacted items in the session that depends on the diversity level between last interacted item of the session. We conduct several experiments to validate our diversity aware model on a real-world dataset. The results show that diversity awareness in the sessions helps to improve the performance of Recommender system in terms of recall and precision evaluation metrics. Also, the proposed method can be applied to other sequential Recommender system methods, including deep-learning based Recommender systems.

In the era of large foundation models, data has become a crucial component in building high-performance AI systems. As the demand for high-quality and large-scale data continues to rise, data copyright protection is attracting increasing attention. In this work, we explore the problem of data watermarking for sequential recommender systems, where a watermark is embedded into the target dataset and can be detected in models trained on that dataset. We focus on two settings: dataset watermarking, which protects the ownership of the entire dataset, and user watermarking, which safeguards the data of individual users. We present a method named Dataset Watermarking for Recommender Systems (DWRS) to address them. We define the watermark as a sequence of consecutive items inserted into normal users' interaction sequences. We define a Receptive Field (RF) to guide the inserting process to facilitate the memorization of the watermark. Extensive experiments on five representative sequential recommendation models and three benchmark datasets demonstrate the effectiveness of DWRS in protecting data copyright while preserving model utility.

Over the last few years, sequential recommender systems have achieved a great success in different applications. In the literature, it is generally believed that items farther away from the recommendation time have a weaker impact on the recommendation results. However, simply considering the distance between the interaction time and the recommendation time would prevent effective user representations. To solve this issue, we propose a Time Interval Aware Collaborative (TIAC) model with self-supervised learning for sequential recommendation. We propose to adjust the attention score learned from the time interval between an interaction time and the recommendation time using a time kernel function to achieve a better user representation. We also introduce self-supervised learning to combine the collaborative information obtained from a graph convolutional network and the sequential information learned from gated recurrent units to further enrich user representation. Extensive experiments on four real-world benchmark datasets show that our proposed TIAC model consistently outperforms state-of-the-art models under various evaluation metrics.KeywordsSequential recommendationAttention mechanismSelf-supervised learningGraph convolutional network

Sequential Recommender Systems (SRSs) predict items of interest for users based on their historical interactions. Two key popular paradigms for SRs are unidirectional Next Item Prediction (NIP) and bidirectional Masked Language Modeling (MLM). NIP performs well in recommendation tasks but is constrained by its reliance on prior information and a rigid temporal order assumption, limiting its ability to capture dynamic personalized preferences. MLM, on the other hand, enhances sequence representation and personalization by leveraging bidirectional context, but its objective is misaligned with recommendation tasks. To this end, we develop MetaSR, a meta-learning-based SR approach that ingeniously combines both paradigms into a unified framework to enhance personalized sequential recommendation. Specifically, we treat the mask prediction task in MLM as the inner-loop task (i.e., the support set), adjusting mask items dynamically to generate sequences that match individual behavioral patterns to capture personalized user preferences. In the outer loop (i.e., the query set), we adopt the NIP paradigm to directly train MetaSR toward the SR target. The joint training bridges the gap between MLM and SRs tasks, also overcoming NIP’s limitations in capturing dynamic personalized preferences. Furthermore, to improve the overall efficiency, we develop a reinforcement learning-based Adaptive Masked Sequence Selection (AMSS) mechanism, automatically selecting the optimal masking prediction tasks within the meta-learning support set to accelerate the meta-learning and achieve better personalization. Experiments on multiple public benchmark datasets show that MetaSR significantly outperforms existing SRs models. The dataset and codes are available at MetaSR .

Top-$N$ sequential recommendation models each user as a sequence of items interacted in the past and aims to predict top-$N$ ranked items that a user will likely interact in a `near future'. The order of interaction implies that sequential patterns play an important role where more recent items in a sequence have a larger impact on the next item. In this paper, we propose a Convolutional Sequence Embedding Recommendation Model (\emph{Caser}) as a solution to address this requirement. The idea is to embed a sequence of recent items into an `image' in the time and latent spaces and learn sequential patterns as local features of the image using convolutional filters. This approach provides a unified and flexible network structure for capturing both general preferences and sequential patterns. The experiments on public datasets demonstrated that Caser consistently outperforms state-of-the-art sequential recommendation methods on a variety of common evaluation metrics.

Recent research has shown that self-attention mechanisms improve Sequential Recommender Systems (SRS) by capturing sequential associations with the interactions. Nevertheless, existing work still needs to address two critical limitations. Firstly, the behavior of users in the original sequences contains various preference signals that are implicit and noisy and hard to reflect the users intentions fully. As a result, it would deteriorate the representation of their true intentions to model all interactions. Secondly, most models only model single-scale interaction sequences and ignore the multi-scale feature relationships of the sequences. In order to address these limitations, the paper proposes MFTSRec (Multi-scale Filter Enhanced Transformer Sequential Recommender), which can weaken those interactions irrelevant to the users intentions from their implicit feedback and adaptively focus on the users multi-scale intentions. Besides, this paper also does extensive experiments on four benchmark datasets and further demonstrates the effectiveness and robustness of MFTSRec compared to the state-of-the-art model.