Modeling Temporal Positive and Negative Excitation for Sequential Recommendation

Sequential recommendation has become increasingly essential in various online services. It aims to model the dynamic preferences of users from their historical interactions and predict their next items. The accumulated user behavior records on real systems could be very long. This rich data brings opportunities to track actual interests of users. Prior efforts mainly focus on making recommendations based on relatively recent behaviors. However, the overall sequential data may not be effectively utilized, as early interactions might affect users' current choices. Also, it has become intolerable to scan the entire behavior sequence when performing inference for each user, since real-world system requires short response time. To bridge the gap, we propose a novel long sequential recommendation model, called Dynamic Memory-based Attention Network (DMAN). It segments the overall long behavior sequence into a series of sub-sequences, then trains the model and maintains a set of memory blocks to preserve long-term interests of users. To improve memory fidelity, DMAN dynamically abstracts each user's long-term interest into its own memory blocks by minimizing an auxiliary reconstruction loss. Based on the dynamic memory, the user's short-term and long-term interests can be explicitly extracted and combined for efficient joint recommendation. Empirical results over four benchmark datasets demonstrate the superiority of our model in capturing long-term dependency over various state-of-the-art sequential models.

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

Sequential recommendation has been widely studied in the recommendation domain since it can capture users’ temporal preferences and provide more accurate and timely recommendations. To address user privacy concerns, the combination of federated learning and sequential recommender systems (FedSeqRec) has gained growing attention. Unfortunately, the performance of FedSeqRec is still unsatisfactory because the models used in FedSeqRec have to be lightweight to accommodate communication bandwidth and clients’ on-device computational resource constraints. Recently, large language models (LLMs) have exhibited strong transferable and generalized language understanding abilities and therefore, in the NLP area, many downstream tasks now utilize LLMs as a service to achieve superior performance without constructing complex models. Inspired by this successful practice, we propose a generic FedSeqRec framework, FELLAS, which aims to enhance FedSeqRec by utilizing LLMs as an external service. Specifically, FELLAS employs an LLM server to provide both item-level and sequence-level representation assistance. The item-level representation service is queried by the central server to enrich the original ID-based item embedding with textual information, while the sequence-level representation service is accessed by each client. However, invoking the sequence-level representation service requires clients to send sequences to the external LLM server. To safeguard privacy, we implement \(d_{\mathcal{X}}\) -privacy satisfied sequence perturbation, which protects clients’ sensitive data with guarantees. Additionally, a contrastive learning-based method is designed to transfer knowledge from the noisy sequence representation to clients’ sequential recommendation models. Furthermore, to empirically validate the privacy protection capability of FELLAS, we propose two interacted item inference attacks, considering the threats posed by the LLM server and the central server acting as curious-but-honest adversaries in cooperation. Extensive experiments conducted on three datasets with two widely used sequential recommendation models demonstrate the effectiveness and privacy-preserving capability of FELLAS.

Adaptive self-supervised learning for sequential recommendation

Deep learning and symbolic learning are two frequently employed methods in Sequential Recommendation (SR). Recent neural-symbolic SR models demonstrate their potential to enable SR to be equipped with concurrent perception and cognition capacities. However, neural-symbolic SR remains a challenging problem due to open issues like representing users and items in logical reasoning. In this paper, we combine the Deep Neural Network (DNN) SR models with logical reasoning and propose a general framework named Sequential Recommendation with Probabilistic Logical Reasoning (short for SR-PLR). This framework allows SR-PLR to benefit from both similarity matching and logical reasoning by disentangling feature embedding and logic embedding in the DNN and probabilistic logic network. To better capture the uncertainty and evolution of user tastes, SR-PLR embeds users and items with a probabilistic method and conducts probabilistic logical reasoning on users' interaction patterns. Then the feature and logic representations learned from the DNN and logic network are concatenated to make the prediction. Finally, experiments on various sequential recommendation models demonstrate the effectiveness of the SR-PLR. Our code is available at https://github.com/Huanhuaneryuan/SR-PLR.

Sequential recommendation models aim to predict the interested items to a user based on his historical behaviors. To train sequential recommenders, implicit feedback data is widely adopted since it is easier to obtain than explicit feedback data. In the setting of implicit feedback, a user’s historical behaviors can be characterized as a chronologically ordered sequence of interacted items. From a perspective of machine learning, the historical interaction sequence and the recommended items can be considered as context and label , respectively, which are usually in one-hot representations in the recommendation models. However, due to the discrete nature, one-hot representations are hard to sufficiently reflect the underlying user preference, and might also contain noise from implicit feedback that will mislead the model training. To solve these issues, we propose a general optimization framework, Multi-View Smoothness (MVS), to enhance the smoothness of sequential recommendation models in both data representations and model learning. Specifically, with the help of a complementary model, we smooth and enrich the one-hot representations of contexts and labels to better depict the underlying user preference (i.e., context smoothness and label smoothness), and devise a model regularization strategy to enforce the neighborhood smoothness of the model itself (i.e., model smoothness). Based on these strategies, we design three regularizers to constrain and improve the training of sequential recommendation models. Extensive experiments on five datasets show that our approach is able to improve the performance of various base models consistently and outperform other regularization training methods.

Sequential recommendation systems which aim to predict the users’ future clicking items are essential in both research and industry services. Based on the contextual characteristics of users, a large number of existing methods exploit abundant information from user behavior sequences and gain users’ interests through expressive sequential models like recurrent neural networks and self-attention mechanism to predict the next item. However, the temporal generative process of behavior sequences, which is crucial to the complex relationships and distributions of users’ dynamic preferences, is rarely considered. In this paper, we propose HawRec (Hawkes process based sequential recommendation), which is a new representation learning approach to model the interacted sequences of users from a temporal point process perspective. In particular, we leverage temporal self-attention mechanism to make an analogy to the self-exciting mechanism of Hawkes process. We bring expressive superiority of neural network into the Hawkes process with non-parametric conditional intensity kernel, resulting in more accurate recommendation and inheriting robustness from the temporal point process. Furthermore, extensive experiments conducted on four public benchmarks demonstrate that our HawRec outperforms various baselines of state-of-the-art sequential recommendation.

Sequential Recommendation (SR) is an important scenario in recommendation tasks. Sequential recommendations model the sequential pattern between item-item or user-item based on a user's recent activity in a time series to predict their next preference. However, existing methods are based only on the conventional Graph Neural Networks (GNN) as a model architecture for adaptive fine-tuning of specific SR tasks. To get better recommendation results, more advanced GNNs can be used as the network architecture of the SR method. This paper introduces graph transformer, a combination of GNN and a good sequential task processing model. Then a cross-sectional comparison is made with the current SR method model and its suitability for application in SR tasks is discussed. The comparison shows that the graph transformer is similar in principle and structure to the current SR models, and requires the addition of some adaptive components to be applied in SR tasks. The superior performance after application can be demonstrated from the results data of the Benchmarking-GNNs and Long-Range Graph Benchmark on the models.

This work presents ai¾?Recommender System RS that relies on distributed recommendation techniques and implicit relations in data. In order to simplify the experience of users, recommender systems pre-select and filter information in which they may be interested in. Users express their interests in items by giving their opinion explicit data and navigating through the web-page implicit data. The Matrix Factorization MF recommendation technique analyze this feedback, but it does not take more heterogeneous data into account. In order to improve recommendations, the description of items can be used to increase the relations among data. Our proposal extends MF techniques by adding implicit relations in ani¾?independent layer. Indeed, using past preferences, we deeply analyze the implicit interest of users in the attributes of items. By using this, we transform ratings and predictions into values, where the term semantic indicates the expansion in the meaning of ratings. The experimentation phase uses MovieLens and IMDb database. We compare our work against ai¾?simple Matrix Factorization technique. Results show accurate personalized recommendations. At least but not at last, both recommendation analysis and semantic analysis can be parallelized, alleviating time processing in large amount of data.

Most sequential recommendation (SR) systems employing graph neural networks (GNNs) only model a user's interaction sequence as a flat graph without hierarchy, overlooking diverse factors in the user's preference. Moreover, the timespan between interacted items is not sufficiently utilized by previous models, restricting SR performance gains. To address these problems, we propose a novel SR system employing a hierarchical graph neural network (HGNN) to model factorial user preferences. Specifically, a timespan-aware sequence graph (TSG) for the target user is first constructed with the timespan among interacted items. Next, all original nodes in TSG are softly clustered into factor nodes, each of which represents a certain factor of the user's preference. At last, all factor nodes' representations are used together to predict SR results. Our extensive experiments upon two datasets justify that our HGNN-based factorial user modeling obtains better SR performance than the state-of-the-art SR models.

IPSRM: An intent perceived sequential recommendation model

An increased interest in sequential recommendation has been observed in recent years. Many models have been proposed to leverage the sequential user-item interaction data, which includes those based on Markov Chain or recurrent neural networks. Most of these models are designed for the scenario where each historical record composed of single item. However, the records could be a subset of items (or session) such as music playlists and baskets in e-commerce applications. How to leverage the session structure to improve the effectiveness of the recommendation system is a challenge. To this end, we propose a MEmory-augmented Attention Network for Sequential recommendation (MEANS), to effectively recommend next items given the sequential session data. The most recent sessions are stored into external memory after a max-pooling operation. The long-term user preference are learned through an attention network which is stacked on the memory layer. Finally, the mixture of long-term and short-term preference is feeded into the prediction layer to make recommendations. Extensive experiments on four real datasets show that MEANS outperforms various state-of-the-art sequential recommendation models.

Self-attention models have achieved the state-of-the-art performance in sequential recommender systems by capturing the sequential dependencies among user–item interactions. However, they rely on adding positional embeddings to the item sequence to retain the sequential information, which may break the semantics of item embeddings due to the heterogeneity between these two types of embeddings. In addition, most existing works assume that such dependencies exist solely in the item embeddings, but neglect their existence among the item features. In our previous study, we proposed a novel sequential recommendation model, i.e., MLP4Rec, based on the recent advances of MLP-Mixer architectures, which is naturally sensitive to the order of items in a sequence because matrix elements related to different positions of a sequence will be given different weights in training. We developed a tri-directional fusion scheme to coherently capture sequential, cross-channel, and cross-feature correlations with linear computational complexity as well as much fewer model parameters than existing self-attention methods. However, the cascading mixer structure, the large number of normalization layers between different mixer layers, and the noise generated by these operations limit the efficiency of information extraction and the effectiveness of MLP4Rec. In this extended version, we propose a novel framework – SMLP4Rec for sequential recommendation to address the aforementioned issues. The new framework changes the flawed cascading structure to a parallel mode, and integrates normalization layers to minimize their impact on the model’s efficiency while maximizing their effectiveness. As a result, the training speed and prediction accuracy of SMLP4Rec are vastly improved in comparison to MLP4Rec. Extensive experimental results demonstrate that the proposed method is significantly superior to the state-of-the-art approaches. The implementation code is available online to ease reproducibility.

Timeliness and contextual accuracy of recommendations are increasingly important when delivering contemporary digital marketing experiences. Conventional recommender systems (RS) suggest relevant but time-invariant items to users by accounting for their past purchases. These recommendations only map to customers’ general preferences rather than a customer’s specific needs immediately preceding a purchase. In contrast, RSs that consider the order of transactions, purchases, or experiences to measure evolving preferences can offer more salient and effective recommendations to customers: Sequential RSs not only benefit from a better behavioral understanding of a user’s current needs but also provide better predictive power. In this paper, we demonstrate and rank the effectiveness of a sequential recommendation system by utilizing a production dataset of over 2.7 million credit card transactions for 46K cardholders. The methodfirst employs an autoencoder on raw transaction data and submits observed transaction encodings to a GRU-based sequential model. The sequential model produces a MAP@1 metric of 47% on the out-of-sample test set, in line with existing research. We also discuss implications for embedding real-timepredictions using the sequential RS into Nexus, a scalable, lowlatency, event-based digital experience architecture.

The sequential recommendation selects and recommends next items for users by modeling their historical interaction sequences, where the chronological order of interactions plays an important role. Most sequential recommendation methods only pay attention to the order information among the interactions and ignore the time intervals information, which leads to the limitations of capturing dynamic user interests. And previous work neglects diversity in order to improve recommendation accuracy. The model Temporal Self-Attention and Multi-Preference Learning (TSAMPL) is proposed to improve sequential recommendation, which learns dynamic and general user interests separately. The proposed temporal gate self-attention network is introduced to learn dynamic user interests, which takes both contextual information and temporal dynamics into account. To model general user interests, we employ a multi-preference matrix to learn users’ multiple types of preferences for improving recommendation diversity. Finally, the interest fusion module combines dynamic user interests (accuracy) and general user interests (diversity) adaptively. The experiments in sequential recommendation confirm our method is superior to all comparison methods, we also study the impact of each component in the model.