Lightweight Self-Attentive Sequential Recommendation

Self-attention models have achieved the state-of-the-art performance in sequential recommender systems by capturing the sequential dependencies among user–item interactions. However, they rely on adding positional embeddings to the item sequence to retain the sequential information, which may break the semantics of item embeddings due to the heterogeneity between these two types of embeddings. In addition, most existing works assume that such dependencies exist solely in the item embeddings, but neglect their existence among the item features. In our previous study, we proposed a novel sequential recommendation model, i.e., MLP4Rec, based on the recent advances of MLP-Mixer architectures, which is naturally sensitive to the order of items in a sequence because matrix elements related to different positions of a sequence will be given different weights in training. We developed a tri-directional fusion scheme to coherently capture sequential, cross-channel, and cross-feature correlations with linear computational complexity as well as much fewer model parameters than existing self-attention methods. However, the cascading mixer structure, the large number of normalization layers between different mixer layers, and the noise generated by these operations limit the efficiency of information extraction and the effectiveness of MLP4Rec. In this extended version, we propose a novel framework – SMLP4Rec for sequential recommendation to address the aforementioned issues. The new framework changes the flawed cascading structure to a parallel mode, and integrates normalization layers to minimize their impact on the model’s efficiency while maximizing their effectiveness. As a result, the training speed and prediction accuracy of SMLP4Rec are vastly improved in comparison to MLP4Rec. Extensive experimental results demonstrate that the proposed method is significantly superior to the state-of-the-art approaches. The implementation code is available online to ease reproducibility.

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

Recent advancements in sequential recommendation have underscored the potential of Large Language Models (LLMs) for enhancing item embeddings. However, existing approaches face three key limitations: 1) the degradation of the semantic space when high-dimensional language embeddings are mapped to lower-dimensional ID embeddings, 2) the underutilization of language embeddings, and 3) the reliance on additional trainable parameters, such as an adapter, to bridge the gap between the semantic and behavior spaces. In this paper, we introduce AlphaFuse, a simple but effective language-guided learning strategy that addresses these challenges by learning ID embeddings within the null space of language embeddings. Specifically, we decompose the semantic space of language embeddings via Singular Value Decomposition (SVD), distinguishing it into a semantic-rich row space and a semantic-sparse null space. Collaborative signals are then injected into the null space, while preserving the rich semantics of the row space. AlphaFuse prevents degradation of the semantic space, integrates the retained language embeddings into the final item embeddings, and eliminates the need for auxiliary trainable modules, enabling seamless adaptation to any sequential recommendation framework. We validate the effectiveness and flexibility of AlphaFuse through extensive experiments on three benchmark datasets, including cold-start user and long-tail settings, showcasing significant improvements in both discriminative and diffusion-based generative sequential recommenders.

Deep learning and symbolic learning are two frequently employed methods in Sequential Recommendation (SR). Recent neural-symbolic SR models demonstrate their potential to enable SR to be equipped with concurrent perception and cognition capacities. However, neural-symbolic SR remains a challenging problem due to open issues like representing users and items in logical reasoning. In this paper, we combine the Deep Neural Network (DNN) SR models with logical reasoning and propose a general framework named Sequential Recommendation with Probabilistic Logical Reasoning (short for SR-PLR). This framework allows SR-PLR to benefit from both similarity matching and logical reasoning by disentangling feature embedding and logic embedding in the DNN and probabilistic logic network. To better capture the uncertainty and evolution of user tastes, SR-PLR embeds users and items with a probabilistic method and conducts probabilistic logical reasoning on users' interaction patterns. Then the feature and logic representations learned from the DNN and logic network are concatenated to make the prediction. Finally, experiments on various sequential recommendation models demonstrate the effectiveness of the SR-PLR. Our code is available at https://github.com/Huanhuaneryuan/SR-PLR.

Sequential recommendation has been widely studied in the recommendation domain since it can capture users’ temporal preferences and provide more accurate and timely recommendations. To address user privacy concerns, the combination of federated learning and sequential recommender systems (FedSeqRec) has gained growing attention. Unfortunately, the performance of FedSeqRec is still unsatisfactory because the models used in FedSeqRec have to be lightweight to accommodate communication bandwidth and clients’ on-device computational resource constraints. Recently, large language models (LLMs) have exhibited strong transferable and generalized language understanding abilities and therefore, in the NLP area, many downstream tasks now utilize LLMs as a service to achieve superior performance without constructing complex models. Inspired by this successful practice, we propose a generic FedSeqRec framework, FELLAS, which aims to enhance FedSeqRec by utilizing LLMs as an external service. Specifically, FELLAS employs an LLM server to provide both item-level and sequence-level representation assistance. The item-level representation service is queried by the central server to enrich the original ID-based item embedding with textual information, while the sequence-level representation service is accessed by each client. However, invoking the sequence-level representation service requires clients to send sequences to the external LLM server. To safeguard privacy, we implement \(d_{\mathcal{X}}\) -privacy satisfied sequence perturbation, which protects clients’ sensitive data with guarantees. Additionally, a contrastive learning-based method is designed to transfer knowledge from the noisy sequence representation to clients’ sequential recommendation models. Furthermore, to empirically validate the privacy protection capability of FELLAS, we propose two interacted item inference attacks, considering the threats posed by the LLM server and the central server acting as curious-but-honest adversaries in cooperation. Extensive experiments conducted on three datasets with two widely used sequential recommendation models demonstrate the effectiveness and privacy-preserving capability of FELLAS.

Sequential recommendation aims to capture users' dynamic interest and predicts the next item of users' preference. Most sequential recommendation methods use a deep neural network as sequence encoder to generate user and item representations. Existing works mainly center upon designing a stronger sequence encoder. However, few attempts have been made with training an ensemble of networks as sequence encoders, which is more powerful than a single network because an ensemble of parallel networks can yield diverse prediction results and hence better accuracy. In this paper, we present Ensemble Modeling with contrastive Knowledge Distillation for sequential recommendation (EMKD). Our framework adopts multiple parallel networks as an ensemble of sequence encoders and recommends items based on the output distributions of all these networks. To facilitate knowledge transfer between parallel networks, we propose a novel contrastive knowledge distillation approach, which performs knowledge transfer from the representation level via Intra-network Contrastive Learning (ICL) and Cross-network Contrastive Learning (CCL), as well as Knowledge Distillation (KD) from the logits level via minimizing the Kullback-Leibler divergence between the output distributions of the teacher network and the student network. To leverage contextual information, we train the primary masked item prediction task alongside the auxiliary attribute prediction task as a multi-task learning scheme. Extensive experiments on public benchmark datasets show that EMKD achieves a significant improvement compared with the state-of-the-art methods. Besides, we demonstrate that our ensemble method is a generalized approach that can also improve the performance of other sequential recommenders. Our code is available at this link: https://github.com/hw-du/EMKD.

Sequential recommendation is a popular task in academic research and close to real-world application scenarios, where the goal is to predict the next action(s) of the user based on his/her previous sequence of actions. In the training process of recommender systems, the loss function plays an essential role in guiding the optimization of recommendation models to generate accurate suggestions for users. However, most existing sequential recommendation techniques focus on designing algorithms or neural network architectures, and few efforts have been made to tailor loss functions that fit naturally into the practical application scenario of sequential recommender systems. Ranking-based losses, such as cross-entropy and Bayesian Personalized Ranking (BPR) are widely used in the sequential recommendation area. We argue that such objective functions suffer from two inherent drawbacks: i) the dependencies among elements of a sequence are overlooked in these loss formulations; ii) instead of balancing accuracy (quality) and diversity, only generating accurate results has been over emphasized. We therefore propose two new loss functions based on the Determinantal Point Process (DPP) likelihood, that can be adaptively applied to estimate the subsequent item or items. The DPP-distributed item set captures natural dependencies among temporal actions, and a quality vs. diversity decomposition of the DPP kernel pushes us to go beyond accuracy-oriented loss functions. Experimental results using the proposed loss functions on three real-world datasets show marked improvements over state-of-the-art sequential recommendation methods in both quality and diversity metrics.

Multi-view spectral clustering, which exploits the complementary information among graphs of diverse views to obtain superior clustering results, has attracted intensive attention recently. However, most existing multi-view spectral clustering methods obtain the clustering partitions in a two-step scheme, i.e., spectral embedding and subsequent k-means. This two-step scheme inevitably seeks sub-optimal clustering results due to the information loss during the two-steps processes. Besides, existing multi-view spectral clustering methods do not jointly utilize the information of graphs and embedding matrices, which also degrades final clustering results. To solve these issues, we propose a unified one-step multi-view spectral clustering method, which integrates the spectral embedding and <formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex>$k$</tex></formula> -means into a unified framework to obtain discrete clustering labels with a one-step strategy. Under the observation that the inner product of the embedding matrix is a low-rank approximation of the graph, we combine graphs and embedding matrices of different views to obtain a unified graph. Then, we directly capture the discrete clustering indicator matrix from the unified graph. Furthermore, we design an effective optimization algorithm to solve the resultant problem. Finally, a set of experiments on various datasets are conducted to verify the effectiveness of the proposed method. The demo code of this work is publicly available at https://github.com/guanyuezhen/UOMvSC.

In the field of sequential recommendation, deep learning--(DL) based methods have received a lot of attention in the past few years and surpassed traditional models such as Markov chain-based and factorization-based ones. However, there is little systematic study on DL-based methods, especially regarding how to design an effective DL model for sequential recommendation. In this view, this survey focuses on DL-based sequential recommender systems by taking the aforementioned issues into consideration. Specifically, we illustrate the concept of sequential recommendation, propose a categorization of existing algorithms in terms of three types of behavioral sequences, summarize the key factors affecting the performance of DL-based models, and conduct corresponding evaluations to showcase and demonstrate the effects of these factors. We conclude this survey by systematically outlining future directions and challenges in this field.

Sequential recommendation systems which aim to predict the users’ future clicking items are essential in both research and industry services. Based on the contextual characteristics of users, a large number of existing methods exploit abundant information from user behavior sequences and gain users’ interests through expressive sequential models like recurrent neural networks and self-attention mechanism to predict the next item. However, the temporal generative process of behavior sequences, which is crucial to the complex relationships and distributions of users’ dynamic preferences, is rarely considered. In this paper, we propose HawRec (Hawkes process based sequential recommendation), which is a new representation learning approach to model the interacted sequences of users from a temporal point process perspective. In particular, we leverage temporal self-attention mechanism to make an analogy to the self-exciting mechanism of Hawkes process. We bring expressive superiority of neural network into the Hawkes process with non-parametric conditional intensity kernel, resulting in more accurate recommendation and inheriting robustness from the temporal point process. Furthermore, extensive experiments conducted on four public benchmarks demonstrate that our HawRec outperforms various baselines of state-of-the-art sequential recommendation.

Adaptive self-supervised learning for sequential recommendation

In contrast to traditional recommender systems which usually pay attention to users' general and long-term preferences, sequential recommendation (SR) can model users' dynamic intents based on their behaviour sequences and suggest the next item(s) to them. However, most of existing sequential models learn the ranking score of an item only based on its relevance property, and the personalized user demands in terms of different learning objectives, such as diversity, tail novelty or recency, which have been played essential roles in multi-objective recommendation (MOR), are often neglected in SR. In this paper, we first discuss the importance of considering multiple different objectives within a learning model for recommender system. Next, to capture users' objective-level preferences by utilizing interactive information in the sequential context, we propose a novel Dynamic Multi-objective Recommendation (DMORec) framework with interactive evolution for SR. In particular, DMORec formulates a dynamic multi-objective optimization task to simultaneously optimize more than two varying objectives in an interactive recommendation process. Moreover, to resolve this optimization task in SR, we develop an evolutionary algorithm with supervised learning approach to obtain the Pareto-optimal solutions of the formulated problem. Comprehensive experiments on four real-world datasets demonstrate the effectiveness of the proposed DMORec for dynamic multi-objective recommendation in sequential recommender systems.

Graph gating-mixer for sequential recommendation

Users' historical activities are usually contained in real-life sequential recommendation systems to predict their future behaviors. In this situation, traditional Factorization Machines (FMs) approaches may be not suitable. Recently, a new surge of interest aims to use recurrent neural networks(RNN) to encode users' dynamic features with temporal characteristics. However, most of these works fail to reproduce computational simplicity of FMs. In this paper, we propose an architecture of extended-FM for sequential recommendation, which presents temporal feature interactions in an explicit way as traditional FM's formula. Our approach also allows us to accomplish computation of the model in linear time. Furthermore, we merge extended-FM into higher-order interaction framework without significant changes to the deeper models themselves. We conduct comprehensive experiments on two real-world datasets. The results demonstrate that extended-FM outperforms traditional FMs as well as deep learning feature combination models on sequential recommendation tasks.

Over the last few years, sequential recommender systems have achieved a great success in different applications. In the literature, it is generally believed that items farther away from the recommendation time have a weaker impact on the recommendation results. However, simply considering the distance between the interaction time and the recommendation time would prevent effective user representations. To solve this issue, we propose a Time Interval Aware Collaborative (TIAC) model with self-supervised learning for sequential recommendation. We propose to adjust the attention score learned from the time interval between an interaction time and the recommendation time using a time kernel function to achieve a better user representation. We also introduce self-supervised learning to combine the collaborative information obtained from a graph convolutional network and the sequential information learned from gated recurrent units to further enrich user representation. Extensive experiments on four real-world benchmark datasets show that our proposed TIAC model consistently outperforms state-of-the-art models under various evaluation metrics.KeywordsSequential recommendationAttention mechanismSelf-supervised learningGraph convolutional network