Future-Aware Diverse Trends Framework for Recommendation

Sequential recommendation is a popular task in academic research and close to real-world application scenarios, where the goal is to predict the next action(s) of the user based on his/her previous sequence of actions. In the training process of recommender systems, the loss function plays an essential role in guiding the optimization of recommendation models to generate accurate suggestions for users. However, most existing sequential recommendation techniques focus on designing algorithms or neural network architectures, and few efforts have been made to tailor loss functions that fit naturally into the practical application scenario of sequential recommender systems. Ranking-based losses, such as cross-entropy and Bayesian Personalized Ranking (BPR) are widely used in the sequential recommendation area. We argue that such objective functions suffer from two inherent drawbacks: i) the dependencies among elements of a sequence are overlooked in these loss formulations; ii) instead of balancing accuracy (quality) and diversity, only generating accurate results has been over emphasized. We therefore propose two new loss functions based on the Determinantal Point Process (DPP) likelihood, that can be adaptively applied to estimate the subsequent item or items. The DPP-distributed item set captures natural dependencies among temporal actions, and a quality vs. diversity decomposition of the DPP kernel pushes us to go beyond accuracy-oriented loss functions. Experimental results using the proposed loss functions on three real-world datasets show marked improvements over state-of-the-art sequential recommendation methods in both quality and diversity metrics.

The sequential recommendation selects and recommends next items for users by modeling their historical interaction sequences, where the chronological order of interactions plays an important role. Most sequential recommendation methods only pay attention to the order information among the interactions and ignore the time intervals information, which leads to the limitations of capturing dynamic user interests. And previous work neglects diversity in order to improve recommendation accuracy. The model Temporal Self-Attention and Multi-Preference Learning (TSAMPL) is proposed to improve sequential recommendation, which learns dynamic and general user interests separately. The proposed temporal gate self-attention network is introduced to learn dynamic user interests, which takes both contextual information and temporal dynamics into account. To model general user interests, we employ a multi-preference matrix to learn users’ multiple types of preferences for improving recommendation diversity. Finally, the interest fusion module combines dynamic user interests (accuracy) and general user interests (diversity) adaptively. The experiments in sequential recommendation confirm our method is superior to all comparison methods, we also study the impact of each component in the model.

Sequential recommendation aims to capture users' dynamic interest and predicts the next item of users' preference. Most sequential recommendation methods use a deep neural network as sequence encoder to generate user and item representations. Existing works mainly center upon designing a stronger sequence encoder. However, few attempts have been made with training an ensemble of networks as sequence encoders, which is more powerful than a single network because an ensemble of parallel networks can yield diverse prediction results and hence better accuracy. In this paper, we present Ensemble Modeling with contrastive Knowledge Distillation for sequential recommendation (EMKD). Our framework adopts multiple parallel networks as an ensemble of sequence encoders and recommends items based on the output distributions of all these networks. To facilitate knowledge transfer between parallel networks, we propose a novel contrastive knowledge distillation approach, which performs knowledge transfer from the representation level via Intra-network Contrastive Learning (ICL) and Cross-network Contrastive Learning (CCL), as well as Knowledge Distillation (KD) from the logits level via minimizing the Kullback-Leibler divergence between the output distributions of the teacher network and the student network. To leverage contextual information, we train the primary masked item prediction task alongside the auxiliary attribute prediction task as a multi-task learning scheme. Extensive experiments on public benchmark datasets show that EMKD achieves a significant improvement compared with the state-of-the-art methods. Besides, we demonstrate that our ensemble method is a generalized approach that can also improve the performance of other sequential recommenders. Our code is available at this link: https://github.com/hw-du/EMKD.

Recent advancements in generative models have positioned them as one of the principal tools for sequential recommendation due to their exceptional sample diversity and generalization capabilities. Among these, diffusion model-based sequential recommenders have achieved remarkable success. However, most existing approaches still face critical challenges, resulting in suboptimal generation quality: (1) They fail to leverage multimodal knowledge for constructing item representations with well-structured distributional characteristics and semantically enriched information; (2) They predominantly rely on discrete diffusion processes, leading to high error accumulation, reduced time efficiency, and constrained controllability in generative sampling. To mitigate these challenges, we propose LSGM4Rec, a novel framework that integrates Large Language Models (LLMs) with advanced multimodal encoding models to establish multimodal fusion embeddings for items. This design ensures distinct distributional characteristics while enabling the incorporation of semantically rich modal features into guidance condition. Furthermore, we pioneer the stochastic differential equations (SDEs) for recommendation, facilitating smooth transitions between data distributions and enabling optimal trade-off between sampling efficiency and generation quality. Extensive experiments on three datasets demonstrate that LSGM4Rec outperforms existing state-of-the-art sequential recommendation methods.

Recent deep sequential recommendation models often struggle to effectively model key characteristics of user behaviors, particularly in handling sequence length variations and capturing diverse interaction patterns. We propose STAR-Rec, a novel architecture that synergistically combines preference-aware attention and state-space modeling through a sequence-level mixture-of-experts framework. STAR-Rec addresses these challenges by: (1) employing preference-aware attention to capture both inherently similar item relationships and diverse preferences (2) utilizing state-space modeling to efficiently process variable-length sequences with linear complexity, and (3) incorporating a mixture-of-experts component that adaptively routes different behavioral patterns to specialized experts, handling both focused category-specific browsing and diverse category exploration patterns. We theoretically demonstrate how the state space model and attention mechanisms can be naturally unified in recommendation scenarios, where SSM captures temporal dynamics through state compression while attention models both similar and diverse item relationships. Extensive experiments on four real-world datasets demonstrate that STAR-Rec consistently outperforms state-of-the-art sequential recommendation methods, particularly in scenarios involving diverse user behaviors and varying sequence lengths. The implementation code is available anonymously online for easy reproducibility.

Sequential recommendations aim to capture users’ preferences from their historical interactions so as to predict the next item that they will interact with. Sequential recommendation methods usually assume that all items in a user’s historical interactions reflect her/his preferences and transition patterns between items. However, real-world interaction data is imperfect in that (i) users might erroneously click on items, i.e., so-called misclicks on irrelevant items, and (ii) users might miss items, i.e., unexposed relevant items due to inaccurate recommendations.

In E-commerce applications, to predict what users will buy next is a crucial mission of sequential recommendation. Most frontier researches build end-to-end training models for sequential recommendation tasks via RNNs, CNNs or attentive models. However, a user’s historical behavior sequence carries more complex contextual information than words. In this paper, we propose a two-stage user modeling framework for sequential recommendation, which is consisted by a Bidirectional Self-attentive Behavior Embedding and a Long/Short-term Sequential Behavior Predictor. Firstly, in order to expand perceivable information, a novel self-attentive behavior embedding method is proposed to learn semantic representations not only for items, but also for other important contextual factors (e.g. actions, categories and time). Then, with the pre-trained behavior embeddings, we propose a personalized memory network for Top-N recommendation. We use recurrent network to encode the short-term intent and learn the personalized long-term memory by a self-attention block. To integrate the long/short-term preferences, we generate the predicted behavior representation by using the present intent as a query to match with user’s historical preferences via attentive memory reader. Finally, we conduct extensive experiments on two benchmark datasets provided by Tmall and Amazon. Compared with state-of-the-art techniques, experimental results demonstrate the effectiveness of our proposed framework.

Top-$N$ sequential recommendation models each user as a sequence of items interacted in the past and aims to predict top-$N$ ranked items that a user will likely interact in a `near future'. The order of interaction implies that sequential patterns play an important role where more recent items in a sequence have a larger impact on the next item. In this paper, we propose a Convolutional Sequence Embedding Recommendation Model (\emph{Caser}) as a solution to address this requirement. The idea is to embed a sequence of recent items into an `image' in the time and latent spaces and learn sequential patterns as local features of the image using convolutional filters. This approach provides a unified and flexible network structure for capturing both general preferences and sequential patterns. The experiments on public datasets demonstrated that Caser consistently outperforms state-of-the-art sequential recommendation methods on a variety of common evaluation metrics.

DistVAE: Distributed Variational Autoencoder for sequential recommendation

Recommendation system plays an important role in online web applications. Sequential recommender further models user short-term preference through exploiting information from latest user-item interaction history. Most of the sequential recommendation methods neglect the importance of ever-changing item popularity. We propose the model from the intuition that items with most user interactions may be popular in the past but could go out of fashion in recent days. To this end, this paper proposes a novel sequential recommendation approach dubbed TRec, TRec learns item trend information from implicit user interaction history and incorporates item trend information into next item recommendation tasks. Then a self-attention mechanism is used to learn better node representation. Our model is trained via pairwise rank-based optimization. We conduct extensive experiments with seven baseline methods on four benchmark datasets, The empirical result shows our approach outperforms other state-of-the-art methods while maintains a superiorly low runtime cost. Our study demonstrates the importance of item trend information in recommendation system designs, and our method also possesses great efficiency which enables it to be practical in real-world scenarios.

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

Group-aware graph neural networks for sequential recommendation

In contrast to traditional recommender systems which usually pay attention to users' general and long-term preferences, sequential recommendation (SR) can model users' dynamic intents based on their behaviour sequences and suggest the next item(s) to them. However, most of existing sequential models learn the ranking score of an item only based on its relevance property, and the personalized user demands in terms of different learning objectives, such as diversity, tail novelty or recency, which have been played essential roles in multi-objective recommendation (MOR), are often neglected in SR. In this paper, we first discuss the importance of considering multiple different objectives within a learning model for recommender system. Next, to capture users' objective-level preferences by utilizing interactive information in the sequential context, we propose a novel Dynamic Multi-objective Recommendation (DMORec) framework with interactive evolution for SR. In particular, DMORec formulates a dynamic multi-objective optimization task to simultaneously optimize more than two varying objectives in an interactive recommendation process. Moreover, to resolve this optimization task in SR, we develop an evolutionary algorithm with supervised learning approach to obtain the Pareto-optimal solutions of the formulated problem. Comprehensive experiments on four real-world datasets demonstrate the effectiveness of the proposed DMORec for dynamic multi-objective recommendation in sequential recommender systems.

Recent research has demonstrated the effectiveness of utilizing contrastive learning for training Transformer-based sequence encoders in sequential recommendation tasks. Items are represented using vectors and the relations between items are measured by the dot product self-attention, the feature representation in sequential recommendation can be enhanced. However, in real-world scenarios, user behavior sequences are unpredictable, and the limitations of dot product-based approaches hinder the complete capture of collaborative transferability. Moreover, the Bayesian personalized ranking (BPR) loss function, commonly utilized in recommendation systems, lacks constraints when considering positive and negative sampled items, potentially leading to suboptimal optimization outcomes. This presents a complex challenge that needs to be addressed. To tackle these issues, this article proposes a novel method involving stochastic self-attention. This article introduces uncertainty into the proposed model by utilizing elliptical Gaussian distribution controlled by mean and covariance vector to explain the unpredictability of items. At the same time, the proposed model combines a Wasserstein self-attention module to compute the positional relationships between items within a sequence in order to effectively incorporate uncertainty into the training process. The Wasserstein self-attention mechanism satisfies the triangular inequality and can not only addresses uncertainty but also promote collaborative transfer learning. Furthermore, embedding a stochastic Gaussian distribution into each item will bring additional uncertainty into the proposed model. Multi-pair contrastive learning relies on high-quality positive samples, and the proposed model combines the cloze task mask and dropout mask mechanisms to generate high-quality positive samples. It demonstrates superior performance and adaptability compared to traditional single-pair contrastive learning methods. Additionally, a dynamic loss reweighting strategy is introduced to balance the cloze task loss and the contrastive loss effectively. We conduct experiments and the results show that the proposed model outperforms the state-of-the-art models, especially on cold start items. For each metric, the hit ratio (HR) and normalized discounted cumulative gain (NDCG) on the Beauty dataset improved by an average of 1.3% and 10.27%, respectively; on the Toys dataset improved by an average of 8.24% and 5.89%, respectively; on the ML-1M dataset improved by an average of 68.62% and 8.22%, respectively; and on the ML-100M dataset improved by an average of 93.57% and 44.87% Our code is available at DOI: 10.5281/zenodo.13634624.

A recommendation system is a set of programs that utilize different methodologies for relevant item selection for the user. Graph neural networks have been extensively used in recent years to improve the quality of recommendations across all domains. A general recommendation system’s main goal is to recommend items to the user accurately, and it frequently prioritizes items that are well-liked or mainstream. If the model concentrates only on one specific item category from the users’ past preferences, then recommendations for the target user will become too obvious. To address this problem, diversity in the recommendation system is introduced. The model IG-DivRS (Item-Enhanced Graph Neural Network for a Diversified Recommendation System) is proposed. Our proposed model uses a Graph Neural Network (GNN) with the user’s interacted and non-interacted item history for diversified recommendation generation. The novelty of our proposed model is to explore the effect of non-interacted items on the target user for diversified recommendation generation. Instead of selecting random non-interacted items for the target user, we apply the DPP(Determinantal Point Process) algorithm to select the non-interacted item appropriately. The detailed experimental analysis shows that our model ID-DivRS outperforms the state-of-the-art model in accuracy and diversity.