Temporal Density-aware Sequential Recommendation Networks with Contrastive Learning

Sequential recommendation aims to capture users' dynamic interest and predicts the next item of users' preference. Most sequential recommendation methods use a deep neural network as sequence encoder to generate user and item representations. Existing works mainly center upon designing a stronger sequence encoder. However, few attempts have been made with training an ensemble of networks as sequence encoders, which is more powerful than a single network because an ensemble of parallel networks can yield diverse prediction results and hence better accuracy. In this paper, we present Ensemble Modeling with contrastive Knowledge Distillation for sequential recommendation (EMKD). Our framework adopts multiple parallel networks as an ensemble of sequence encoders and recommends items based on the output distributions of all these networks. To facilitate knowledge transfer between parallel networks, we propose a novel contrastive knowledge distillation approach, which performs knowledge transfer from the representation level via Intra-network Contrastive Learning (ICL) and Cross-network Contrastive Learning (CCL), as well as Knowledge Distillation (KD) from the logits level via minimizing the Kullback-Leibler divergence between the output distributions of the teacher network and the student network. To leverage contextual information, we train the primary masked item prediction task alongside the auxiliary attribute prediction task as a multi-task learning scheme. Extensive experiments on public benchmark datasets show that EMKD achieves a significant improvement compared with the state-of-the-art methods. Besides, we demonstrate that our ensemble method is a generalized approach that can also improve the performance of other sequential recommenders. Our code is available at this link: https://github.com/hw-du/EMKD.

Most sequential recommendation (SR) systems employing graph neural networks (GNNs) only model a user's interaction sequence as a flat graph without hierarchy, overlooking diverse factors in the user's preference. Moreover, the timespan between interacted items is not sufficiently utilized by previous models, restricting SR performance gains. To address these problems, we propose a novel SR system employing a hierarchical graph neural network (HGNN) to model factorial user preferences. Specifically, a timespan-aware sequence graph (TSG) for the target user is first constructed with the timespan among interacted items. Next, all original nodes in TSG are softly clustered into factor nodes, each of which represents a certain factor of the user's preference. At last, all factor nodes' representations are used together to predict SR results. Our extensive experiments upon two datasets justify that our HGNN-based factorial user modeling obtains better SR performance than the state-of-the-art SR models.

The sequential recommendation selects and recommends next items for users by modeling their historical interaction sequences, where the chronological order of interactions plays an important role. Most sequential recommendation methods only pay attention to the order information among the interactions and ignore the time intervals information, which leads to the limitations of capturing dynamic user interests. And previous work neglects diversity in order to improve recommendation accuracy. The model Temporal Self-Attention and Multi-Preference Learning (TSAMPL) is proposed to improve sequential recommendation, which learns dynamic and general user interests separately. The proposed temporal gate self-attention network is introduced to learn dynamic user interests, which takes both contextual information and temporal dynamics into account. To model general user interests, we employ a multi-preference matrix to learn users’ multiple types of preferences for improving recommendation diversity. Finally, the interest fusion module combines dynamic user interests (accuracy) and general user interests (diversity) adaptively. The experiments in sequential recommendation confirm our method is superior to all comparison methods, we also study the impact of each component in the model.

Personalized recommendation plays a central role in many online content sharing platforms. To provide quality micro-video recommendation service, it is of crucial importance to consider the interactions between users and items (i.e. micro-videos) as well as the item contents from various modalities (e.g. visual, acoustic, and textual). Existing works on multimedia recommendation largely exploit multi-modal contents to enrich item representations, while less effort is made to leverage information interchange between users and items to enhance user representations and further capture user's fine-grained preferences on different modalities. In this paper, we propose to exploit user-item interactions to guide the representation learning in each modality, and further personalized micro-video recommendation. We design a Multi-modal Graph Convolution Network (MMGCN) framework built upon the message-passing idea of graph neural networks, which can yield modal-specific representations of users and micro-videos to better capture user preferences. Specifically, we construct a user-item bipartite graph in each modality, and enrich the representation of each node with the topological structure and features of its neighbors. Through extensive experiments on three publicly available datasets, Tiktok, Kwai, and MovieLens, we demonstrate that our proposed model is able to significantly outperform state-of-the-art multi-modal recommendation methods.

Sequential recommendation has become increasingly essential in various online services. It aims to model the dynamic preferences of users from their historical interactions and predict their next items. The accumulated user behavior records on real systems could be very long. This rich data brings opportunities to track actual interests of users. Prior efforts mainly focus on making recommendations based on relatively recent behaviors. However, the overall sequential data may not be effectively utilized, as early interactions might affect users' current choices. Also, it has become intolerable to scan the entire behavior sequence when performing inference for each user, since real-world system requires short response time. To bridge the gap, we propose a novel long sequential recommendation model, called Dynamic Memory-based Attention Network (DMAN). It segments the overall long behavior sequence into a series of sub-sequences, then trains the model and maintains a set of memory blocks to preserve long-term interests of users. To improve memory fidelity, DMAN dynamically abstracts each user's long-term interest into its own memory blocks by minimizing an auxiliary reconstruction loss. Based on the dynamic memory, the user's short-term and long-term interests can be explicitly extracted and combined for efficient joint recommendation. Empirical results over four benchmark datasets demonstrate the superiority of our model in capturing long-term dependency over various state-of-the-art sequential models.

Sequential Recommender Systems (SRSs) predict items of interest for users based on their historical interactions. Two key popular paradigms for SRs are unidirectional Next Item Prediction (NIP) and bidirectional Masked Language Modeling (MLM). NIP performs well in recommendation tasks but is constrained by its reliance on prior information and a rigid temporal order assumption, limiting its ability to capture dynamic personalized preferences. MLM, on the other hand, enhances sequence representation and personalization by leveraging bidirectional context, but its objective is misaligned with recommendation tasks. To this end, we develop MetaSR, a meta-learning-based SR approach that ingeniously combines both paradigms into a unified framework to enhance personalized sequential recommendation. Specifically, we treat the mask prediction task in MLM as the inner-loop task (i.e., the support set), adjusting mask items dynamically to generate sequences that match individual behavioral patterns to capture personalized user preferences. In the outer loop (i.e., the query set), we adopt the NIP paradigm to directly train MetaSR toward the SR target. The joint training bridges the gap between MLM and SRs tasks, also overcoming NIP’s limitations in capturing dynamic personalized preferences. Furthermore, to improve the overall efficiency, we develop a reinforcement learning-based Adaptive Masked Sequence Selection (AMSS) mechanism, automatically selecting the optimal masking prediction tasks within the meta-learning support set to accelerate the meta-learning and achieve better personalization. Experiments on multiple public benchmark datasets show that MetaSR significantly outperforms existing SRs models. The dataset and codes are available at MetaSR .

Modeling user preferences from users' historical sequences is one of the core problems of sequential recommendation. Previous studies only considered transition patterns between user-items, ignoring transition patterns between item features and item-item interactions. Recently, there has been interest in integrating knowledge graphs as auxiliary information into sequential recommendation. Most of the existing methods deal with the heterogeneous information in the knowledge graph in a coarse-grained manner. We believe that fine-grained processing of information in knowledge graphs can help recommendation systems understand changes in user preferences, i.e. dividing hetero-geneous information into item-to-item relationships and item-to-attribute relationships. In this paper, we propose a dual-level self-attention network for sequential recommendation. Specifically, we divide the knowledge graph heterogeneous information about items into relation level and attribute level, representing item-item relationship and item-attribute relationship, respectively. Afterwards, the self-attention network is used to learn user preferences at dual-level, respectively. Then, the outputs of the above dual levels are integrated for next item recommendation. Based on extensive experiments on three real-world data sets, our model achieves significant improvements compared to state-of-the-art baseline methods.

Sequential recommendation aims to predict the next item which interests users via modeling their interest in items over time. Most of the existing works on sequential recommendation model users’ dynamic interest in specific items while overlooking users’ static interest revealed by some static attribute information of items, e.g., category, brand. Moreover, existing works often only consider the positive excitation of a user’s historical interactions on his/her next choice on candidate items while ignoring the commonly existing negative excitation, resulting in insufficiently modeling dynamic interest. The overlook of static interest and negative excitation will lead to incomplete interest modeling and thus impedes the recommendation performance. To this end, in this paper, we propose modeling both static interest and negative excitation for dynamic interest to further improve the recommendation performance. Accordingly, we design a novel Static-Dynamic Interest Learning (SDIL) framework featured with a novel Temporal Positive and Negative Excitation Modeling (TPNE) module for accurate sequential recommendation. TPNE is specially designed for comprehensively modeling dynamic interest based on temporal positive and negative excitation learning. Extensive experiments on three real-world datasets show that SDIL can effectively capture both static and dynamic interest and outperforms state-of-the-art baselines.

Sequential recommendations aim to capture users’ preferences from their historical interactions so as to predict the next item that they will interact with. Sequential recommendation methods usually assume that all items in a user’s historical interactions reflect her/his preferences and transition patterns between items. However, real-world interaction data is imperfect in that (i) users might erroneously click on items, i.e., so-called misclicks on irrelevant items, and (ii) users might miss items, i.e., unexposed relevant items due to inaccurate recommendations.

Predicting the next interaction of a short-term interaction session is a challenging task in session-based recommendation. Almost all existing works rely on item transition patterns, and neglect the impact of user historical sessions while modeling user preference, which often leads to non-personalized recommendation. Additionally, existing personalized session-based recommenders capture user preference only based on the sessions of the current user, but ignore the useful item-transition patterns from other user's historical sessions. To address these issues, we propose a novel Heterogeneous Global Graph Neural Networks (HG-GNN) to exploit the item transitions over all sessions in a subtle manner for better inferring user preference from the current and historical sessions. To effectively exploit the item transitions over all sessions from users, we propose a novel heterogeneous global graph that contains item transitions of sessions, user-item interactions and global co-occurrence items. Moreover, to capture user preference from sessions comprehensively, we propose to learn two levels of user representations from the global graph via two graph augmented preference encoders. Specifically, we design a novel heterogeneous graph neural network (HGNN) on the heterogeneous global graph to learn the long-term user preference and item representations with rich semantics. Based on the HGNN, we propose the Current Preference Encoder and the Historical Preference Encoder to capture the different levels of user preference from the current and historical sessions, respectively. To achieve personalized recommendation, we integrate the representations of the user current preference and historical interests to generate the final user preference representation. Extensive experimental results on three real-world datasets show that our model outperforms other state-of-the-art methods.

User preferences are usually dynamic in real-world recommender systems, and a user»s historical behavior records may not be equally important when predicting his/her future interests. Existing recommendation algorithms -- including both shallow and deep approaches -- usually embed a user»s historical records into a single latent vector/representation, which may have lost the per item- or feature-level correlations between a user»s historical records and future interests. In this paper, we aim to express, store, and manipulate users» historical records in a more explicit, dynamic, and effective manner. To do so, we introduce the memory mechanism to recommender systems. Specifically, we design a memory-augmented neural network (MANN) integrated with the insights of collaborative filtering for recommendation. By leveraging the external memory matrix in MANN, we store and update users» historical records explicitly, which enhances the expressiveness of the model. We further adapt our framework to both item- and feature-level versions, and design the corresponding memory reading/writing operations according to the nature of personalized recommendation scenarios. Compared with state-of-the-art methods that consider users» sequential behavior for recommendation, e.g., sequential recommenders with recurrent neural networks (RNN) or Markov chains, our method achieves significantly and consistently better performance on four real-world datasets. Moreover, experimental analyses show that our method is able to extract the intuitive patterns of how users» future actions are affected by previous behaviors.

Sequential recommendation aims to predict the next item based on users' historical interaction behaviors.Most existing methods enhance representations by generating different views of user sequences via random augmentation and maximizing their mutual information.However, user interaction behaviors are influenced by underlying intents, and random augmentation may alter the original intents of users.Furthermore, the embedding representations based on Transformer exhibit high similarity, which can obscure short-term intents.To address these challenges, we propose an approach called Diffusion Model-based Intent Contrastive Learning for Sequential Recommendation (DICLR).DICLR leverages contextual information to guide the diffusion model in generating positive samples that are consistent in intent, and uses the Gated Linear Unit (GLU) to enhance context understanding and denoising effects.Additionally, DICLR uses the Fourier transform to separate the frequency-domain features of user behavior, thereby capturing both long-term and short-term user intents.DICLR integrates the recommendation task with intent contrastive learning and diffusion tasks within a multi-task learning framework.Experiments conducted on four datasets demonstrate the superiority of the proposed model.

Sequential recommendation (SR) predicts the next item by analyzing users' historical interaction data.However, it may lead to the homogenization of recommended items due to reliance on historical data.User sequences may also reflect latent intents influenced by factors such as seasonality or age.To address these challenges, we propose a model named Stochastic Embeddings in Multi-Intent Contrastive Learning for Sequential Recommendation (SICLR).SICLR uses mean and covariance embeddings to represent stochastic embeddings of sequential intents, leveraging Wasserstein distance to measure the distance between distributions.This approach can capture the uncertainty and non-stationarity in the distributions of intents.In addition, SICLR employs a multi-perspective of local and global viewpoints for intent contrastive learning, which enhances the capture of latent intent discrepancies and interest drifts.Furthermore, SICLR also integrates the recommendation task with local and global intent contrastive learning tasks.This can improve the recommendation performance of the model.The proposed model demonstrates improved performance metrics across three realworld datasets.It shows robustness and generalization under different training parameters.

In recommendation systems, items of interest are often classified into categories such as genres of movies. Existing research has shown that diversified recommendations can improve real user experience. However, most existing methods do not consider the fact that users’ levels of interest (i.e., user preferences) in different categories usually vary, and such user preferences are not reflected in the diversified recommendations. We propose an algorithm that considers user preferences for different categories when recommending diversified results, and refer to this problem as personalized recommendation diversification. In the proposed algorithm, a model that captures user preferences for different categories is optimized jointly toward both relevance and diversity. To provide the proposed algorithm with informative training labels and effectively evaluate recommendation diversity, we also propose a new personalized diversity measure. The proposed measure overcomes limitations of existing measures in evaluating recommendation diversity: existing measures either cannot effectively handle user preferences for different categories, or cannot evaluate both relevance and diversity at the same time. Experiments using two real-world datasets confirm the superiority of the proposed algorithm, and show the effectiveness of the proposed measure in capturing user preferences.

With the popularity of location-based social networks such as Weibo and Twitter, there are many records of points of interest (POIs) showing when and where people have visited certain locations. From these records, next POI recommendation suggests the next POI that a target user might want to visit based on their check-in history and current spatio-temporal context. Current next POI recommendation methods mainly apply different deep learning models to capture user preferences by learning the nonlinear relations between POIs and user preference and pay little attention to mining or using the information that explicitly reflects user preference. In contrast, this paper proposes to utilize data that explicitly reflect user preference and include these data in a deep learning-based process to better capture user preference. Based on the self-attention network, this paper utilizes the attributes of the month of the check-ins and the categories of check-ins during this time, which indicate the periodicity of the user’s work and life and can reflect the habits of users. Moreover, considering that distance has a significant impact on a user’s decision of whether to visit a POI, we used a filter to remove candidate POIs that were more than a certain distance away when recommending the next POIs. We use check-in data from New York City (NYC) and Tokyo (TKY) as datasets, and experiments show that these improvements improve the recommended performance of the next POI. Compared with the state-of-the-art methods, the proposed method improved the recall rate by 7.32% on average.