Capturing User Interests from Data Streams for Continual Sequential Recommendation

Sequential recommender systems, as a specialized branch of recommender systems that can capture users’ dynamic preferences for more accurate and timely recommendations, have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Federated Sequential Recommender Systems (FedSeqRecs), in which a public sequential recommender model is shared and frequently transmitted between a central server and clients to achieve collaborative learning. Although these solutions mitigate user privacy to some extent, they present two significant limitations that affect their practical usability: (1) They require a globally shared sequential recommendation model. However, in real-world scenarios, the recommendation model constitutes a critical intellectual property for platform and service providers. Therefore, service providers may be reluctant to disclose their meticulously developed models. (2) The communication costs are high as they correlate with the number of model parameters. This becomes particularly problematic as the current FedSeqRec will be inapplicable when sequential recommendation marches into a large language model era. To overcome the above challenges, this article proposes a parameter transmission-free federated sequential recommendation framework (PTF-FSR), which ensures both model and data privacy protection to meet the privacy needs of service providers and system users alike. Furthermore, since PTF-FSR only transmits prediction results under privacy protection, which are independent of model sizes, this new federated learning architecture can accommodate more complex and larger sequential recommendation models. Extensive experiments conducted on three widely used recommendation datasets, employing various sequential recommendation models from both ID-based and ID-free paradigms, demonstrate the effectiveness and generalization capability of our proposed framework. To facilitate future research in this direction, we release our code at https://github.com/hi-weiyuan/PTF-FSR .

[Objective] Construct a spatial and temporal integration of the Qin Dynasty historical celebrities knowledge map, in order to lay the foundation for information retrieval and visualization when customers learn the Qin Dynasty historical knowledge. [Methods] We use the “seven-step method” and the Protégé tool to construct the ontology model of the Qin Dynasty historical celebrities with spatial and temporal correlation from the top down, and then construct the spatial and temporal fusion knowledge map instance through knowledge extraction, attribute fusion and knowledge storage, and finally implant the model into the network, so that we can provide the users with an accurate, intuitive, high-capacity, fast, and in-depth spatial and temporal correlation historical knowledge map. Finally, the model is implanted into the network to provide users with accurate, intuitive, high-capacity, fast, and deeply spatio-temporally related popularization services. [Results] The constructed knowledge graph is stored and visualized by Neo4j, supports Cypher language for knowledge query, and realizes accurate promotion in three ways: webpage, SVG vector image of knowledge graph, and short video. [Limitations] As the historical knowledge is more cumbersome, data acquisition takes a long time while ensuring correctness. [Conclusion] Compared with traditional knowledge mapping, knowledge mapping based on spatio-temporal fusion integrates the spatio-temporal factors associated with events into the model, and the events shown in the mapping are more abundant, so that the users can understand the events and their spatio-temporal backgrounds clearly, accurately, intuitively, and quickly through various network terminals, and it is more efficient and more acceptable than the traditional popularization methods.

Sequential Recommendation (SR) is an important scenario in recommendation tasks. Sequential recommendations model the sequential pattern between item-item or user-item based on a user's recent activity in a time series to predict their next preference. However, existing methods are based only on the conventional Graph Neural Networks (GNN) as a model architecture for adaptive fine-tuning of specific SR tasks. To get better recommendation results, more advanced GNNs can be used as the network architecture of the SR method. This paper introduces graph transformer, a combination of GNN and a good sequential task processing model. Then a cross-sectional comparison is made with the current SR method model and its suitability for application in SR tasks is discussed. The comparison shows that the graph transformer is similar in principle and structure to the current SR models, and requires the addition of some adaptive components to be applied in SR tasks. The superior performance after application can be demonstrated from the results data of the Benchmarking-GNNs and Long-Range Graph Benchmark on the models.

Approximately 10% of all newborns require resuscitation at birth. Training healthcare providers in standardised formal neonatal resuscitation training (SFNRT) programmes may improve neonatal outcomes. Substantial healthcare resources are expended on SFNRT. To determine whether SFNRT programmes reduce neonatal mortality and morbidity, improve acquisition and retention of knowledge and skills, or change teamwork and resuscitation behaviour. We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, PREMEDLINE, EMBASE, CINAHL, Web of Science and the Oxford Database of Perinatal Trials, ongoing trials and conference proceedings in April 2014 and updated in March 2015. Randomised or quasi-randomised trials including cluster-randomised trials, comparing a SFNRT with no SFNRT, additions to SFNRT or types of SFNRT, and reporting at least one of our specified outcomes. Two authors extracted data independently and performed statistical analyses including typical risk ratio (RR), risk difference (RD), mean difference (MD), and number needed to treat for an additional beneficial outcome (NNTB) or an additional harmful outcome (NNTH) (all with 95% confidence intervals (CI)). We analysed cluster-randomised trials using the generic inverse variance and the approximate analysis methods. We identified two community-based and three manikin-based trials that assessed the effect of SFNRT compared with no SFNRT. Very low quality evidence from one study suggested improvement in acquisition of knowledge (RR 5.96, 95% CI 3.60 to 9.87) and skills (RR 170, 95% CI 10.8 to 2711) and retention of knowledge (RR 3.60, 95% CI 2.43 to 5.35) and the other study suggested improvement in resuscitation and behavioural scores.We identified three community-based cluster-randomised trials in developing countries comparing SFNRT with basic resuscitation training (Early Newborn Care). In this setting, there was moderate quality evidence that SFNRT decreased early neonatal mortality (typical RR 0.88, 95% CI 0.78 to 1.00; 3 studies, 66,162 neonates) and when analysed by the approximate analysis method (typical RR 0.85, 95% CI 0.75 to 0.96; RD -0.0044, 95% CI -0.0082 to -0.0006; NNTB 227, 95% CI 122 to 1667). Low quality evidence from one trial showed that SFNRT may decrease 28-day mortality (typical RR 0.55, 95% CI 0.33 to 0.91) but the effect on late neonatal mortality was more uncertain (typical RR 0.47, 95% CI 0.20 to 1.11). None of our a priori defined neonatal morbidities were reported. We did not identify any randomised studies in the developed world.We identified two trials that compared SFNRT with team training to SFNRT. Teamwork training of physician trainees with simulation may increase any teamwork behaviour (assessed by frequency) (MD 2.41, 95% CI 1.72 to 3.11) and decrease resuscitation duration (MD -149.54, 95% CI -214.73 to -84.34) but may lead to little or no difference in Neonatal Resuscitation Program (NRP) scores (MD 1.40, 95% CI -2.02 to 4.82; 98 participants, low quality evidence).We identified two trials that compared SFNRT with booster courses to SFNRT. It is uncertain whether booster courses improve retention of resuscitation knowledge (84 participants, very low quality evidence) but may improve procedural and behavioural skills (40 participants, very low quality evidence).We identified two trials on decision support tools, one on a cognitive aid that did not change resuscitation scores and the other on an electronic decision support tool that improved the frequency of correct decision making on positive pressure ventilation, cardiac compressions and frequency of fraction of inspired oxygen (FiO2) adjustments (97 participants, very low quality evidence). SFNRT compared to basic newborn care or basic newborn resuscitation, in developing countries, results in a reduction of early neonatal and 28-day mortality. Randomised trials of SFNRT should report on neonatal morbidity including hypoxic ischaemic encephalopathy and neurodevelopmental outcomes. Innovative educational methods that enhance knowledge and skills and teamwork behaviour should be evaluated.

In the field of sequential recommendation, deep learning--(DL) based methods have received a lot of attention in the past few years and surpassed traditional models such as Markov chain-based and factorization-based ones. However, there is little systematic study on DL-based methods, especially regarding how to design an effective DL model for sequential recommendation. In this view, this survey focuses on DL-based sequential recommender systems by taking the aforementioned issues into consideration. Specifically, we illustrate the concept of sequential recommendation, propose a categorization of existing algorithms in terms of three types of behavioral sequences, summarize the key factors affecting the performance of DL-based models, and conduct corresponding evaluations to showcase and demonstrate the effects of these factors. We conclude this survey by systematically outlining future directions and challenges in this field.

Recent advancements in generative models have positioned them as one of the principal tools for sequential recommendation due to their exceptional sample diversity and generalization capabilities. Among these, diffusion model-based sequential recommenders have achieved remarkable success. However, most existing approaches still face critical challenges, resulting in suboptimal generation quality: (1) They fail to leverage multimodal knowledge for constructing item representations with well-structured distributional characteristics and semantically enriched information; (2) They predominantly rely on discrete diffusion processes, leading to high error accumulation, reduced time efficiency, and constrained controllability in generative sampling. To mitigate these challenges, we propose LSGM4Rec, a novel framework that integrates Large Language Models (LLMs) with advanced multimodal encoding models to establish multimodal fusion embeddings for items. This design ensures distinct distributional characteristics while enabling the incorporation of semantically rich modal features into guidance condition. Furthermore, we pioneer the stochastic differential equations (SDEs) for recommendation, facilitating smooth transitions between data distributions and enabling optimal trade-off between sampling efficiency and generation quality. Extensive experiments on three datasets demonstrate that LSGM4Rec outperforms existing state-of-the-art sequential recommendation methods.

Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called Beyond Self-Attention for Sequential Recommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.

Sequential recommendation is a popular task in academic research and close to real-world application scenarios, where the goal is to predict the next action(s) of the user based on his/her previous sequence of actions. In the training process of recommender systems, the loss function plays an essential role in guiding the optimization of recommendation models to generate accurate suggestions for users. However, most existing sequential recommendation techniques focus on designing algorithms or neural network architectures, and few efforts have been made to tailor loss functions that fit naturally into the practical application scenario of sequential recommender systems. Ranking-based losses, such as cross-entropy and Bayesian Personalized Ranking (BPR) are widely used in the sequential recommendation area. We argue that such objective functions suffer from two inherent drawbacks: i) the dependencies among elements of a sequence are overlooked in these loss formulations; ii) instead of balancing accuracy (quality) and diversity, only generating accurate results has been over emphasized. We therefore propose two new loss functions based on the Determinantal Point Process (DPP) likelihood, that can be adaptively applied to estimate the subsequent item or items. The DPP-distributed item set captures natural dependencies among temporal actions, and a quality vs. diversity decomposition of the DPP kernel pushes us to go beyond accuracy-oriented loss functions. Experimental results using the proposed loss functions on three real-world datasets show marked improvements over state-of-the-art sequential recommendation methods in both quality and diversity metrics.

Adaptive self-supervised learning for sequential recommendation

Sequential recommendation has been widely studied in the recommendation domain since it can capture users’ temporal preferences and provide more accurate and timely recommendations. To address user privacy concerns, the combination of federated learning and sequential recommender systems (FedSeqRec) has gained growing attention. Unfortunately, the performance of FedSeqRec is still unsatisfactory because the models used in FedSeqRec have to be lightweight to accommodate communication bandwidth and clients’ on-device computational resource constraints. Recently, large language models (LLMs) have exhibited strong transferable and generalized language understanding abilities and therefore, in the NLP area, many downstream tasks now utilize LLMs as a service to achieve superior performance without constructing complex models. Inspired by this successful practice, we propose a generic FedSeqRec framework, FELLAS, which aims to enhance FedSeqRec by utilizing LLMs as an external service. Specifically, FELLAS employs an LLM server to provide both item-level and sequence-level representation assistance. The item-level representation service is queried by the central server to enrich the original ID-based item embedding with textual information, while the sequence-level representation service is accessed by each client. However, invoking the sequence-level representation service requires clients to send sequences to the external LLM server. To safeguard privacy, we implement \(d_{\mathcal{X}}\) -privacy satisfied sequence perturbation, which protects clients’ sensitive data with guarantees. Additionally, a contrastive learning-based method is designed to transfer knowledge from the noisy sequence representation to clients’ sequential recommendation models. Furthermore, to empirically validate the privacy protection capability of FELLAS, we propose two interacted item inference attacks, considering the threats posed by the LLM server and the central server acting as curious-but-honest adversaries in cooperation. Extensive experiments conducted on three datasets with two widely used sequential recommendation models demonstrate the effectiveness and privacy-preserving capability of FELLAS.

Recent years have witnessed great success in deep learning-based sequential recommendation (SR), which can provide more timely and accurate recommendations. One of the most effective deep SR architectures is to stack high-performance residual blocks, e.g., prevalent self-attentive and convolutional operations, for capturing long- and short-range dependence of sequential behaviors. By carefully revisiting previous models, we observe: 1) simple architecture modification of gating each residual connection can help us train deeper SR models and yield significant improvements; 2) compared with self-attention mechanism, stacking of convolution layers also can cover each item of the whole sequential behaviors and achieve competitive or even superior performance.

In previous chapters we have considered the nature, conditions, and underlying properties of meaningful verbal reception learning and retention. We have also concluded that various specifiable properties of existing cognitive structure are the most important single class of factors influencing the acquisition, retention, and transferability of knowledge. Now we are ready to operationalize these cognitive structure variables, to cite and consider available research evidence regarding their effects on subject-matter learning and retention, and to suggest how they may be optimally manipulated and applied to instructional practices so as to maximize students’ acquisition and retention of knowledge.

Mothers' beliefs about the role of parents and of teachers in the child's acquisition of knowledge in five domains were investigated in a sample of families from Tanzania and from the United States. Tanzanian and US mothers differed in their views of the importance of parents and teachers in the child's acquisition of language, sociopolitical, and historical knowledge, but not in the domains of science or mathematics knowledge. These differences were linked in predictable ways to the cultural traditions of the two countries. Teacher ratings of children's classroom behaviours were also examined across cultures and in relation to mothers' beliefs. Tanzanian children were rated as more dependent than the US children were, and as less independent, creative, and intelligent. United States and Tanzanian mothers' beliefs about the teachers' role in children's historical knowledge showed the most consistent relationship to children's classroom behaviours. For Tanzanian children's classroom behaviour, several other maternal beliefs were related to classroom ratings in addition to beliefs relative to the acquisition of historical knowledge.

Deep learning and symbolic learning are two frequently employed methods in Sequential Recommendation (SR). Recent neural-symbolic SR models demonstrate their potential to enable SR to be equipped with concurrent perception and cognition capacities. However, neural-symbolic SR remains a challenging problem due to open issues like representing users and items in logical reasoning. In this paper, we combine the Deep Neural Network (DNN) SR models with logical reasoning and propose a general framework named Sequential Recommendation with Probabilistic Logical Reasoning (short for SR-PLR). This framework allows SR-PLR to benefit from both similarity matching and logical reasoning by disentangling feature embedding and logic embedding in the DNN and probabilistic logic network. To better capture the uncertainty and evolution of user tastes, SR-PLR embeds users and items with a probabilistic method and conducts probabilistic logical reasoning on users' interaction patterns. Then the feature and logic representations learned from the DNN and logic network are concatenated to make the prediction. Finally, experiments on various sequential recommendation models demonstrate the effectiveness of the SR-PLR. Our code is available at https://github.com/Huanhuaneryuan/SR-PLR.

The sequential recommendation system predicts user's future operations based on their historical interaction information and achieves good performance in recent work. However, when applying to the task of recommendation in software crowdsourcing platform, the accuracy of the previous recommendation models is significantly reduced because of the sparse interactive data and dynamic item list in the platform. The Generative Ranking based Sequential Recommendation Model (GRS) is proposed to solve the problems mentioned above. The generative layer is introduced into a translation-based recommendation model to prevent overfitting problem. By generating latent vector in feature space, the interpolation between encoded points is highly reduced and the model is adapted to achieve better performance by embedding auxiliary features into the model. The efficiency and feasibility of the model is validated by the experiment in different datasets extracted from crowdsourcing platforms.