DUAL TRANSFORMER ENCODERS FOR SESSION-BASED RECOMMENDATION

Session-based recommendation aims to predict the next item that is most likely to be clicked by an anonymous user, based on his/her clicking sequence within one visit. It becomes an essential function of many recommender systems since it protects privacy. However, as the accumulated session records keep increasing, it becomes challenging to model the user interests since they would drift when the time span is large. Efforts have been devoted to handling dynamic user interests by modeling all historical sessions at one time or conducting offline retraining regularly. These solutions are far from practical requirements in terms of efficiency and capturing timely user interests. To this end, we propose a memory-efficient framework - TASRec. It constructs a graph for each day to model the relations among items. Thus, the same item on different days could have different neighbors, corresponding to the drifting user interests. We design a tailored graph neural network to embed this dynamic graph of items and learn temporal augmented item representations. Based on this, we leverage a sequential neural architecture to predict the next item of a given sequence. Experiments on real-world datasets demonstrate that TASRec outperforms state-of-the-art session-based recommendation methods.

Session-based recommendation (SBR) nowadays plays a vital role in many online services, aiming to predict users' next action based on anonymous sessions. Recent research of GNNs-based methods models a session as a graph via investigating complex transitions of items in a session. However, these methods do not consider sequential information of the session when aggregating item embeddings to form a session-level embedding. Most methods consider not all previous but the last one item as the interest of a user, which restricts the performance of the model. To address this problem, we propose a model named Sequence-Aware Graph Neural Network (SA-GNN) for session-based recommendation. In SA-GNN, we design a sequence-aware attention to adaptively weigh the previous items to generate a session-level embedding, which greatly improves the representation ability of the model. Also, to improve the representation ability of the item embeddings, SA-GNN harnesses the power of self-attention within the GNN layer to capture both transitions between adjacent items and long-range dependencies among all items in a session. In empirical evaluations on three public recommendation datasets, our method consistently outperforms an extensive of state-of-the-art session-based recommendation methods.

Gated graph spiking neural P network for session-based recommendation

Session-based recommendation aims to predict user preferences based on anonymous behavior sequences. Recent research on session-based recommendation systems has mainly focused on utilizing attention mechanisms on sequential patterns, which has achieved significant results. However, most existing studies only consider individual items in a session and do not extract information from continuous items, which can easily lead to the loss of information on item transition relationships. Therefore, this paper proposes a session-based recommendation algorithm (SGT) based on Gated Recurrent Unit (GRU) and Transformer, which captures user interests by learning continuous items in the current session and utilizes all item transitions on sessions in a more refined way. By combining short-term sessions and long-term behavior, user dynamic preferences are captured. Extensive experiments were conducted on three session-based recommendation datasets, and compared to the baseline methods, both the recall rate Recall@20 and the mean reciprocal rank MRR@20 of the SGT algorithm were improved, demonstrating the effectiveness of the SGT method.

Session-based recommendation uses past clicks and interaction sequences from anonymous users to predict the next item most likely to be clicked. Predicting the user’s subsequent behavior in online transactions becomes a problem mainly due to the lack of user information and limited behavioral information. Existing methods, such as recurrent neural network (RNN)-based models that model user’s past behavior sequences and graph neural network (GNN)-based models that capture potential relationships between items, miss different time intervals in the past behavior sequence and can only capture certain types of user interest patterns due to the characteristics of neural networks. Graphic models created to improve the current session reduce the model’s success due to the addition of irrelevant items. Moreover, attention mechanisms in recent approaches have been insufficient due to weak representations of users and products. In this study, we propose a model based on the combination of skip-gram and transformer (SkipGT) to solve the above-mentioned drawbacks in session-based recommendation systems. In the proposed method, skip-gram both captures chained user interest in the session thread through item-specific subreddits and learns complex interaction information between items. The proposed method captures short-term and long-term preference representations to predict the next click with the help of a transformer. The transformer in our proposed model overcomes many limitations in turn-based models and models longer contextual connections between items more effectively. In our proposed model, by giving the transformer trained item embeddings from the skip-gram model as input, the transformer has better performance because it does not learn item representations from scratch. By conducting extensive experiments with three real-world datasets, we confirm that SkipGT significantly outperforms state-of-the-art solutions with an average MRR score of 5.58%.

With the rapid development of recommendation models and device computing power, device-based recommendation has become an important research area due to its better real-time performance and privacy protection. Previously, Transformer-based sequential recommendation models have been widely applied in this field because they outperform Recurrent Neural Network (RNN)-based recommendation models in terms of performance. However, as the length of interaction sequences increases, Transformer-based models introduce significantly more space and computational overhead compared to RNN-based models, posing challenges for device-based recommendation. To balance real-time performance and high performance on devices, we propose Device-Cloud Collaborative Correction Framework for On-Device Recommendation (CoCorrRec). CoCorrRec uses a self-correction network (SCN) to correct parameters with extremely low time cost. By updating model parameters during testing based on the input token, it achieves performance comparable to current optimal but more complex Transformer-based models. Furthermore, to prevent SCN from overfitting, we design a global correction network (GCN) that processes hidden states uploaded from devices and provides a global correction solution. Extensive experiments on multiple datasets show that CoCorrRec outperforms existing Transformer-based and RNN-based device recommendation models in terms of performance, with fewer parameters and lower FLOPs, thereby achieving a balance between real-time performance and high efficiency. Code is available at https: //github.com/Yuzt-zju/CoCorrRec.

Session-based recommendation aims to predict a user's actions at the next timestamp based on anonymous sessions. Previous work mainly focuses on the transition relationship between items that the user interacted with during an ongoing session. They generally fail to pay enough attention to the importance of the items involved in these interactions in terms of their relevance to user's main intent. In this paper, we propose a Session-based Recommendation approach with an Importance Extraction Module, i.e., SR-IEM, that considers both a user's long-term and recent behavior in an ongoing session. We employ a modified self-attention mechanism to estimate item importance in a session, which is then used to predict user's long-term preference. Item recommendations are produced by combining the user's long-term preference and their current interest as conveyed by the last item they interacted with. Comprehensive experiments are conducted on two publicly available benchmark datasets. The proposed SR-IEM model outperforms start-of-the-art baselines in terms of Recall and MRR for the task of session-based recommendation. In addition, compared to state-of-the-art models, SR-IEM has a reduced computational complexity.

This study presents a transformer-based power prediction model for Raman-amplified wavelength division multiplexing (WDM) systems. Leveraging the transformer's self-attention mechanism, the model effectively captures global channel interactions, achieving a maximum prediction error of only 0.122 dB in a 200 km bidirectional Raman system with 47 channels and 12 pumps. The differential evolution (DE) technique was employed to optimize the hyperparameters of the models, further enhancing their performance. The comparative analysis was conducted to evaluate the performance of the two models under their respective optimal configurations. The model demonstrates robust generalization under varying pump powers and surpasses convolutional neural network (CNN)-based approaches in accuracy and interpretability.

To evaluate the diagnostic performance of a Transformer-based deep learning model that integrates real-world laboratory test indicators for differential diagnosis of ovarian cancer. The clinical data and 99 laboratory test indicators were retrospectively collected from patients with ovarian cancer and benign ovarian lesions admitted to Department of Obstetrics and Gynecology of Tongji Hospital between January 1, 2012 and April 4, 2021. A feature selection algorithm based on ANOVA F-test was used on the training set to identify 20 key features. Each case was then converted into a unified embedded vector using a tabular data Transformer. An improved stacked Transformer model was then trained to encode these feature vectors. The proposed model was compared with multiple traditional machine learning methods. The evaluation metrics included the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity. Five-fold cross-validation was performed to assess the generalization ability and robustness of the model. Five-fold cross-validation showed that the Transformer-based deep learning model achieved the best performance in predicting ovarian cancer with an AUC of 0.931, an accuracy of 0.813, a sensitivity of 0.833, and a specificity of 0.865. The proposed Transformer-based model demonstrates high accuracy and generalization capability in predicting ovarian cancer, and may thus offer a assistance in clinical diagnosis of ovarian tumors.

Top-$N$ sequential recommendation models each user as a sequence of items interacted in the past and aims to predict top-$N$ ranked items that a user will likely interact in a `near future'. The order of interaction implies that sequential patterns play an important role where more recent items in a sequence have a larger impact on the next item. In this paper, we propose a Convolutional Sequence Embedding Recommendation Model (\emph{Caser}) as a solution to address this requirement. The idea is to embed a sequence of recent items into an `image' in the time and latent spaces and learn sequential patterns as local features of the image using convolutional filters. This approach provides a unified and flexible network structure for capturing both general preferences and sequential patterns. The experiments on public datasets demonstrated that Caser consistently outperforms state-of-the-art sequential recommendation methods on a variety of common evaluation metrics.

Recommending relevant items of interest for a user is the main purpose of the recommendation system. In the past, those systems achieve the recommended list based on long-term user profiles. However, personal data privacy is becoming a big challenge recently. Thus, the recommendation system needs to reduce the dependence on user profiles while preserving high accuracy on the recommendation. Session-based recommendation is a recently proposed approach for the recommendation system to overcome the issue of user profiles dependency. The relevance of the problem is quite high and has triggered interest among researchers in observing the activities of users. It increased several proposals for session-based recommendation algorithms that aim to predict the next actions. In this paper, we would like to compare the performance of such algorithms by using various datasets and evaluation metrics. A deep learning approach named GRU4REC (Hidasi et al. in Session-based recommendations with recurrent neural networks, 2015) and simpler methods are included in our comparison. Real-world datasets from three different domains are included in our experiment. Our experiments reveal that in some cases of numerous unpopular items dataset, GRU4REC’s performance is lower than expected. However, its performance is significantly increased after applying our proposed sampling method. Therefore, our obtained results suggested that there is still room for improving deep learning session-based recommendation algorithms.

Session-based Recommendation (SBR) explores dynamic user interests based on short anonymous sessions. The performance of traditional ID-collaborative SBR models is constrained by the limited session length. To this end, we focus on Multi-modal Session-based Recommendation (MMSBR), which aims to leverage modality knowledge to promote short-term user interest modeling. Previous studies cannot solve the MMSBR problem well due to issues as: (I1) Modal-invariant and specific dependencies are difficult to disentangle and unify. (I2) Multiple modal transitions produce mutual effects. (I3) Modal-aware personalized preference discrepancy exists. Thus, we propose a modality-aware diffusion-based framework MDSD with subspace disentanglement, which decouples multi-modal collaborations and provides a new paradigm that integrates modality representation generation into next-item prediction. We first employ modality-disentangled consistency graphs based on affinity semantics to extract the unified modal subspace. Then we design a cross-modal contrastive attention fusion that explores cross-modality interplay, enhancing modal alignment and consistency. To interpret personalized interest discrepancy, we realize the preference-guided multi-modal diffusion, which combines modality-aware interest generation with prediction based on collaborative signals, thus providing comprehensive recommendation results. Extensive experiments on real-world datasets demonstrate the effectiveness of MDSD.

Session-based recommendations aim to predict a user’s next click based on the user’s current and historical sessions, which can be applied to shopping websites and APPs. Existing session-based recommendation methods cannot accurately capture the complex transitions between items. In addition, some approaches compress sessions into a fixed representation vector without taking into account the user’s interest preferences at the current moment, thus limiting the accuracy of recommendations. Considering the diversity of items and users’ interests, a personalized interest attention graph neural network (PIA-GNN) is proposed for session-based recommendation. This approach utilizes personalized graph convolutional networks (PGNN) to capture complex transitions between items, invoking an interest-aware mechanism to activate users’ interest in different items adaptively. In addition, a self-attention layer is used to capture long-term dependencies between items when capturing users’ long-term preferences. In this paper, the cross-entropy loss is used as the objective function to train our model. We conduct rich experiments on two real datasets, and the results show that PIA-GNN outperforms existing personalized session-aware recommendation methods.

Ontology learning from unstructured text has become a critical task for knowledge-driven applications in Big Data and Artificial Intelligence. While significant advances have been made in the automatic extraction of concepts and relations using neural and Transformer-based models, the generation of formal Description Logic axioms required for constructing logically consistent and computationally tractable ontologies remains largely underexplored. This paper puts forward a novel pipeline for automated axiom generation through schema mapping. Our paper introduces three key innovations: a deterministic mapping framework that guarantees logical consistency (unlike stochastic Large Language Models); guaranteed formal consistency verified by OWL reasoners (unaddressed by prior statistical methods); and a transparent, scalable bridge from neural extractions to symbolic logic, eliminating manual post-processing. Technically, the pipeline builds upon the outputs of a Transformer-based fusion model for joint concept and relation extraction. We then map lexical relational phrases to formal ontological properties through a lemmatization-based schema alignment step. Entity typing and hierarchical induction are then employed to infer class structures, as well as domain and range constraints. Using RDFLib and structured data processing, we transform the extracted triples into both assertional (ABox) and terminological (TBox) axioms expressed in Description Logic. Experimental evaluation on benchmark datasets (Conll04 and NYT) demonstrates the efficacy of the approach, with expert validation showing high acceptance rates (>95%) and reasoners confirming zero inconsistencies. The pipeline thus establishes a reliable, scalable foundation for automated ontology learning, advancing the field from extraction to formally verifiable knowledge base construction.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.