A Contextual Recurrent Collaborative Filtering framework for modelling sequences of venue checkins

Recurrent neural networks (RNN) are ubiquitous computing systems for sequences and multivariate time-series data. While several robust RNN architectures are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose treating RNN as dynamical systems and correlating hyperparameters with accuracy through Lyapunov spectral analysis, a methodology designed explicitly for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its Latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the Latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for predicting the accuracy to which RNN would converge when training is complete. We conclude that the representation of RNN through the Lyapunov spectrum, along with AeLLE, provides a novel method for the organization and interpretation of variants of RNN architectures.

This paper examines and evaluates the effectiveness of recurrent neural network (RNN) architectures for natural language knowledge. The paper starts by imparting historical data on recurrent neural networks and their software to various responsibilities. It then goes on to talk about diverse RNN architectures, such as lengthy quick-time period reminiscence (LSTM), Gated Recurrent Unit (GRU) and Bonior Recurrent Neural network (BONN). The paper then focuses on the efficacy of each RNN structure for textual content expertise obligations. A sequence of experiments is carried out to degree the overall performance of the one-of-a-kind fashions, the use of 3 preferred datasets-Wiki text, Squad and IMDB, as well as two newly created datasets. The paper analyses the empirical consequences of the various RNN architectures and finds that LSTM fashions are normally satisfactory performers, although BONN architectures can also be aggressive in some cases. The paper additionally looks at numerous techniques to enhance performance, which include pre-processing and hyper-parameter tuning. The paper concludes that, whilst RNN architectures are powerful for herbal language know-how responsibilities, the exact architecture and parameters can also need to be chosen cautiously, relying on the project and dataset to hand.

With the advances in technology and smart devices, more and more attention has been paid to model spatial correlations, temporal dynamics, and friendship influence over point-of-interest (POI) checkins. Besides directly capturing general user's checkin behavior, existing works mostly highlight the intrinsic feature of POIs, i.e., spatial and temporal dependency. Among them, the family of methods based on Markov chain can capture the instance-level interaction between a pair of POI checkins, while recurrent neural network (RNN) based approaches (state-of-the-art) can deal with flexible length of checkin sequence. However, the former is not good at capturing high-order POI transition dependency, and the latter cannot distinguish the exact contribution of each POI in a historical checkin sequence. Moreover, in recurrent neural networks, local and global information is propagated along the sequence through one bottleneck i.e., hidden states only.In this work, we design a novel model to enforce contextual constraints on sequential data by designing a spatial and temporal attention mechanisms over recurrent neural network that leverages the importance of POIs visited by users in given time interval and geographical distance in successive checkins. Attention mechanism helps us to learn which POIs bounded by time difference and spatial distance in user checkin history are important for the prediction of next POI. Moreover, we also consider periodicity and friendship influence in our model design. Experimental results on two real location based social networks Gowalla, and BrightKite show that our proposed method outperforms the existing state-of-the-art deep neural network methods for next POI prediction and understanding user transition behavior. We also analyze the sensitivity of parameters including context window for capturing sequential effect, temporal context window for estimating temporal attention and spatial context window for estimating spatial attention respectively.

The objective of the next POI recommendation is using the historical check-in sequences of users to learn the preferences and habits of users, providing a list of POIs that users will be inclined to visit next. Then, there are some limitations in existing POI recommendation algorithms. On the one hand, after obtaining the user’s preferences for the current period, if we consider the entire historical check-in sequence, including future check-in information, it is susceptible to the influence of noisy data, thereby reducing the accuracy of recommendations. On the other hand, the current methods generally rely on modeling long- and short-term preferences within a fixed time window, which possibly leads to an inability to capture users’ behavior characteristics at different time scales. As a result, we proposed a Multimodal Temporal Fusion for Next POI Recommendation(MTFNR). Firstly, to understand users’ preferences and habits at different periods, multiple hypergraph neural networks are constructed to analyze user behavior patterns at different stages, and in order to avoid introducing interference factors, only the check-in sequences visited in the current period are considered to reduce the impact of noise on the model’s recommendation performance. Secondly, modeling the next POI recommendation task through the fusion of time information and long- and short-term preferences in order to gain a more comprehensive understanding of users’ preferences and habits, enhance the timeliness of recommendations, and improve the accuracy of recommendations. Lastly, introducing spatio-temporal interval information into the GRU model, capturing dependencies in sequences to improve the overall performance of the model. Extensive experiments on the real LBSN datasets demonstrated the superior performance of the MTFNR model. The experimental results indicate that Top-10 recall improved 2.81% to 15.97% compared to current methods.

Next point-of-interest (POI) recommendation is a critical task in location-based social networks, yet remains challenging due to a high degree of variation and personalization exhibited in user movements. In this work, we explore the latent hierarchical structure composed of multi-granularity short-term structural patterns in user check-in sequences. We propose a Spatio-Temporal context AggRegated Hierarchical Transformer (STAR-HiT) for next POI recommendation, which employs stacked hierarchical encoders to recursively encode the spatio-temporal context and explicitly locate subsequences of different granularities. More specifically, in each encoder, the global attention layer captures the spatio-temporal context of the sequence, while the local attention layer performed within each subsequence enhances subsequence modeling using the local context. The sequence partition layer infers positions and lengths of subsequences from the global context adaptively, such that semantics in subsequences can be well preserved. Finally, the subsequence aggregation layer fuses representations within each subsequence to form the corresponding subsequence representation, thereby generating a new sequence of higher-level granularity. The stacking of hierarchical encoders captures the latent hierarchical structure of the check-in sequence, which is used to predict the next visiting POI. Extensive experiments on three public datasets demonstrate that the proposed model achieves superior performance while providing explanations for recommendations.

Point of Interest (POI) recommendation is to use the user check-in data in location-based Social Networks (LBSN) to predict the next Location that the user will visit. With the rapid development of LBSN, users' check-in information (rating information, geographic location, check-in sequence, social network, etc.) has become more and more accessible. How to effectively fuse and model these multi-source heterogeneous data is crucial to personalized POI recommendations. This paper presents a POI recommendation model based on improved factorization machine and Bert. By integrating social influence on the basis of factorization machine, Bert was used to extract the check-in sequence features. Experiments on real data sets show that our model performs better than the traditional POI recommended model.

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

In this article, we propose a new behavioral thermal modeling method for fast building performance analysis, which is critical for energy-efficient smart building control and management. The new approach is based on two recurrent neutral network architecture to obtain the compact nonlinear thermal models for complicated building. We start with a more realistic building simulation program, EnergyPlus, from Department of Energy, to model some practical buildings such as office buildings and data centers. EnergyPlus can model the various time-series inputs to a building such as ambient temperature, heating, ventilation, and air-conditioning (HVAC) inputs, power consumption of electronic equipment, lighting and number of occupants in a room sampled in each hour and produce resulting temperature traces of zones (rooms). In this work, we apply two recurrent neural network (RNN) architectures to build the non-linear compact thermal model of the building: one is non-linear state-space RNN architecture (NLSS), which has global feedbacks, and the other one is Elman's RNN architecture (ELNN), which has local feedbacks in each layer. We give a simple formula to calculate the RNN layer number, layer size to configure RNN architecture to avoid overfitting and underfitting problems. A cross-validation based training technique is further applied to improve predictable accuracy of models. Experimental results from a case study of three buildings show that ELNN and NLSS can both build very accurate building thermal models for the 2-zone and 5-zone building cases: both of them have average errors from around 1% to 1.5% for the two buildings. For the more complex 6-zone building case, ELNN outperforms NLSS with maximum errors 16% against 23%. But both methods have 2.2% average errors.

Lots of multilayer information, such as the spatio-temporal privacy check-in data, is accumulated in the location-based social network (LBSN). When using the collaborative filtering algorithm for LBSN location recommendation, one of the core issues is how to improve recommendation performance by combining the traditional algorithm with the multilayer information. The existing approaches of collaborative filtering use only the sparse user-item rating matrix. It entails high computational complexity and inaccurate results. A novel collaborative filtering-based location recommendation algorithm called LGP-CF, which takes spatio-temporal privacy information into account, is proposed in this paper. By mining the users check-in behavior pattern, the dataset is segmented semantically to reduce the data size that needs to be computed. Then the clustering algorithm is used to obtain and narrow the set of similar users. User-location bipartite graph is modeled using the filtered similar user set. Then LGP-CF can quickly locate the location and trajectory of users through message propagation and aggregation over the graph. Through calculating users similarity by spatio-temporal privacy data on the graph, we can finally calculate the rating of recommendable locations. Experiments results on the physical clusters indicate that compared with the existing algorithms, the proposed LGP-CF algorithm can make recommendations more accurately.

Quantifying Gross Primary Production (GPP) is fundamental for understanding terrestrial carbon dynamics, particularly in forests. The overarching question we address here is whether integrating remote sensing (RS) with deep learning (DL) methodologies can enhance the estimation of daily forest GPP on a European scale.The Eddy Covariance (EC) method, although widely used to infer ecosystem-scale estimates of GPP from in situ CO2 exchange measurements, suffers from limited global coverage. When EC data are not available, RS data are often employed to estimate GPP by establishing statistical relationships with in situ observations. Recently, Machine Learning (ML) strategies, particularly involving RS and meteorological inputs, have been used for estimating GPP continuously in space and time.However, the potential of DL techniques, particularly those exploiting the sequential characteristics of time series data (i.e. recurrent neural network architectures) in estimating daily forest GPP has not been comprehensively explored. This includes their performance during photosynthetic downregulation (e.g. during climate extremes such as droughts) and an in-depth examination of the importance of the utilised features.This study presents a comparative analysis of three recurrent neural network architectures—Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), and Long-Short Term Memory (LSTMs)—for daily forest GPP estimation across ICOS ecosystem stations. We assess their performance throughout the entire year, during the growing season, and under photosynthetic downregulation events. This assessment utilises RS inputs —optical data from Sentinel-2, Land Surface Temperature (LST) from MODIS, and radar data from Sentinel-1—, combined with potential radiation data. Furthermore, we analysed the importance of these features to provide insights into the complex interactions and dependencies when estimating GPP.Our results indicate that all three architectures yield similar accuracy for full period and growing season GPP estimations, with mean NRMSE values of 0.136 and 0.170, respectively. All models exhibit increased errors while estimating GPP during photosynthetic downregulation events and their performance varies notably, with LSTMs showing the best results (NRMSE=0.202), followed by RNNs (NRMSE=0.214), and GRUs exhibiting the least efficacy (NRMSE=0.276). Additionally, our study underscores the importance of potential radiation as a critical feature influencing the GPP response, with LST data proving particularly valuable when estimating GPP during photosynthetic downregulation events, more so than optical or radar data.These findings suggest that recurrent neural network architectures, especially LSTMs, are effective in daily GPP estimations, with slight performance decreases during climate extreme conditions. The study also reveals the efficacy of combining RS data with potential radiation for accurate forest GPP estimation, with LST data being especially crucial when estimating GPP during photosynthetic downregulation events.Our research paves the way for further exploration into recurrent models and innovative architectures, with an emphasis on overcoming the challenges in modelling climate-induced GPP extremes.

Funding Information: V. CONCLUSIONS This work proposed four classification models based on LSTM RNNs to classify SIP dialogs. The detection probability was evaluated based on experimental data. To detect abnormal SIP dialogs, we have adopted classification features computed from the output of the LSTM RNN model and two different classification schemes were proposed. A semi-supervised scheme is shown to reach higher performance, achieving a detection probability of 99.45%, thus confirming the effective utility of the proposed methodology to detect abnormal SIP sequences in a short period of time. ACKNOWLEDGEMENTS This work was funded by Fundac¸ão para a Ciência e Tecnologia, under the projects InfoCent-IoT (PTDC/EEI-TEL/30433/2017), CoSHARE (PTDC/EEI-TEL/30709/2017), and RFSense (UIDB/50008/2020).

Next Point of Interest (POI) recommendation is an important aspect of information feeds for Location Based Social Networks (LSBNs). The boom in LSBN platforms such as Foursquare, Twitter, and Yelp has motivated a considerable amount of research focused on POI recommendations within the last decade. Inspired by the success of deep neural networks in many fields, researchers are increasingly interested in using neural networks such as Recurrent Neural Network (RNN) to make POI recommendation. Compared to traditional methods like Factorizing Personalized Markov Chain (FPMC) and Tensor Factorization (TF), neural network methods show great improvement in general sequences prediction. However, the user’s personal preference, which is crucial for personalized POI recommendation, is not addressed well in existing works. Moreover, the user’s personal preference is dynamic rather than static, which can guide predictions in different temporal and spatial contexts. To this end, we propose a new deep neural network model called Personal Dynamic Preference Neural Network(PDPNN). The core of the PDPNN model includes two parts: one part learns the user’s personal long-term preferences from the historical trajectories, and the other part learns the user’s short-term preferences from the current trajectory. By introducing a similarity function that evaluates the similarity between spatiotemporal contexts of user’s current trajectory and historical trajectories, PDPNN learns the user’s personal dynamic preference from user’s long-term and short-term preferences. We conducted experiments on three real-world datasets, and the results show that our model outperforms current well-known methods.

The problem of high sensitivity in modeling is well known. Small perturbations in the model parameters may result in large, undesired changes in the model behavior. A number of authors have considered the issue of sensitivity in feedforward neural networks from a probabilistic perspective. Less attention has been given to such issues in recurrent neural networks. In this article, we present a new recurrent neural network architecture, that is capable of significantly improved parameter sensitivity properties compared to existing recurrent neural networks. The new recurrent neural network generalizes previous architectures by employing alternative discrete-time operators in place of the shift operator normally used. An analysis of the model demonstrates the existence of parameter sensitivity in recurrent neural networks and supports the proposed architecture. The new architecture performs significantly better than previous recurrent neural networks, as shown by a series of simple numerical experiments.

A personalized e-learning recommender system can help learners in finding learning resources that suits their needs. Collaborative filtering (CF) generates recommendations to learners by leveraging the preferences of group of similar learners. In this paper, we put forward that in addition to the traditional similarity in recommending the E- learning resources (like book, subjects, teachers), other factors such as experience and trust have an important role in generating effective recommendations. So, we considered both experience and trustworthiness of learners to develop a collaborative filtering framework, called CF-EXP-TR scheme, for e-learning recommendations. A two level filtering methodology is proposed which shows that recommendation of learning resources is taken by those learners who are more experienced as well as high trustworthy. The experimental results show that incorporation of experience and trust concepts into collaborative framework, CF-EXP-TR indeed improves the recommendation accuracy and establishes that our scheme is better than traditional Pearson based collaborative filtering, CF-PR.KeywordsRecommender SystemCollaborative FilteringE-learningTrust

MOOCs (Massive Open Online Courses) are definitely one of the best approach to support the international agenda about inclusive and equitable education and lifelong learning opportunities for all (SDG4) [1]. A great deal universities and institutions offer valuable free courses to their numerous students and to people around the word through MOOC platforms. However, because of huge number of learners and data generated, learner’s behaviour in those platforms remain a kind of black box for learners themselves and for courses instructors who are supposed to tutor or monitor learners in the learning process. Therefore, learner do not receive sufficient support from instructors and from their peers, during the learning process [2]. This is one the main reasons that lead to high dropout, low completion and success rate observed in the MOOCs. Many research work have suggested descriptive, predictive and prescriptive models to address this issue, but most of these models focus on predicting dropout, completion and/or success, and do not generally pay enough attention to one of the key step (learner behaviour), that comes before, and can explain dropping out and failure. Our research aims to develop a deep learning model to predict learner behaviour (learner interactions) in the learning process, in order to equip learners and course instructors with insight understanding of the learner behaviour in the learning process. This specific paper will focus on analysing relevant algorithms to develop such model. For this analysis, we used data from UNESCO-IICBA (International Institute for Capacity Building in Africa) MOOC platform, designed for teacher training in Africa, and then we examine many types of features: geographical, social behavioural and learning behavioural features. Learner’s behaviour being a time series Big data, we built the predictive model using Deep Learning algorithms for better performance and accuracy (Thanks to the power of deep learning) compared to baseline Machine learning algorithms. Time series data is best handled by recurrent neural networks (RNN) [3], so, we choose RNN and implemented/tested the three main architectures of RNN: Simple RNNs, GRU (Gated Recurrent Unit) RNNs and LSTM (Long short-term memory) RNNs. The models were trained using L2 Regularization, based on the predictions results, we unexpectedly found model with simple RNNs produced the best performance and accuracy on the dataset used than the other RNN architectures. We had couple of observations, example: we saw a correlation between video viewing and quiz behaviour and the participation of the learner to the learning process. This observation could allow teachers to provide meaningful support and guidance to at risk or less active students. We also observed that, the shorter the video or the quiz, the more the viewer. We conclude that we could use learner video or quiz viewing behaviour to predict his behaviour concerning other MOOC contents. These results suggest the need of deeper research on educational video and educational quiz designing for MOOCs.