Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation

Not having access to compact and meaningful representations is known to significantly increase the complexity of reinforcement learning (RL). For this reason, it can be useful to perform state representation learning (SRL) before tackling RL tasks. However, obtaining a good state representation can only be done if a large diversity of transitions is observed, which can require a difficult exploration, especially if the environment is initially reward-free. To solve the problems of exploration and SRL in parallel, we propose a new approach called XSRL (eXploratory State Representation Learning). On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a k-step learning progress bonus to form the maximization objective of a discovery policy. This results in a policy that seeks complex transitions from which the trained models can effectively learn. Our experimental results show that the approach leads to efficient exploration in challenging environments with image observations, and to state representations that significantly accelerate learning in RL tasks.

Multi-Agent Deterministic Policy Gradient (MADDPG) is a very useful algorithm in multi-agent domains. We analyze the algorithm and find that MADDPG uses deep neural networks (DNNs) as a Q function model. One advantage of DNNs is that they can build very complex processing functions to handle high-dimensional input. However, the disadvantage of this end-to-end learning is that it usually requires a lot of data, which is not always available for real-world control applications. In this paper, a new algorithm, State Representation Learning Multi-Agent Deep Deterministic Policy Gradient (SRL-MADPPG), is proposed that combines MADDPG with state representation learning which uses DNNs as a function fitting. i.e., model learning network is used to pre-train the first layer of the actor and critic networks, then the actor and critic learn from the state representation instead of the raw observations. Simulation result shows that the SRL-MADDPG algorithm improves the final performance in comparison with the end-to-end learning.

Camera Selection Decisions (CSD) are highly useful for several applications in a multi-camera network. For example, CSD benefit multi-camera target tracking by reducing the number of candidate cameras to look for the target’s next location. The correct candidate cameras, decreases the number of false Re-ID queries as well as the computation time. Also, in multi-camera trajectory forecasting (MCTF) to predict where a person will re-appear in the camera network along with the transition time. These applications require a large amount of annotated data for training. In this paper, we use state-representation learning with a reinforcement learning based policy to effectively and efficiently make camera selection decisions. We further demonstrate that by using learned state representations, as opposed to hand-crafted state variables, we are able to achieve state-of-the-art results on camera selection, while reducing the training time for the RL policy. Along with this, we use a reward function that helps to reduce the amount of supervision in training the policy in a semi-supervised way. We report our results on four datasets: NLPR_MCT, DukeMTMC, CityFlow, and WNMF dataset. We show that an RL policy reduces unnecessary Re-ID queries and therefore the false alarms, scales well to larger camera networks, and is target-agnostic.

Causal feature selection using a knowledge graph combining structured knowledge from the biomedical literature and ontologies: A use case studying depression as a risk factor for Alzheimer’s disease

State representation learning for control: An overview

Currently, deep reinforcement learning is widely used to transform the recommendation process into a sequential task. Despite this, the recommendation system continues to be plagued by data sparsity and cold start issues as a result of high online costs and a sparse user-item rating matrix. To address the aforementioned issues, this article proposes a social multi-agent reinforcement learning framework (MATR). Two agents collaborate within the framework to enhance the performance of conventional solutions. Specifically, one agent is responsible for capturing the dynamic preferences of users based on their interaction with the system. The other agent leverages a ubiquitous social network to minimise cold starts and data sparsity for some users. In addition, we designed a module for learning state representations from social networks and user rating matrices. To optimise the use of social networks in the state representation module, in addition to first-order social neighbour features, we used trust inference to discover high-level social neighbour features, and we integrated Feature aggregation modelling state representation with graph convolutional neural networks and attention mechanisms. Our method has been shown to be effective and advantageous using actual data sets.

Spectrum intervention based invariant causal representation learning for single-domain generalizable medical image segmentation.

In two experiments, unusual phenomena (spontaneous destruction of objects in an empty wooden box) were demonstrated to adult participants living in rural communities in Mexico. These were accompanied by actions which had no physical link to the destroyed object but could suggest either scientifically based (the effect of an unknown physical device) or non-scientifically based (the effect of a 'magic spell') causal explanations of the event. The results were compared to the results of the matching two experiments from the earlier study made in Britain. The expectation that scientifically based explanations would prevail in British participants' judgments and behaviours, whereas Mexican participants would be more tolerant toward magical explanations, received only partial support. The prevalence of scientific explanations over magical explanations was evident in British participants' verbal judgments but not in Mexican participants' judgments. In their behavioural responses under the low-risk condition, British participants rejected magical explanations more frequently than did Mexican participants. However, when the risk of disregarding the possible causal effect of magic was increased, participants in both samples showed an equal degree of credulity in the possible effect of magic. The data are interpreted in terms of the relationships between scientific and 'folk' representations of causality and object permanence.

The increasing popularity of social networks indicates that the vast amounts of data contained within them could be useful in various implementations, including recommendation systems. Interests and research publications on deep learning-based recommendation systems have largely increased. This study aimed to identify, summarize, and assess studies related to the application of deep learning-based recommendation systems on social media platforms to provide a systematic review of recent studies and provide a way for further research to improve the development of deep learning-based recommendation systems in social environments. A total of 32 papers were selected from previous studies in five of the major digital libraries, including Springer, IEEE, ScienceDirect, ACM, Scopus, and Web of Science, published between 2016 and 2020. Results revealed that even though RS has received high coverage in recent years, several obstacles and opportunities will shape the future of RS for researchers. In addition, social recommendation systems achieving high accuracy can be built by using a combination of techniques that incorporate a range of features in SRS. Therefore, the adoption of deep learning techniques in developing social recommendation systems is undiscovered.

Through well-designed optimization paradigm and deep neural networks as feature extractor, deep reinforcement learning (DRL) algorithms learn optimal policy on discrete and continuous action space. However, such capability is restricted by the low sampling efficiency. By inspecting the importance of feature extraction in DRL, we find that state feature learning is one of the key obstacles for sampling efficiently. To this end, we propose a new state representation learning scheme with adjacent state consistency loss (ASC loss). The loss is based on the hypothesis that the distance between adjacent states is smaller than that of far apart ones since scenes in videos generally evolve smoothly. We exploit ASC loss as an assistant of RL loss in the training phase to boost the state feature learning, and make evaluation on existing DRL algorithms as well as behavioral cloning algorithm. Experiments on Atari games and MuJoCo continuous control tasks demonstrate the effectiveness of our scheme.

Applications based on electroencephalography (EEG) signals suffer from the mutual contradiction of high classification performance vs. low cost. The nature of this contradiction makes EEG signal reconstruction with high sampling rates and sensitivity challenging. Conventional reconstruction algorithms lead to loss of the representative details of brain activity and suffer from remaining artifacts because such algorithms only aim to minimize the temporal mean-squared-error (MSE) under generic penalties. Instead of using temporal MSE according to conventional mathematical models, this paper introduces a novel reconstruction algorithm based on generative adversarial networks with the Wasserstein distance (WGAN) and a temporal-spatial-frequency (TSF-MSE) loss function. The carefully designed TSF-MSE-based loss function reconstructs signals by computing the MSE from time-series features, common spatial pattern features, and power spectral density features. Promising reconstruction and classification results are obtained from three motor-related EEG signal datasets with different sampling rates and sensitivities. Our proposed method significantly improves classification performances of EEG signals reconstructions with the same sensitivity and the average classification accuracy improvements of EEG signals reconstruction with different sensitivities. By introducing the WGAN reconstruction model with TSF-MSE loss function, the proposed method is beneficial for the requirements of high classification performance and low cost and is convenient for the design of high-performance brain computer interface systems.

Learning to rank has been intensively studied and has shown great value in many fields, such as web search, question answering and recommender systems. This paper focuses on listwise document ranking, where all documents associated with the same query in the training data are used as the input. We propose a novel ranking method, referred to as WassRank, under which the problem of listwise document ranking boils down to the task of learning the optimal ranking function that achieves the minimum Wasserstein distance. Specifically, given the query level predictions and the ground truth labels, we first map them into two probability vectors. Analogous to the optimal transport problem, we view each probability vector as a pile of relevance mass with peaks indicating higher relevance. The listwise ranking loss is formulated as the minimum cost (the Wasserstein distance) of transporting (or reshaping) the pile of predicted relevance mass so that it matches the pile of ground-truth relevance mass. The smaller the Wasserstein distance is, the closer the prediction gets to the ground-truth. To better capture the inherent relevance-based order information among documents with different relevance labels and lower the variance of predictions for documents with the same relevance label, ranking-specific cost matrix is imposed. To validate the effectiveness of WassRank, we conduct a series of experiments on two benchmark collections. The experimental results demonstrate that: compared with four non-trivial listwise ranking methods (i.e., LambdaRank, ListNet, ListMLE and ApxNDCG), WassRank can achieve substantially improved performance in terms of nDCG and ERR across different rank positions. Specifically, the maximum improvements of WassRank over LambdaRank, ListNet, ListMLE and ApxNDCG in terms of nDCG@1 are 15%, 5%, 7%, 5%, respectively.

Heterogeneous relational reasoning in knowledge graphs with reinforcement learning

Recommendation System (RS) has been broadly utilized in various areas and discovers product recommendations during an active user interaction in E-Commerce sites. Tremendous growth of users and products in recent years has faced some key challenges. There are numerous online sites that present many decisions to the user at once, which is strenuous. Moreover, finding active user or right product is an important task in RS. Existing works have been proposed to recommend a product by considering user inclination and socio-demographic behaviour. In this paper, we propose a Hybrid Action-Related K-Nearest Neighbour similarity (HAR-KNN) recommender that consolidates the simplicity of hybrid filtering to enrich user behaviour matrix with formation of the vector of features. It will classify the features using race classifiers from both quality and quantity aspects. The proposed approach also addresses the problems of the previous methods to efficiently evaluate user preference on products and balance feature analysis. The K-NN classification method has been qualified online and real-time to find user behaviour data coordinating to a specific user group containing the relationship between the similarity of many users and target users from a huge amount of data. The proposed experimental result is evaluated based on measures such as Mean Absolute Error (MAE), Mean Square Error (MSE) and Root Mean Squared Error (RMSE) with the lowest error of 0.7165, 0.7201 and 0.7322 separately. High predictive measures like Precision (P), Recall (R) and F1 are found to have values 0.8501, 0.2201 and 0.3507 respectively.

Credit scoring is extensively used by credit industries and financial institutions for financial decision-making. It is a way to assess the risk associated with an applicant based on historical data. However, the historical data may have large number of redundant and noisy features which could affect performance of credit scoring models. Main focus of this paper is to develop a hybrid credit scoring model by combining the feature selection and multi-layer ensemble classifier framework to improve the prediction performance of credit scoring model. The proposed hybrid credit scoring model uses hybrid binary particle swarm optimization and gravitational search algorithm (BPSOGSA) for feature selection and multi-layer ensemble classifier framework with five heterogeneous classifiers. A novel V-shaped transfer function for BPSOGSA is also designed for effective feature selection, which is used to transform the continuous search space to binary search space. Also, a novel fitness function for BPSOGSA is proposed to calculate the fitness value for each search agent. Further, multi-layer ensemble classifier framework along with a novel aggregation function is designed based on generalized convex function. The proposed hybrid credit scoring model is validated using Australian, German-categorical, German-numerical and Japanese credit scoring datasets. The experimental results on all the datasets demonstrate that the proposed credit scoring model outperforms other methods such as random forest and ensemble frameworks, namely majority voting, layered majority voting, weighted voting and layered weighted voting in terms of accuracy, sensitivity, G-measure and ROC characteristics.