Policy Transfer via Markov Logic Networks

Domain- and task-specific transfer learning for medical segmentation tasks

What matters in a transferable neural network model for relation classification in the biomedical domain?

Increasingly higher demands are being made on the capacity‐limited cognitive capabilities of human operators as they strive to maintain situation awareness (i.e., understanding “what is going on”) and performance in complex tasks. In the current study we asked whether: (a) training administered via a mobile phone‐based app could improve multitasking and (b) improved multitasking in the app would generalize to improved performance and situation awareness in a simulated air traffic control task (ATC). Participants completed the ATC task before and after multiple sessions of app‐based multitasking training or control training. Multitasking on the app improved across training sessions. However, this did not lead to improved performance or situation awareness, or workload reduction, relative to control training on the ATC task. These outcomes indicate that app‐based multitasking training based on repetition of a single training task will not necessarily yield generalizable benefits to human performance in other complex dynamic tasks.

Transfer learning in reinforcement learning has been an active area of research over the past decade. In transfer learning, training on a source task is leveraged to speed up or otherwise improve learning on a target task. This paper presents the more ambitious problem of curriculum learning in reinforcement learning, in which the goal is to design a sequence of source tasks for an agent to train on, such that final performance or learning speed is improved. We take the position that each stage of such a curriculum should be tailored to the current ability of the agent in order to promote learning new behaviors. Thus, as a first step towards creating a curriculum, the trainer must be able to create novel, agent-specific source tasks. We explore how such a space of useful tasks can be created using a parameterized model of the domain and observed trajectories on the target task. We experimentally show that these methods can be used to form components of a curriculum and that such a curriculum can be used successfully for transfer learning in 2 challenging multiagent reinforcement learning domains.

Object recognition from images in a home environment is challenging since the object usually has low resolution in the image and the scene is usually cluttered. However, many objects have specific functions to the user and the interactions between the user and the object provides useful contextual information to recognize the object. In this paper, we use Markov logic network (MLN) to model such context information as relationship between the objects and user activities. We demonstrate that Markov logic network provides a flexible way in the syntax of first-order logic to incorporate relational context information. It is also a probabilistic graphical model which handles uncertainty in the knowledge base, observations and decisions. In our experiment, objects in the living room and kitchen in a home are recognized based on only user's activity. The user's activity is analyzed from images of cameras installed in the home. Relationship between user activity and objects is defined in a knowledge base with MLN. Experiments show that objects in the home can be recognized irrespective of their position, size and appearance in the image.

Transfer learning refers to the transfer of knowledge or information from a relevant source task to a target task. However, most existing works assume both tasks are sampled from a stationary task distribution, thereby leading to the sub-optimal performance for dynamic tasks drawn from a non-stationary task distribution in real scenarios. To bridge this gap, in this paper, we study a more realistic and challenging transfer learning setting with dynamic tasks, i.e., source and target tasks are continuously evolving over time. We theoretically show that the expected error on the dynamic target task can be tightly bounded in terms of source knowledge and consecutive distribution discrepancy across tasks. This result motivates us to propose a generic meta-learning framework L2E for modeling the knowledge transferability on dynamic tasks. It is centered around a task-guided meta-learning problem with a group of meta-pairs of tasks, based on which we are able to learn the prior model initialization for fast adaptation on the newest target task. L2E enjoys the following properties: (1) effective knowledge transferability across dynamic tasks; (2) fast adaptation to the new target task; (3) mitigation of catastrophic forgetting on historical target tasks; and (4) flexibility in incorporating any existing static transfer learning algorithms. Extensive experiments on various image data sets demonstrate the effectiveness of the proposed L2E framework.

Intelligent Decision Support System for Dementia Care Through Smart Home

Markov Logic Networks (MLNs) represent relational knowledge using a combination of first-order logic and probabilistic models. In this paper, we develop an approach to explain the results of probabilistic inference in MLNs. Unlike approaches such as LIME and SHAP that explain black-box classifiers, explaining M LN inference is harder since the data is interconnected. We develop an explanation framework that computes importance weights for MLN formulas based on their influence on the marginal likelihood. However, it turns out that computing these importance weights exactly is a hard problem and even approximate sampling methods are unreliable when the MLN is large resulting in non-interpretable explanations. Therefore, we develop an approach where we reduce the large MLN into simpler coalitions of formulas that approximately preserve relational dependencies and generate explanations based on these coalitions. We then weight explanations from different coalitions and combine them into a single explanation. Our experiments illustrate that our approach generates more interpretable explanations in several text processing problems as compared to other state-of-the-art methods.

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $$\varvec{\aleph }$$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing its average reward. We show that when the agents share their observations the mutual regret of all agents is smaller by a factor of $$\varvec{\sqrt{\aleph }}$$ compared to the case when each agent has to rely on the information collected by itself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.

Statistical Relational Models are state-of-the-art representation formalisms at the intersection of logical and statistical machine learning. One of the most promising models is Markov Logic (ML) which combines Markov networks (MNs) and first-order logic by attaching weights to first-order formulas and using these as templates for features of MNs. MAP inference in ML is the task of finding the most likely state of a set of output variables given the state of the input variables and this problem is NP-hard. In this paper we present an algorithm for this inference task based on the Iterated Local Search (ILS) and Robust Tabu Search (RoTS) metaheuristics. The algorithm performs a biased sampling of the set of local optima by using RoTS as a local search procedure and repetitively jumping in the search space through a perturbation operator, focusing the search not on the full space of solutions but on a smaller subspace defined by the solutions that are locally optimal for the optimization engine. We show through extensive experiments in real-world domains that it improves over the state-of-the-art algorithm in terms of solution quality and inference time.

ABSTRACTWith diverse online media emerging, there is a growing concern of sentiment classification problem. At present, text sentiment classification mainly utilizes supervised machine learning methods, which feature certain domain dependency. On the basis of Markov logic networks (MLNs), this study proposed a cross-domain multi-task text sentiment classification method rooted in transfer learning. Through many-to-one knowledge transfer, labeled text sentiment classification, knowledge was successfully transferred into other domains, and the precision of the sentiment classification analysis in the text tendency domain was improved. The experimental results revealed the following: (1) the model based on a MLN demonstrated higher precision than the single individual learning plan model. (2) Multi-task transfer learning based on Markov logical networks could acquire more knowledge than self-domain learning. The cross-domain text sentiment classification model could significantly improve the precision and efficiency of text sentiment classification.

Reinforcement learning promises high performance in complex tasks as well as low online storage and computation cost. However, the trial-and-error learning approach of reinforcement learning could explore unsafe behavior in the search for an optimal solution. Run time assurance (RTA) approaches can be applied to monitor behavior and ensure safety constraint satisfaction during reinforcement learning. This paper investigates the effect of RTA on reinforcement learning training performance in terms of training efficiency, safety constraint satisfaction, control efficiency, task efficiency, and training duration. For the purposes of demonstration, a custom reinforcement learning environment is created where the objective is to develop a policy that moves a satellite into docking position with another satellite in a two-dimensional relative-motion reference frame. Six different policies are trained. The first features no RTA, the second features no RTA but a higher penalty for safety violations, and four others use different RTA techniques to enforce a dynamic velocity constraint during training. The trained policies are analyzed with standardized test points. It is shown that the policies trained without RTA do not learn to adhere to the constraint, whereas all policies trained with RTA do learn to adhere to the constraint. Although more complex RTA frameworks can be better for operational use, it is found that a simple RTA framework provides the best overall results for reinforcement learning training.

We propose a system to recognize objects with a camera network in a smart home. Recognizing objects in a home environment from images is challenging, due to the variation in object appearances such as chairs, as well as the clutters in the scene. Therefore, we propose to recognize objects through user interactions. A hierarchical activity analysis is first performed in the system to recognize fine-grained activities including eating, typing, cutting etc. The object-activity relationship is encoded in the knowledge base of a Markov logic network (MLN). MLN has the advantage of encoding relationships in an intuitive way with first-order logic syntax. It can also deal with both soft and hard constraints by associating weights to the formulas in the knowledge base. With activity observations, the defined MLN is grounded and turned into a dynamic Bayesian network (DBN) to infer object type probabilities. We expedite inference by decomposing the MLN into smaller separate domains that relates to the active activity. Experimental results are presented with our testbed smart home environment.

Markov logic networks (MLNs) reconcile two opposing schools in machine learning and artificial intelligence: causal networks, which account for uncertainty extremely well, and first-order logic, which allows for formal deduction. An MLN is essentially a first-order logic template to generate Markov networks. Inference in MLNs is probabilistic and it is often performed by approximate methods such as Markov chain Monte Carlo (MCMC) Gibbs sampling. An MLN has many regular, symmetric structures that can be exploited at both first-order level and in the generated Markov network. We analyze the graph structures that are produced by various lifting methods and investigate the extent to which quantum protocols can be used to speed up Gibbs sampling with state preparation and measurement schemes. We review different such approaches, discuss their advantages, theoretical limitations, and their appeal to implementations. We find that a straightforward application of a recent result yields exponential speedup compared to classical heuristics in approximate probabilistic inference, thereby demonstrating another example where advanced quantum resources can potentially prove useful in machine learning.

Transfer learning enables machine learning system to use previously acquired knowledge from source task on target task. Source task and target task should be correlated. Successful transfer learning would improve system performance on target task especially when target data is small. In this paper, the source task is Indonesian part-of-speech (POS) tagging system and the target task is Indonesian named entity recognition (NER) tagging system. A bidirectional gated recurrent unit and conditional random field (GRU-CRF) sequence labeller is crafted as the model for the feature representation transfer to be applied. We compared F1 evaluation results between system that used transfer and system that did not use transfer and saw an improvement around 0.02 on the transferred model when 100% training data was used and around 0.15 when 1% training data was used.