Using Classifier System with / without Genetic Algorithm in Robotics Behaviors

Enhancing Humans Trust in Robots through Explanations

Bidirectional Human-Robot Learning: Imitation and Skill Improvement

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

This issue is dedicated to Stewart Wilson both in recognition of his achievements and in appreciation for his guidance of the field. Not only has his research transformed the field, but his support, encouragement, and advice have been invaluable to many of us individually and to the community as a whole. Learning Classifier Systems (LCS) are rule-based machine learning algorithms introduced by Holland (see [2, 4–6]) that learn a population of IF-THEN rules that specify ‘‘IF x happens THEN do (or predict) y’’. In 1995 Stewart Wilson, already a leading LCS researcher, published a paper entitled ‘‘Classifier Fitness based on Accuracy’’ [16] that introduced the XCS algorithm. This paper proved a turning point for the field as XCS and its derivatives rapidly became the main focus of LCS research, and continue to be so today. We organised this special issue of Evolutionary Intelligence to mark the 20th anniversary of this landmark paper, and to serve as post-proceedings for IWLCS 2014: the Seventeenth International Workshop on Learning Classifier Systems. The annual IWLCS meetings are the yearly highlight of the LCS calendar and have no doubt contributed to the strong sense of community in the LCS field. For 2015 IWLCS has been rebranded as the more descriptive IWERML: the International Workshop on Evolutionary Rule-based Machine Learning. In 1994, the year before introducing XCS, Wilson introduced ZCS, the ‘‘Zeroth-order Classifier System’’ [15]. Wilson felt the growing complexity of the LCS concept was hindering progress and ZCS was an attempt to strip the LCS down to its essentials while retaining a working system. XCS built on the minimalist ZCS with two radical changes: a switch to accuracy-based fitness and, based on work by Booker [1], the addition of a niche Genetic Algorithm (GA) that strongly favours general rules. This combination was a hit: the niche GA favours general rules but accuracy-based fitness insists strictly that they make good predictions. This combination drives XCS to learn rules that are as general as possible while remaining accurate. Bull’s article in this issue provides much more on the history of LCS prior to and following XCS [2]. The famous ‘‘accuracy-based fitness’’ of XCS needs some explanation. The earlier ZCS featured ‘‘strengthbased fitness’’, in which the fitness of a rule in the GA was derived from its strength, a measure of the amount of reward the rule received. Consequently, fit rules in ZCS are those that receive a lot of reward. In contrast, XCS rules are fit if they make consistently accurate predictions about the reward they receive. This has the counterintuitive consequence that XCS rules that consistently take a bad action can be fit, since they are consistent. However, this poses no problem when it comes time for XCS to choose an action, since that can be done using the magnitude of the reward predicted by each rule. Why does XCS retain rules whose action it does not use? This ‘‘complete map’’ of the state/ & Muhammad Iqbal muhammad.iqbal@ecs.vuw.ac.nz

To manifest anticipatory behaviour that goes beyond simple stimulus-response, classifier systems must evolve internal reasoning processes based on couplings via internal messages. A major challenge that has been encountered in engendering internal reasoning processes in classifier systems has been the discovery and maintenance of long classifier chains. This paper proposes a modified version of the traditional classifier system, called the delayed action classifier system (DACS), devised specifically for learning of anticipatory or predictive behaviour. DACS operates by delaying the action (i.e. posting of messages) of appropriately tagged, matched classifiers by a number of execution cycles which is encoded on the classifier. Since classifier delays are encoded on the classifier genome, a GA is able to explore simultaneously the spaces of actions and delays. Results of experiments comparing DACS to a traditional classifier system in terms of the dynamics of classifier reinforcement and system performance using the bucket brigade are presented and examined. Experiments comparing DACS with a traditional classifier system, which appear encouraging, for a simple prediction problem are described and considered. Areas for further work using the delayed-action classifier notion are suggested and briefly discussed.KeywordsGenetic AlgorithmClassifier SystemTraditional ClassifierMarginal PayoffLetter SequenceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Learning classifier systems: New models, successful applications

Classifier systems and genetic algorithms

Robot shaping: developing autonomous agents through learning

If complexity is your problem, learning classifier systems (LCSs) may offer a solution. These rule-based, multifaceted, machine learning algorithms originated and have evolved in the cradle of evol...

Data mining, and specifically supervised data classification, is a key application area for Learning Classifier Systems (LCS). Scaling to larger classification problems, especially to higher dimensional problems, is a key challenge. Ensemble based approaches can be applied to LCS to address scalability issues. To this end a rough set based ensemble of LCS is proposed, which relies on a pre-processed feature partitioning step to train multiple LCS on feature subspaces. Each base classifier in the ensemble is a Michigan style supervised LCS. The traditional genetic algorithm based rule evolution is replaced by a differential evolution based rule discovery, to improve generalisation capabilities of LCS. A voting mechanism is then used to generate output for test instances. This paper describes the proposed ensemble algorithm in detail, and compares its performance with different versions of base LCS on a number of benchmark classification tasks. Analysis of computational time and model accuracy show the relative merits of the ensemble algorithm and base classifiers on the tested data sets. The rough set based ensemble learning approach and differential evolution based rule searching out-perform the base LCS on classification accuracy over the data sets considered. Results also show that small ensemble size is sufficient to obtain good performance.

This paper presented three disease diagnosis systems using pattern recognition based on genetic algorithm (GA) and neural networks. All systems dealt with feature selection and classification. GA chose subsets of features for the input of the classifier (neural network) and the accuracy of the classifier determined the percentage of effectiveness of each subset of features. The classifiers using in this paper were general regression neural network (GRNN), radial basis function (RBF) and radial basis network exact fit (RBEF). It uses breast cancer and hepatitis disease datasets taken from UCI machine learning database as medical dataset. The system performances were estimated by classification accuracy and they were compared with similar methods without feature selection. Key words: Pattern recognition, genetic algorithm, neural networks, classification.

This dissertation addresses two significant issues in the effective application of machine learning algorithms and models for the physical and engineering sciences. The first is the broad challenge of automated modeling of data across different processes in a physical system. The second is the dilemma of obtaining insightful interpretations on the relationships between the inputs and outcome of a system as inferred from complex, black box machine learning models.Automated Geometric Shape Deviation Modeling for Additive Manufacturing SystemsAdditive manufacturing systems possess an intrinsic capability for one-of-a-kind manufacturing of a vast variety of shapes across a wide spectrum of processes. One major issue in AM systems is geometric accuracy control for the inevitable shape deviations that arise in AM processes. Current effective approaches for shape deviation control in AM involve the specification of statistical or machine learning deviation models for additively manufactured products. However, this task is challenging due to the constraints on the number of test shapes that can be manufactured in practice, and limitations on user efforts that can be devoted for learning deviation models across different shape classes and processes in an AM system. We develop an automated, Bayesian neural network methodology for comprehensive shape deviation modeling in an AM system. A fundamental innovation in this machine learning method is our new and connectable neural network structures that facilitate the transfer of prior knowledge and models on deviations across different shape classes and AM processes. Several case studies on in-plane and out-of-plane deviations, regular and free-form shapes, and different settings of lurking variables serve to validate the power and broad scope of our methodology, and its potential to advance high-quality manufacturing in an AM system.Interpretable Machine LearningMachine learning algorithms and models constitute the dominant set of predictive methods for a wide range of complex, real-world processes. However, interpreting what such methods effectively infer from data is difficult in general. This is because their typical black box natures possess a limited ability to directly yield insights on the underlying relationships between inputs and the outcome for a process. We develop methodologies based on new predictive comparison estimands that effectively enable one to ``mine’’ machine learning models, in the sense of (a) interpreting their inferred associations between inputs and/or functional forms of inputs with the outcome, (b) identifying the inputs that they effectively consider relevant, and (c) interpreting the inferred conditional and two-way associations of the inputs with the outcome. We establish Fisher consistent estimators, and their corresponding standard errors, for our new estimands under a condition on the inputs' distributions. The significance of our predictive comparison methodology is demonstrated with a wide range of simulation and case studies that involve Bayesian additive regression trees, neural networks, and support vector machines. Our extended study of interpretable machine learning for AM systems demonstrates how our method can contribute to smarter advanced manufacturing systems, especially as current machine learning methods for AM are lacking in their ability to yield meaningful engineering knowledge on AM processes.

Genetic algorithms are employed in classifier systems in order to discover new classifiers. The paper formalises this rule discovery or learning problem for classifier systems and uses methods of computational complexity theory to analyse its inherent difficulty. It is proved that two distinct learning problems are NP-complete, i.e. not likely to be solvable efficiently. The practical relevance of these theoretical results is briefly discussed.KeywordsGenetic AlgorithmLearning TaskClassifier ConditionClassifier SystemLearning ProblemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Learning Classifier Systems (LCS) evolve IF-THEN rules for classification and control tasks. The earliest Michigan-style LCS used a panmictic Genetic Algorithm (GA) (in which all rules compete for selection) but newer ones tend to use a niche GA (in which only a certain subset of rules compete for selection at any one time). The niche GA was thought to be advantageous in all learning tasks, but recent research suggests it has difficulties when the rules composing the solution overlap. Furthermore, the niche GA's effects are implicit, making it difficult study, and fixed, which prevents tuning its performance. Given these issues, we set out on a long-term project to reevaluate the niche GA. This work is our starting point and in it we address the implicit and unquantified effects of the niche GA by building a mathematical model of the probability of rule selection. This model reveals a number of insights into the components of rule fitness, particularly the bonus for rule generality and penalty for overlaps, both previously unquantified. These theoretical results are our primary contribution. However, to demonstrate one way to apply this theory, we then introduce a new variant of the UCS algorithm, which uses a hybrid panmictic/niche GA. Preliminary results suggest, unexpectedly, that the niche GA may have even more drawbacks than previously thought.

The integration of independently-developed applications into an efficient system, particularly in a distributed setting, is the core issue addressed in this work. Cooperation between researchers across various field boundaries in order to solve complex problems has become commonplace. Due to the multidisciplinary nature of such efforts, individual applications are developed independent of the integration process. The integration of individual applications into a fully-functioning architecture is a complex and multifaceted task. This thesis extends a component-based architecture, previously developed by the authors, to allow the integration of various software applications which are deployed in a distributed setting. The test bed for the framework is the EU project XPERO, the goal of which is robot learning by experimentation. The task at hand is the integration of the required applications, such as planning of experiments, perception of parametrized features, robot motion control and knowledge-based learning, into a coherent cognitive architecture. This allows a mobile robot to use the methods involved in experimentation in order to learn about its environment. To meet the challenge of developing this architecture within a distributed, heterogeneous environment, the authors specified, defined, developed, implemented and tested a component-based architecture called XPERSIF. The architecture comprises loosely-coupled, autonomous components that offer services through their well-defined interfaces and form a service-oriented architecture. The Ice middleware is used in the communication layer. Its deployment facilitates the necessary refactoring of concepts. One fully specified and detailed use case is the successful integration of the XPERSim simulator which constitutes one of the kernel components of XPERO.The results of this work demonstrate that the proposed architecture is robust and flexible, and can be successfully scaled to allow the complete integration of the necessary applications, thus enabling robot learning by experimentation. The design supports composability, thus allowing components to be grouped together in order to provide an aggregate service. Distributed simulation enabled real time tele-observation of the simulated experiment. Results show that incorporating the XPERSim simulator has substantially enhanced the speed of research and the information flow within the cognitive learning loop.