Teaching agents with corrective human feedback for challenging problems

COACH (COrrective Advice Communicated by Humans), a new interactive learning framework that allows non-expert humans to shape a policy through corrective advice, using a binary signal in the action domain of the agent, is proposed. One of the main innovative features of COACH is a mechanism for adaptively adjusting the amount of human feedback that a given action receives, taking into consideration past feedback. The performance of COACH is compared with the one of TAMER (Teaching an Agent Manually via Evaluative Reinforcement), ACTAMER (Actor-Critic TAMER), and an autonomous agent trained using SARSA(?) in two reinforcement learning problems. COACH outperforms all other learning frameworks in the reported experiments. In addition, results show that COACH is able to transfer successfully human knowledge to agents with continuous actions, being a complementary approach to TAMER, which is appropriate for teaching in discrete action domains.

Microarray technology has generated vast amounts of gene expression data with distinct patterns. Based on the premise that genes of correlated functions tend to exhibit similar expression patterns, various machine learning methods have been applied to capture these specific patterns in microarray data. However, the discrepancy between the rich expression profiles and the limited knowledge of gene functions has been a major hurdle to the understanding of cellular networks. To bridge this gap so as to properly comprehend and interpret expression data, we introduce Relevance Feedback to microarray analysis and propose an interactive learning framework to incorporate the expert knowledge into the decision module. In order to find a good learning method and solve two intrinsic problems in microarray data: high dimensionality and small sample size, we also propose a semi-supervised learning algorithm: Kernel Discriminant-EM (KDEM). This algorithm efficiently utilizes a large set of unlabeled data to compensate for the insufficiency of a small set of labeled data and it extends linear algorithm in Discriminant-EM (DEM) to kernel algorithm to handle nonlinearly separable data in a lower dimensional space. Relevance Feedback technique and KDEM together construct an efficient and effective interactive semi-supervised learning framework for microarray analysis. Extensive experiments on the yeast cell cycle regulation data set and Plasmodium falciparum red blood cell cycle data set show the promise of this approach.

One of the biggest challenges in intelligent robotics is to build robots that can understand and use language. Such robots will be a part of our everyday life; at the same time, they can be of great help to investigate the complex mechanism of language acquisition by infants in constructive approach. To this end, we think that the practical long-term on-line concept/word learning algorithm for robots and the interactive learning framework are the key issues to be addressed. In this paper we develop a practical on-line learning algorithm that solves three remaining problems in our previous study. We also propose an interactive learning framework, in which the proposed on-line learning algorithm is embedded. The main contribution of this paper is to develop such a practical learning framework, and we test it on a real robot platform to show its potential toward the ultimate goal.

(2004). Robo-Erectus: a low-cost autonomous humanoid soccer robot. Advanced Robotics: Vol. 18, No. 7, pp. 717-720.

Various domain users are increasingly leveraging real-time social media data to gain rapid situational awareness. However, due to the high noise in the deluge of data, effectively determining semantically relevant information can be difficult, further complicated by the changing definition of relevancy by each end user for different events. The majority of existing methods for short text relevance classification fail to incorporate users' knowledge into the classification process. Existing methods that incorporate interactive user feedback focus on historical datasets. Therefore, classifiers cannot be interactively retrained for specific events or user-dependent needs in real-time. This limits real-time situational awareness, as streaming data that is incorrectly classified cannot be corrected immediately, permitting the possibility for important incoming data to be incorrectly classified as well. We present a novel interactive learning framework to improve the classification process in which the user iteratively corrects the relevancy of tweets in real-time to train the classification model on-the-fly for immediate predictive improvements. We computationally evaluate our classification model adapted to learn at interactive rates. Our results show that our approach outperforms state-of-the-art machine learning models. In addition, we integrate our framework with the extended Social Media Analytics and Reporting Toolkit (SMART) 2.0 system, allowing the use of our interactive learning framework within a visual analytics system tailored for real-time situational awareness. To demonstrate our framework's effectiveness, we provide domain expert feedback from first responders who used the extended SMART 2.0 system.

The electricity market has provided a complex economic environment, and consequently has increased the requirement for advancement of learning methods. In the agent-based modeling and simulation framework of this economic system, the generation company's decision-making is modeled using reinforcement learning. Existing learning methods that model the generation company's strategic bidding behavior are not adapted to the non-stationary and non-Markovian environment involving multidimensional and continuous state and action spaces. This paper proposes a reinforcement learning method to overcome these limitations. The proposed method discovers the input space structure through the self-organizing map, exploits learned experience through Roth-Erev reinforcement learning and explores through the actor critic map. Simulation results from experiments show that the proposed method outperforms Simulated Annealing Q-Learning and Variant Roth-Erev reinforcement learning. The proposed method is a step towards more realistic agent learning in Agent-based Computational Economics.

This paper presents a novel approach for acquiring dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass (CoM). In our approach, we combine both a model-based CoM controller and a model-free reinforcement learning (RL) method to acquire dynamic whole-body movements in humanoid robots. (i) To cope with high dimensionality, we use a model-based CoM controller as a basic controller that derives joint angular velocities from the desired CoM velocity. The balancing issue can also be considered in the controller. (ii) The RL method is used to acquire a controller that generates the desired CoM velocity based on the current state. To demonstrate the effectiveness of our approach, we apply it to a ball-punching task on a simulated humanoid robot model. The acquired whole-body punching movement was also demonstrated on Fujitsu's Hoap-2 humanoid robot.

Learning to solve complex manipulation tasks from visual observations is a dominant challenge for real-world robot learning. Although deep reinforcement learning algorithms have recently demonstrated impressive results in this context, they still require an impractical amount of time-consuming trial-and-error iterations. In this work, we consider the promising alternative paradigm of interactive learning in which a human teacher provides feedback to the policy during execution, as opposed to imitation learning where a pre-collected dataset of perfect demonstrations is used. Our proposed CEILing (Corrective and Evaluative Interactive Learning) framework combines both corrective and evaluative feedback from the teacher to train a stochastic policy in an asynchronous manner, and employs a dedicated mechanism to trade off human corrections with the robot’s own experience. We present results obtained with our framework in extensive simulation and real-world experiments to demonstrate that CEILing can effectively solve complex robot manipulation tasks directly from raw images in less than one hour of real-world training.

We propose a new `Mark-Ant-Walk' algorithm for robust and efficient covering of continuous domains by ant-like robots with very limited capabilities. The robots can mark places visited with pheromone marks and sense the level of the pheromone in their local neighborhood. In the case of multiple robots, these pheromone marks can be sensed by all robots and provide the only way of (indirect) communication between the robots. The robots are assumed to be memoryless, and to have no information besides that mentioned above. Despite the robots' simplicity we show that they are able, by running a very simple rule of behavior, to ensure efficient covering of arbitrary connected domains, including non-planar and multidimensional domains. The novelty of our algorithm lies in the fact that, unlike previously proposed methods, our algorithm works on continuous domains without relying on some induced underlying graph effectively reducing the problem to a discrete case of graph covering. In addition, we demonstrate various benefits of our algorithm such as ability to cope with arbitrary initial pheromone profile and a bounded constant time between two successive visits of the robot at the same location, making it suitable for patrolling. Finally, we provide a new theoretical bound on covering time of a wide family of such mark and walk algorithms.

Multi-label learning algorithms have attracted more and more attention as of recent. This is mainly because real-world data is generally associated with multiple and non-exclusive labels, which could correspond to different objects, scenes, actions, and attributes. In this paper, we consider the following challenging multi-label stream scenario: the new labels emerge continuously in the changing environments, and are assigned to the previous data. In this setting, data mining solutions must be able to learn the new concepts and avoid catastrophic forgetting simultaneously. We propose a novel continual and interactive feature distillation-based learning framework (CIFDM), to effectively classify instances with novel labels. We utilize the knowledge from the previous tasks to learn new knowledge to solve the current task. Then, the system compresses historical and novel knowledge and preserves it while waiting for new emerging tasks. CIFDM consists of three components: 1) a knowledge bank that stores the existing feature-level compressed knowledge, and predicts the observed labels so far; 2) a pioneer module that aims to learn and predict new emerged labels based on knowledge bank.; 3) an interactive knowledge compression function which is used to compress and transfer the new knowledge to the bank, and then apply the current compressed knowledge to initialize the label embedding of the pioneer for the next task.

Spinal Cord Injury (SCI) can have a devastating impact on motor, sensory and autonomic functions. Its consequences extend far beyond the loss of physical mobility, having profound effects on psychological health and quality of life. While no established treatment exists, neurotechnologies such as epidural electrical stimulation (EES) have shown promise by restoring standing and walking people with SCI. However, the clinical adoption of EES is hindered by its dependence on expert-driven, time-consuming programming of individualized stimulation patterns. This thesis first introduces an interactive machine learning framework using Gaussian process (GP) based Bayesian optimization (BO) to automatically identify stimulation settings that generate isolated functional movements, creating a stimulation library to support subsequent physical therapy. For this, we formulated a reward function, based on kinematic and electromyography (EMG) signals, tailored to lower-limb motor tasks. To map the input space to this function, our kernel used a distance metric capable of handling both numeric and non numeric parameters present in EES. Yet, for more complex movements, we need concurrent active stimulations. This, however, might cause parameters to interact with each other in a detrimental way. Because the relative coactivation between muscles was the key feature to resolve this problem, we implemented a composite approach whereby the GP models the muscles directly. To validate the methodology, a medical-grade programming interface was built to let the algorithm autonomously adjust stimulation settings. Tests in individuals with complete and incomplete SCI showed that the framework reliably identified effective parameters. To advance integration of this framework into clinical practice, the second part of the thesis involved implanting a brain-spine digital bridge in participants with chronic motor complete SCI, combining electrocorticography (ECoG) recording with spinal cord stimulation. Because these patients have no residual movement, optimized stimulation protocols were critical. Using the developed framework together with a decoding algorithm, both participants were able to walk overground with the assistance of a front-wheel walker. Moreover, by leveraging ECoG signals, we investigated the potential of brain-derived feedback as a direct reward signal for optimization. This strategy circumvents the need for external sensors and manual reward definitions, paving the way for adaptive neuroprosthetic systems guided directly by the patient's brain activity. The final part of the thesis extends EES beyond lower-limb mobility by establishing a brain-cervical spinal cord digital bridge in individuals with chronic incomplete tetraplegia. To target hand and arm movements, stimulation libraries were created through human-guided optimization and by adapting the existing framework with a reward function focused solely on kinematics. Preliminary tests in one participant showed that, the algorithm successfully identified stimulation settings that support useful motor outcomes for rehabilitation. Collectively, this thesis shows that automated optimization frameworks can be applied across different contexts, paving the way for more accessible and personalized neurorehabilitation therapies.

Interactive feature learning framework for mask-occluded face recognition

In this paper, we introduce ACAT-G, an interactive dialogue learning framework that incorporates constant human feedback into fine-tuning language models in order to assist conditioned dialog generation. The system takes in a limited amount of input from a human and generates personalized response corresponding to the context of the conversation within natural dialog time-frame. By combining inspirations from online learning, reinforcement learning, and large scale language models, we expect this project to provide a foundation for human-in-the-loop conditional dialog generation tasks.

This letter proposes a new idea to improve learning efficiency in reinforcement learning (RL) with the actor-critic method used as a muscle controller for posture stabilization of the human arm. Actor-critic RL (ACRL) is used for simulations to realize posture controls in humans or robots using muscle tension control. However, it requires very high computational costs to acquire a better muscle control policy for desirable postures. For efficient ACRL, we focused on embodiment that is supposed to potentially achieve efficient controls in research fields of artificial intelligence or robotics. According to the neurophysiology of motion control obtained from experimental studies using animals or humans, the pedunculopontine tegmental nucleus (PPTn) induces muscle tone suppression, and the midbrain locomotor region (MLR) induces muscle tone promotion. PPTn and MLR modulate the activation levels of mutually antagonizing muscles such as flexors and extensors in a process through which control signals are translated from the substantia nigra reticulata to the brain stem. Therefore, we hypothesized that the PPTn and MLR could control muscle tone, that is, the maximum values of activation levels of mutually antagonizing muscles using different sigmoidal functions for each muscle; then we introduced antagonism function models (AFMs) of PPTn and MLR for individual muscles, incorporating the hypothesis into the process to determine the activation level of each muscle based on the output of the actor in ACRL. ACRL with AFMs representing the embodiment of muscle tone successfully achieved posture stabilization in five joint motions of the right arm of a human adult male under gravity in predetermined target angles at an earlier period of learning than the learning methods without AFMs. The results obtained from this study suggest that the introduction of embodiment of muscle tone can enhance learning efficiency in posture stabilization disorders of humans or humanoid robots.

Abstract: The proposed theme of this paper is how "Arts" and "Science," with the inclusion of the topic of "Health", can be combined through a story based on research about innovation in the Past. The goal of this paper is threefold. First, to present a research-based story about Jacob Bruce (1669–1735), a scholar of Scottish descent born in Moscow, and his innovation – the prototype of Artificial Intelligence, humanoid robot. Second, to illustrate the new experimental interdisciplinary method of storytelling - SMS (Stories based on Music about Scientists), a novel approach that integrates Arts (Music, Poem, Digital images), Science (Historical Research about Innovations of the Past), and Culture (cultural heritage with metaphor of a philosophical soul development). Third, to explain how this method can be applied. SMS method created in 2012 uses the Avant-garde genre of Music with the integration of interdisciplinary knowledge, holistic approach, and implies a deeper level of integration of various disciplines. The theme of this SMS is a song-story about the beautiful humanoid girl-robot (but with a different technology of the 17th century), which serves as a prototype of the modern homunculus or AI created by Jacob Bruce. AI humanoid already resembled a natural human being by the end of the 17th century. The author explains and discusses how and why SMS can be used as an Innovative Teaching and Learning method, related to Science and Arts (similar to STEAM approach), with the power of Lifelong Learning (LLL), and pedagogical didactics across branches of sciences, thus stimulating innovations and discoveries. Keywords: Arts and Science; Innovations; STEAM/SMS (Stories based on Music about Scientists) method; Jacob Bruce (1669–1735); Innovative Teaching and Learning; Health