EWA Model with Recency Effect and Limited Memory

When multiple participants work on a workflow that represents a large, collaborative activity, it is important to have a well defined process to determine the portions of the workflow that each participant is responsible for executing. In this paper, we describe a process and related algorithms required to assign tasks in a workflow, to hosts that are willing to carry out the execution of these tasks, and thereby contributing to the completion of the activity. This problem is a stylized form of the multi-processor scheduling algorithm which has been shown to be NP-Hard. Further complicating the issue is that we are targeting our approach to mobile ad hoc networks, where hosts are physically mobile, communication links are frequently interrupted, and spatiotemporal considerations become increasingly important. We describe a distributed approach to task allocation in mobile ad hoc networks that employs heuristics to assign tasks in a workflow to mobile hosts based on their capabilities and their mobility patterns. We have implemented our algorithms in the context of CiAN, a workflow management system (WfMS) supporting collaborations in a mobile environment. In addition, we also present performance data of our algorithm and compare it to naive and brute force approaches.

Modern computer architectures have evolved from uni-processor platforms to multi-processor and multi-core platforms, but traditional real-time distributed middleware such as RT-CORBA has not kept pace with that evolution. To address those issues, this paper describes the design and implementation of MCFlow, a new real-time distributed middleware for dependent task graphs running on multi-core platforms. MCFlow provides the following contributions to the state of the art in real-time middleware: (1) it provides an efficient C++ based component model through which computations can be configured flexibly for execution within a single core, across cores of a common host, or spanning multiple hosts; (2) it allows optimizations for inter-component communication to avoid data copying without sacrificing the parallel executability of data dependent tasks; (3) it strictly separates timing and functional concerns of an application so that they can evolve and can be configured independently; and (4) it provides a novel event dispatching architecture that uses lock free algorithms to avoid mutex locking and reduce memory contention, CPU context switching, and priority inversion. We also present an empirical evaluation that demonstrates the efficacy of our approach. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 The Design and Implementation of MCFlow: a Real-time Multi-core Aware Middleware for Dependent Task Graphs Huang-Ming Huang, Christopher Gill, Chengyang Lu Department of Computer Science and Engineering, Washington University St. Louis, MO, USA {hh1, cdgill, lu}@cse.wustl.edu Abstract—Modern computer architectures have evolved from uni-processor platforms to multi-processor and multi-core platforms, but traditional real-time distributed middleware such as RT-CORBA has not kept pace with that evolution. To address those issues, this paper describes the design and implementation of MCFlow, a new real-time distributed middleware for dependent task graphs running on multi-core platforms. MCFlow provides the following contributions to the state of the art in real-time middleware: (1) it provides an efficient C++ based component model through which computations can be configured flexibly for execution within a single core, across cores of a common host, or spanning multiple hosts; (2) it allows optimizations for inter-component communication to avoid data copying without sacrificing the parallel executability of data dependent tasks; (3) it strictly separates timing and functional concerns of an application so that they can evolve and can be configured independently; and (4) it provides a novel event dispatching architecture that uses lock free algorithms to avoid mutex locking and reduce memory contention, CPU context switching, and priority inversion. We also present an empirical evaluation that demonstrates the efficacy of our approach.Modern computer architectures have evolved from uni-processor platforms to multi-processor and multi-core platforms, but traditional real-time distributed middleware such as RT-CORBA has not kept pace with that evolution. To address those issues, this paper describes the design and implementation of MCFlow, a new real-time distributed middleware for dependent task graphs running on multi-core platforms. MCFlow provides the following contributions to the state of the art in real-time middleware: (1) it provides an efficient C++ based component model through which computations can be configured flexibly for execution within a single core, across cores of a common host, or spanning multiple hosts; (2) it allows optimizations for inter-component communication to avoid data copying without sacrificing the parallel executability of data dependent tasks; (3) it strictly separates timing and functional concerns of an application so that they can evolve and can be configured independently; and (4) it provides a novel event dispatching architecture that uses lock free algorithms to avoid mutex locking and reduce memory contention, CPU context switching, and priority inversion. We also present an empirical evaluation that demonstrates the efficacy of our approach. Keywords-component; middleware

Fine-grained hardware protection could provide a powerful and effective means for isolating untrusted code. However, previous techniques for providing fine-grained protection in hardware have lead to poor performance. Legba has been proposed as a new caching architecture, designed to reduce the granularity of protection, without slowing down the processor. Unfortunately, the designers of Legba have not attempted an implementation. Instead, all of their analysis is based purely on simulations. We present an implementation of the Legba design on a MIPS Core Processor, along with an analysis of our observations and results. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Implementing Legba: Fine-Grained Memory Protection Aaron Sheffield, Ross T. Sowell, Mike Wilson Department of Computer Science & Engineering Washington University in St. Louis One Brookings Drive, Campus Box 1045 St. Louis, Missouri 63130-4899 {ajs6, mlw2}@cec.wustl.edu rsowell@gmail.com Abstract—Fine-grained hardware protection could provide a powerful and effective means for isolating untrusted code. However, previous techniques for providing fine-grained protection in hardware have lead to poor performance. Legba has been proposed as a new caching architecture, designed to reduce the granularity of protection, without slowing down the processor. Unfortunately, the designers of Legba have not attempted an implementation. Instead, all of their analysis is based purely on simulations. We present an implementation of the Legba design on a MIPS Core Processor, along with an analysis of our observations and results.Fine-grained hardware protection could provide a powerful and effective means for isolating untrusted code. However, previous techniques for providing fine-grained protection in hardware have lead to poor performance. Legba has been proposed as a new caching architecture, designed to reduce the granularity of protection, without slowing down the processor. Unfortunately, the designers of Legba have not attempted an implementation. Instead, all of their analysis is based purely on simulations. We present an implementation of the Legba design on a MIPS Core Processor, along with an analysis of our observations and results.

The field of manifold learning provides powerful tools for parameterizing high-dimensional data points with a small number of parameters when this data lies on or near some manifold. Images can be thought of as points in some high-dimensional image space where each coordinate represents the intensity value of a single pixel. These manifold learning techniques have been successfully applied to simple image sets, such as handwriting data and a statue in a tightly controlled environment. However, they fail in the case of natural image sets, even those that only vary due to a single degree of freedom, such as a person walking or a heart beating. Parameterizing data sets such as these will allow for additional constraints on traditional computer vision problems such as segmentation and tracking. This dissertation explores the reasons why classical manifold learning algorithms fail on natural image sets and proposes new algorithms for parameterizing this type of data. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 WASHINGTON UNIVERSITY THE HENRY EDWIN SEVER GRADUATE SCHOOL DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING MANIFOLD LEARNING FOR NATURAL IMAGE SETS by Richard M. Souvenir, M.S. Computer Science, B.S. Applied Science Prepared under the direction of Professor Robert Pless A dissertation presented to the Henry Edwin Sever Graduate School of Washington University in partial fulfillment of the requirements for the degree of DOCTOR OF SCIENCE August 2006 Saint Louis, Missouri WASHINGTON UNIVERSITY THE HENRY EDWIN SEVER GRADUATE SCHOOL DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

Recent advances in High-Throughput Sequencing (HTS) technology have greatly facilitated the researches in bioinformatics field. With the ultra-high sequencing speed and improved base-calling accuracy, Illumina Genome Analyzer is currently the most widely used platform in the field. To use the raw reads generated from the sequencing machine, the 3’ adapter sequence attached to the real read in the process of ligation needs to be correctly trimmed. This is often done by some inhouse scripts or different packages with various parameters. They either use the Smith-Waterman algorithm or search for an exact match of the 3’ adapter sequence. In this report, I investigated methodologies as well as the strengths and weaknesses of five representative mainstream adapter trimming tools in order to suggest a direction for other researchers. Furthermore, four sets of detailed analysis were performed to evaluate the performances of these tools. I demonstrated that my adapter trimming method is flexible, accurate and efficient for Next Generation Sequencing (NGS) analysis. Type of Report: MS Project Report Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Master’s Project Report LCScanner: An Efficient and Accurate Trimming Tool for Illumina Next Generation Sequencing Reads by Xiang Zhou Master of Engineering in Computer Science Washington University in St. Louis, 2012 Research Advisor: Professor Weixiong Zhang

Protein databases have grown exponentially over the last decade. This exponential growth has made extracting valuable information from these databases increasingly time consuming. This project presents a new method of accelerating a commonly used program for performing similarity searching on protein databases, BLASTP. This project describes the design and implementation of Mercury BLASTP, a customized hardware accelerated variant of BLASTP. This project focuses on the gapped alignment stage of Mercury BLASTP and provides design details and implementation results. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Acceleration of Gapped Alignment in BLASTP Using the Mercury System Brandon Harris bbh2@wustl.edu Abstract Protein databases have grown exponentially over the last decade. This exponential growth has made extracting valuable information from these databases increasingly time consuming. This project presents a new method of accelerating a commonly used program for performing similarity searching on protein databases, BLASTP. This project describes the design and implementation of Mercury BLASTP, a customized hardware accelerated variant of BLASTP. This project focuses on the gapped alignment stage of Mercury BLASTP and provides design details and implementation results.Protein databases have grown exponentially over the last decade. This exponential growth has made extracting valuable information from these databases increasingly time consuming. This project presents a new method of accelerating a commonly used program for performing similarity searching on protein databases, BLASTP. This project describes the design and implementation of Mercury BLASTP, a customized hardware accelerated variant of BLASTP. This project focuses on the gapped alignment stage of Mercury BLASTP and provides design details and implementation results.

Identification of groups of functionally related genes from high throughput gene expression data is an important step towards elucidating gene functions at a global scale. Most existing approaches treat gene expression data as points in a metric space, and apply conventional clustering algorithms to identify sets of genes that are close to each other in the metric space. However, they usually ignore the topology of the underlying biological networks. In this paper, we propose a network-based clustering method that is biologically more realistic. Given a gene expression data set, we apply a rank-based transformation to obtain a sparse co-expression network, and use a novel spectral clustering algorithm to identify natural community structures in the network, which correspond to gene functional modules. We have tested the method on two large-scale gene expression data sets in yeast and Arabidopsis, respectively. The results show that the clusters identified by our method on these datasets are functionally richer and more coherent than the clusters from the standard k-means clustering algorithm. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Discovering Functional Modules by Clustering Gene Co-expression Networks Jianhua Ruan and Weixiong Zhang Department of Computer Science and Department of Genetics Washington University in St. Louis, St. Louis, MO 63130, USA jruan@cse.wustl.edu, zhang@cse.wustl.edu

Service provisioning has gained significant attention as a promising programming model for heterogeneous wireless sensor networks. Its key idea is to exploit the decoupling of service providers and consumers to enable platform-independent applications that are dynamically bound to platform-specific services. We explore novel adaptive service binding strategies that are able to cope with network dynamics and to promote energy conservation. To achieve this goal, we developed policies and algorithms that automatically switch providers in response to network topology changes and adapt application behavior when opportunities for energy savings surface. The latter is accomplished by providing limited information about the energy consumption associated with using various services, by systematically exploiting opportunities for sharing service invocations, and by exploiting the broadcast nature of wireless communication in WSNs. The policies and algorithms have been implemented and evaluated on two disparate WSN platforms, the TelosB and Imote2. Empirical results show that adaptive service provisioning can significantly increase service availability and enable energy-aware service binding decisions that result in increased energy efficiency, while imposing minimal additional burden on the application, service, and device developers. Applications involving medical patient monitoring and structural health monitoring are used in the evaluation process. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Adaptive Service Provisioning for Wireless Sensor Networks Chien-Liang Fok, Gruia-Catalin Roman and Chenyang Lu Washington University in St. Louis {liang, roman, lu}@cse.wustl.edu Abstract—Service provisioning has gained significant attention as a promising programming model for heterogeneous wireless sensor networks. Its key idea is to exploit the decoupling of service providers and consumers to enable platform-independent applications that are dynamically bound to platform-specific services. We explore novel adaptive service binding strategies that are able to cope with network dynamics and to promote energy conservation. To achieve this goal, we developed policies and algorithms that automatically switch providers in response to network topology changes and adapt application behavior when opportunities for energy savings surface. The latter is accomplished by providing limited information about the energy consumption associated with using various services, by systematically exploiting opportunities for sharing service invocations, and by exploiting the broadcast nature of wireless communication in WSNs. The policies and algorithms have been implemented and evaluated on two disparate WSN platforms, the TelosB and Imote2. Empirical results show that adaptive service provisioning can significantly increase service availability and enable energyaware service binding decisions that result in increased energy efficiency, while imposing minimal additional burden on the application, service, and device developers. Applications involving medical patient monitoring and structural health monitoring are used in the evaluation process, demonstrating the system’s efficacy.Service provisioning has gained significant attention as a promising programming model for heterogeneous wireless sensor networks. Its key idea is to exploit the decoupling of service providers and consumers to enable platform-independent applications that are dynamically bound to platform-specific services. We explore novel adaptive service binding strategies that are able to cope with network dynamics and to promote energy conservation. To achieve this goal, we developed policies and algorithms that automatically switch providers in response to network topology changes and adapt application behavior when opportunities for energy savings surface. The latter is accomplished by providing limited information about the energy consumption associated with using various services, by systematically exploiting opportunities for sharing service invocations, and by exploiting the broadcast nature of wireless communication in WSNs. The policies and algorithms have been implemented and evaluated on two disparate WSN platforms, the TelosB and Imote2. Empirical results show that adaptive service provisioning can significantly increase service availability and enable energyaware service binding decisions that result in increased energy efficiency, while imposing minimal additional burden on the application, service, and device developers. Applications involving medical patient monitoring and structural health monitoring are used in the evaluation process, demonstrating the system’s efficacy.

OF THE DISSERTATION Application-Specific Memory Subsystems by Joseph G. Wingbermuehle Doctor of Philosophy in Computer Science Washington University in St. Louis, 2015 Professor Roger D. Chamberlain, Chair The disparity in performance between processors and main memories has led computer architects to incorporate large cache hierarchies in modern computers. These cache hierarchies are designed to be general-purpose in that they strive to provide the best possible performance across a wide range of applications. However, such a memory subsystem does not necessarily provide the best possible performance for a particular application. Although general-purpose memory subsystems are desirable when the work-load is unknown and the memory subsystem must remain fixed, when this is not the case a custom memory subsystem may be beneficial. For example, in an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA) designed to run a particular application, a custom memory subsystem optimized for that application would be desirable. In addition, when there are tunable parameters in the memory subsystem, it may make sense to change these parameters depending on the application being run. Such a situation arises today with FPGAs and, to a lesser extent, GPUs, and it is plausible that general-purpose computers will begin to support greater flexibility in the memory subsystem in the future. In this dissertation, we first show that it is possible to create application-specific memory subsystems that provide much better performance than a general-purpose memory subsystem. In addition, we show a way to discover such memory subsystems automatically using a superoptimization technique on memory address traces gathered from applications. This allows one to generate a custom memory subsystem with little effort. xi We next show that our memory subsystem superoptimization technique can be used to optimize for objectives other than performance. As an example, we show that it is possible to reduce the number of writes to the main memory, which can be useful for main memories with limited write durability, such as flash or Phase-Change Memory (PCM). Finally, we show how to superoptimize memory subsystems for streaming applications, which are a class of parallel applications. In particular, we show that, through the use of ScalaPipe, we can author and deploy streaming applications targeting FPGAs with superoptimized memory subsystems. ScalaPipe is a domain-specific language (DSL) embedded in the Scala programming language for generating streaming applications that can be implemented on CPUs and FPGAs. Using the ScalaPipe implementation, we are able to demonstrate actual performance improvements using the superoptimized memory subsystem with applications implemented in hardware.

With the development of new sequencing techniques, whole genomes of many species have become available. This huge amount of data gives rise to new opportunities and challenges. These new sequences provide valuable information on relationships among species, e.g. genome recombination and conservation. One of the principal ways to investigate such information is multiple sequence alignment (MSA). Currently, there is large amount of MSA data on the internet, such as the UCSC genome database, but how to effectively use this information to solve classical and new problems is still an area lacking of exploration. In this thesis, we explored how to use this information in four problems, i.e. sequence orthology search problem, multiple alignment improvement problem, short read mapping problem, and genome rearrangement inference problem. For the first problem, we developed a EM algorithm to iteratively align a query with a multiple alignment database with the information from a phylogeny relating the query species and the species in the multiple alignment. We also infer the query's location in the phylogeny. We showed that by doing alignment and phylogeny inference together, we can improve the accuracies for both problems. For the second problem, we developed an optimization algorithm to iteratively refine the multiple alignment quality. Experiment results showed our algorithm is very stable in term of resulting alignments. The results showed that our method is more accurate than existing methods, i.e. Mafft, Clustal-O, and Mavid, on test data from three sets of species from the UCSC genome database. For the third problem, we developed a model, PhyMap, to align a read to a multiple alignment allowing mismatches and indels. PhyMap computes local alignments of a query sequence against a fixed multiple-genome alignment of closely related species. PhyMap uses a known phylogenetic tree on the species in the multiple alignment to improve the quality of its computed alignments while also estimating the placement of the query on this tree. Both theoretical computation and experiment results show that our model can differentiate between orthologous and paralogous alignments better than other popular short read mapping tools (BWA, BOWTIE and BLAST). For the fourth problem, we gave a simple genome recombination model which can express insertions, deletions, inversions, translocations and inverted translocations on aligned genome segments. We also developed an MCMC algorithm to infer the order of the query segments. We proved that using any Euclidian metrics to measure distance between two sequence orders in the tree optimization goal function will lead to a degenerated solution where the inferred order will be the order of one of the leaf nodes. We also gave a graph-based formulation of the problem which can represent the probability distribution of the order of the query sequences.

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

This paper presents an ethologically inspired actionselection mechanism, part of a larger architecture for selfanimated artificial animals (agents) that communicate emotions to each other, influencing each other’s behaviour. The problem of action selection is that of choosing at each moment in time the most appropriate action out of a repertoire of possible actions. Action selection algorithms have been proposed by both ethologists and computer scientists. Models suggested by ethologists are usually at a conceptual level, while those of computer scientists (with some exceptions [3] ) generally do not take into account classical ethology theories. According several ethologists [2], a hierarchical structure represents an essential organising principle of complex behaviours. We have chosen to add emotion to a hierarchical action selection mechanism so that behaviour shows persistence (emotion acting as a short term memory [2]) and to avoid dithering between competing behaviours, namely the herding group behaviour, and the individual behaviour of grazing.

Machine Learning is a modern and actively developing field of computer science, devoted to extracting and estimating dependencies from empirical data. It combines such fields as statistics, optimization theory and artificial intelligence. In practical tasks, the general aim of Machine Learning is to construct algorithms able to generalize and predict in previously unseen situations based on some set of examples. Given some finite information, Machine Learning provides ways to exract knowledge, describe, explain and predict from data. Kernel Methods are one of the most successful branches of Machine Learning. They allow applying linear algorithms with well-founded properties such as generalization ability, to non-linear real-life problems. Support Vector Machine is a well-known example of a kernel method, which has found a wide range of applications in data analysis nowadays. In many practical applications, some additional prior knowledge is often available. This can be the knowledge about the data domain, invariant transformations, inner geometrical structures in data, some properties of the underlying process, etc. If used smartly, this information can provide significant improvement to any data processing algorithm. Thus, it is important to develop methods for incorporating prior knowledge into data-dependent models. The main objective of this thesis is to investigate approaches towards learning with kernel methods using prior knowledge. Invariant learning with kernel methods is considered in more details. In the first part of the thesis, kernels are developed which incorporate prior knowledge on invariant transformations. They apply when the desired transformation produce an object around every example, assuming that all points in the given object share the same class. Different types of objects, including hard geometrical objects and distributions are considered. These kernels were then applied for images classification with Support Vector Machines. Next, algorithms which specifically include prior knowledge are considered. An algorithm which linearly classifies distributions by their domain was developed. It is constructed such that it allows to apply kernels to solve non-linear tasks. Thus, it combines the discriminative power of support vector machines and the well-developed framework of generative models. It can be applied to a number of real-life tasks which include data represented as distributions. In the last part of the thesis, the use of unlabelled data as a source of prior knowledge is considered. The technique of modelling the unlabelled data with a graph is taken as a baseline from semi-supervised manifold learning. For classification problems, we use this apporach for building graph models of invariant manifolds. For regression problems, we use unlabelled data to take into account the inner geometry of the input space. To conclude, in this thesis we developed a number of approaches for incorporating some prior knowledge into kernel methods. We proposed invariant kernels for existing algorithms, developed new algorithms and adapted a technique taken from semi-supervised learning for invariant learning. In all these cases, links with related state-of-the-art approaches were investigated. Several illustrative experiments were carried out on real data on optical character recognition, face image classification, brain-computer interfaces, and a number of benchmark and synthetic datasets.

OF THE DISSERTATION In vivo function of Otopetrin 1 in the vestibular sensory epithelium by Euysoo Kim Doctor of Philosophy in Biology and Biomedical Sciences (Molecular and Cell Biology) Washington University in St. Louis, 2009 Professor David M. Ornitz, Chairperson Otopetrin family genes encode multi-transmembrane domain proteins with three highly conserved domains. In mice, three Otopetrin paralogues are found. One of its members, Otopetrin 1 (Otop1) has been previously shown to be essential for the formation of otoconia in the vestibular system of the inner ear. Otoconia are calcium carbonate biominerals that are required for normal balance and the sensation of linear acceleration with respect to gravity. The mechanism by which OTOP1 mediates otoconia biosynthesis is not known, but the ability of OTOP1 to modulate [Ca]i in response to purinergic signals in heterologous systems suggest that OTOP1 may be involved in concentrating [Ca] within the sensory epithelium and/or the globular substance vesicles, which are secreted from the epithelium. In this study, we generated null alleles of Otop1 (Otop1) and Otop2 (Otop2) in mice to investigate the in vivo functions of these genes. The otoconial agenesis phenotype in Otop1 mice suggests that the most sensitive and important role for OTOP1 during development is mediating otoconia formation. X-gal staining and immunohistochemical analysis reveal that Otop1 is expressed in the developing and adult supporting cells in the non-striola region of the maculae. Within these sensory epithelia, OTOP1 localizes near the apical end. In primary

Solving the Credit Assignment Problem: The interaction of Explicit and Implicit learning with Internal and External State Information Wai-Tat Fu (wfu@cmu.edu) Human Factors Division and Beckman Institute University of Illinois at Urbana-Champaign 1 Airport Road, Savoy, IL 61874, USA John R. Anderson (ja+@cmu.edu) Department of Psychology Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 USA recognition of external state information (signs on the walls). Indeed, many have argued that real-world skills often involve the interplay between cognition (internal), perception, and action (external) that the understanding of these interactive skills requires careful study of how internal (memory) and external information (cues in the environment) are processed in the learning processes (Ballard, 1997; Fu & Gray, 2000; 2004; Gray & Fu, 2004; Larkin, 1989; Gray, Sims, Fu, & Schoelles, in press). The navigation problem above is an example of one of the most difficult situations in skill learning: when the learner has to perform a sequence of actions but only gets feedback on their success at the end of the sequence (e.g., when the destination is reached). This creates a credit-assignment problem, in which the learner has to assign credits to earlier actions that are responsible for eventual success. When actions are interdependent, either memory of previous actions or recognition of the correct problem state in the external environment is required to properly assign credits to the appropriate actions. In this article, we present results from an experiment in which we study how people learn to solve the credit-assignment problem in a simple but challenging example of such a situation. Our focus is on the recent proposal that humans exhibit two distinct learning processes and we apply it to learning of action sequences with delayed feedback: an explicit process (with awareness) that requires memory for actions and outcomes, and an implicit process (without awareness) that does not require such memory. We will first review research in some related areas that informed the design of our experiment. Abstract In most problem-solving activities, feedback is received at the end of an action sequence. This creates a credit-assignment problem where the learner must associate the feedback with earlier actions, and the interdependencies of actions require the learner to either remember past choices of actions (internal state information) or rely on external cues in the environment (external state information) to select the right actions. We investigated the nature of explicit and implicit learning processes in the credit-assignment problem using a probabilistic sequential choice task with and without external state information. We found that when explicit memory encoding was dominant, subjects were faster to select the better option in their first choices than in the last choices; when implicit reinforcement learning was dominant subjects were faster to select the better option in their last choices than in their first choices. However, implicit reinforcement learning was only successful when distinct external state information was available. The results suggest the nature of learning in credit assignment: an explicit memory encoding process that keeps track of internal state information and a reinforcement-learning process that uses state information to propagate reinforcement backwards to previous choices. However, the implicit reinforcement learning process is effective only when the valences can be attributed to the appropriate states in the system – either internally generated states in the cognitive system or externally presented stimuli in the environment. Introduction Consider a person navigating in a large office building. The person has to decide when to turn left or right at various hallway intersections. The sequence of decisions is interdependent – e.g., turning left at a particular hallway intersection will affect the decisions at the next intersections. The person may therefore need to keep track of previous actions to inform what actions to take in the future. In reality, memory of previous actions (internal state information) may not be necessary as people can explicitly seek information in the environment (external state information) to know where one is located or which direction to go to reach a destination (Fu & Gray, 2006). Learning to navigate is therefore likely to involve both the retention of internal state information (memory) and the Explicit and Implicit Learning Probability Learning and Classification There have been numerous studies on the learning of the probabilistic relationship between choices and their consequences. The simplest situation is the probability- learning experiment in which subjects guess which of the alternatives occurs and then receives feedback on their guesses (e.g., Estes, 1964). One robust finding is that subjects often “probability match”; that is, they will choose a particular alternative with the same probability that it is reinforced (e.g., Friedman et al., 1964). This leads many to propose that probability matching is the result of an implicit