SCARL: Attentive Reinforcement Learning-Based Scheduling in a Multi-Resource Heterogeneous Cluster

Reinforcement Learning with Foregone Payoff Information in Normal Form Games

Grid Computing is a widespread distributed computing environment that consists of heterogeneous resources and it provides huge computational power for the large scale distributed application. In this job scheduling is an essential objective to prioritize the jobs based on the job characteristics. It is a big challenge to design and implement an efficient Job Scheduling algorithm and the Job Scheduling model. This paper proposes the priority based Job Scheduling algorithm with Nonpreemptive Priority (NPRP) Job Scheduling model. The priority based job scheduling algorithm is the combination of Backfill algorithm and Earliest Deadline based algorithm. The proposed model has been integrated with Care Resource Broker (CRB) which supports virtualized grid environment. The proposed model has been integrated with CRB helps to make the intelligent scheduling decisions for prioritizing the jobs and also the proposed model minimizes the turnaround time, waiting time and response time of the jobs available in the queue and the user's quality of service has been maximized in the heterogenous and virtualized grid environment.

The development and maturation of multisensory processing and integration depend on the brain’s ability to detect patterns and adjust behaviour based on feedback. However, the developmental dynamics of these mechanisms are not yet fully understood. This study examines neural and behavioural differences in multisensory statistical and reinforcement learning processes by comparing adults and children, and further investigates these processes across two learning tasks of varying difficulty in children. Using a discriminative choice and a match recognition multisensory associative learning task with embedded statistical stimulus regularities, 28 adults (19.0–30.9 years) and 2 groups of children (N = 28 and N = 29, 8.5–12.8 years) learned novel, non-linguistic associations of audio–visual or tactile–visual stimuli. Behaviourally, accuracy increased and reaction time decreased over time in all groups, and these learning effects were stronger in adults than in children and for audio–visual than for tactile–visual learning. Computational modelling revealed that adults employed more advanced learning strategies, with greater sensitivity to value differences, and more deterministic choice patterns. Behaviourally, reaction times were slower for more surprising trials across all groups. Neuroimaging showed that statistical surprise modulated activity in the lateral prefrontal cortex, intraparietal sulcus, and anterior cingulate cortex, while reward prediction errors were linked to striatal, medial frontal, and hippocampal regions. Reward prediction errors were consistent between adults and children and between task variants, whereas neural surprise processing emerged only in the match recognition task and showed developmental differences, indicating immaturity in children’s ability to detect statistical regularities. These findings highlight the importance of task design when comparing learning differences across developmental groups and showcase how reinforcement and statistical learning can be studied in parallel.

Target searching is crucial in real-world scenarios such as search and rescue in disaster sites and battlefield target reconnaissance. Unmanned aerial vehicles (UAVs) are an ideal technical solution for target searching in large-scale and high-risk areas because they are agile, low cost, and able to collaborate and carry different sensors. In complex scenarios like battlefields, due to the lack of communication infrastructures and the intensive interference, UAVs often operate in communication denied environments. As a result, fast and reliable communication channels between UAVs and ground operators are difficult to establish. Thus, in such conditions, UAVs must be able to complete tasks autonomously and intelligently, without receiving real-time commands from the operators. With the rapid advances in artificial intelligence, reinforcement learning has shown potentiality for solving continuous decision problems. The target searching problem studied in this paper falls into this category and is suitable for adopting reinforcement learning technologies. However, the feasibility of reinforcement learning in UAV-based target searching in communication denied environments is not clear and, thus, requires in-depth investigations. As a pilot study in this direction, this paper models the target searching problem in communication denied and confrontation situations and proposes a simulation environment based on this model. Extensive experiments are conducted to answer the following questions. (1) Can reinforcement learning be applied in target searching by multi-UAVs in communication denied environments? (2) What are the advantages and disadvantages of different reinforcement learning algorithms in solving this problem? (3) How the degree of communication denial influences the performance of these algorithms? The current mainstream reinforcement learning technologies are adopted to perform simulations, whose results are analyzed quantitatively, leading to the following observations. (1) Reinforcement learning can effectively solve target searching problems for multi-UAVs in communication denied environments. (2) Compared with other algorithms, an autonomous decision-making UAV cluster based on a deep Q-network (DQN)exhibits the best problem-solving ability. (3) The algorithm performance changes with the degree of communication denial but remains largely stable when the communication condition varies.

Reinforcement learning is a framing of enabling agents to learn from interaction with environments. It has focused generally on Markov decision process (MDP) domains, but a domain may be non-Markovian in the real world. In this paper, we introduce a new description of macro-actions with tree structure in reinforcement learning. The macro-action is an action control structure which provides an agent with control which applies a collection of related microscopic actions as a single action unit. And we propose a simple method for dynamically acquiring macro-actions from the experiences of agents during reinforcement learning process.

In this paper, we introduce low-priority cut-in (moratorium) to chain form reinforcement learning, which we proposed as Simple Reinforcement Learning for a reinforcement learning agent that has small memory. In the real world, learning is difficult because there are an infinite number of states and actions that need a large number of stored memory and learning time. To solve the problem, better estimated values are categorized as GOOD in the reinforcement learning process. Additionally, the alignment sequence of estimated values is changed because they are regarded as an important sequence themselves. However, the method is heavily affected by the action policy. If an agent tends to search many states, its memory overflows with low-value data. Thus, low-priority cut-in (moratorium) enhances the method in order to solve this problem. We conducted some simulations and observed the influence of our methods. Several simulation results show good influence on learning.

Reinforcement Learning (RL) agents have great successes in solving tasks with large observation and action spaces from limited feedback. Still, training the agents is data-intensive and there are no guarantees that the learned behavior is safe and does not violate rules of the environment, which has limitations for the practical deployment in real-world scenarios. This paper discusses the engineering of reliable agents via the integration of deep RL with constraint-based augmentation models to guide the RL agent towards safe behavior. Within the constraints set, the RL agent is free to adapt and explore, such that its effectiveness to solve the given problem is not hindered. However, once the RL agent leaves the space defined by the constraints, the outside models can provide guidance to still work reliably. We discuss integration points for constraint guidance within the RL process and perform experiments on two case studies: a strictly constrained card game and a grid world environment with additional combinatorial subgoals. Our results show that constraint-guidance does both provide reliability improvements and safer behavior, as well as accelerated training.

The supervisory control of robots is a very demanding application. In the context of robots control in assistive environments, it is important that the robot user is able to give commands to robots in a way that is easy and intuitive. There are several tasks that can be achieved using robots under assistive environments. It is challenging to efficiently control multiple robots / robots with degrees of freedom with a simple/intuitive interface by a single operator. In this proposal, we propose the use of Reinforcement Learning for intuitive interface mapping. Based on interaction with the environments, we can determine the optimal interface mapping through the process of Reinforcement Learning. The novelty of this paper is the use of changing reward functions based on qualitative performance evaluation for the Reinforcement Learning algorithm. In this paper, we show that the use of proposed reward functions can result in optimal/intuitive interface mapping for multiple robots / robots with degrees of freedom control applications.

It is common to find a mixture of both long batch jobs and latency-sensitive short jobs in enterprise data centers. Recently hybrid job schedulers emerge as attractive alternatives of conventional centralized job schedulers. In this paper, we conduct trace-driven experiments to study the job-completion-delay performance of two representative hybrid job schedulers (Hawk and Eagle), and find that short jobs still encounter long latency issues due to fluctuating bursty nature of workloads. To this end, we propose Dice, a general performance optimization framework for hybrid job schedulers, to alleviate the high job-completion-delay problem of short jobs. Dice is composed of two simple yet effective techniques: Elastic Sizing and Opportunistic Preemption. Both Elastic Sizing and Opportunistic Preemption keep track of the task waiting times of short jobs. When the mean task waiting time of short jobs is high, Elastic Sizing dynamically and adaptively increases the short partition size to prioritize short jobs over long jobs. On the other hand, Opportunistic Preemption preempts resources from long tasks running in the general partition on demand, so as to mitigate the head-of-line blocking problem of short jobs. We enhance the two schedulers with Dice and evaluate Dice performance improvement in our prototype implementation. Experiment results show that Dice achieves 50.9%, 54.5%, and 43.5% improvement on 50th-percentile, 75th-percentile, and 90th-percentile job completion delays of short jobs in Hawk respectively, as well as 33.2%, 74.1%, and 85.3% improvement on those in Eagle respectively under the Google trace, at low performance costs to long jobs.

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

Autonomous driving technology is constantly developing to a higher level of complex scenes, and there is a growing demand for the utilization of end-to-end data-driven control. However, the end-to-end path tracking process often encounters challenges in learning efficiency and generalization. To address this issue, this paper designs a deep deterministic policy gradient (DDPG)-based reinforcement learning strategy that integrates imitation learning and feedforward exploration in the path following process. In imitation learning, the path tracking control data generated by the model predictive control (MPC) method is used to train an end-to-end steering control model of a deep neural network. Another feedforward exploration behavior is predicted by road curvature and vehicle speed, and adds it and imitation learning to the DDPG reinforcement learning to obtain decision-making experience and action prediction behavior of the path tracking process. In the reinforcement learning process, imitation learning is used to update the pre-training parameters of the actor network, and a feedforward steering technique with random noise is adopted for strategy exploration. In the reward function, a hierarchical progressive reward form and a constrained objective reward function referring to MPC are designed, and the actor-critic network architecture is determined. Finally, the path tracking performance of the designed method is verified by comparing various training results, simulations, and HIL tests. The results show that the designed method can effectively utilize pre-training and feedforward prior experience to obtain optimal path tracking performance of an autonomous vehicle, and has better generalization ability than other methods. This study provides an efficient control scheme for improving the end-to-end control performance of autonomous vehicles.

Most existing person re-identification(Re-ID) approaches achieve superior results based on the assumption that a large amount of pre-labelled data is usually available and can be put into training phrase all at once. However, this assumption is not applicable to most real-world deployment of the Re-ID task. In this work, we propose an alternative reinforcement learning based human-in-the-loop model which releases the restriction of pre-labelling and keeps model upgrading with progressively collected data. The goal is to minimize human annotation efforts while maximizing Re-ID performance. It works in an iteratively updating framework by refining the RL policy and CNN parameters alternately. In particular, we formulate a Deep Reinforcement Active Learning (DRAL) method to guide an agent (a model in a reinforcement learning process) in selecting training samples on-the-fly by a human user/annotator. The reinforcement learning reward is the uncertainty value of each human selected sample. A binary feedback (positive or negative) labelled by the human annotator is used to select the samples of which are used to fine-tune a pre-trained CNN Re-ID model. Extensive experiments demonstrate the superiority of our DRAL method for deep reinforcement learning based human-in-the-loop person Re-ID when compared to existing unsupervised and transfer learning models as well as active learning models.

With the complexity and increasing demand of the power grid, more efficient scheduling methods are needed. Reinforcement learning, as an artificial intelligence technology, provides adaptive decision-making solutions that can optimize resource allocation and job scheduling based on the actual state and task requirements of the power grid. The experimental results indicate that the system has achieved significant performance improvement in different power grid operation scenarios. In terms of resource utilization, due to the higher utilization rate of the algorithm in this paper compared to the other two algorithms, the scheduling success rate is correspondingly higher than the other two algorithms. The success rate of the algorithm in this paper can reach 90.24%, which is 22.18% and 22.72% higher than GA (Genetic Algorithm) and DL (Deep Learning). Finally, in the cost-effectiveness ratio experiment, it is evident that the cost-effectiveness ratio of the algorithm proposed in this paper is much higher than that of the other two algorithms. This demonstrates the potential advantages of reinforcement learning algorithms in power grid spatial job scheduling. Future research can further explore more complex power grid operation and maintenance scenarios and algorithm optimization to continuously improve the efficiency and reliability of power grid operation and maintenance. The construction of an automatic scheduling and visualization system for power grid spatial operations provides strong support for the sustainable development of the power grid industry and the integration of renewable energy.

The electric vehicle (EV) has been popular in recent years, which also brings huge challenges to the distribution network due to its energy instability. In order to consider the economic factors of dispatching these distributed renewable resources, the voltage variation is also important. A novel model‐free method is put forward for collaborative management of EV resources of aggregators in the distribution network. The economic costs and physical network constraints for this energy management issue are considered at the same time. A Multiagent Deep Deterministic Policy Gradient (MADDPG) algorithm is applied to learn the cooperative energy control strategies. A transfer learning technique is used to fine‐tune the trained policy when more aggregators join in the network. The proposed method can achieve close results to the traditional optimization methods, while it takes less than one second to take control actions, making it is more suitable for real‐time online energy management. Compared to other advanced reinforcement learning (RL) models, numerical simulations conducted on IEEE test cases greatly illustrate the effectiveness and superiority of the proposed method.

Large earthquakes (EQs) occur at surprising loci and timing, and their descriptions remain a long-standing enigma. Finding answers by traditional approaches or recently emerging machine learning (ML)-driven approaches is formidably difficult due to data scarcity, interwoven multiple physics, and absent first principles. This paper develops a novel artificial intelligence (AI) framework that can transform raw observational EQ data into ML-friendly new features via basic physics and mathematics and that can self-evolve in a direction to better reproduce short-term large EQs. An advanced reinforcement learning (RL) architecture is placed at the highest level to achieve self-evolution. It incorporates transparent ML models to reproduce magnitude and spatial location of large EQs (\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$M_w \\ge $$\\end{document} 6.5) weeks before of the failure. Verifications with 40-year EQs in the western U.S. and comparisons against a popular EQ forecasting method are promising. This work will add a new dimension of AI technologies to large EQ research. The developed AI framework will help establish a new database of all EQs in terms of ML-friendly new features and continue to self-evolve in a direction of better reproducing large EQs.