Scheduling Workflows with Budget Constraints

Scheduling is absolutely the resource management. A group of interdependent jobs/tasks forms the workflow application and scheduling is to map the jobs/tasks on to the collection of heterogeneous resources available in a massive geographic spread. Most complicated applications consist of interdependent jobs that coordinate to solve a problem. The completion of a particular job is the criterion function essentially to meet in order to start the execution of those jobs that depend upon it [1]. This kind of workflow application may be represented in the form of a Directed Acyclic Graph (DAG). Grid Workflow is such an application and is modeled by DAG. This paper proposes a tool that generates Directed Acyclic Graph through Randomizer, which helps in solving the scheduling problem among the dependent tasks by considering the parameters, computation cost (COMPCost) of the nodes and the communication cost (COMMCost) between the nodes. This tool is developed in Java, considering it as a platform independent and web authoring application developer. The task dependencies are made random, the computation cost and communication cost are also randomly allocated by the randomizer. The output generated by the tool includes (i) a visual component of an actual DAG,(ii) a table with complete information on task, its predecessors, COMPCost, COMMCost and (iii) detailed description about the number of levels, number of tasks at each level, identification of a tasks in a level and relationship between the nodes.

Grid Computing is a newly emerging technology that enables users to share a large number of computing resources distributed over a network. Due to that the grid computational resources are not storable, the advance reservation of these resources are necessary for users to request resources from multiple scheduling systems at a specific time. In this paper, we use auction-based scheduling method for resource reservation. We propose a variant version of traditional ascending auction for automatic resource reservation and introduce two novel heuristic strategies to guide agents on participating actions. We also compare our bidding strategies with several other different bidding strategies for the computational resource reservation in different scenarios. The results of experiments show that our heuristic bidding strategies outperforms those methods in those cases.

This paper presents a mathematical model to evaluate the performance of grid resources when availability of the resources is taken into account. The proposed model uses continuous time Markov chains (CTMCs) to model the failure-repair behavior of a grid resource. In grid computing environment, a resource not only may fail during task execution, but also it can cancel its membership at any time. Hence, the proposed CTMC considers the availability of a grid resource to a grid user in both failure and membership refusal situations. After modeling the availability of the resource, the mean sojourn time of grid tasks in each of the availability states is estimated. Assigning the mean sojourn times of the tasks as performance levels to each of the CTMC's states, a Markov reward model (MRM) representing the combined performance and availability measures is obtained. Computing the cumulative state probability of the CTMC and multiplying reward rates of the MRM's states to each of the corresponding state probabilities, the expected accumulated sojourn time of grid tasks in each of the grid resources is achieved. An illustrative example is presented and the results obtained from the proposed model are reported in cases where various scheduling disciplines are considered inside the grid resource to simultaneously service grid and local tasks.

Metaheuristics algorithms show very good performance in solving various job scheduling problems in computational grid systems. However, due to the complexity and heterogeneous nature of resources in grid computing, stand-alone algorithm is not capable to find a good quality solution in reasonable time. This study proposes a hybrid algorithm, specifically ant colony system and genetic algorithm to solve the job scheduling problem. The high level hybridization algorithm will keep the identity of each algorithm in performing the scheduling task. The study focuses on static grid computing environment and the metrics for optimization are the makespan and flowtime. Experiment results show that the proposed algorithm outperforms other stand-alone algorithms such as ant system, genetic algorithms, and ant colony system for makespan. However, for flowtime, ant system and genetic algorithm perform better.

In this paper, a hierarchical ensemble classification approach that utilizes a Directed Acyclic Graph (DAG) structure is proposed as a solution to the multi-class classification problem. Two main DAG structures are considered: (i) rooted DAG, and (ii) non-rooted DAG. The main challenges that are considered in this paper are: (i) the successive misclassification issue associated with hierarchical classification, and (i) identification of the starting node within the non-rooted DAG approach. To address these issues the idea is to utilize Bayesian probability values to: select the best starting DAG node, and to dictate whether single or multiple paths should be followed within the DAG structure. The reported experimental results indicated that the proposed DAG structure is more effective than when using a simple binary tree structure for generating a hierarchical classification model.

BackgroundDirected acyclic graphs (DAGs) are an effective means of presenting expert-knowledge assumptions when selecting adjustment variables in epidemiology, whereas the change-in-estimate procedure is a common statistics-based approach. As DAGs imply specific empirical relationships which can be explored by the change-in-estimate procedure, it should be possible to combine the two approaches. This paper proposes such an approach which aims to produce well-adjusted estimates for a given research question, based on plausible DAGs consistent with the data at hand, combining prior knowledge and standard regression methods.MethodsBased on the relationships laid out in a DAG, researchers can predict how a collapsible estimator (e.g. risk ratio or risk difference) for an effect of interest should change when adjusted on different variable sets. Implied and observed patterns can then be compared to detect inconsistencies and so guide adjustment-variable selection.ResultsThe proposed approach involves i. drawing up a set of plausible background-knowledge DAGs; ii. starting with one of these DAGs as a working DAG, identifying a minimal variable set, S, sufficient to control for bias on the effect of interest; iii. estimating a collapsible estimator adjusted on S, then adjusted on S plus each variable not in S in turn (“add-one pattern”) and then adjusted on the variables in S minus each of these variables in turn (“minus-one pattern”); iv. checking the observed add-one and minus-one patterns against the pattern implied by the working DAG and the other prior DAGs; v. reviewing the DAGs, if needed; and vi. presenting the initial and all final DAGs with estimates.ConclusionThis approach to adjustment-variable selection combines background-knowledge and statistics-based approaches using methods already common in epidemiology and communicates assumptions and uncertainties in a standardized graphical format. It is probably best suited to areas where there is considerable background knowledge about plausible variable relationships. Researchers may use this approach as an additional tool for selecting adjustment variables when analyzing epidemiological data.

The Directed Acyclic Graph (DAG) is a graph representing causal pathways for informing the conduct of an observational study. The use of DAGs allows transparent communication of a causal model between researchers and can prevent over-adjustment biases when conducting causal inference, permitting greater confidence and transparency in reported causal estimates. In the era of ‘big data’ and increasing number of observational studies, the role of the DAG is becoming more important. Recent best-practice guidance for constructing a DAG with reference to the literature has been published in the ‘Evidence synthesis for constructing DAGs’ (ESC-DAG) protocol. We aimed to assess adherence to these principles for DAGs constructed within perioperative literature. Following registration on the International Prospective Register of Systematic Reviews (PROSPERO) and with adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting framework for systematic reviews, we searched the Excerpta Medica dataBASE (Embase), the Medical Literature Analysis and Retrieval System Online (MEDLINE) and Cochrane databases for perioperative observational research incorporating a DAG. Nineteen studies were included in the final synthesis. No studies demonstrated any evidence of following the mapping stage of the protocol. Fifteen (79%) fulfilled over half of the translation and integration one stages of the protocol. Adherence with one stage did not guarantee fulfilment of the other. Two studies (11%) undertook the integration two stage. Unmeasured variables were handled inconsistently between studies. Only three (16%) studies included unmeasured variables within their DAG and acknowledged their implication within the main text. Overall, DAGs that were constructed for use in perioperative observational literature did not consistently adhere to best practice, potentially limiting the benefits of subsequent causal inference. Further work should focus on exploring reasons for this deviation and increasing methodological transparency around DAG construction.

Rooted phylogenetic networks, or more generally, directed acyclic graphs (DAGs), are widely used to model species or gene relationships that traditional rooted trees cannot fully capture, especially in the presence of reticulate processes or horizontal gene transfers. Such networks or DAGs are typically inferred from observable data (e.g., genomic sequences of extant species), providing only an estimate of the true evolutionary history. However, these inferred DAGs are often complex and difficult to interpret. In particular, many contain vertices that do not serve as least common ancestors (LCAs) for any subset of the underlying genes or species, thus may lack direct support from the observable data. In contrast, LCA vertices are witnessed by historical traces justifying their existence and thus represent ancestral states substantiated by the data. To reduce unnecessary complexity and eliminate unsupported vertices, we aim to simplify a DAG to retain only LCA vertices while preserving essential evolutionary information. In this paper, we characterize LCA-relevant and lca-relevant DAGs, defined as those in which every vertex serves as an LCA (or unique LCA) for some subset of taxa. We introduce methods to identify LCAs in DAGs and efficiently transform any DAG into an LCA-relevant or lca-relevant one while preserving key structural properties of the original DAG or network. This transformation is achieved using a simple operator “⊖” that mimics vertex suppression.

In this paper, we propose READYS, a reinforcement learning algorithm for the dynamic scheduling of computations modeled as a Directed Acyclic Graph (DAGs). Our goal is to develop a scheduling algorithm in which allocation and scheduling decisions are made at runtime, based on the state of the system, as performed in runtime systems such as StarPU or ParSEC. Reinforcement Learning is a natural candidate to achieve this task, since its general principle is to build step by step a strategy that, given the state of the system (the state of the resources and a view of the ready tasks and their successors in our case), makes a decision to optimize a global criterion. Moreover, the use of Reinforcement Learning is natural in a context where the duration of tasks (and communications) is stochastic. We propose READYS that combines Graph Convolutional Networks (GCN) with an Actor-Critic Algorithm (A2C): it builds an adaptive representation of the scheduling problem on the fly and learns a scheduling strategy, aiming at minimizing the makespan. A crucial point is that READYS builds a general scheduling strategy which is neither limited to only one specific application or task graph nor one particular problem size, and that can be used to schedule any DAG. We focus on different types of task graphs originating from linear algebra factorization kernels (CHOLESKY, LU, QR) and we consider heterogeneous platforms made of a few CPUs and GPUs. We first propose to analyze the performance of READYS when learning is performed on a given (platform, kernel, problem size) combination. Using simulations, we show that the scheduling agent obtains performances very similar or even superior to algorithms from the literature, and that it is especially powerful when the scheduling environment contains a lot of uncertainty. We additionally demonstrate that our agent exhibits very promising generalization capabilities. To the best of our knowledge, this is the first paper which shows that reinforcement learning can really be used for dynamic DAG scheduling on heterogeneous resources.

Directed acyclic graphs (DAGs) play a large role in the modern approach to causal inference. DAGs describe the relationship between measurements taken at various discrete times including the effect of interventions. The causal mechanisms, on the other hand, would naturally be assumed to be a continuous process operating over time in a cause–effect fashion. How does such immediate causation, that is causation occurring over very short time intervals, relate to DAGs constructed from discrete observations? We introduce a time-continuous model and simulate discrete observations in order to judge the relationship between the DAG and the immediate causal model. We find that there is no clear relationship; indeed the Bayesian network described by the DAG may not relate to the causal model. Typically, discrete observations of a process will obscure the conditional dependencies that are represented in the underlying mechanistic model of the process. It is therefore doubtful whether DAGs are always suited to describe causal relationships unless time is explicitly considered in the model. We relate the issues to mechanistic modeling by using the concept of local (in)dependence. An example using data from the Swiss HIV Cohort Study is presented.

Cancer progression models such as Oncogenetic Tress [1], Conjunctive Bayesian Networks [2], or CAPRI [3], try to infer possible restrictions in the order of accumulation of mutations using cancer cross-sectional data. These restrictions would arise from epistatic interactions between mutations. The Directed Acyclic Graphs (DAGs) returned by these methods predict the genotypes that can and cannot be observed. Since fitness landscapes are shaped by those same epistatics interactions and also specify what genotypes are accessible and inaccessible [4] we would expect a close correspondence between a fitness landscape and the DAGs inferred from evolutionary processes in that landscape. I will show here, however, that there is a many-to- many mapping between fitness landscapes and DAGs inferred from cancer progression models. I show that many fitness landscapes cannot be represented by a cancer progression DAG in the sense that no DAG can codify the genotypes that are accessible under that landscape and I propose two measures of “DAG-non-representability”. Next, using 500 different random fitness lanscapes (with 100 begin perfectly DAG-representable) I simulate cancer progression using different mutation rates and sampling regimes; I analyze 20 replicates (of 1000 genotypes) under each condition with CAPRI[3] and CBN[2]. I show that, even under constant mutation and sampling regime conditions, data obtained from the same landscape lead to inferring widely different DAGs (e.g., median Gini-Simpson index of 0.5); this one-landscape-to-many-DAGs phenomenon increases with DAG-non-representability. I also show the one-DAG-to-many-landscapes phenomenon where the same DAG is frequently inferred from very different fitness landscapes (e.g., > 50% difference in accessible genotypes). Finally, I “reverse landscape engineer” the pancreatic cancer data used in [6]: using 190 random fitness landscapes, I show that genotype frequencies similar to the empirically observed one can be obtained under fitness landscapes that are very different from each other and that produce data that lead to the inference of DAGs that are also widely different among themselves (e.g., pairwise difference in edges of DAGs > 50%). I discuss the implications of these results for the interpretation and applied use of cancer progression models. Novelty and significance This paper explicitly establishes a connection between the inference of cancer progression models and the fitness landscapes that condition that cancer progression. It shows there is a many-to- many mapping between them, even under large sample sizes, that can limit their interpretability.

Adjusting for Selection Effects in Epidemiologic Studies

During the past decade, the scientific community has witnessed the rapid accumulation of gene sequence data and data related to physiology and biochemistry of organisms. Bioinformatics tools used for efficient and computationally intensive analysis of genetic sequences require large-scale computational resources to accommodate the growing data. Grid computational resources such as the Open Science Grid and TeraGrid have proved useful for scientific discovery. GADU is a high-throughput computational system developed to automate the steps involved in accessing the Grid resources for running bioinformatics applications. This paper describes the requirements for building an automated scalable system such as GADU that can run a job simultaneously on different grids. The paper describes the resource-independent configuration of GADU using the Pegasus-based virtual data system that helps in using heterogeneous grid resources. The paper also highlights the features implemented to make GADU a gateway to computationally intensive bioinformatics applications on the Grid

The scheduling of tasks in a parallel program is an NP-complete problem, where scheduling tasks over multiple processing units requires an effective strategy to maximize the exploitation of the parallel hardware. Several studies focus on the scheduling of parallel programs described into DAGs (Directed Acyclic Graphs). However, this representation does not describe a multithreaded program suitably. This paper shows the structure and semantics of a DCG, an abstraction which describes a multithreaded program, and proposes standards to map structures found in DAGs into segments of a DCG. A graph grammar has been developed to perform the proposed transformation and case studies using DAGs found in the literature validate the transformation process. Besides the automatic translation and precise definition of the mapping, the use of a formal language also allowed the verification of the existence and uniqueness of the out coming model.

Directed Gaussian graphical models with toric vanishing ideals