Cloud Infrastructure Automation for Scientific Workflows

Scientific workflows are one of the well-established pillars of modern large-scale computational science. More recently, scientists have started to leverage machine learning (ML) capabilities in their workflows, leading to a new category of scientific workflows, denoted as scientific ML workflows. ML is not only about training and inference, modern ML workflows also involve complex data processing steps before the training can start, which are not often accounted for in most performance studies. In this work, we consider scientific ML workflows, from data pre-processing to training, inference, and model evaluation. We aim to explore (i) how scientific ML workflows differ from more traditional scientific workflows and; (ii) how we can characterize ML workflows both in terms of execution time and data movements when executing on a target cloud platform. We select three representative workflows, ranging from image classification to natural language processing and image segmentation, which have been executed using the academic cloud platform, Chameleon. We build four realistic deployment scenarios for each workflow, which stress data movements during workflow executions. Then, we compare the performance observed when utilizing these different configurations and study how different settings impact overall workflows performance and efficiency when running on cloud infrastructures. Finally, we summarize our findings and discuss performance impacts when augmenting scientific workflows with ML techniques and how traditional workflow management systems can improve their support for such workflows.

Enforcing QoS in scientific workflow systems enacted over Cloud infrastructures

Scientific Workflows (SWFs) are widely used to model applications in e-Science. In this programming model, scientific applications are described as a set of tasks that have dependencies among them. During the last decades, the execution of scientific workflows has been successfully performed in the available computing infrastructures (supercomputers, clusters and grids) using software programs called Workflow Management Systems (WMSs), which orchestrate the workload on top of these computing infrastructures. However, because each computing infrastructure has its own architecture and each scientific applications exploits efficiently one of these infrastructures, it is necessary to organize the way in which they are executed. \nWMSs need to get the most out of all the available computing and storage resources. Traditionally, scientific workflow applications have been extensively deployed in high-performance computing infrastructures (such as supercomputers and clusters) and grids. But, in the last years, the advent of cloud computing infrastructures has opened the door of using on-demand infrastructures to complement or even replace local infrastructures. However, new issues have arisen, such as the integration of hybrid resources or the compromise between infrastructure reutilization and elasticity, everything on the basis of cost-efficiency.\nThe main contribution of this thesis is an ad-hoc solution for managing workflows exploiting the capabilities of cloud computing orchestrators to deploy resources on demand according to the workload and to combine heterogeneous cloud providers (such as on-premise clouds and public clouds) and traditional infrastructures (supercomputers and clusters) to minimize costs and response time. The thesis does not propose yet another WMS, but demonstrates the benefits of the integration of cloud orchestration when running complex workflows. The thesis shows several configuration experiments and multiple heterogeneous backends from a realistic comparative genomics workflow called Orthosearch, to migrate memory-intensive workload to public infrastructures while keeping other blocks of the experiment running locally. The running time and cost of the experiments is computed and best practices are suggested.

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

Towards a model-driven transformation framework for scientific workflows

Scientific workflow technology has become a de facto standard in e-Science. It provides a powerful unifying platform that allows scientists to build arbitrarily complicated applications by combining predefined components. With a wide range of components, with the functions ranging from acquiring data inputs from sensors, querying databases, performing data-mining, performing execution, through to visualising the results, this paradigm allows scientists to automate, manage and execute various steps in scientific research. Although powerful, scientific workflow systems still require some programming knowledge, which might not be optimal for many scientists. Recently, scientific workflows have been integrated into the Web, creating workflow-based science gateways. These gateways allow scientists to launch large-scale computational experiments without concern for the underlying complexity of the infrastructure. This approach delivers scientific workflows to more scientists and is becoming increasingly popular. To date, much workflow-based science gateway development effort has been invested in creating user-friendly and secure interfaces to setup parameters, execute workflows and collect results. However, interaction between users and executing workflows has not received a lot of attention. Scientific workflows have typically been executed in batch mode. This model does not work with human-in-the-loop workflows, which require continuous interaction between users and executing workflows. A fundamental contribution of this thesis is to show that gateways can support complex human- workflow interactions, facilitating monitoring and steering of the workflow execution as it progresses. This is important because scientific workflows increasingly require user decisions and interactions at various steps; human decision-making is an important component of real-world computational science. In order to achieve this goal, the thesis presents the design, implementation and demonstration of WorkWays, an interactive workflow-based science gateway for supporting development and execution of human-in-the-loop scientific workflows. WorkWays consists of two distinct frameworks with well defined APIs and bindings: a framework that enables human- workflow interaction and a framework for workflow development and execution.

We present an end-to-end solution for automation of scientific workflow deployment and execution on distributed computing infrastructures. The solution integrates de-facto standard and widely adopted tools, including Terraform and Kubernetes, with our HyperFlow workflow management system. In such a solution, infrastructure providers have abstracted away thanks to generic Kubernetes layer. However, we also support other computing infrastructures, both containerized such as Kubernetes or Amazon ECS, and non-containerized, e.g. Amazon Lambda, in a single unified approach. The resulting solution enables execution of hybrid workflows that utilize multiple computing infrastructures and significantly lowers the complexity related to management of repeatable infrastructures for the execution of scientific workflows and conducting scientific workflow research.

I/O intensive jobs such as stage-in, stage-out or data clean-up jobs account for significant time in execution of scientific workflows. Workflow managers typically add these data management operations as supporting jobs to computational tasks with scheduling emphasis on compute jobs only. We present the integration of the Pegasus Workflow Management System with a Policy Based Data Placement Service (PDPS) to reduce overall workflow execution time. Pegasus delegates all data staging jobs to PDPS, which schedules and executes stage-in jobs based on selected data placement policies and simply executes stage-out and clean-up jobs independent of the workflow execution state. We measure the impact of using PDPS with Pegasus first with the Montage workflow, and then with a synthetic workflow. We enforce two policies and demonstrate the advantage of using PDPS for asynchronous data placement for scientific workflows. Our results show that the influence of PDPS on the overall workflow runtimes is dependent on the data characteristics of the executable workflow and the data placement policy being enforced.

Many computation-intensive scientific applications feature complex workflows of distributed computing modules with intricate execution dependencies. Such scientific workflows must be mapped and executed in shared environments to support distributed scientific collaborations. We formulate workflow mapping as an optimization problem for latency minimization, whose difficulty essentially arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension. We conduct a rigorous analysis of the resource sharing dynamics in workflow executions, which constitutes the base for a workflow mapping algorithm to minimize the end-to-end delay. The correctness of the dynamics analysis is verified in comparison with an approximate solution, a dynamic system simulation program, and a real network deployment, and the performance superiority of the proposed mapping solution is illustrated by extensive comparisons with existing methods using both simulations and experiments.

In distributed computing an escalating trend is Cloud computing which mainly focuses on directly delivering computational power to the end-user regardless of when and where. Delivering of services to the users is there such as Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS). Cloud computing is having an IaaS model which is a desired platform for workflow execution of scientific applications for example bioinformatics, astronomy, geophysics etc. Hence, several researches are recommencing scientific workflows execution in Infrastructure as a Service cloud computing environment. An algorithm ‘Rank-Based Task Scheduling Algorithm’ (RBTSA) which is proposed by Shilpa Rana et al. have future research scope that is the performance of RBTSA can be tested by calculating the rank of virtual machines by considering different characteristic of bandwidth, physical memory etc. of virtual machines which is considered in this paper. Herein paper an algorithm termed as ‘Multi Criteria Rank-Based Task Scheduling Algorithm’ (MCRBTSA) is planned for scientific workflows tasks scheduling to increase resources utilization in IaaS cloud computing environment. The experimental simulation of research work which is mentioned above is done on ‘WorkflowSim-1.0’ and scientific workflow execution results of proposed MCRBTSA Algorithm are contrasted with the existing ‘Rank-Based Task Scheduling Algorithm’ (RBTSA). Experimentally it shows that MCRBTSA which is proposed have utilization of resources better while scheduling workflow tasks as compared to existing RBTSA and workflow makespan reduction is also there.

A Q-learning approach for the autoscaling of scientific workflows in the Cloud

Scheduling large-scale and resource-intensive workflows in cloud infrastructure is one of the main challenges for cloud service providers (CSPs). Cloud infrastructure is more efficient when virtual machines and other resources work up to their full potential. The main factor that influences the quality of cloud services is the distribution of workflow on virtual machines (VMs). Scheduling tasks to VMs depends on the type of workflow and mechanism of resource allocation. Scientific workflows include large-scale data transfer and consume intensive resources of cloud infrastructures. Therefore, scheduling of tasks from scientific workflows on VMs requires efficient and optimized workflow scheduling techniques. This paper proposes an optimised workflow scheduling approach that aims to improve the utilization of cloud resources without increasing execution time and execution cost.

Scientific workflows are widely used to automate scientific data analysis and often involve computationally intensive processing of large datasets on compute clusters. As such, their execution tends to be long-running and resource-intensive, resulting in substantial energy consumption and, depending on the energy mix, carbon emissions. Meanwhile, a wealth of carbon-aware computing methods have been proposed, yet little work has focused specifically on scientific workflows, even though they present a substantial opportunity for carbon-aware computing because they are often significantly delay tolerant, efficiently interruptible, highly scalable and widely heterogeneous. In this study, we first exemplify the problem of carbon emissions associated with running scientific workflows, and then show the potential for carbon-aware workflow execution. For this, we estimate the carbon footprint of seven real-world Nextflow workflows executed on different cluster infrastructures using both average and marginal carbon intensity data. Furthermore, we systematically evaluate the impact of carbon-aware temporal shifting, and the pausing and resuming of the workflow. Moreover, we apply resource scaling to workflows and workflow tasks. Finally, we report the potential reduction in overall carbon emissions, with temporal shifting capable of decreasing emissions by over 80%, and resource scaling capable of decreasing emissions by 67%.

We consider the problem of optimizing the execution of data-intensive scientific workflows in the Cloud. We address the problem under the following scenario. The tasks of the workflows communicate through files; the output of a task is used by another task as an input file and if these tasks are assigned on different execution sites, a file transfer is necessary. The output files are to be stored at a site. Each execution site is to be assigned a certain percentage of the files and tasks. These percentages, called target weights, are pre-determined and reflect either user preferences or the storage capacity and computing power of the sites. The aim is to place the data files into and assign the tasks to the execution sites so as to reduce the cost associated with the file transfers, while complying with the target weights. To do this, we model the workflow as a hypergraph and with a hypergraph-partitioning-based formulation, we propose a heuristic which generates data placement and task assignment schemes simultaneously. We report simulation results on a number of real-life and synthetically generated scientific workflows. Our results show that the proposed heuristic is fast, and can find mappings and assignments which reduce file transfers, while respecting the target weights.

In recent decades, the prevalence of the Internet and Semantic Web technologies has shifted the traditional scientific journal publishing framework towards the digital environment. In support of this, ontologies on digital publishing and new forms of granular provenance modeling have been built to support digital publishing. These fine-grained technologies facilitate the decomposition of traditional science articles in constituent machine-readable parts that are linked not only with one another, but also to other related fine-grained parts of knowledge on the Web following the Linked Data principles. However, these resulting digital artifacts of fine-grained knowledge are static objects that do not take dynamic processes, or scientific workflows, into account. Additionally, scientific workflows are important because they directly produce and consume digital artifacts. In this project, we enable the decentralized execution of scientific workflows of digital artifacts across platforms such that individual steps of single workflows can be distributed. By considering these scientific workflows, we can further find new dimensions with respect to the quality and impact of digital artifacts. In our preliminary results we have developed a model that is able to support Linked Data Notifications to demonstrate the feasibility of our approach.