A Performance Characterization of Scientific Machine Learning Workflows

Scientific workflow technology has become a de facto standard in e-Science. It provides a powerful unifying platform that allows scientists to build arbitrarily complicated applications by combining predefined components. With a wide range of components, with the functions ranging from acquiring data inputs from sensors, querying databases, performing data-mining, performing execution, through to visualising the results, this paradigm allows scientists to automate, manage and execute various steps in scientific research. Although powerful, scientific workflow systems still require some programming knowledge, which might not be optimal for many scientists. Recently, scientific workflows have been integrated into the Web, creating workflow-based science gateways. These gateways allow scientists to launch large-scale computational experiments without concern for the underlying complexity of the infrastructure. This approach delivers scientific workflows to more scientists and is becoming increasingly popular. To date, much workflow-based science gateway development effort has been invested in creating user-friendly and secure interfaces to setup parameters, execute workflows and collect results. However, interaction between users and executing workflows has not received a lot of attention. Scientific workflows have typically been executed in batch mode. This model does not work with human-in-the-loop workflows, which require continuous interaction between users and executing workflows. A fundamental contribution of this thesis is to show that gateways can support complex human- workflow interactions, facilitating monitoring and steering of the workflow execution as it progresses. This is important because scientific workflows increasingly require user decisions and interactions at various steps; human decision-making is an important component of real-world computational science. In order to achieve this goal, the thesis presents the design, implementation and demonstration of WorkWays, an interactive workflow-based science gateway for supporting development and execution of human-in-the-loop scientific workflows. WorkWays consists of two distinct frameworks with well defined APIs and bindings: a framework that enables human- workflow interaction and a framework for workflow development and execution.

The characteristics of cloud computing such as on-demand provisioning of virtual machines in a pay-as-you-go manner have attracted more and more scientific workflows deploying on cloud platforms. Since there are many types of virtual machines which are charged by time intervals, the difficulties of resource provisioning hinders efficient execution of scientific workflows on cloud platforms. To address the challenge, a novel task backfill based scientific workflow scheduling strategy is proposed in this paper. The strategy will use task backfill algorithm to aggregate multiple tasks on a virtual machine instance with suitable performance and fill idle time slot of virtual machines with single tasks, improving resource utilization without affecting the overall performance. Experimental results demonstrate that in comparison with widely used HEFT and IC-PCPD2 strategy, the proposed strategy can effectively reduce the execution cost of the scientific workflows and improve resource utilization while satisfying the deadline constraint.

Many computation-intensive scientific applications feature complex workflows of distributed computing modules with intricate execution dependencies. Such scientific workflows must be mapped and executed in shared environments to support distributed scientific collaborations. We formulate workflow mapping as an optimization problem for latency minimization, whose difficulty essentially arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension. We conduct a rigorous analysis of the resource sharing dynamics in workflow executions, which constitutes the base for a workflow mapping algorithm to minimize the end-to-end delay. The correctness of the dynamics analysis is verified in comparison with an approximate solution, a dynamic system simulation program, and a real network deployment, and the performance superiority of the proposed mapping solution is illustrated by extensive comparisons with existing methods using both simulations and experiments.

Scientific Workflows (SWfs) have revolutionized how scientists in various domains of science conduct their experiments. The management of SWfs is performed by complex tools that provide support for workflow composition, monitoring, execution, capturing, and storage of the data generated during execution. In some cases, they also provide components to ease the visualization and analysis of the generated data. During the workflow’s composition phase, programs must be selected to perform the activities defined in the workflow specification. These programs often require additional parameters that serve to adjust the program’s behavior according to the experiment’s goals. Consequently, workflows commonly have many parameters to be manually configured, encompassing even more than one hundred in many cases. Wrongly parameters’ values choosing can lead to crash workflows executions or provide undesired results. As the execution of data- and compute-intensive workflows is commonly performed in a high-performance computing environment e.g., (a cluster, a supercomputer, or a public cloud), an unsuccessful execution configures a waste of time and resources. In this article, we present FReeP—Feature Recommender from Preferences, a parameter value recommendation method that is designed to suggest values for workflow parameters, taking into account past user preferences. FReeP is based on Machine Learning techniques, particularly in Preference Learning. FReeP is composed of three algorithms, where two of them aim at recommending the value for one parameter at a time, and the third makes recommendations for n parameters at once. The experimental results obtained with provenance data from two broadly used workflows showed FReeP usefulness in the recommendation of values for one parameter. Furthermore, the results indicate the potential of FReeP to recommend values for n parameters in scientific workflows.

Machine learning (ML) has become an increasingly popular tool for use in neurosurgical research. The number of publications and interest in the field have recently seen significant expansion in both quantity and complexity. However, this also places a commensurate burden on the general neurosurgical readership to appraise this literature and decide if these algorithms can be effectively translated into practice. To this end, the authors sought to review the burgeoning neurosurgical ML literature and to develop a checklist to help readers critically review and digest this work. The authors performed a literature search of recent ML papers in the PubMed database with the terms "neurosurgery" AND "machine learning," with additional modifiers "trauma," "cancer," "pediatric," and "spine" also used to ensure a diverse selection of relevant papers within the field. Papers were reviewed for their ML methodology, including the formulation of the clinical problem, data acquisition, data preprocessing, model development, model validation, model performance, and model deployment. The resulting checklist consists of 14 key questions for critically appraising ML models and development techniques; these are organized according to their timing along the standard ML workflow. In addition, the authors provide an overview of the ML development process, as well as a review of key terms, models, and concepts referenced in the literature. ML is poised to become an increasingly important part of neurosurgical research and clinical care. The authors hope that dissemination of education on ML techniques will help neurosurgeons to critically review new research better and more effectively integrate this technology into their practices.

Towards a model-driven transformation framework for scientific workflows

Machine learning application in the oil and gas industry is rapidly becoming popular and in recent years has been applied in the optimization of production for various reservoirs. The objective of this paper is to evaluate the efficacy of advanced machine learning algorithms in reservoir production optimization. A 3-D geological model was constructed based on permeability calculated using a machine learning technique which involved different architectures of algorithms tested using a 5-fold cross-validation to decide the best machine learning algorithm. Sensitivity analysis and a subsequent history matching were conducted using a machine learning workflow. The aquifer properties, permeability heterogeneity in different directions and relative permeability were the control variables assessed. Field development scenarios were exploited with the objective to optimize cumulative oil recovery. The impact of using a normal depletion plan to a secondary recovery plan using waterflooding was investigated. Different injection well placement locations, well patterns as well as the possibility of converting existing oil producing wells to water injection wells were exploited. Considering the outcome of an economic analysis, the optimum development strategy was realized as an outcome for the optimization process. Prior to forecasting cumulative oil production using artificial neural network (ANN) for the optimization process on the generated surrogate model, a sensitivity analysis was performed where the well location, injection rates and bottomhole pressure of both the producer and injector wells were specified as control variables. The water cut as part of the optimization process was utilized as a secondary constraint. Forecasting was performed for a 15-year period. The history-matching results from the constructed geological model showed that the oil rate, water rate, bottom hole pressure, and average reservoir pressure were matched within a 10% deviation from the observed data. In this study, the ANN optimizer was found to provide the best results for the field cumulative oil production. Using a secondary recovery development plan was observed to significantly increase the cumulative oil production. A machine learning based proxy model was built for the prediction of cumulative oil production to reduce computational time. In this study, we propose an approach applied to reservoir production optimization utilizing a machine learning workflow. This was accomplished by utilizing a surrogate model which was calibrated with a number of training simulations and then optimized using advanced machine learning algorithms. A detailed economic analysis was also conducted showing the impact of a variety of field development strategies.

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

Enabled by a rich ecosystem of Machine Learning (ML) libraries, programming using learned models , i.e., Software-2.0 , has gained substantial adoption. However, we do not know what challenges developers encounter when they use ML libraries. With this knowledge gap, researchers miss opportunities to contribute to new research directions, tool builders do not invest resources where automation is most needed, library designers cannot make informed decisions when releasing ML library versions, and developers fail to use common practices when using ML libraries. We present the first large-scale quantitative and qualitative empirical study to shed light on how developers in Software-2.0 use ML libraries, and how this evolution affects their code. Particularly, using static analysis we perform a longitudinal study of 3,340 top-rated open-source projects with 46,110 contributors. To further understand the challenges of ML library evolution, we survey 109 developers who introduce and evolve ML libraries. Using this rich dataset we reveal several novel findings. Among others, we found an increasing trend of using ML libraries: The ratio of new Python projects that use ML libraries increased from 2% in 2013 to 50% in 2018. We identify several usage patterns including the following: (i) 36% of the projects use multiple ML libraries to implement various stages of the ML workflows, (ii) developers update ML libraries more often than the traditional libraries , (iii) strict upgrades are the most popular for ML libraries among other update kinds, (iv) ML library updates often result in cascading library updates, and (v) ML libraries are often downgraded (22.04% of cases). We also observed unique challenges when evolving and maintaining Software-2.0 such as (i) binary incompatibility of trained ML models and (ii) benchmarking ML models. Finally, we present actionable implications of our findings for researchers, tool builders, developers, educators, library vendors, and hardware vendors.

Background Previous studies have described machine learning (ML) models to predict how human readers would score disease activity in UC using the endoscopic Mayo Score (eMS). So far, none employed deep human annotation that considers all the endoscopic features making up the eMS. Here we report the results of an ML model that is trained on eMS features using centrally read endoscopies. Methods 793 full-length videos were obtained from 249 patients with UC who participated in NCT02589665, a phase 2 trial with mirikizumab in patients with UC and associated with centrally read (single reader) eMS (CReMS) as the primary dataset. After cleaning for usable frames, the data were split into training, validation and testing subsets. The ML workflow consisted of annotation, segmentation, and classification (e.g., erosions, ulcers, erythema, vascular pattern, and bleeding). Human image classification and segmentation with bounding boxes and was subjected to quality control adjudicated by one of three IBD specialists, generating more than 60,000 eMS-relevant annotation labels. The model was evaluated on a test set of 147 videos using the CReMS, and a consensus set of 94 test videos, where CReMS and annotator reported eMS (AReMS) were in agreement without adjudication. The primary objective of the model was a categorical prediction of endoscopically inactive disease (eMS 0 & 1) compared with active disease (eMS 2 & 3). The secondary objectives of the model were to predict endoscopic healing (eMS 0) and to predict severe disease (eMS 3). Results The model performances are in Table 1. On the full test set of 147 videos, the model predicted inactive disease compared with active disease with an accuracy of 84%, positive predictive value (PPV) of 80%, and negative predictive value (NPV) of 85%. In the subset of 94 videos with CreMS and AReMS consensus, the model predicted inactive disease compared with active disease with an accuracy of 89%, PPV 87%, and NPV of 90%. In this same subset, the model predicted endoscopic healing and severe disease with an accuracy of 95% and 85%, PPVs of 86% and 82% and NPVs of 95% and 87%, respectively. For the secondary objectives in the full set of 147 videos, the model predicted endoscopic healing and severe disease with an accuracy of 90% and 80%, PPVs of 44% and 86%, and NPVs of 95% and 86%, respectively. Conclusion We have developed a ML predictive model of the eMS in UC using centrally read videos and demonstrate excellent distinction between active and inactive disease, and clear discrimination between other levels of endoscopic activity. We propose that this unique ML approach to endoscopic assessment be considered as a substitute to human central reading in future clinical trials.

Scientific workflows are widely used to automate scientific data analysis and often involve computationally intensive processing of large datasets on compute clusters. As such, their execution tends to be long-running and resource-intensive, resulting in substantial energy consumption and, depending on the energy mix, carbon emissions. Meanwhile, a wealth of carbon-aware computing methods have been proposed, yet little work has focused specifically on scientific workflows, even though they present a substantial opportunity for carbon-aware computing because they are often significantly delay tolerant, efficiently interruptible, highly scalable and widely heterogeneous. In this study, we first exemplify the problem of carbon emissions associated with running scientific workflows, and then show the potential for carbon-aware workflow execution. For this, we estimate the carbon footprint of seven real-world Nextflow workflows executed on different cluster infrastructures using both average and marginal carbon intensity data. Furthermore, we systematically evaluate the impact of carbon-aware temporal shifting, and the pausing and resuming of the workflow. Moreover, we apply resource scaling to workflows and workflow tasks. Finally, we report the potential reduction in overall carbon emissions, with temporal shifting capable of decreasing emissions by over 80%, and resource scaling capable of decreasing emissions by 67%.

Most of the large-scale scientific experiments modeled as scientific workflows produce a large amount of data and require workflow parallelism to reduce workflow execution time. Some of the existing Scientific Workflow Management Systems (SWfMS) explore parallelism techniques - such as parameter sweep and data fragmentation. In those systems, several computing resources are used to accomplish many computational tasks in homogeneous environments, such as multiprocessor machines or cluster systems. Cloud computing has become a popular high performance computing model in which (virtualized) resources are provided as services over the Web. Some scientists are starting to adopt the cloud model in scientific domains and are moving their scientific workflows (programs and data) from local environments to the cloud. Nevertheless, it is still difficult for the scientist to express a parallel computing paradigm for the workflow on the cloud. Capturing distributed provenance data at the cloud is also an issue. Existing approaches for executing scientific workflows using parallel processing are mainly focused on homogeneous environments whereas, in the cloud, the scientist has to manage new aspects such as initialization of virtualized instances, scheduling over different cloud environments, impact of data transferring and management of instance images. In this paper we propose SciCumulus, a cloud middleware that explores parameter sweep and data fragmentation parallelism in scientific workflow activities (with provenance support). It works between the SWfMS and the cloud. SciCumulus is designed considering cloud specificities. We have evaluated our approach by executing simulated experiments to analyze the overhead imposed by clouds on the workflow execution time.

We present a solution for cloud infrastructure automation for scientific workflows. Unlike existing approaches, our solution is based on widely adopted tools, such as Terraform, and achieves a strict separation of two concerns: infrastructure description and provisioning vs. workflow description. At the same time it enables a comprehensive integration with a given cloud infrastructure, i.e. such wherein workflow execution can be managed by the cloud. The solution is integrated with our HyperFlow workflow management system and evaluated by demonstrating its use in experiments related to auto-scaling of scientific workflows in two types of cloud infrastructures: containerized Infrastructure-as-a-Service (IaaS) and Function-as-a-Service (FaaS). Experimental evaluation involves deployment and execution of a test workflow in Amazon ECS/Docker cluster and on a hybrid of Amazon ECS and AWS Lambda. The results show that our solution not only helps in the creation of repeatable infrastructures for scientific computing but also greatly facilitates automation of research experiments related to the execution of scientific workflows on advanced computing infrastructures.

In recent decades, the prevalence of the Internet and Semantic Web technologies has shifted the traditional scientific journal publishing framework towards the digital environment. In support of this, ontologies on digital publishing and new forms of granular provenance modeling have been built to support digital publishing. These fine-grained technologies facilitate the decomposition of traditional science articles in constituent machine-readable parts that are linked not only with one another, but also to other related fine-grained parts of knowledge on the Web following the Linked Data principles. However, these resulting digital artifacts of fine-grained knowledge are static objects that do not take dynamic processes, or scientific workflows, into account. Additionally, scientific workflows are important because they directly produce and consume digital artifacts. In this project, we enable the decentralized execution of scientific workflows of digital artifacts across platforms such that individual steps of single workflows can be distributed. By considering these scientific workflows, we can further find new dimensions with respect to the quality and impact of digital artifacts. In our preliminary results we have developed a model that is able to support Linked Data Notifications to demonstrate the feasibility of our approach.