Improving Workflow Design Using Abstract Provenance Graphs

Structured design, consistency analysis and failure reasoning of business workflows with activity-control templates and causal ordering

Design and Verification of Time-Constrained Workflows With Multi-Level Security

The design of nuclear power plant electrical systems is characterized by an extensive and intricate body of standards, strong cross-disciplinary coupling, and stringent safety and compliance requirements. Traditional digital design workflows face significant limitations in knowledge reuse, intelligent decision support, and quality assurance. At the same time, general-purpose large language models (LLMs) exhibit hallucinations and lack reliability in safety-critical engineering domains. To address these challenges, this paper proposes an intelligent nuclear power electrical design methodology that integrates knowledge-graph-enhanced retrieval-augmented generation (Graph-RAG) with a multi-agent system (MAS). First, a domain ontology covering IEC/GB standards, equipment parameter libraries, and design rules is constructed to provide a structured knowledge backbone. Second, a graph-guided hybrid retrieval strategy that combines semantic retrieval with graph path reasoning is designed to enhance retrieval accuracy and contextual relevance. Third, a multi-agent collaboration architecture—comprising requirement analysis, intelligent recommendation, automatic document generation, consistency checking, format review, and interactive question answering agents—is developed. Task orchestration is implemented through a directed acyclic graph (DAG), enabling parallel and coordinated execution of complex design workflows. Theoretical analysis and scenario-based validation indicate that the proposed approach substantially reduces hallucination rates, improves design efficiency, and ensures the standard compliance of design outputs. Each design decision becomes traceable to explicit standard clauses and equipment parameters, demonstrating the feasibility of the method for the intelligent transformation of nuclear power electrical design and its potential transferability to other complex engineering systems.

Scientific workflows are one of the well-established pillars of modern large-scale computational science. More recently, scientists have started to leverage machine learning (ML) capabilities in their workflows, leading to a new category of scientific workflows, denoted as scientific ML workflows. ML is not only about training and inference, modern ML workflows also involve complex data processing steps before the training can start, which are not often accounted for in most performance studies. In this work, we consider scientific ML workflows, from data pre-processing to training, inference, and model evaluation. We aim to explore (i) how scientific ML workflows differ from more traditional scientific workflows and; (ii) how we can characterize ML workflows both in terms of execution time and data movements when executing on a target cloud platform. We select three representative workflows, ranging from image classification to natural language processing and image segmentation, which have been executed using the academic cloud platform, Chameleon. We build four realistic deployment scenarios for each workflow, which stress data movements during workflow executions. Then, we compare the performance observed when utilizing these different configurations and study how different settings impact overall workflows performance and efficiency when running on cloud infrastructures. Finally, we summarize our findings and discuss performance impacts when augmenting scientific workflows with ML techniques and how traditional workflow management systems can improve their support for such workflows.

This paper presents HyperFlow: an approach to workflow programming which combines the advantages of a declarative workflow description and low-level scripting programming. The workflow execution model of HyperFlow is based on a formal model of computation - Process Networks. The execution environment is implemented on the basis of a widely adopted runtime platform node.js. Workflow programming benefits from such an approach in multiple ways, including leveraging a large programming ecosystem with many developers, reusable software packages and learning resources; elimination of shim nodes from the workflow graph; and increased reusability of workflow processing components. The HyperFlow workflow programming approach and its advanced capabilities are presented. The HyperFlow engine is also briefly described. Four example workflow applications from various domains, including flood threat assessment, are studied as a demonstration of the HyperFlow programming approach and a comparison with related solutions.

Execution time estimation for workflow scheduling

Limited resources are available to timely inspect and maintain the aging civil infrastructure across the United States. Reality capturing technologies, such as laser scanning, is replacing visual inspection and manual surveying for improved data qualities and reduced resource requirements, while bringing challenges of timely processing terabytes of spatial data. Even using state-of-art 3D reverse engineering environments, inspectors need to manually select data processing algorithms, compose and configure data processing workflows, and verify the correctness of these workflows. Such manual design and execution of spatial data processing workflows are tedious, and result in sub-optimal workflows that do not fully utilize time and resources for producing accurate and detailed spatial information needed by domain applications. This paper proposes a computational framework that will assist in the infrastructure inspection process through streamlined spatial data processing workflow generation, execution, and optimization. Based on previous studies on spatial information query, spatial data processing, and building information modeling (BIM), the authors are exploring the feasibility of automatically generating and optimizing spatial data processing workflows based on formalized representations of these workflows.

Biological experiments often require a series of precisely timed operations, and small variations in treatment can result in inconsistent or biased results. To handle multiple samples in parallel with precise temporal resolution, experimentalists may stagger treatments by initiating the workflow of one sample during the wait or incubation time of another. However, as the number of samples processed in parallel and the number of operations increase, it becomes increasingly difficult to identify and execute valid treatment regimens that permit the handling of each sample. To address this, I developed StaggR, an interactive web application that calculates and visualizes compatible staggering intervals for parallelized execution of identical processing workflows. This tool provides a user-friendly interface for defining protocol operations, durations, and wait times. It can automatically calculate the shortest possible conflict-free interval for initiating sample treatments, or allow users to simulate specific intervals to explore potential treatment regimens or bottlenecks. Using StaggR, users of any experience level can rapidly generate complete, color-coded experimental schedules, visualize these workflows in an easy-to-read chart, and execute them using a built-in timer displaying a treatment schedule with live updates. The experimental designs can be saved, shared, and re-imported, ensuring full reproducibility and user control. The application of StaggR is expected to expedite the design and throughput of complex experimental workflows while maximizing reproducibility.

Grid computing platforms provide middleware and services for coordinating the use of data and computational resources available throughout the network. Grids are used to implement a wide range of distributed applications and systems, including frameworks for distributed data mining and knowledge discovery. This chapter presents a framework we developed to support the execution of knowledge discovery workflows in Grid environments by executing data mining and computation intelligence algorithms on a set of Grid nodes. Our framework is an extension of Weka, an open-source toolkit for data mining and knowledge discovery, and makes use of Web Service technologies to access Grid resources and distribute the computation. We present the implementation of the framework and show through some applications how it supports the design of knowledge discovery workflows and their execution on a Grid.

The scale of scientific applications becomes increasingly large not only in computation, but also in data. Many of these applications also concern inter-related tasks with data dependencies, hence, they are scientific workflows. The efficient coordination of executing/running scientific workflows is of great practical importance. The core of such coordination is scheduling and resource allocation. In this paper, we present three scheduling heuristics for running large-scale, data-intensive scientific workflows in clouds. In particular, the three heuristic algorithms are designed to leverage slot queue threshold, data locality and data prefetching, respectively. We also demonstrate how these heuristics can be collectively used to tackle different issues in running "data-intensive" workflows in clouds although each of these heuristics can be used independently. The practicality of our algorithms has been realized by actually implementing and incorporating them into our workflow execution system (DEWE). Using Montage, an astronomical image mosaic engine, as an example workflow, and Amazon EC2 as the cloud environment, we evaluate the performance of our heuristics in terms primarily of completion time (make span). We also scrutinize workflow execution showing different execution phases to identify their impact on performance. Our algorithms scale well and reduce make span by up to 27%.

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

In-situ processing alleviates the gap between computation and I/O capabilities by performing data analysis close to the data source. With simulation data varying in size and content during workflow execution, it becomes necessary for insitu processing to support resource elasticity, i.e., the ability to change resource configurations such as the number of computing nodes/processes during workflow execution. An elastic job may dynamically adjust resource configurations; it may use a few resources at the beginning and more resources towards the end of the job when interesting data appears. However, it is hard to predict a priori how many computing nodes/processes need to be added/removed during the workflow execution to adapt to changing workflow needs. How to efficiently guide elasticity operations, such as growing or shrinking the number of processes used for in-situ analysis during workflow execution, is an open-ended research question. In this paper, we present an adaptive elasticity policy that adopts workflow runtime information collected online to predict how to trigger the addition and removal of processes in order to minimize in-situ processing overheads. We integrate the presented elasticity policy into a staging-based elastic workflow and evaluate its efficiency in multiple elasticity scenarios. The results indicate that an adaptive elasticity policy can save overhead in finding a proper resource configuration, when compared with a static policy that uses a fixed number of processes for each rescaling operation. Finally, we discuss multiple existing research opportunities of elastic insitu processing from different aspects.

Genarris is an open source Python package for generating random molecular crystal structures with physical constraints for seeding crystal structure prediction algorithms and training machine learning models. Here we present a new version of the code, containing several major improvements. A MPI-based parallelization scheme has been implemented, which facilitates the seamless sequential execution of user-defined workflows. A new method for estimating the unit cell volume based on the single molecule structure has been developed using a machine-learned model trained on experimental structures. A new algorithm has been implemented for generating crystal structures with molecules occupying special Wyckoff positions. A new hierarchical structure check procedure has been developed to detect unphysical close contacts efficiently and accurately. New intermolecular distance settings have been implemented for strong hydrogen bonds. To demonstrate these new features, we study two specific cases: benzene and glycine. Genarris finds the experimental structures of the two polymorphs of benzene and the three polymorphs of glycine. Program summaryProgram Title: Genarris 2.0Program Files doi:http://dx.doi.org/10.17632/grx6mz4pjn.1Licensing provisions: BSD-3 ClauseProgramming language: Python, CExternal routines/libraries: Spglib, ASE, pymatgen, SciPy, mpi4py, scikit-learn, PyTorch, FHI-aims.Nature of problem: Molecular crystal structure prediction.Solution method: Genarris 2.0 generates molecular crystal structures over the 230 space groups, on general and special Wyckoff positions, using physical constraints. Down-sampling of the generated structures may be performed subsequently, based on molecular crystal packing descriptors and an unsupervised machine learning algorithm. Lastly, ab initio structure relaxation may be performed for the final pool. Depending on the user-defined workflow implemented, Genarris may be used to generate diverse molecular crystal datasets to seed evolutionary algorithms or to train machine learning algorithms or as a standalone crystal structure prediction method.Restrictions: For crystal structure generation, the molecule of interest must be semi-rigid with no bond rotational degrees of freedom.Unusual features: Genarris 2.0 is a highly distributed program, making use of MPI for Python parallelization. The user has the ability to design and implement workflows by executing a user-defined list of procedures. Genarris 2.0 offers new features including a machine learning model for estimating the molecular volume in the solid state from the single molecule structure, structure generation in special Wyckoff positions of space groups, hierarchical structure checks including rigorous treatment of non-orthogonal structures, and clustering and down-selection workflows combining first principles simulations with machine learning.

Scientific workflow technology has become a de facto standard in e-Science. It provides a powerful unifying platform that allows scientists to build arbitrarily complicated applications by combining predefined components. With a wide range of components, with the functions ranging from acquiring data inputs from sensors, querying databases, performing data-mining, performing execution, through to visualising the results, this paradigm allows scientists to automate, manage and execute various steps in scientific research. Although powerful, scientific workflow systems still require some programming knowledge, which might not be optimal for many scientists. Recently, scientific workflows have been integrated into the Web, creating workflow-based science gateways. These gateways allow scientists to launch large-scale computational experiments without concern for the underlying complexity of the infrastructure. This approach delivers scientific workflows to more scientists and is becoming increasingly popular. To date, much workflow-based science gateway development effort has been invested in creating user-friendly and secure interfaces to setup parameters, execute workflows and collect results. However, interaction between users and executing workflows has not received a lot of attention. Scientific workflows have typically been executed in batch mode. This model does not work with human-in-the-loop workflows, which require continuous interaction between users and executing workflows. A fundamental contribution of this thesis is to show that gateways can support complex human- workflow interactions, facilitating monitoring and steering of the workflow execution as it progresses. This is important because scientific workflows increasingly require user decisions and interactions at various steps; human decision-making is an important component of real-world computational science. In order to achieve this goal, the thesis presents the design, implementation and demonstration of WorkWays, an interactive workflow-based science gateway for supporting development and execution of human-in-the-loop scientific workflows. WorkWays consists of two distinct frameworks with well defined APIs and bindings: a framework that enables human- workflow interaction and a framework for workflow development and execution.

Many computation-intensive scientific applications feature complex workflows of distributed computing modules with intricate execution dependencies. Such scientific workflows must be mapped and executed in shared environments to support distributed scientific collaborations. We formulate workflow mapping as an optimization problem for latency minimization, whose difficulty essentially arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension. We conduct a rigorous analysis of the resource sharing dynamics in workflow executions, which constitutes the base for a workflow mapping algorithm to minimize the end-to-end delay. The correctness of the dynamics analysis is verified in comparison with an approximate solution, a dynamic system simulation program, and a real network deployment, and the performance superiority of the proposed mapping solution is illustrated by extensive comparisons with existing methods using both simulations and experiments.