Streamflow Programming Model for Data Streaming in Scientific Workflows

Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams

Scientific workflow has become an important tool used by scientists to operate distributed resources in current scientific research. However, the gap between scientific domain and workflow specification has increased the difficulty of adopting scientific workflow. Further, the lack of collaborative mechanism has gradually become a major obstacle for scientists to share scientific workflow. In this paper, we introduced an open-source tool for scientific workflow modeling, called Orchestra Designer. It solves above problems by isolating domain tasks from workflow specification and providing a service-oriented collaboration bus. Meanwhile, Orchestra Designer supports the design of cross domains scientific workflows.

Internet of Things (IoT) systems include many smart devices that continuously generate massive spatio-temporal data, which can be difficult to process. These continuous data streams need to be stored smartly so that query searches are efficient. In this work, we propose an efficient method, in the fog-cloud computing architecture, to index continuous and heterogeneous data streams in metric space. This method divides the fog layer into three levels: clustering, clusters processing and indexing. The Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used to group the data from each stream into homogeneous clusters at the clustering fog level. Each cluster in the first data stream is stored in the clusters processing fog level and indexed directly in the indexing fog level in a Binary tree with Hyperplane (BH tree). The indexing of clusters in the subsequent data stream is determined by the coefficient of variation (CV) value of the union of the new cluster with the existing clusters in the cluster processing fog layer. An analysis and comparison of our experimental results with other results in the literature demonstrated the effectiveness of the CV method in reducing energy consumption during BH tree construction, as well as reducing the search time and energy consumption during a k Nearest Neighbor (kNN) parallel query search.

This article proposes an approach to rely on the standard operators of relational algebra (including grouping and ag- gregation) for processing complex event without requiring window specifications. In this way the approach can pro- cess complex event queries of the kind encountered in appli- cations such as emergency management in metro networks. This article presents Temporal Stream Algebra (TSA) which combines the operators of relational algebra with an analy- sis of temporal relations at compile time. This analysis de- termines which relational algebra queries can be evaluated against data streams, i. e. the analysis is able to distinguish valid from invalid stream queries. Furthermore the analysis derives functions similar to the pass, propagation and keep invariants in Tucker's et al. \Exploiting Punctuation Seman- tics in Continuous Data Streams. These functions enable the incremental evaluation of TSA queries, the propagation of punctuations, and garbage collection. The evaluation of TSA queries combines bulk-wise and out-of-order processing which makes it tolerant to workload bursts as they typically occur in emergency management. The approach has been conceived for efficiently processing complex event queries on top of a relational database system. It has been deployed and tested on MonetDB.

Scientific workflow technology has become a de facto standard in e-Science. It provides a powerful unifying platform that allows scientists to build arbitrarily complicated applications by combining predefined components. With a wide range of components, with the functions ranging from acquiring data inputs from sensors, querying databases, performing data-mining, performing execution, through to visualising the results, this paradigm allows scientists to automate, manage and execute various steps in scientific research. Although powerful, scientific workflow systems still require some programming knowledge, which might not be optimal for many scientists. Recently, scientific workflows have been integrated into the Web, creating workflow-based science gateways. These gateways allow scientists to launch large-scale computational experiments without concern for the underlying complexity of the infrastructure. This approach delivers scientific workflows to more scientists and is becoming increasingly popular. To date, much workflow-based science gateway development effort has been invested in creating user-friendly and secure interfaces to setup parameters, execute workflows and collect results. However, interaction between users and executing workflows has not received a lot of attention. Scientific workflows have typically been executed in batch mode. This model does not work with human-in-the-loop workflows, which require continuous interaction between users and executing workflows. A fundamental contribution of this thesis is to show that gateways can support complex human- workflow interactions, facilitating monitoring and steering of the workflow execution as it progresses. This is important because scientific workflows increasingly require user decisions and interactions at various steps; human decision-making is an important component of real-world computational science. In order to achieve this goal, the thesis presents the design, implementation and demonstration of WorkWays, an interactive workflow-based science gateway for supporting development and execution of human-in-the-loop scientific workflows. WorkWays consists of two distinct frameworks with well defined APIs and bindings: a framework that enables human- workflow interaction and a framework for workflow development and execution.

Owing to Big Data popularity, the metrics Variety, Volume and Velocity (V3), are gaining importance in large scale data intensive applications. Complex Event Processing (CEP) is an efficient solution for identifying events of interest on data streams arriving from geographically distributed heterogeneous sources in near-real time. CEP is capable of handling large variety of data with high computational velocity and delivers better solution compared to other existing techniques. However, handling large volume data streams still poses challenge in CEP systems. Hence, a middleware is required to manage huge streams of data on scalable distributed environment. This paper proposes Scalable Complex Event Processing (SCALACEP) framework for managing voluminous data streams. This paper addresses challenges such as state management and efficient rule distribution for design of distributed framework for CEP using the proposed SCALACEP. This paper also proposes a novel way of indexing CEP rules based on Geometric series. The proposed indexing is used for CEP Rule allocation and CEP rule pruning in the SCALACEP system. SCALACEP is evaluated and compared with other existing systems in the constrained virtualized environment and found to give better results in terms of throughput and reduction in Latency and Multicast.

Maritime surveillance systems are of particular importance for the smooth and safe operation of maritime traffic. Such systems must efficiently analyze the continuous stream of incoming ship movement data to provide an up-to-date picture of the situation at sea. Complex Event Processing (CEP) is a software technology dedicated to the analysis of data streams in real time and seems to be promising for maritime surveillance as well. CEP is based on rules that usually have to be formulated manually by domain experts. Recently, several approaches of learning CEP rules have been proposed. However, all these approaches are limited to standard CEP rule languages which are not well suited for describing maritime situations. In this paper, we extend Bat4CEP, an innovative bat-inspired approach to automatic CEP rule learning, to meet the specific needs of the maritime domain. In particular, we extend the rule language of Bat4CEP to include temporal-spatial operators and domain-specific functions. This extended rule language makes it possible to express complex maritime facts in a simple and understandable form. Furthermore, it is shown how the proposed rule language extensions can be integrated into the Bat4CEP learning approach. The effectiveness of the approach is demonstrated through extensive experiments with real maritime data streams.

BackgroundScientific workflows management systems are increasingly used to specify and manage bioinformatics experiments. Their programming model appeals to bioinformaticians, who can use them to easily specify complex data processing pipelines. Such a model is underpinned by a graph structure, where nodes represent bioinformatics tasks and links represent the dataflow. The complexity of such graph structures is increasing over time, with possible impacts on scientific workflows reuse. In this work, we propose effective methods for workflow design, with a focus on the Taverna model. We argue that one of the contributing factors for the difficulties in reuse is the presence of "anti-patterns", a term broadly used in program design, to indicate the use of idiomatic forms that lead to over-complicated design. The main contribution of this work is a method for automatically detecting such anti-patterns, and replacing them with different patterns which result in a reduction in the workflow's overall structural complexity. Rewriting workflows in this way will be beneficial both in terms of user experience (easier design and maintenance), and in terms of operational efficiency (easier to manage, and sometimes to exploit the latent parallelism amongst the tasks).ResultsWe have conducted a thorough study of the workflows structures available in Taverna, with the aim of finding out workflow fragments whose structure could be made simpler without altering the workflow semantics. We provide four contributions. Firstly, we identify a set of anti-patterns that contribute to the structural workflow complexity. Secondly, we design a series of refactoring transformations to replace each anti-pattern by a new semantically-equivalent pattern with less redundancy and simplified structure. Thirdly, we introduce a distilling algorithm that takes in a workflow and produces a distilled semantically-equivalent workflow. Lastly, we provide an implementation of our refactoring approach that we evaluate on both the public Taverna workflows and on a private collection of workflows from the BioVel project.ConclusionWe have designed and implemented an approach to improving workflow structure by way of rewriting preserving workflow semantics. Future work includes considering our refactoring approach during the phase of workflow design and proposing guidelines for designing distilled workflows.

Complex event processing (CEP) is a promising technology, which can evaluate the business logic from a large number of continuous real-time data streams. There are a few issues that are still remai...

Information filtering is a crucial task in a world where data is generated steadily and at a high rate, helping users in distinguishing relevant from irrelevant content. This requires efficient processing of continuous streams of textual data. Event processing allows for real time processing of data streams. But up to now event processing has mainly been investigated in the context of business transaction-oriented domains like logistics or finance, but not explicitly in terms of text stream processing and information filtering. The growth of applications that analyze social media streams lets such an approach appear reasonable. Therefore we propose a common vocabulary represented by a text domain event model as well as a reference architecture for text stream processing and information filtering, in order to facilitate the implementation and the assessment of event processing applications for text streams. In addition we describe results from actual use cases that employ this architecture.

The methods of approximate query processing have been proposed for analytics over high-speed data streams, which compact continuous streams into a space-constrained sketch and provide reliable estimates for different queries. Count-Min (CM) is the state-of-the-art sketching structure supporting many queries with error-guaranteed estimates under limited space. However, we need to create a counter table beforehand in CM according to the size of data streams, while it is usually unpredictable for dynamic data streams. In this paper, we proposed an approach, called Dynamic Count-Min sketch (DCM), which is appropriate for dynamic data set and can provide accurate estimates for point query and self-join size query. Our approach constitutes incremental CM sketches and allocates space in a pay-as-you-go manner. Our mathematical analysis and substantial experiments both show that our approach is appropriate for data sets with dynamic or skewed inputs and can provide error-guaranteed estimates with less space compared to CM.

Scientists carry out complex scientific data analyses by managing and executing many related computational steps. Typically, scientists find a type of analysis relevant to their data, implement it step by step to try it out, and run many variants as they explore different datasets or method configurations. These processes are often done manually and are prone to error, slowing the pace of discoveries. Scientific workflows have emerged as a formalism to represent how the individual steps work and how they relate to the overall process. Workflows can be published, discovered, and reused to make data analysis processes more efficient through automation and assistance. In this talk, I will argue that integrating case-based reasoning techniques with workflows research would result in improved approaches to workflow sharing, retrieval, and adaptation. I will describe our initial work on semantic workflow matching using labeled graphs and knowledge intensive similarity measures. Furthermore, I will argue that if scientists followed a case-based approach more closely, scientific results would be more easily inspectable and reproducible. Through scientific workflows and case-based reasoning, scientific data analysis could be made more efficient and more rigorous.Keywordscase-based reasoningscientific data analysis

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

This paper discusses ongoing research on scientific workflows at the Institute of Computing, University of Campinas (IC - UNICAMP) Brazil. Our projects with bio-scientists have led us to develop a scientific workflow infrastructure named WOODSS. This framework has two main objectives in mind: to help scientists to specify and annotate their models and experiments; and to document collaborative efforts in scientific activities. In both contexts, workflows are annotated and stored in a database. This "annotated scientific workflow" database is treated as a repository of (sometimes incomplete) approaches to solving scientific problems. Thus, it serves two purposes: allows comparison of distinct solutions to a problem, and their designs; and provides reusable and executable building blocks to construct new scientific workflows, to meet specific needs. Annotations, moreover, allow further insight into methodology, success rates, underlying hypotheses and other issues in experimental activities.The many research challenges faced by us at the moment include: the extension of this framework to the Web, following Semantic Web standards; providing means of discovering workflow components on the Web for reuse; and taking advantage of planning in Artificial Intelligence to support composition mechanisms. This paper describes our efforts in these directions, tested over two domains - agro-environmental planning and bioinformatics.

Geospatially Enabled Scientific Workflows offer a promising toolset to help researchers in the earth observation domain with many aspects of the scientific process. One such aspect is that of access to distributed earth observation data and computing resources. Earth observation research often utilizes large datasets requiring extensive CPU and memory resources in their processing. These resource intensive processes can be chained; the sequence of processes (and their provenance) makes up a scientific workflow. Despite the exponential growth in capacity of desktop computers, their resources are often insufficient for the scientific workflow processing tasks at hand. By integrating distributed computing capabilities into a geospatially enabled scientific workflow environment, it is possible to provide researchers with a mechanism to overcome the limitations of the desktop computer. Most of the effort on extending scientific workflows with distributed computing capabilities has focused on the web services approach, as exemplified by the OGC's Web Processing Service and by GRID computing. The approach to leveraging distributed computing resources described in this article uses instead remote objects via RPyC and the dynamic properties of the Python programming language. The Vistrails environment has been extended to allow for geospatial processing through the EO4Vistrails package (http://code.google.com/p/eo4vistrails/). In order to allow these geospatial processes to be seamlessly executed on distributed resources such as cloud computing nodes, the Vistrails environment has been extended with both multi‐tasking capabilities and distributed processing capabilities. The multi‐tasking capabilities are required in order to allow Vistrails to run side‐by‐side processes, a capability it does not currently have. The distributed processing capabilities are achieved through the use of remote objects and mobile code through RPyC.