Addressing the Shimming Problem in Big Data Scientific Workflows

Scientific workflows benefit from the cloud computing paradigm, which offers access to virtual resources provisioned on pay-as-you-go and on-demand basis. Minimizing resources costs to meet user’s budget is very important in a cloud environment. Several optimization approaches have been proposed to improve the performance and the cost of data-intensive scientific Workflow Scheduling (DiSWS) in cloud computing. However, in the literature, the majority of the DiSWS approaches focused on the use of heuristic and metaheuristic as an optimization method. Furthermore, the tasks hierarchy in data-intensive scientific workflows has not been extensively explored in the current literature. Specifically, in this paper, a data-intensive scientific workflow is represented as a hierarchy, which specifies hierarchical relations between workflow tasks, and an approach for data-intensive workflow scheduling applications is proposed. In this approach, first, the datasets and workflow tasks are modeled as a conditional probability matrix (CPM). Second, several data transformation and hierarchical clustering are applied to the CPM structure to determine the minimum number of virtual machines needed for the workflow execution. In this approach, the hierarchical clustering is done with respect to the budget imposed by the user. After data transformation and hierarchical clustering, the amount of data transmitted between clusters can be reduced, which can improve cost and makespan of the workflow by optimizing the use of virtual resources and network bandwidth. The performance and cost are analyzed using an extension of Cloudsim simulation tool and compared with existing multi-objective approaches. The results demonstrate that our approach reduces resources cost with respect to the user budgets.

Scientific workflows are widely used to automate scientific data analysis and often involve computationally intensive processing of large datasets on compute clusters. As such, their execution tends to be long-running and resource-intensive, resulting in substantial energy consumption and, depending on the energy mix, carbon emissions. Meanwhile, a wealth of carbon-aware computing methods have been proposed, yet little work has focused specifically on scientific workflows, even though they present a substantial opportunity for carbon-aware computing because they are often significantly delay tolerant, efficiently interruptible, highly scalable and widely heterogeneous. In this study, we first exemplify the problem of carbon emissions associated with running scientific workflows, and then show the potential for carbon-aware workflow execution. For this, we estimate the carbon footprint of seven real-world Nextflow workflows executed on different cluster infrastructures using both average and marginal carbon intensity data. Furthermore, we systematically evaluate the impact of carbon-aware temporal shifting, and the pausing and resuming of the workflow. Moreover, we apply resource scaling to workflows and workflow tasks. Finally, we report the potential reduction in overall carbon emissions, with temporal shifting capable of decreasing emissions by over 80%, and resource scaling capable of decreasing emissions by 67%.

Service-oriented workflow provides a mechanism to automate the coordination of a sequence of tasks within a process, whereas each task is presented as a Web service and integrated into the workflow. It has become increasingly important to have an appropriate workflow platform to manage the scientific workflow lifecycle. In order to address this need, we propose to use Pipeline Pilot to manage the lifecycle of grid-based scientific workflow using SOA approach. Pipeline Pilot is a commercial software package dedicated to scientific research. It is reliable, lightweight and comprehensive. We then describe a case study of how we employ Pipeline Pilot to manage a workflow lifecycle for calculating material dielectric properties in grid environment. We also discuss lessons we learned in managing lifecycle of a service-oriented scientific workflow within e-Science.

The objective of this paper was to propose a multi-agents model for automating big data processing, to improve the process of decision-making and urban planning. The huge amounts of collected data from different domains, such as urban management and remote sensing, are characterized as big data with a spatial component. Smart data is the approach to deal with big data characteristics and challenges by focusing on the Value aspect. The focus on smart data on the relevant data and the mechanism of automation and collaboration of the agents, will contribute to the efficient automating for big data analytics and processing. The proposed approach is based on a collection of agents, and adopt the concept of smart data, this paradigm focus on the aspect value from the big data and help to retrieve the useful information from the large volumes of data by intelligent processing. The proposed model describes the functionalities of the agents. The agents receive data in real time, perform the operations of storing data, pre-processing, streaming processing and batch processing and finally transfer the results of analysis to the services and applications. Machine learning techniques can be used to enhance the aspect of cognition of the agents; through a case study, we used supervised learning methods to build a classification model to support the process of urban planning by predicting the type of habitat adequate for the population based on their demographic and socio-economic characteristics.KeywordsSmart dataBig dataMulti-agent systemAutomationMachine learningUrban planningBig data analyticsDecision-making

The advancement of science is increasingly intertwined with complex computational processes [1]. Scientific workflows are at the heart of this evolution, acting as essential orchestrators for a vast range of experiments. Specifically, these workflows are central to the field of computational Earth Sciences, where they orchestrate a diverse range of activities, from cloud-based data preprocessing pipelines in environmental modeling to intricate multi-facility instrument-to-edge-to-HPC computational frameworks for seismic data analysis and geophysical simulations [2]. The emergence of continuum and cross-facility workflows marks a significant evolution in computational sciences [3]. Continuum workflows represent continuous computing access required for analysis pipelines, while cross-facility workflows extend across multiple sites, integrating experiments andcomputing facilities. These cross-facility workflows, crucial for real-time applications, offer resiliency and stand as solutions for the demands of continuum workflows. Addressing continuum and cross-facility computing requires a focus on data, ensuring workflow systems are equipped to handle diverse datarepresentations and storage systems. As we navigate the computing continuum, the pressing needs of contemporary scientific applications in Earth Sciences call for a dual approach: the recalibration of existing systems and the innovation of new workflow functionalities. This recalibration involves optimizing data-intensive operations andincorporating advanced algorithms for spatial data analysis, while innovation may entail the integration of machine learning techniques for predictive modeling and real-time data processing in earth sciences. We offer a comprehensive overview of cutting-edge advancements in this dynamic realm, with a focus on computational Earth Sciences, including managing the increasing volume and complexity of geospatial data, ensuring the reproducibility of large-scale simulations, and adapting workflows to leverage emerging computational architectures.   [1] Ferreira da Silva, R., Casanova, H., Chard, K., Altintas, I., Badia, R. M., Balis, B., Coleman, T., Coppens, F., Di Natale, F., Enders, B., Fahringer, T., Filgueira, R., et al. (2021). A Community Roadmap for Scientific Workflows Research and Development. 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), 81–90. DOI: 10.1109/WORKS54523.2021.00016 [2] Badia Sala, R. M., Ayguadé Parra, E., & Labarta Mancho, J. J. (2017). Workflows for science: A challenge when facing the convergence of HPC and big data. Supercomputing frontiers and innovations, 4(1), 27-47. DOI: 10.14529/jsfi170102 [3] Antypas, K. B., Bard, D. J., Blaschke, J. P., Canon, R. S., Enders, B., Shankar, M. A., ... & Wilkinson, S. R. (2021, December). Enabling discovery data science through cross-facility workflows. In 2021 IEEE International Conference on Big Data (Big Data) (pp. 3671-3680). IEEE. DOI: 10.1109/BigData52589.2021.967142

Most of the domains of science are facing an explosion in the data that they have to collect and process in order to conduct research. This is true for both: scientific domains dealing with experimental data, e.g. biology, sociology, astronomy etc, but also scientific domains dealing with simulation data, e.g. seismology, physics etc. To maximize the potential outcome of scientific data analysis, respective data management applications need to fulfil the following coarse tasks: fast on-demand data processing, and effective storage and consolidation of diverse data collections. These two tasks are in general hard to realize because of: (a) the big data size, (b) the diversity of data formats, (c) their conceptual dependencies, (d) disperse data locations, and (e) intensive and systematic nature of scientific queries. We present the characteristics of big scientific data collections and their necessities in terms of data management. Based on this discussion, we discuss the structure of a framework for the processing and consolidation of heterogeneous scientific data collections. Such a framework aims to mediate between the user and a set of available data management technologies, such as relational DBMSs, key-value stores and column stores, in order to efficiently direct data management operations (insertions, updates) and especially requests (queries) to the appropriate data management application. The framework aims to distribute, dissect, and schedule data management actions, as well as integrate results, in a way that reduces response time. This entails the accommodation of methods for the selective parallelism and serialization depending on partial results and response times. Also, this entails the accommodation of methods for the gradual alteration of data formats and storage, e.g. storage of semi-structured data or raw data in files into relational databases. Furthermore, we discuss the processing of scientific query bulks or workflows with the possibility to retrieve early partial results and calibrate query parameters.

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid, which offers supercomputing power through shared distributed resources. To execute a scientific workflow in the Grid, tasks within the workflow that represent steps in the scientific process are assigned to Grid resources for execution. To ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources. Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows which focus on parametric study and parameter optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise. In the thesis, we propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. We evaluate the Besom algorithm under a variety of conditions. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks. The Besom scheduling algorithm advances the ability to schedule parallel execution of parameter sweep workflows in the Grid. The outcomes of this thesis justify continuing research in this area to increase our understanding of scheduling multiple Grid workflow instances and to provide support to those involved in parametric study and scientific workflow management.

Scientific applications are more and more faced with very large volumes of data and complex, resource-intensive workflows that process or analyze these data. The recent interest in web services and service-oriented architectures has strongly facilitated the development of individual workflow activities as well as their composition and the distributed execution of complete workflows. However, in many applications concurrent scientific workflows may be served by multiple competing providers, with each of them offering only limited resources. At the same time, these workflows need to be executed in a predictable manner, with dedicated Quality of Service guarantees. In this paper, we introduce an approach to Advance Resource Reservation for service-oriented complex scientific workflows that optimize resource consumption based on user-defined criteria (e.g., cost or time). It exploits optimization techniques using genetic algorithms for finding optimal or near-optimal allocations in a distributed system. The approach takes into account the locality of services and in particular enforces constraints imposed by control or data flow dependencies within workflows. Finally, we provide a comprehensive evaluation of the effectiveness of the proposed approach.

Scheduling is a process that manages the workflow tasks during execution on different resources. Virtual infrastructure is a dynamic mapping of system resources to applications in order to maximize its utilization. In today's technological world, cloud has taken a long stride on the success towards maximum throughput as well as highest qualitative services to its consumers. Yet, approaches for maximizing the utilization of cloud resources are at peak demand. Each cloud service provider focuses on maximum utilization with minimum consumption of cloud resources, although managing and providing computational resources to maximum number of users and to execute such huge applications is a challenging one. In this paper, a scheduling algorithm with name TBW (Tabu Bayesian Whale Optimization) has been proposed. Basically, the algorithm is used to target the improvement in scheduling of scientific workflows. The complete framework has firstly used a ranking algorithm named distributed HEFT ranking and then applied TBW algorithm on ranked tasks of input workflows. The work has been executed for five scientific workflows LIGO, MONTAGE, Epigenomics, SIPHT and Cybershake. TBW is using tabu method on workflow tasks for fast local search in cloud system, and Bayesian Optimization is used to find out best possible combinations of resources where tasks are mapped and then whale optimization maps the tasks on the resources in a smart way. In the whole process, total execution time and cost parameters are minimized under deadline constraints.KeywordsCloud VMsTask mappingTabu searchBayesian optimizationWhale optimization

In distributed computing an escalating trend is Cloud computing which mainly focuses on directly delivering computational power to the end-user regardless of when and where. Delivering of services to the users is there such as Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS). Cloud computing is having an IaaS model which is a desired platform for workflow execution of scientific applications for example bioinformatics, astronomy, geophysics etc. Hence, several researches are recommencing scientific workflows execution in Infrastructure as a Service cloud computing environment. An algorithm ‘Rank-Based Task Scheduling Algorithm’ (RBTSA) which is proposed by Shilpa Rana et al. have future research scope that is the performance of RBTSA can be tested by calculating the rank of virtual machines by considering different characteristic of bandwidth, physical memory etc. of virtual machines which is considered in this paper. Herein paper an algorithm termed as ‘Multi Criteria Rank-Based Task Scheduling Algorithm’ (MCRBTSA) is planned for scientific workflows tasks scheduling to increase resources utilization in IaaS cloud computing environment. The experimental simulation of research work which is mentioned above is done on ‘WorkflowSim-1.0’ and scientific workflow execution results of proposed MCRBTSA Algorithm are contrasted with the existing ‘Rank-Based Task Scheduling Algorithm’ (RBTSA). Experimentally it shows that MCRBTSA which is proposed have utilization of resources better while scheduling workflow tasks as compared to existing RBTSA and workflow makespan reduction is also there.

This paper explores the specialized nature of research-orient\- ed web applications that enable interactions with and the visual analysis of ``Big Data,'' i.e., large, heterogeneous scientific datasets. We introduce a pragmatic methodology for the design and evaluation of scientific workflows in research-oriented web applications. Through an in-depth usability study of the CoGe web application, a system that provides a rich set of tools for exploring genomic datasets, we demonstrate: how to identify bottlenecks in multi-step tasks; how to analyze these bottlenecks in order to provide effective solutions for improving user experience; and how these solutions may more generally apply to similar research-oriented websites in other scientific domains that also enable scientific workflows. Specifically, we provide details regarding: our user interviews, the visualization system we created to analyze complex tasks associated with scientific workflows, and how this analysis directly leads to suggestions for improvements in the current implementation of the CoGe web application. A follow-up study was carried out which indicates that our suggestions improved the ability of CoGe users to navigate and complete custom workflows, leading us to believe that our approach could also be applied to other research-oriented web applications that utilize scientific datasets.

Managing structural genomic workflows using Web services

Toward self-describing and workflow integrated Earth system models: A coupled atmosphere-ocean modeling system application

Energy-efficient execution of the scientific workflow is a challenging task in cloud computing that demands high-performance computing to process growing datasets. Due to the interdependency of tasks in the scientific workflow applications, energy-efficient resource allocation is vital for large-scale applications running on heterogeneous physical machines. Thus, this paper proposes a hybrid heuristic algorithm based energy-efficient cloud computing service (HH-ECO) that offers a significant solution for resource allocation, task scheduling, and optimization of scientific workflows. To ensure the energy-efficient execution, the HH-ECO focuses on executing non-dominant workflow tasks through adaptive mutation and energy-aware migration strategy. HH-ECO adopts the chaotic based particle swarm optimization (C-PSO) principle to optimize the resource allocation, task scheduling, and resource migration by generating the global best plans without local convergence. C-PSO with adaptive mutation avoids the deterioration of global optima while finding the best host to place the virtual machine and ensures an appropriate resource allocation plan. By considering the workflow task precedence relationships during C-PSO based task scheduling, the novel hybrid heuristic method efficiently solves the multi-objective combinatorial optimization problem without dominance among the workflow tasks. The Cloudsim based simulation study delivers superior results compared to the existing methods such as the hybrid heuristic workflow scheduling algorithm (HHWS) and distributed dynamic VM management (DDVM). The proposed approach significantly improves the optimal makespan to 38.27% and energy conservation to 38.06% compared to the existing methods.

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.