SAGA: A standardized access layer to heterogeneous Distributed Computing Infrastructure

A growing number of very simple parallel applications can benefit from the availability of very large computing resource pools that are exploited in an opportunistic way to considerably speed up the processing of these applications. In particular, many applications can scale out to take advantage of the availability of thousands, or even millions, of processors that can be simultaneously used. Some large-scale distributed computing infrastructures (DCI) have already been successfully assembled to this end. However, these DCI are normally designed to cater to specific kinds of applications, and their assemblage is not only costly, but also time consuming. We propose a novel architecture, named OddCI (On-demand DCI), for the cheap, fast and on-demand instantiation and dismantle of very large DCI. These characteristics allow application users to dynamically commission DCI that can cost-effectively be customized to their particular needs, and disposed after being used. Our approach leverages on broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as mobile phones, and digital television (DTV) receivers. We discuss how the proposed architecture can be implemented on top of a network of DTV receivers, providing evidences of both the technical, as well as the operational feasibility of the OddCI architecture.

Nowadays visualization-based knowledge discovery can play an important role in astrophysics. Collaborative visualization can enable multiple users to share visualization experiences, e.g. by interacting simultaneously with astrophysical datasets giving feedback on what other participants are doing/seeing. Further, workflow-driven applications allow reproduction of specific visualization results, a challenging task as selecting suitable visualization parameters may not be a straightforward process. This paper presents VisIVO Science Gateway, a web-based workflow-enabled framework integrating large-scale, multidimensional datasets and applications for visualization and data filtering on Distributed Computing Infrastructures (DCIs). Advanced users are able to create, change, invoke, and monitor workflows while standard users are provided with easy-to-use customised web interfaces hiding all technical aspects of the visualization algorithms and DCI configurations.

Virtual high-throughput screening (vHTS) is an invaluable method in modern drug discovery. It permits screening large datasets or databases of chemical structures for those structures binding possibly to a drug target. Virtual screening is typically performed by docking code, which often runs sequentially. Processing of huge vHTS datasets can be parallelized by chunking the data because individual docking runs are independent of each other. The goal of this work is to find an optimal splitting maximizing the speedup while considering overhead and available cores on Distributed Computing Infrastructures (DCIs). We have conducted thorough performance studies accounting not only for the runtime of the docking itself, but also for structure preparation. Performance studies were conducted via the workflow-enabled science gateway MoSGrid (Molecular Simulation Grid). As input we used benchmark datasets for protein kinases. Our performance studies show that docking workflows can be made to scale almost linearly up to 500 concurrent processes distributed even over large DCIs, thus accelerating vHTS campaigns significantly.

Allocation of computing resources is a crucial issue when dealing with a huge number of tasks to be completed according to a given deadline and cost constraints. The task scheduling to several resources (e.g. grid, cloud or a supercomputer) with different characteristics is not trivial, especially if a trade-off in terms of time and cost is considered. We propose an allocation approach able to fulfill the given requirements about time and cost through the use of optimizing techniques and an adaptive behavior. Simulated productions of tasks have been run in order to evaluate the characteristics of the proposed approach.

In this paper, we address job scheduling in Distributed Computing Infrastructures, that is a loosely coupled network of autonomous acting High Performance Computing systems. In contrast to the common approach of mutual workload exchange, we consider the more intuitive operator's viewpoint of load-dependent resource reconfiguration. In case of a site's over-utilization, the scheduling system is able to lease resources from other sites to keep up service quality for its local user community. Contrary, the granting of idle resources can increase utilization in times of low local workload and thus ensure higher efficiency. The evaluation considers real workload data and is done with respect to common service quality indicators. For two simple resource exchange policies and three basic setups we show the possible gain of this approach and analyze the dynamics in workload-adaptive reconfiguration behavior.

WS-PGRADE/gUSE is a gateway framework that offers a set of high-level grid and cloud services by which interoperation between grids, clouds, and scientific user communities can be achieved. gUSE is also a workflow system that enables scientific communities to compose and execute a series of computational or data manipulation steps in a scientific application on Distributed Computing Infrastructures (DCIs). The current chapter summarizes the most important features of WS-PGRADE/gUSE.

The Cherenkov Telescope Array (CTA) - an array of several tens of Cherenkov telescopes - is the next generation of ground-based instrument in the field of very high energy gamma-ray astronomy. The CTA observatory is expected to produce a main data stream for permanent storage of the order of 1-to-5 GB/s for about 1000 hours of observation per year, thus producing a total data volume of the order of several PB per year. The CPU time needed to calibrate and process one hour of data taking will be of the order of some thousands CPU hours with current technology. The high data rate of CTA, together with the large computing power requirements for Monte Carlo (MC) simulations, need dedicated computing resources. Massive MC simulations are needed to study the physics of cosmic-ray atmospheric showers as well as telescope response and performance for different detectors and layout configurations. Given these large storage and computing requirements, the Grid approach is well suited, and a vast number of MC simulations are already running on the European Grid Infrastructure (EGI). In order to optimize resource usage and to handle all production and future analysis activities in a coherent way, a high-level framework with advanced functionalities is desirable. For this purpose we have preliminarly evaluated the DIRAC framework for distributed computing and tested it for the CTA workload and data management systems. In this paper we present a possible implementation of a Distributed Computing Infrastructure (DCI) Computing Model for CTA as well as the benchmark test results of DIRAC.

OF THE THESIS Pilot based frameworks for Weather Research Forecasting by Dinesh Prasanth Ganapathi Thesis Director: Dr. Shantenu Jha The Weather Research Forecasting (WRF) domain consists of complex workflows that demand the use of Distributed Computing Infrastructure (DCI). Weather forecasting requires that weather researchers use different set of initial conditions and one or a combination of physics models on the same set of input data. For these type of simulations an ensemble based computing approach becomes imperative. Most DCIs have local job-schedulers that have no smart way of dealing with the execution of an ensemble type of computational problem as the job-schedulers are built to cater to the bare essentials of resource allocation. This means the weather scientists have to submit multiple jobs to the job-scheduler. In this dissertation we use Pilot-Job based tools to decouple work-load submission and resource allocation therefore streamlining the complex workflows in Weather Research and Forecasting domain and reduce their overall time to completion. We also achieve location independent job execution, data movement, placement and processing. Next, we create the necessary enablers to run an ensemble of tasks bearing the capability to run on multiple heterogeneous distributed computing resources there by creating the opportunity to minimize the overall time consumed in running the models. Our experiments show that the tools developed exhibit very good, strong and weak scaling characteristics. These results bear the potential to change the way weather researchers are submitting traditional WRF jobs to the

The availability of large quantities of processors is a crucial enabler of many-task computing. Voluntary computing systems have proven that it is possible to build computing platforms with millions of nodes to support the execution of embarrassingly parallel applications. These systems, however, lack the flexibility of more traditional grid infrastructures. On the other hand, flexible infrastructures currently available can gather only dozens of thousands nodes. We propose a novel architecture for generic Distributed Computing Infrastructures (DCI) that can be instantiated on demand to be, at the same time, flexible and highly-scalable. Bringing the scalability from voluntary computing, the flexibility from grid computing and the elasticity from cloud computing in a single arrangement, our proposal allows for fast setup, fast initialization and fast dismantle of customized DCI supported by both dedicated and shared underlying infrastructures. Our approach leverages broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as PDA, mobile phones and Digital TV receivers, using both opportunistic and non-opportunistic models. We show the feasibility of the proposed architecture by implementing it atop a digital television system. We also assess the performance of such system and show that it can be used to execute several classes of many-tasks computing applications with very high efficiency, substantially decreasing their response time.

Distributed computing owes its success to the development of infrastructure, middleware, and standards (e.g., CORBA) to support interoperability. The computing community has slowly recognized the need to protect information and has begun to develop commercial security infrastructures and standards. The US Government must protect national security information against unauthorized information flow. To support MLS distributed computing, a MLS infrastructure must be built that enables information sharing among users at different classification levels. This infrastructure should provide MLS services for protection of classified information and use both the emerging distributed computing and commercial security infrastructures, when possible. The resulting infrastructure will enable users to integrate commercial information technology products into their systems. In this paper, we examine the philosophy that has led to successful distributed computing among heterogeneous, autonomous components and propose an analogous approach for MLS distributed computing. We identify some services that are required to support MLS distributed computing, argue that these services are needed regardless of the MLS architecture used, present an approach for designing these services, and provide design guidance for a critical building block of the MLS infrastructure.

In this paper we present the concept of a scalable job centric monitoring infrastructure.The overall performance of this distributed, layer based architecturecalled SLAte can be increased by installing additional servers to adapt to thedemands of the monitored resources and users. Another important aspect is tooffer a uniform global view on all data which are stored distributed to providean easy access for users or visualisation tools. Additionally we discus the impactof these uniform access layer on scalability.

This chapter provides a brief overview of the UNICORE grid middleware and its utilization in the large Distributed Computing Infrastructure. UNICORE framework, in its recent version implements key grid standards and specifications. The system architecture and capabilities, such as security, workflow and data management are described. The installation of the UNICORE environment in the PL-Grid is presented. Special attention is given to the integration of the UNICORE middleware with the PL-Grid authentication and authorization framework which allows for uniform infrastructure and user management across different middlewares. The solutions for monitoring and accounting of the UNICORE infrastructure is presented.

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures.

The current state of much of the Wuhan pneumonia virus (severe acute respiratory syndrome coronavirus 2 [SARS-CoV-2]) research shows a regrettable lack of data sharing and considerable analytical obfuscation. This impedes global research cooperation, which is essential for tackling public health emergencies and requires unimpeded access to data, analysis tools, and computational infrastructure. Here, we show that community efforts in developing open analytical software tools over the past 10 years, combined with national investments into scientific computational infrastructure, can overcome these deficiencies and provide an accessible platform for tackling global health emergencies in an open and transparent manner. Specifically, we use all SARS-CoV-2 genomic data available in the public domain so far to (1) underscore the importance of access to raw data and (2) demonstrate that existing community efforts in curation and deployment of biomedical software can reliably support rapid, reproducible research during global health crises. All our analyses are fully documented at https://github.com/galaxyproject/SARS-CoV-2.

Machine learning for Big Data analytics in plants.