Introduction to the WS-PGRADE/gUSE Science Gateway Framework

Neuroimaging is a field that benefits from distributed computing infrastructures (DCIs) to perform data processing and analysis, which is often achieved using grid workflow systems. Collaborative research in neuroimaging requires ways to facilitate exchange between different groups, in particular to enable sharing, re-use and interoperability of applications implemented as workflows. The SHIWA project provides solutions to facilitate sharing and exchange of workflows between workflow systems and DCI resources. In this paper we present and analyse how the SHIWA platform was used to implement various usage scenarios in which workflow exchange supports collaboration in neuroscience. The SHIWA platform and the implemented solutions are described and analysed from the user perspective, in this case the workflow developers and the neuroscientists. We conclude that the platform in its current form is valuable for the foreseen usage scenarios, and we identify remaining challenges concerning management of multiple credentials and data transfers across DCIs.

A growing number of very simple parallel applications can benefit from the availability of very large computing resource pools that are exploited in an opportunistic way to considerably speed up the processing of these applications. In particular, many applications can scale out to take advantage of the availability of thousands, or even millions, of processors that can be simultaneously used. Some large-scale distributed computing infrastructures (DCI) have already been successfully assembled to this end. However, these DCI are normally designed to cater to specific kinds of applications, and their assemblage is not only costly, but also time consuming. We propose a novel architecture, named OddCI (On-demand DCI), for the cheap, fast and on-demand instantiation and dismantle of very large DCI. These characteristics allow application users to dynamically commission DCI that can cost-effectively be customized to their particular needs, and disposed after being used. Our approach leverages on broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as mobile phones, and digital television (DTV) receivers. We discuss how the proposed architecture can be implemented on top of a network of DTV receivers, providing evidences of both the technical, as well as the operational feasibility of the OddCI architecture.

Cloud Computing has gained a lot of popularity in recent years because of the flexibility that it offers. In addition, there seems to be a rising interest in combining Parallel Computing, Cloud Computing and Big Data to create large scale scientific applications. WS-PGRADE is a gateway framework that allows users to create such applications by defining them as scientific workflows. This paper investigates how workflow systems and science gateways, such as WS-PGRADE, can be extended with data processing capabilities of Hadoop based on the MapReduce paradigm in the cloud. Analysis shows the methods described to integrate Hadoop with workflows and science gateways work well in different scenarios and can be used to create massively parallel applications for scientific analysis of Big Data.

The rapid advancement in the collection of environmental data, geospatial analysis and modeling software calls for new GIS-enabled cyberinfrastructure (CI) capable of support large scale integrated data and modeling activities. Although the HUBzero platform is gaining popularity as a portal development framework for many science and engineering communities, it has limited success in earth and environmental science domains, mainly due to its lack of native support for large geospatial data management and GIS functionalities in the Rappture toolkit. In this paper, we describe our initial work on bringing integrated GIS data and modeling capabilities into the HUBzero platform. We describe the overall architecture of our CI solution which is scalable, extensible and portable to other gateway framework as well. Two prototype applications are presented as a proof-of-concept. Coupling the scalable GIS capabilities with a wealth of tools for user participation and community building on the HUBzero platform, we hope to help new science communities in utilizing this platform.

Nowadays visualization-based knowledge discovery can play an important role in astrophysics. Collaborative visualization can enable multiple users to share visualization experiences, e.g. by interacting simultaneously with astrophysical datasets giving feedback on what other participants are doing/seeing. Further, workflow-driven applications allow reproduction of specific visualization results, a challenging task as selecting suitable visualization parameters may not be a straightforward process. This paper presents VisIVO Science Gateway, a web-based workflow-enabled framework integrating large-scale, multidimensional datasets and applications for visualization and data filtering on Distributed Computing Infrastructures (DCIs). Advanced users are able to create, change, invoke, and monitor workflows while standard users are provided with easy-to-use customised web interfaces hiding all technical aspects of the visualization algorithms and DCI configurations.

Data-intensive scientific applications are posing many challenges in distributed computing systems. In the scientific field, the application data are expected to double every year over the next decade and further. With this continuing data explosion, high performance computing systems are needed to store and process data efficiently, and workflow technologies are facilitated to automate these scientific applications. Scientific workflows are typically very complex. They usually have a large number of tasks and need a long time for execution. Running scientific workflow applications usually need not only high performance computing resources but also massive storage. The emergence of cloud computing technologies offers a new way to develop scientific workflow systems. Scientists can upload their data and launch their applications on the scientific cloud workflow systems from everywhere in the world via the Internet, and they only need to pay for the resources that they use for their applications. As all the data are managed in the cloud, it is easy to share data among scientists. This kind of model is very convenient for users, but remains a big challenge to the system. This paper proposes several research topics of data management in scientific cloud workflow systems, and discusses their research methodologies and state-of-the-art solutions.

An important part of enabling the future information infrastructure is to provide convenient services for application users that make such an environment highly functional and easy to use for the non-expert. In other words, the scientist should be able to focus on the science but should have sophisticated tools at hand that enable new and enhanced modalities in which science can be performed. We describe a general architecture that provides access to scientific applications. We demonstrate how such services can interact with each other and be reused by the scientific application architect to orchestrate Grid services. Our architecture is implemented by reusing concepts, infrastructures, and middleware from Web and Grid services, as well as from the newest Globus Toolkit and the Java CoG Kit. We apply our general architecture to a specific application: Active Thermochemical Tables. This application demonstrates a number of interesting and future-oriented uses, such as the need for batch processing, interactive and collaborative steering, use of multiple platforms,visualization through large displays, and access via a portal framework. Besides the innovative use of the Grid and Web services, we also provide a novel algorithmic contribution to scientific disciplines that use thermochemical tables. Specifically, we modified the original approach to constructing thermochemical tables to include an iterative process of refinement leading to increased accuracy. We have designed a variety of access environments including a shell, a Grid desktop, and a portal for accessing the set of services provided, which include the display of network dependencies between the reactions a chemist may be interested in and interactive querying of associated species data. Modalities that need to be supported are the protection of intellectual property, the creation of metadata and information shared with and gathered by the community, and easy access to the service by the application scientists.

Distributed Computing Infrastructure is characterized by interfaces that are heterogeneous—syntactically and semantically. SAGA represents the most comprehensive community effort to date to address the heterogeneity by defining a simple, uniform access layer. In this paper, we describe the basic concepts underpinning its design and development. We also discuss RADICAL-SAGA which is the most widely used implementation of SAGA.

Virtual high-throughput screening (vHTS) is an invaluable method in modern drug discovery. It permits screening large datasets or databases of chemical structures for those structures binding possibly to a drug target. Virtual screening is typically performed by docking code, which often runs sequentially. Processing of huge vHTS datasets can be parallelized by chunking the data because individual docking runs are independent of each other. The goal of this work is to find an optimal splitting maximizing the speedup while considering overhead and available cores on Distributed Computing Infrastructures (DCIs). We have conducted thorough performance studies accounting not only for the runtime of the docking itself, but also for structure preparation. Performance studies were conducted via the workflow-enabled science gateway MoSGrid (Molecular Simulation Grid). As input we used benchmark datasets for protein kinases. Our performance studies show that docking workflows can be made to scale almost linearly up to 500 concurrent processes distributed even over large DCIs, thus accelerating vHTS campaigns significantly.

Allocation of computing resources is a crucial issue when dealing with a huge number of tasks to be completed according to a given deadline and cost constraints. The task scheduling to several resources (e.g. grid, cloud or a supercomputer) with different characteristics is not trivial, especially if a trade-off in terms of time and cost is considered. We propose an allocation approach able to fulfill the given requirements about time and cost through the use of optimizing techniques and an adaptive behavior. Simulated productions of tasks have been run in order to evaluate the characteristics of the proposed approach.

In this paper, we address job scheduling in Distributed Computing Infrastructures, that is a loosely coupled network of autonomous acting High Performance Computing systems. In contrast to the common approach of mutual workload exchange, we consider the more intuitive operator's viewpoint of load-dependent resource reconfiguration. In case of a site's over-utilization, the scheduling system is able to lease resources from other sites to keep up service quality for its local user community. Contrary, the granting of idle resources can increase utilization in times of low local workload and thus ensure higher efficiency. The evaluation considers real workload data and is done with respect to common service quality indicators. For two simple resource exchange policies and three basic setups we show the possible gain of this approach and analyze the dynamics in workload-adaptive reconfiguration behavior.

The Cherenkov Telescope Array (CTA) - an array of several tens of Cherenkov telescopes - is the next generation of ground-based instrument in the field of very high energy gamma-ray astronomy. The CTA observatory is expected to produce a main data stream for permanent storage of the order of 1-to-5 GB/s for about 1000 hours of observation per year, thus producing a total data volume of the order of several PB per year. The CPU time needed to calibrate and process one hour of data taking will be of the order of some thousands CPU hours with current technology. The high data rate of CTA, together with the large computing power requirements for Monte Carlo (MC) simulations, need dedicated computing resources. Massive MC simulations are needed to study the physics of cosmic-ray atmospheric showers as well as telescope response and performance for different detectors and layout configurations. Given these large storage and computing requirements, the Grid approach is well suited, and a vast number of MC simulations are already running on the European Grid Infrastructure (EGI). In order to optimize resource usage and to handle all production and future analysis activities in a coherent way, a high-level framework with advanced functionalities is desirable. For this purpose we have preliminarly evaluated the DIRAC framework for distributed computing and tested it for the CTA workload and data management systems. In this paper we present a possible implementation of a Distributed Computing Infrastructure (DCI) Computing Model for CTA as well as the benchmark test results of DIRAC.

OF THE THESIS Pilot based frameworks for Weather Research Forecasting by Dinesh Prasanth Ganapathi Thesis Director: Dr. Shantenu Jha The Weather Research Forecasting (WRF) domain consists of complex workflows that demand the use of Distributed Computing Infrastructure (DCI). Weather forecasting requires that weather researchers use different set of initial conditions and one or a combination of physics models on the same set of input data. For these type of simulations an ensemble based computing approach becomes imperative. Most DCIs have local job-schedulers that have no smart way of dealing with the execution of an ensemble type of computational problem as the job-schedulers are built to cater to the bare essentials of resource allocation. This means the weather scientists have to submit multiple jobs to the job-scheduler. In this dissertation we use Pilot-Job based tools to decouple work-load submission and resource allocation therefore streamlining the complex workflows in Weather Research and Forecasting domain and reduce their overall time to completion. We also achieve location independent job execution, data movement, placement and processing. Next, we create the necessary enablers to run an ensemble of tasks bearing the capability to run on multiple heterogeneous distributed computing resources there by creating the opportunity to minimize the overall time consumed in running the models. Our experiments show that the tools developed exhibit very good, strong and weak scaling characteristics. These results bear the potential to change the way weather researchers are submitting traditional WRF jobs to the

The availability of large quantities of processors is a crucial enabler of many-task computing. Voluntary computing systems have proven that it is possible to build computing platforms with millions of nodes to support the execution of embarrassingly parallel applications. These systems, however, lack the flexibility of more traditional grid infrastructures. On the other hand, flexible infrastructures currently available can gather only dozens of thousands nodes. We propose a novel architecture for generic Distributed Computing Infrastructures (DCI) that can be instantiated on demand to be, at the same time, flexible and highly-scalable. Bringing the scalability from voluntary computing, the flexibility from grid computing and the elasticity from cloud computing in a single arrangement, our proposal allows for fast setup, fast initialization and fast dismantle of customized DCI supported by both dedicated and shared underlying infrastructures. Our approach leverages broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as PDA, mobile phones and Digital TV receivers, using both opportunistic and non-opportunistic models. We show the feasibility of the proposed architecture by implementing it atop a digital television system. We also assess the performance of such system and show that it can be used to execute several classes of many-tasks computing applications with very high efficiency, substantially decreasing their response time.

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.