Data placement in widely distributed environments

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

Large-scale data transport for data-intensive sciences is a complex multidimensional challenge. The challenge includes optimizing the end-to-end Big Data movement performance in real-time, supporting direct remote data access using NVMe over Fabrics (NVMeoF) and deploying to existing research platforms. AIDTN is the first effort to provide a unique AI system designed to incorporate NVMe over Fabrics (NVMeoF) and optimize coordination among multiple components supporting large-scale, multi-domain Wide Area Network (WAN) data-intensive science. AIDTN's research objective is to integrate next-generation storage architecture using NVMeoF, specialized network design using high-performance network appliances, Data Transfer Nodes (DTNs), catalysts in driving data transport, and a unique AI system explicitly designed for high-performance data movement challenges. AIDTN is the first system that uses network and system features to predict the end-to-end performance of high-performance data movement and further extends the model with NVMe-specific features for NVMeoF remote data access. As a result, AIDTN improves data movement performance by up to 284% while minimizing packet loss compared to other heuristics approaches. It also has a prediction error rate as low as 0.16 compared to AI models with the only network (error rate = 0.29) or network and system features (error rate = 0.19).

Network-on-Chip (NoC) is a critical hardware supporting on-chip data movement in multicore and many-core processors. Data movement is predicted to become a major power-consuming operation compared to computation as the technology scales. This paper shows that unifying router power gating with data placement significantly reduces dynamic and static power needed for moving data in NoC for memory hierarchy. Region-based data placement enables to localize private data traffic and to concentrate shared data traffic in one region of NoC, which shapes traffic in a well-behaved way and increases power gating opportunities. In this regard, a dimensionally power-gated router with a region-based routing algorithm is proposed to reduce router static power and performance/energy overheads in power gating. Full-system evaluation using SPEComp benchmarks shows that the dynamic power gating management achieves NoC power savings by 46% and the static power gating management improves energy-efficiency by 20%.

Many science facilities, such as the Department of Energy's Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide-area network (WAN) is a major problem inhibiting collaboration. Next-generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the endsystem hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimental systems, analysis and visualization clusters, in addition to wide-area data movers. These competing uses often induce temporary, but significant, I/O load imbalances on the storage system, which impact the performance of all the users. The problem is a serious concern because some resources are more expensive (e.g. super computers) or have time-critical deadlines (e.g. experimental data from a light source), but parallel file systems handle all requests fairly even if some storage servers are under heavy load. This paper investigates the problem of competing workloads accessing the parallel file system and how the performance of wide-area data movement can be improved in these environments. First, we study the I/O load imbalance problems using actual I/O performance data collected from the Spider storage system at the Oak Ridge Leadership Computing Facility. Second, we present I/O optimization solutions with layout-awareness on end-system hosts for bulk data movement. With our evaluation, we show that our I/O optimization techniques can avoid the I/O congested disk groups, improving storage I/O times on parallel storage systems for terabit data movement.

The core activity within the newly created H2020 project eXtreme DataCloud project will be the policy-driven orchestration of federated data management for data intensive sciences like High Energy Physics, Astronomy, Photon and Life Science. Well-known experts in this field will work on combining already established data management and orchestration tools to provide a highly scalable solution supporting the entire European Scientific Landscape. The work will cover Data Life Cycle Management as well as smart data placement on meta data, including storage availability, network bandwidth and data access patterns. Mechanisms will be put in place to trigger computational resources based on data ingestion and data movements. This paper presents the first architecture of this endeavor.

We present the design and implementation of Data Jockey, a data management system for HPC multi-tiered storage systems. As a centralized data management control plane, Data Jockey automates bulk data movement and placement for scientific workflows and integrates into existing HPC storage infrastructures. Data Jockey simplifies data management by eliminating human effort in programming complex data movements, laying datasets across multiple storage tiers when supporting complex workflows, which in turn increases the usability of multitiered storage systems emerging in modern HPC data centers.Specifically, Data Jockey presents a new data management scheme called “goal driven data management” that can automatically infer low-level bulk data movement plans from declarative high-level goal statements that come from the lifetime of iterative runs of scientific workflows. While doing so, Data Jockey aims to minimize data wait times by taking responsibility for datasets that are unused or to be used, and aggressively utilizing the capacity of the upper, higher performant storage tiers.We evaluated a prototype implementation of Data Jockey under a synthetic workload based on a year’s worth of Oak Ridge Leadership Computing Facility’s (OLCF) operational logs. Our evaluations suggest that Data Jockey leads to higher utilization of the upper storage tiers while minimizing the programming effort of data movement compared to human involved, per-domain adhoc data management scripts.

Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightly couple their domain specific applications with a post processing application.The paper presents a solution for the data transfer between MPI applications using a dedicated wide area network (WAN) terabit infrastructure. The proposed solution is based on parallel access to data files and the Message Passing Interface (MPI) over the Common Communication Infrastructure (CCI) for the data transfer over a routed infrastructure. In the context of this research, the Energy Sciences Network (ESnet) of the U.S. Department of Energy (DOE) is targeted for the transfer of data between DOE national laboratories.

The RiskMod Model is comprised of a set of risk simulation models, collectively referred to as RiskSim; a set of computer programs that manages data referred to as Data Management Procedures; and RevSim, a model that calculates net revenues. RiskMod interacts with the AURORA Model, the RAM2007, and the ToolKit Model during the process of performing the Risk Analysis Study. AURORA is the computer model being used to perform the Market Price Forecast Study (see Market Price Forecast Study, WP-07-E-BPA-03); the RAM2007 is the computer model being used to calculate rates (see Wholesale Power Rate Development Study, WP-07-E-BPA-05); and the ToolKit is the computer model being used to develop the risk mitigation package that achieves BPA's 92.6 percent TPP standard (see Section 3 in the Risk Analysis Study, WP-07-E-BPA-04). Variations in monthly loads, resources, natural gas prices, forward market electricity prices, transmission expenses, and aluminum smelter benefit payments are simulated in RiskSim. Monthly spot market electricity prices for the simulated loads, resources, and natural gas prices are estimated by the AURORA Model. Data Management Procedures facilitate the format and movement of data that flow to and/or from RiskSim, AURORA, and RevSim. RevSim estimates net revenues using risk data from RiskSim, spot market electricity prices from AURORA, loads and resources data from the Load Resource Study, WP-07-E-BPA-01, various revenues from the Revenue Forecast component of the Wholesale Power Rate Development Study, WP-07-E-BPA-05, and rates and expenses from the RAM2007. Annual average surplus energy revenues, purchased power expenses, and section 4(h)(10)(C) credits calculated by RevSim are used in the Revenue Forecast and the RAM2007. Heavy Load Hour (HLH) and Light Load Hour (LLH) surplus and deficit energy values from RevSim are used in the Transmission Expense Risk Model. Net revenues estimated for each simulation by RevSim are input into the ToolKit Model to develop the risk mitigation package that achieves BPA's 92.6 percent TPP standard. The processes and interaction between each of the models and studies are depicted in Graph 1.

Advanced Artificial Intelligence (AI) systems that process vast feature sets can be designed using Deep Neural Networks (DNN). The popularity and prevalence of DNN based tasks are mainly due to their ability to achieve state of the art accuracy. However, DNN based systems are computationally intensive with heavy on-chip and off-chip data movement demands. Efficient hardware accelerators with systematized interconnection platform support handling this insistent computation and fast communication requirements. With rise in accelerator complexity, it is important to identify and solve communication overheads imposed due to heavy data movement in the system. Need for increased data transfers come as a consequence of complex computations and structure of data storage hierarchy in different DNN models. Multi-core system with underlying Network on Chip (NoC) framework is a promising design choice for implementing DNN based systems with diverse layers. Data movement plays a vital role in deciding performance of such systems. Analysing metrics that affect latency and energy/power efficiency of system is thus crucial in deciding performance enhancement of these DNN inference accelerators. This paper quantifies the communication overheads associated with accelerating different DNN models which use NoC as its basic communication platform.

Logistical Networking1 can be defined as the global optimisation and scheduling of data storage, data movement, and computation. It is a technology for shared network storage that allows an easy scaling in terms of the size of the user community, the aggregate quantity of storage that can be allocated, and the distribution breadth of service nodes across network borders.After describing the base concepts of Logistical Networking, we will introduce the Internet Backplane Protocol, a middleware for managing and using remote storage through allocation of primitive “byte arrays”, showing a semantic in between buffer block and common files. As this characteristic can be too limiting for a large number of applications, we developed the exNode, that can be defined, in two words, as an inode for the for network distributed files. We will introduce then the Logistical Backbone, or L-Bone, is a distributed set of facilities that aim to provide high-performance, location- and application-independent access to storage for network and Grid applications of all kind.KeywordsLogistical NetworkingIBPstorage-enable Internet

In this paper, we propose the China-US international data placement laboratory (iDPL) based on an inter-continental testbed for data placement research. iDPL is able to support various data placement research due to its scalability and flexibility in deploying the experiments in the real network environment. The core design of iDPL leverages reliable workflow management and lightweight I/O protocol to allow complex experiment setup and on-the-fly experiment deployment. It is also extensible to plugin different network profiling tools such as iperf. We expect the powerful measurement capability of iDPL promotes research study on the intelligent data placement policies which adapt to the uncertainty of the wide-area network and guarantee the quality of service (QoS) of the big data applications. As a case study, we setup a set of data placement experiments to measure the end-to-end network performance constantly among several sites between China and US using different data placement tools. The experiments have been running for more than one year, and its measurement data is public available (http://mickey.buaa.edu.cn:8080/). We believe the measurement data is valuable for both network and big data researchers to understand the performance disparity between the raw network and the actual data placement, which provides useful insights to design big data applications with performance awareness. We encourage more researchers to deploy their own data placement experiments on iDPL, expediting the research direction of intelligent data placement with real network environment.

Over the last few years the Grid Computing research community has become interested in developing data intensive applications for the Grid. These applications face significant challenges because their widely distributed nature makes it difficult to access data with reasonable speed. In order to address this problem, we feel that the Grid community needs to develop and explore data movement challenges that represent problems encountered in these applications. In this paper, we will identify two such problems that we have dubbed the Livny Problem and the Plank-Beck Problem. We will also present data movement scheduling techniques that we have developed to address these problems.

While regular algorithms, characterized by operations on dense matrices and arrays, have long been the mainstay of scientific, high-performance computing, irregular algorithms, which feature unpredictable accesses to pointer-based data structures, are becoming increasingly common in high performance computing, arising in graph analysis, data mining and visualization, among other domains. Unfortunately, the defining characteristics of irregular applications, their dynamic, unpredictable, data-dependent access patterns and data layouts, make achieving high performance on large scale systems difficult. Scaling applications to peta- and exa-scale requires carefully controlling communication and data movement and placement, an inherently difficult task when access patterns and data layouts are unpredictable! Most irregular applications that attain high performance must be painstakingly hand-written and hand-tuned, with few common principles or paradigms uniting various implementations and easing future development. Despite the increasing importance of irregular applications, there is little programmer knowledge, and even less compiler ability, devoted to optimizing them. This project aims to solve these problems. By allowing programmers to write irregular applications in high level forms, with at most a few annotations highlighting key structural properties, programmers can focus on developing their algorithms and methods. The compiler and run-time system can take on the tedious task of optimizing the application for execution at large scales, and can automatically provide efficient implementations. This will provide portability and ease maintenance for existing irregular applications, but, more importantly, open up whole new domains of computational science to large-scale, high-performance simulation codes.

Efficient data management in a large-scale epidemiology research project

During the full cycle of the emergency management, Civil Protection operative procedures involve many actors belonging to several institutions playing different roles. In this context the sharing of information is a vital requirement to make correct and effective decisions. Therefore a European-wide technological infrastructure providing a distributed and coordinated access to different kinds of resources (data, information, services, expertise, etc.) could enhance existing Civil Protection applications and even enable new ones. In the recent years Grid technologies have reached a mature state providing a platform for secure and coordinated resource sharing between the participants in the so-called Virtual Organizations. Moreover the Earth and Space Sciences Informatics provide the conceptual tools for modelling the geospatial information shared in Civil Protection applications during its entire life cycle. Therefore a European Civil Protection e-infrastructure could be based on a Grid platform enhanced with Earth Sciences specific services. However Civil Protection applications stress the requirements of Earth Sciences research applications, for example in terms of real-time support. Therefore a set of high-level services specifically tailored for such applications must be built on top of the Grid platform. As a result of a requirement analysis, the FP6 project CYCLOPS has proposed an architectural framework for the future European Civil Protection e-Infrastructure. In this architecture a layer of high-level services tailored to Civil Protection applications is built on top of the EGEE Grid middleware. This architectural approach has been tested implementing a prototype of a grid-enabled RISICO, the application for wild fire risk assessment used by the Italian Civil Protection.