Software Design and Implementation for MapReduce across Distributed Data Centers

In this paper, we describe a use case applying a scientific workflow system and a distributed file system to improve the performance of telescope data processing. The application is pipeline processing of data generated by Hyper Suprime-Cam (HSC) which is a focal plane camera mounted on the Subaru telescope. In this paper, we focus on the scalability of parallel I/O and core utilization. The IBM Spectrum Scale (GPFS) used for actual operation has a limit on scalability due to the configuration using storage servers. Therefore, we introduce the Gfarm file system which uses the storage of the worker node for parallel I/O performance. To improve core utilization, we introduce the Pwrake workflow system instead of the parallel processing framework developed for the HSC pipeline. Descriptions of task dependencies are necessary to further improve core utilization by overlapping different types of tasks. We discuss the usefulness of the workflow description language with the function of scripting language for defining complex task dependency. In the experiment, the performance of the pipeline is evaluated using a quarter of the observation data per night (input files: 80 GB, output files: 1.2 TB). Measurements on strong scaling from 48 to 576 cores show that the processing with Gfarm file system is more scalable than that with GPFS. Measurement using 576 cores shows that our method improves the processing speed of the pipeline by 2.2 times compared with the method used in actual operation.

Large data sets, so-called big data, are no longer only a problem for web companies. Advances in data acquisition and telecommunications have created huge amounts of data. Storing, managing and processing this data are critical issues today. Gaia, an ESA mission for which CNES is a data processing center, will face the challenge of dealing with an end mission volume of one Petabyte stored in tables containing eighty billion rows. Unlike other big databases already in operation, the project involves complex data accesses and intensive computing. In this paper we will briefly present the studied products. Then we will describe the data management system based on Hadoop we have designed, which is a batch execution framework (MapReduce paradigm) with an underlying file system (HDFS) optimized for task execution. We will conclude by explaining how we have solved the many key questions raised by this conceptual migration: data access, workflow orchestration, designing a powerful Hadoop cluster taking into account its integration and the budget.

With the development of the new generation of High Energy Physics (HEP) experiments, huge amounts of data are being generated. Efficient parallel algorithms/frameworks and High IO throughput are key to meet the scalability and performance requirements of HEP offline data analysis. Though Hadoop has gained a lot of attention from scientific community for its scalability and parallel computing framework for large data sets, it’s still difficult to make HEP data processing tasks run directly on Hadoop. In this paper we investigate the application of Hadoop to make HEP jobs run on it transparently. Particularly, we discuss a new mechanism to support HEP software to random access data in HDFS. Because HDFS is streaming data stored only supporting sequential write and append. It cannot satisfy HEP jobs to random access data. This new feature allows the Map/Reduce tasks to random read/write on the local file system on data nodes instead of using Hadoop data streaming interface. This makes HEP jobs run on Hadoop possible. We also develop diverse MapReduce model for HEP jobs such as Corsika simulation, ARGO detector simulation and Medea++ reconstruction. And we develop a toolkit for users to submit/query/remove jobs. In addition, we provide cluster monitoring and account system to benefit to the system availability. This work has been in production for HEP experiment to gain about 40,000 CPU hours per month since September, 2016.

This paper proposes a design and implementation of an MPI-IO implementation of the Gfarm file system, called MPI-IO/Gfarm. The Gfarm file system is a global file system that federates the local storage of compute nodes among several clusters. It has a scale-out architecture designed to support distributed data-intensive computing. However Gfarm file system does not achieve scalable performance in the case of parallel writes to a single file, a typical file operation in MPI-IO. This paper proposes an optimization technique to improve the parallel write performance to a single file. In the evaluation, MPI-IO/Gfarm achieves scalable parallel I/O performance.

The ROMIO implementation of the MPI-IO standard provides a portable infrastructure for use on top of a variety of underlying storage targets. These targets vary widely in their capabilities, and in some cases additional effort is needed within ROMIO to support all MPI-IO semantics. Two aspects of the interface that can be problematic to implement are MPI-IO atomic mode and the shared file pointer access routines. Atomic mode requires enforcing strict consistency semantics, and shared file pointer routines require communication and coordination in order to atomically update a shared resource. For some file systems, native locks may be used to implement these features, but not all file systems have lock support. In this work, we describe algorithms for implementing efficient mutex locks using MPI-1 and the one-sided capabilities from MPI-2. We then show how these algorithms may be used to implement both MPI-IO atomic mode and shared file pointer methods for ROMIO without requiring any features from the underlying file system. We show that these algorithms can outperform traditional file system lock approaches. Because of the portable nature of these algorithms, they are likely useful in a variety of situations where distributed locking or coordination is needed in the MPI-2 environment.

In recent years, there was a growing interest in improving the utilization of supercomputers by running applications of experiments at the Large Hadron Collider (LHC) at CERN when idle cores cannot be assigned to traditional HPC jobs. At the same time, the upcoming LHC machine and detector upgrades will produce some 60 times higher data rates and challenge LHC experiments to use so far untapped compute resources. LHC experiment applications are tailored to run on high-throughput computing resources and they have a different anatomy than HPC applications. LHC applications comprise a core framework that allows hundreds of researchers to plug in their specific algorithms. The software stacks easily accumulate to many gigabytes for a single release. New releases are often produced on a daily basis. To facilitate the distribution of these software stacks to world-wide distributed computing resources, LHC experiments use a purpose-built, global, POSIX file system, the CernVM File System. CernVM-FS pre-processes data into content-addressed, digitally signed Merkle trees and it uses web caches and proxies for data distribution. Fuse-mounted files system clients on the compute nodes load and cache on demand only the small fraction of files needed at any given moment. In this paper, we report on problems and lessons learned in the deployment of CernVM-FS on supercomputers such as the supercomputers at NERSC in Berkeley, at LRZ in Munich, and at CSCS in Lugano. We compare CernVM-FS to a shared software area on a traditional HPC storage system and to container-based systems.

MPI-IO/GPFS is a prototype implementation of the I/O chapter of the Message Passing Interface (MPI) 2 standard. It uses the IBM General Parallel File System (GPFS) as the underlying file system. This paper describes the features of this prototype which support its high performance. The use of hints allows tailoring the use of the file system to the application needs.

With the advent of the High-Luminosity Large Hadron Collider (HL-LHC) era, high energy physics (HEP) event selection will require new approaches to rapidly and accurately analyze vast databases. The current study addresses the enormity of HEP databases in an unprecedented manner—a quantum search using Grover’s Algorithm (GA) on an unsorted database, ATLAS Open Data, from the ATLAS detector. A novel method to identify rare events at 13 TeV in CERN’s LHC using quantum computing (QC) is presented. As indicated by the Higgs boson decay channel Hrightarrow ZZ^*rightarrow 4l, the detection of four leptons in one event may be used to reconstruct the Higgs boson and, more importantly, evince Higgs boson decay to some new phenomena, such as Hrightarrow ZZ_d rightarrow 4l. Searching the dataset for collisions resulting in detection of four leptons using a Jupyter Notebook, a classical simulation of GA, and several quantum computers with multiple qubits, the current application was found to make the proper selection in the unsorted dataset. Quantum search efficacy was analyzed for the incoming HL-LHC by implementing the QC method on multiple classical simulators and IBM’s quantum computers with the IBM Qiskit Open Source Software. The current QC application provides a novel, high-efficiency alternative to classical database searches, demonstrating its potential utility as a rapid and increasingly accurate search method in HEP.

Programming development tools are a vital component for understanding the behavior of parallel applications. Event tracing is a principal ingredient to these tools, but new and serious challenges place event tracing at risk on extreme-scale machines. As the quantity of captured events increases with concurrency, the additional data can overload the parallel file system and perturb the application being observed. In this work we present a solution for event tracing on extreme-scale machines. We enhance an I/O forwarding software layer to aggregate and reorganize log data prior to writing to the storage system, significantly reducing the burden on the underlying file system. Furthermore, we introduce a sophisticated write buffering capability to limit the impact. To validate the approach, we employ the Vampir tracing toolset using these new capabilities. Our results demonstrate that the approach increases the maximum traced application size by a factor of 5× to more than 200,000 processes.

Wireless sensor networks (WSN) have unique challenges and constraints. Sensor nodes e.g. have tough memory limitations. However, the latest advances in WSN research direct for an implementation of lightweight versions of Internet protocols like IPv6, TCP, and HTTP on sensor nodes. These protocols have challenging requirements. Especially, memory consumption of these protocols is often higher than the physical RAM that microcontrollers have integrated. Therefore, we suggest an approach for virtual memory providing more memory than the available RAM. As microcontrollers do not include a memory management unit the usage of memory is implemented in cooperative fashion based on the C standard library function malloc and free. We suggest an underlying file system and a hardware abstraction layer to support various external or internal memory devices like Flash or EEPROM. In this work in progress we present an API, some implementation details and preliminary results including future work.

Monte Carlo event generators are nowadays indispensable tools for predictions based on first principles in high energy physics, and they represent one of the mainstays of particle physics research at the Large Hadron Collider. In the dawn of the high luminosity upgrade of the Large Hadron Collider, there is a push to more complex signatures and higher accuracy, rendering the generation of simulated events more expensive. At the same time, there are strict limitations on the computational budget. In this situation, the efficiency of event generators can be identified as a key issue. The recent rapid advancement of machine learning tools, first and foremost deep neural networks, and their successes in diverse applications make them a promising choice in addressing this challenge. In this thesis, I consider two central building blocks of event generation that represent bottlenecks in typical applications. The first is the sampling of phase space configurations such that their distribution closely approximates a given target. For this I present two new approaches, one based on normalizing flows and the other on nested sampling. The second is the unweighting of event samples, that is the generation of unit weight events that contribute equally to the total scattering cross-section. To accelerate the unweighting process, I present an unbiased unweighting method based on fast and accurate neural network surrogates for the event weights. Furthermore, I show how a surrogate optimized for the factorization properties of the corresponding matrix elements can significantly improve the performance for suitable processes. All methods are evaluated by means of examples, which are oriented towards realistic applications. It is also discussed how the different approaches could be combined and what opportunities there are for further developments.

In January 2018, the China Reform and Development Committee (led by President Jinping Xi) had the meeting, and passed the plan of “Chinese Initiated International Large Scientific Plan and Large Scientific Project”. In March 2018, it was released by Chinese government. The 14th five years plan will start in 2021 and it is very important to have a historical review of China’s large science projects in order to help in future large science projects’ planning and implementation, such as Circular Electron Positron Collider (CEPC), as Higgs factory proposed in September 2012 by Chinese scientists after the Higgs boson was discovered on July 4, 2012 at CERN on Large Hadron Collider (LHC). In November 2018, the Conceptual Design Report (CDR) of CEPC was released formally and Technical Design Report (TDR) phase started in 2019. It is planned that CEPC TDR will be completed at the end of 2022 and CEPC will be proved by Chinese government during 14th five year plan and completed around 2030. As China’s first large science project, Beijing Electron Positron Collider (BEPC), which was completed in October 1988, its success and experiences are very precious and useful for the realization of CEPC. In 1993, the year of 20th anniversary of Institute of High Energy Physics (IHEP), Chinese Academy of Sciences (CAS), Professor Jialin Xie, Academician of CAS and the first director of BEPC, has written an article recalling the planning, design, R&D and construction of BEPC. In this article, Jialin Xie recalled firstly the Beijing Proton Synchrotron (BPS), so called “87 Project”, in late 1970’s, which was a <sc>50 GeV</sc> synchrotron, and Jialin Xie was the project leader of BPS. During BPS planning and R&D, many trips were to Fermi National Accelerator Laboratory (FNAL), Brookhaven National Laboratory (BNL) and CERN. At the end of 1970’s, due to Chinese economy situation, BPS was suspended. During a collaboration meeting initiated by Professor T. D. Lee of Columbia University and Nobel Prize winner in 1957, between China and USA held at FNAL in March 1981, Professor Wolfgan K. H. Panofsky from SLAC proposed to Chinese delegation a <sc>2.2 GeV</sc> electron positron collider instead of proton synchrotron as BPS. After careful discussions among scientists and the following evaluations from CAS, it was finally decided that China will build Beijing Electron Positron Collider, so called BEPC. BEPC was started in 1984 and completed in October 1988. In the article of Jialin Xie, he recalled many detailed difficulties in design, R&D and construction phases, and many supports from international collaboration with US scientists and government under China-US scientific collaboration agreement. This article revealed many decision making process, design philosophy, technology choice and R&D issues, team training, project management in time and cost controls, etc. He also recalled the doubts and objection voices towards BEPC. BEPC has not only laid a solid ground but also accumulated many precious experiences for the high energy physics and many other large science projects which follow in China. By recalling, keeping and carrying forwards these precious experiences and pioneering spirits, it is believed that CEPC as a successor of BEPC, will be realized successfully in the future. In addition to high energy physics project CEPC, other two long term key directions are exploration to Manned Mars Project and Chinese Fussion Experimental Test Reactor (CFETRE), and three of them should be the strategic parallel Chinese large science and technology projects in the future.

MapReduce is increasingly becoming a popular framework, and a potent programming model. The most popular open source implementation of MapReduce, Hadoop, is based on the Hadoop Distributed File System (HDFS). However, as HDFS is not POSIX compliant, it cannot be fully leveraged by applications running on a majority of existing HPC environments such as Teragrid and NERSC. These HPC environments typically support globally shared file systems such as NFS and GPFS. On such resourceful HPC infrastructures, the use of Hadoop not only creates compatibility issues, but also affects overall performance due to the added overhead of the HDFS. This paper not only presents a MapReduce implementation directly suitable for HPC environments, but also exposes the design choices for better performance gains in those settings. By leveraging inherent distributed file systems' functions, and abstracting them away from its MapReduce framework, MARIANE (MApReduce Implementation Adapted for HPC Environments) not only allows for the use of the model in an expanding number of HPC environments, but also allows for better performance in such settings. This paper shows the applicability and high performance of the MapReduce paradigm through MARIANE, an implementation designed for clustered and shared-disk file systems and as such not dedicated to a specific MapReduce solution. The paper identifies the components and trade-offs necessary for this model, and quantifies the performance gains exhibited by our approach in distributed environments over Apache Hadoop in a data intensive setting, on the Magellan test bed at the National Energy Research Scientific Computing Center (NERSC).

As the rapid development of hardware and network technology, cloud computing has become an important research topic. For applications of large-scale data processing, such as data warehouse, Map Reduce is the most famous platform for parallel data processing in cloud computing. To support the star-join queries in data warehouse, Scatter-Gather-Merge (SGM) proposes an efficient algorithm on the Map Reduce framework. However, SGM supports only the equi-join queries. Nonequi-join queries may cause SGM to fail. In this paper, we propose a method to cope with theta-join queries, i.e., both equi-join and nonequi-join queries. Our proposed method uses a novel manipulation of keys for partitioning data. The key manipulation matches up the Map Reduce paradigm, and makes theta-join queries workable on the Map Reduce platform. Our experimental results show that the proposed method achieves similar performance to SGM, but our method supports more join-query types. Our method performs even better than SGM in some query types of high data selectivity.

The CERN Large Hadron Collider (LHC) is designed to collide proton beams of unprecedented energy, in order to extend the frontiers of high-energy particle physics. During the first very successful running period in 2010--2013, the LHC was routinely storing protons at 3.5--4 TeV with a total beam energy of up to 146 MJ, and even higher stored energies are foreseen in the future. This puts extraordinary demands on the control of beam losses. An un-controlled loss of even a tiny fraction of the beam could cause a superconducting magnet to undergo a transition into a normal-conducting state, or in the worst case cause material damage. Hence a multi-stage collimation system has been installed in order to safely intercept high-amplitude beam protons before they are lost elsewhere. To guarantee adequate protection from the collimators, a detailed theoretical understanding is needed. This article presents results of numerical simulations of the distribution of beam losses around the LHC that have leaked out of the collimation system. The studies include tracking of protons through the fields of more than 5000 magnets in the 27 km LHC ring over hundreds of revolutions, and Monte-Carlo simulations of particle-matter interactions both in collimators and machine elements being hit by escaping particles. The simulation results agree typically within a factor 2 with measurements of beam loss distributions from the previous LHC run. Considering the complex simulation, which must account for a very large number of unknown imperfections, and in view of the total losses around the ring spanning over 7 orders of magnitude, we consider this an excellent agreement. Our results give confidence in the simulation tools, which are used also for the design of future accelerators.