<title>Codesign and high-performance computing: scenes and crisis</title>

Usually, scientific applications outlive the lifespan of the High Performance Computing (HPC) systems for which they are initially developed. The innovations in the HPC systems’ hardware and parallel programming standards drive the modernization of HPC applications so that they continue being performant. While such code modernization efforts may not be challenging for HPC experts and well-funded research groups, many domain-experts and students may find it challenging to adapt their applications for the latest HPC systems due to lack of expertise, time, and funds. The challenges of such domain-experts and students can be mitigated by providing them high-level tools for code modernization and migration. A brief overview of two such high-level tools is presented in this chapter. These tools support the code modernization and migration efforts by assisting users in parallelizing their applications and porting them to HPC systems with high-bandwidth memory. The tools are named as: Interactive Parallelization Tool (IPT) and Interactive Code Adaptation Tool (ICAT). Such high-level tools not only improve the productivity of their users and the performance of the applications but they also improve the utilization of HPC resources.

For decades, Lawrence Livermore National Laboratory (LLNL) has been engaged in significant research, development, and support for software to enable scientific computing and, particularly, the use of high performance computing (HPC) in the NNSA mission space. In particular, the move in the mid-1990’s to simulation as a leading component of stockpile stewardship through the ASCI and the successor ASC programs, as well as the need for reliable data acquisition and control software for the National Ignition Facility, have been important drivers in building expertise in production-quality software development at LLNL. LLNL has also been a leader in the DOE SciDAC FASTMath Institute and the DOE Exascale Computing Project (ECP), both of which have striven to make scientific computing software – in particular, the enabling technologies underpinning simulation capabilities – more widely adopted and sustainable. As such, we believe that our experience can inform the broader goal of software stewardship for scientific and high-performance computing. LLNL strongly supports the formation of a new DOE ASCR program element in software stewardship and sustainment. Historically, DOE ASCR has funded applied mathematics and computer science research that has led to the development of important new capabilities and algorithms that are expressed as artifacts in research software. Such frameworks, libraries, and tools have seldom been directly funded to address the important issues of code maintenance, documentation, robustness, and community building. Software engineering and support have typically been done on the side in support of the ASCR-driven research products. DOE funding priorities have been slow to recognize that good software engineering, the kind that ensures research investments have more adoption and longevity, requires significant resources. Based upon our experiences, we have prepared this response to highlight the concerns and issues we believe to be important as DOE ASCR considers its role in scientific software stewardship. We believe that role is important and will require a significant investment of new funding to legitimately support the technologies past and future DOE ASCR investments have and will produce to facilitate their uptake and adoption in the broader scientific computing community. Following a summary of our involvement in scientific software development, the remainder our response is organized around the nine topics specifically identified in the RFI.

While High Performance Computing (HPC) is critical for scientific breakthroughs and AI development, its growing energy use (often dependent on fossil fuels) requires a detailed environmental impact analysis. This study addresses the ever-increasing ecological challenges posed by high-performance computing (HPC) data centers by developing a model to quantify and predict their energy consumption and carbon emissions. Our model estimates the annual energy consumption of HPC data centers globally using data on the size and capacity of small and massive data centers. It distinguishes between theoretical maximum and actual energy consumption based on average utilization. The results show an upward trend in carbon emissions associated with HPC, which are projected to reach 5.81 x 1017 grams of carbon dioxide by 2030. As a result, we propose a series of recommendations to reduce the environmental impact of HPC at both the technical and policy levels in favor of promoting a healthy and sustainable environment.

Modern High Performance Computing (HPC) applications process very large amounts of data. A critical research challenge lies in transporting input data to the HPC center from a number of distributed sources, e.g., scientific experiments and web repositories, etc., and offloading the result data to geographically distributed, intermittently available end-users, often over under-provisioned connections. Such end-user data services are typically performed using point-to-point transfers that are designed for well-endowed sites and are unable to reconcile the center's resource usage and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. To overcome these inefficiencies, decentralized HPC data services are emerging as viable alternatives. In this paper, we develop and enhance such distributed data services by designing CATCH, a Cloud-based Adaptive data Transfer service for HPC. CATCH leverages a bevy of cloud storage resources to orchestrate a decentralized data transport with fail-over capabilities. Our results demonstrate that CATCH is a feasible approach, and can help improve the data transfer times at the HPC center by as much as 81.1\% for typical HPC workloads.

High performance computing is essential for simulating complex problems using agent-based modelling (ABM). Researchers are hindered by complexities of porting models on parallel platforms and time taken to run large simulations on a single machine. This paper presents FLAME framework, the only supercomputing framework which automatically produces parallelisable code to execute on different parallel hardware architectures. FLAME's inherent parallelism allows large number of agents to be simulated in less time than comparable simulation frameworks. The framework also handles the parallelisation of model code allowing modellers to run simulations on number of supported architectures. The framework has been well tested in various disciplines like biology and economics projects yielding successful research results like project EURACE, where the European economy was modelled using agents. More recently FLAME has been ported to consumer NVIDIA Graphics Processing Units (GPUs) allowing parallel performance equal to that of grid architectures with the ability to perform real time visualisation.

High Performance Computing (HPC) has been studied and used in the scientific community for decades. The Message Passing Interface was first introduced in 1992. Similarly, commercial businesses have been relying on High Throughput Computing (HTC) for the past two decades. Mapreduce platforms became popular with the advent of Very Large Databases (VLDBs) and Big Data. We are now seeing the convergence between HPC and HTC to provide faster and cheaper parallel computation. The emergence of MPI as a scalable and viable parallel platform along with the acceptance of Mapreduce to tackle large data sets now opens the door to a host of new applications particularly in biomedical, public health, scientific, and health informatics research. This convergence is making it possible to have every device be a parallel node. In this paper we explore this convergence and a method for creating dynamic self-assembling clusters using commodity hardware and mobile devices.

The solution of large eigenproblems is involved in many scientific and engineering applications when for instance, stability analysis is a concern. For large simulation in material physics or thermo-acoustics, the calculation can last for many hours on large parallel platforms. On future large-scale systems, the mean time between failures (MTBF) of the system is expected to decrease so that many faults could occur during the solution of large eigenproblems. Consequently, it becomes critical to design parallel eigensolvers that can survive faults. In that framework, we investigate the relevance of approaches relying on numerical techniques, which might be combined with more classical techniques for real large-scale parallel implementations. Because we focus on numerical remedies we do not consider parallel implementations nor parallel experiments but only numerical experiments. We assume that a separate mechanism ensures the fault detection and that a system layer provides support for setting back the environment (processes,. . .) in a running state. Once the system is in a running state, after a fault, our main objective is to provide robust resilient schemes so that the eigensolver may keep converging in the presence of the fault without restarting the calculation from scratch. For this purpose, we extend the interpolation-restart (IR) strategies initially introduced for the solution of linear systems in a previous work to the solution of eigenproblems in this paper. For a given numerical scheme, the IR strategies consist of extracting relevant spectral information from available data after a fault. After data extraction, a well-selected part of the missing data is regenerated through interpolation strategies to constitute a meaningful input to restart the numerical algorithm. One of the main features of this numerical remedy is that it does not require extra resources, i.e., computational unit or computing time, when no fault occurs. In this paper, we revisit a few state-of-the-art methods for solving large sparse eigenvalue problems namely the Arnoldi methods, subspace iteration methods and the Jacobi-Davidson method, in the light of our IR strategies. For each considered eigensolver, we adapt the IR strategies to regenerate as much spectral information as possible. Through extensive numerical experiments, we study the respective robustness of the resulting resilient schemes with respect to the MTBF and to the amount of data loss via qualitative and quantitative illustrations. 1. Introduction. The computation of eigenpairs (eigenvalues and eigenvectors) of large sparse matrices is involved in many scientific and engineering applications such as when stability analysis is a concern. To name a few, it appears in structural dynamics, thermodynamics, thermo-acoustics, quantum chemistry. With the permanent increase of the computational power of high performance computing (HPC) systems by using a larger and larger number of CPU cores or specialized processing units, HPC applications are increasingly prone to faults. To guarantee fault tolerance, two classes of strategies are required. One for the fault detection and the other for fault correction. Faults such as computational node crashes are obvious to detect while silent faults may be challenging to detect. To cope with silent faults, a duplication strategy is commonly used for fault detection [18, 39] by comparing the outputs, while triple modular redundancy (TMR) is used for fault detection and correction [34, 37]. However, the additional computational resources required by such replication strategies may represent a severe penalty. Instead of replicating computational resources, studies [7, 36] propose a time redundancy model for fault detection. It consists in repeating computation twice on the same resource. The advantage of time redundancy models is the flexibility at application level; software developers can indeed select only a set of critical instructions to protect. Recomputing only some instructions instead of the whole application lowers the time redundancy overhead [25]. In some numerical simulations, data naturally satisfy well defined mathematical properties. These properties can be efficiently exploited for fault detection through a periodical check of the numerical properties during computation [10]. Checkpoint/restart is the most studied fault recovery strategy in the context of HPC systems. The common checkpoint/restart scheme consists in periodically saving data onto a reliable storage device such as a remote disk. When a fault occurs, a rollback is performed to the point of the most recent and consistent checkpoint. According to the implemented checkpoint strategy, all processes

: Project goals were to develop techniques of continual reallocation of resources to maintain application performance despite statically unpredictable change in resource demands. Research was targeted to multiple application systems executing on HPC (High Performance Computing) platforms. This project built on the results of a previous program, called Adaptive Resource Allocation (ARA). In ARA, Honeywell developed techniques for dynamic reallocation of resources to single parallel applications, structured as multi-pipelines, executing on a high performance parallel machine. They extended ARA results to systems with multiple applications and multiple machines connected over a network. In October 1997 DARPA merged the technical effort on this project with the RTARM project funded under Quorum. This did not affect the core statement of work for ARM, but led to extension of its completion date. ARM focused on developing an approach based on adaption models, and addressed best-effort resource allocation in an environment with partitionable rather than shared resources. Parallel HPG platforms were de-emphasized in favor of general distributed computing platforms. Results from ARM are being integrated into RTARM. The layered architecture of ARM has given way to a hierarchical architecture characterized by uniformity across different levels. The MPI-based communication infrastructure in ARM has given way to a CORBA ORB infrastructure. While ARM implementation was targeted to Unix machine connected over Ethernet, the target platform for PTARM consists of Windows NT machines networked over ATM.

Voluminous data (Multispectral, Hyperspectral) from Variety of sensors (Airborne sensors, space borne sensors) with Velocity (high temporal resolution) when used for decision making to support natural disasters such as earthquakes, floods, oil-spills etc., for near real time accurate responses, is a problem that needs Big Data Analytics. To gain rapid insight from this big data, high performance computing (HPC) with some scalable solution that reduces the execution time are in extreme demand. To serve this real time need, scalable hybrid parallelism approach based on state of art multi-core GPUs and Message Passing Interface (MPI) is explored for analyzing remote sensing disaster data. Spatio-temporal remote sensing data of oil-spill at Gulf of Mexico captured by LANDSAT 7 ETM+ is considered for analysis. The core objective includes performance evaluation of the analysis process across various parallel implementation platforms.

As the scales of parallel applications and platforms increase the negative impact of communication latencies on performance becomes large. Fortunately, modern High Performance Computing (HPC) systems can exploit low-latency topologies of high-radix switches. In this context, we propose the use of random shortcut topologies, which are generated by augmenting classical topologies with random links. Using graph analysis we find that these topologies, when compared to non-random topologies of the same degree, lead to drastically reduced diameter and average shortest path length. The best results are obtained when adding random links to a ring topology, meaning that good random shortcut topologies can easily be generated for arbitrary numbers of switches. Using flit-level discrete event simulation we find that random shortcut topologies achieve throughput comparable to and latency lower than that of existing non-random topologies such as hypercubes and tori. Finally, we discuss and quantify practical challenges for random shortcut topologies, including routing scalability and larger physical cable lengths.

Tremendous computational resources are required to compute complex physical simulations. Unfortunately computers able to provide such computational power are difficult to program, especially since the rise of heterogeneous hardware architectures. This makes it particularly challenging to exploit efficiently and sustainably supercomputers resources. We think that model-driven engineering can help us tame the complexity of high-performance scientific computing software development by separating the different concerns such as mathematics, parallelism, or validation. The principles of our approach, named MDE4HPC, stem from this idea. In this paper, we describe the High-Performance Computing Modeling Language (HPCML), a domain-specific modeling language at the center of this approach.

Tensor algebra operations such as contractions in computational chemistry consume a significant fraction of the computing time on large-scale computing platforms. The widespread use of tensor contractions between large multi-dimensional tensors in describing electronic structure theory has motivated the development of multiple tensor algebra frameworks targeting heterogeneous computing platforms. In this paper, we present Tensor Algebra for Many-body Methods (TAMM), a framework for productive and performance-portable development of scalable computational chemistry methods. TAMM decouples the specification of the computation from the execution of these operations on available high-performance computing systems. With this design choice, the scientific application developers (domain scientists) can focus on the algorithmic requirements using the tensor algebra interface provided by TAMM, whereas high-performance computing developers can direct their attention to various optimizations on the underlying constructs, such as efficient data distribution, optimized scheduling algorithms, and efficient use of intra-node resources (e.g., graphics processing units). The modular structure of TAMM allows it to support different hardware architectures and incorporate new algorithmic advances. We describe the TAMM framework and our approach to the sustainable development of scalable ground- and excited-state electronic structure methods. We present case studies highlighting the ease of use, including the performance and productivity gains compared to other frameworks.

This paper introduces a fundamentally new machine paradigm, which takes into account that hardware has become soft. Along with a compilation technique, this paper introduces its implementation and application by illustrating the rDPA (reconfigurable data path array), an integrated circuit having been designed at Kaiserslautern and submitted for fabrication. This is the first implementation of a field-programmable data path array (FPDPA), which uses configurable ALU blocks (CABs) instead of CLBs (configurable logic blocks). This novel platform, provides highly efficient means of instruction level parallelism: for many applications by orders of magnitude more efficient than the traditional parallelism used in all scenes of high performance computing.

The symposium ParaFPGA focuses on parallel techniques using FPGAs as accelerator in high performance computing. The green computing aspects of low power consumption at high performance were somewhat tempered by long design cycles and hard programmability issues. However, in recent years FPGAs have become new contenders as versatile compute accelerators because of a growing market interest, extended application domains and maturing high-level synthesis tools. The keynote paper highlights the historical and modern approaches to high-level FPGA programming and the contributions cover applications such as NP-complete satisfiability problems and convex hull image processing as well as performance evaluation, partial reconfiguration and systematic design exploration.

<p>The complexity of the climate system calls for a combined approach of different knowledge areas. For that, increasingly larger projects need a coordinate effort that fosters an active collaboration between members. On the other hand, although the continuous improvement of numerical models and larger observational data availability provides researchers with a growing amount of data to analyze, the need for greater resources to host, access, and evaluate them efficiently through High Performance Computing (HPC) infrastructures is growing more than ever. Finally, the thriving emphasis on FAIR data principles [1] and the easy reproducibility of evaluation workflows also requires a framework that facilitates these tasks. Freva (Free Evaluation System Framework [2, 3]) is an efficient solution to handle customizable evaluation systems of large research projects, institutes or universities in the Earth system community [4-6] over the HPC environment and in a centralized manner.</p><p> </p><p>Freva is a scientific software infrastructure for standardized data and analysis tools (plugins) that provides all its available features both in a shell and web environment. Written in python, is equipped with a standardized model database, an application-programming interface (API) and a history of evaluations, among others:</p><ul><li>An implemented metadata system in SOLR with its own search tool allows scientists and their plugins to retrieve the required information from a centralized database. The databrowser interface satisfies the international standards provided by the Earth System Grid Federation (ESGF, e.g. [7]).</li> <li>An API allows scientific developers to connect their plugins with the evaluation system independently of the programming language. The connected plugins are able to access from and integrate their results back to the database, allowing for a concatenation of plugins as well. This ecosystem increases the number of scientists involved in the studies, boosting the interchange of results and ideas. It also fosters an active collaboration between plugin developers.</li> <li>The history and configuration sub-system stores every analysis performed with Freva in a MySQL database. Analysis configurations and results can be searched and shared among the scientists, offering transparency and reproducibility, and saving CPU hours, I/O, disk space and time.</li> </ul><p>Freva efficiently frames the interaction between different technologies thus improving the Earth system modeling science.</p><p> </p><p>This framework has undergone major refactoring and restructuring of the core that will also be discussed. Among others:</p><ul><li>Major core Python update (2.7 to 3.9).</li> <li>Easier deployment and containerization of the framework via Docker.</li> <li>More secure system configuration via Vault integration.</li> <li>Direct Freva function calls via python client (e.g. for jupyter notebooks).</li> <li>Improvements in the dataset incorporation.</li> </ul><p><strong> </strong></p><p><strong>