Experimental Verification and Analysis of Dynamic Loop Scheduling in Scientific Applications

In heterogeneous environments, dynamic scheduling algorithms are a powerful tool towards performance improvement of scientific applications via load balancing. However, these scheduling techniques employ heuristics that require prior knowledge about workload via profiling resulting in higher overhead as problem sizes and number of processors increase. In addition, load imbalance may appear only at run-time, making profiling work tedious and sometimes even obsolete. Recently, the integration of dynamic loop scheduling algorithms into a number of scientific applications has been proven effective. This paper reports on performance improvements obtained by integrating the Adaptive Weighted Factoring, a recently proposed dynamic loop scheduling technique that addresses these concerns, into two scientific applications: computational field simulation on unstructured grids, and N-Body simulations. Reported experimental results confirm the benefits of using this methodology, and emphasize its high potential for future integration into other scientific applications that exhibit substantial performance degradation due to load imbalance.

©2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Faced with a backlog of nuclear waste and weapons plutonium, as well as an ever-increasing public concern about safety and environmental issues associated with conventional nuclear reactors, many countries are studying new, accelerator-driven technologies that hold the promise of providing safe and effective solutions to these problems. Proposed projects include accelerator transmutation of waste (ATW), accelerator-based conversion of plutonium (ABC), accelerator-driven energy production (ADEP), and accelerator production of tritium (APT). Also, next-generation spallation neutron sources based on similar technology will play a major role in materials science and biological science research. The design of accelerators for these projects will require a major advance in numerical modeling capability. For example, beam dynamics simulations with approximately 100 million particles will be needed to ensure that extremely stringent beam loss requirements (less than a nanoampere per meter) can be met. Compared with typical present-day modeling using 10,000–100,000 particles, this represents an increase of 3–4 orders of magnitude. High performance computing (HPC) platforms make it possible to perform such large scale simulations, which require 10’s of GBytes of memory. They also make it possible to perform smaller simulations in a matter of hours that would require months to run on a single processor workstation. This paper will describe how HPC platforms can be used to perform the numerically intensive beam dynamics simulations required for development of these new accelerator-driven technologies.

Dynamic loop scheduling (DLS) algorithms provide application-level load balancing of loop iterates, with the goal of maximizing application performance on the underlying system. These methods use run-time information regarding the performance of the application's execution (for which irregularities change over time). Many DLS methods are based on probabilistic analyses, and therefore account for unpredictable variations of application and system related parameters. Scheduling scientific and engineering applications in large-scale distributed systems (possibly shared with other users) makes the problem of DLS even more challenging. Moreover, the chances of failure, such as processor or link failure, are high in such large-scale systems. In this paper, we employ the hierarchical approach for three DLS methods, and propose metrics for quantifying their robustness with respect to variations of two parameters (load and processor failures), for scheduling irregular applications in large-scale heterogeneous distributed systems.

The state-of-the-art for auditing and reproducing scientific applications on high-performance computing (HPC) systems is through a data provenance subsystem. While recent advances in data provenance lie in reducing the performance overhead and improving the user's query flexibility, the fidelity of data provenance is often overlooked: there is no such way to ensure that the provenance data itself has not been fabricated or falsified. This paper advocates leveraging blockchains to deliver immutable and autonomous data provenance services such that scientific discoveries are trustworthy. The challenges for adopting blockchains to HPC include designing a new blockchain architecture compatible with the HPC platforms and, more importantly, a set of new consensus protocols for scientific applications atop blockchains. To this end, we have designed the proof-of-scalable-traceability (POST) protocol and implemented it in a blockchain prototype, namely SciChain, the very first practical blockchain system for provenance services on HPC. We evaluated SciChain by comparing it with multiple state-of-the-art systems; experimental results showed that SciChain guaranteed trustworthy data provenance while incurring orders of magnitude lower overhead than existing solutions.

Fifth generation (5G) systems are considered as the future of telecommunications and data processing. 5G systems are envisaged to increase current network capacity by 1000-fold. In addition to increasing spectral efficiency and utilizing wider bandwidths, ultra network densification with wireless small cells is considered a significant capacity-enhancing method for meeting the ever-increasing demand of 5G data traffic requirements. The enormous amounts of data traffic generated in inter-small-cell 5G backhaul networks require efficient routing protocols to speed up the routing decisions, while ensuring high data rates, low latency, and low power consumption requirements. Therefore, a parallel routing protocol is proposed to speed up routing decisions in 5G backhaul networks, using high performance computing (HPC). We explore the efficiency of utilizing HPC to manage and speed up the parallel routing protocol using different communication network sizes. Our numerical results indicate that our HPC implementation achieves a routing speed-up of 37× for large network size (2048).

An important research area in statistical computing is found in the literature for vector functional coefficient autoregressive models, a special case of vector nonlinear time series. Methods used are computationally intensive. As a result, analyses and simulations can run into weeks, or even months. Statisticians have been known to base empirical results on a relatively small number of simulation replications, sacrificing precision, accuracy and reliability of results in the interest of time and productivity. The simulations are amenable for parallelization; however, parallel computing technology has not yet been widely used in this specific research area. This paper proposes an approach to the parallelization of statistical simulation codes to address the challenge of long running times, without resorting to extensive code revisions. This approach takes advantage of recent advances in dynamic loop scheduling on workstation clusters to achieve high performance, even with the presence of unpredictable load imbalance factors. Preliminary results of applying this approach in the simulation of normal white noise and threshold autoregressive model obtains efficiencies in the range 95–98% on 8–64 processors.

This research addresses the pressing need to optimize Python-based social science applications for high-performance computing (HPC)systems, emphasizing the combined use of task and data parallelism techniques. The paper delves into a substantial body of research,recognizing Python’s interpreted nature as a challenge for efficient social science data processing. The paper introduces a Pythonprogram that exemplifies the proposed methodology. This program uses task parallelism with multi-processing and data parallelismwith dask to optimize data processing workflows. It showcases how researchers can effectively manage large datasets and intricatecomputations on HPC systems. The research offers a comprehensive framework for optimizing Python-based social science applicationson HPC systems. It addresses the challenges of Python’s performance limitations, data-intensive processing, and memory efficiency.Incorporating insights from a rich literature survey, it equips researchers with valuable tools and strategies for enhancing the efficiencyof their social science applications in HPC environments.

This report provides documentation for the completion of the Sandia Level II milestone 'Develop feedback system for intelligent dynamic resource allocation to improve application performance'. This milestone demonstrates the use of a scalable data collection analysis and feedback system that enables insight into how an application is utilizing the hardware resources of a high performance computing (HPC) platform in a lightweight fashion. Further we demonstrate utilizing the same mechanisms used for transporting data for remote analysis and visualization to provide low latency run-time feedback to applications. The ultimate goal of this body of work is performance optimization in the face of the ever increasing size and complexity of HPC systems.

Sparse matrix-sparse vector (SpMSpV) multiplication is one of the fundamental and important operations in many high-performance scientific and engineering applications. The inherent irregularity and poor data locality lead to two main challenges to scaling SpMSpV over high-performance computing (HPC) systems: (i) a large amount of redundant data limits the utilization of bandwidth and parallel resources; (ii) the irregular access pattern limits the exploitation of computing resources. This paper proposes a fine-grained parallel SpMSpV (fgSpMSpV) framework on Sunway TaihuLight supercomputer to alleviate the challenges for large-scale real-world applications. First,fgSpMSpVadopts an MPI\( + \)OpenMP\( +X \)parallelization model to exploit the multi-stage and hybrid parallelism of heterogeneous HPC architectures and accelerate both pre-/post-processing and main SpMSpV computation. Second,fgSpMSpVutilizes an adaptive parallel execution to reduce the pre-processing, adapt to the parallelism and memory hierarchy of the Sunway system, while still tame redundant and random memory accesses in SpMSpV, including a set of techniques like the fine-grained partitioner, re-collection method, and Compressed Sparse Column Vector (CSCV) matrix format. Third,fgSpMSpVuses several optimization techniques to further utilize the computing resources.fgSpMSpVon the Sunway TaihuLight gains a noticeable performance improvement from the key optimization techniques with various sparsity of the input. Additionally,fgSpMSpVis implemented on an NVIDIA Tesal P100 GPU and applied to the breath-first-search (BFS) application.fgSpMSpVon a P100 GPU obtains the speedup of up to\( 134.38\times \)over the state-of-the-art SpMSpV algorithms, and the BFS application usingfgSpMSpVachieves the speedup of up to\( 21.68\times \)over the state-of-the-arts.

High-performance computing (HPC) is focused on providing large-scale compute capacity to scientific applications. HPC schedulers tend to be optimized for large parallel batch jobs and, as such, often overlook the requirements of other scientific applications. In this work, we propose a cloud-inspired HPC scheduling model that aims to capture application performance and requirement models (Application Aware - A2) and dynamically resize malleable application resource allocations to be able to support applications with critical performance or deadline requirements. (Low Latency allocation - L2). The proposed model incorporates measures to improve data-intensive applications performance on HPC systems and is derived from a set of cloud scheduling techniques that are identified as applicable in HPC environments. The model places special focus on dynamically malleable applications; data-intensive applications that support dynamic resource allocation without incurring severe performance penalties; which are proposed for fine-grained backfilling and dynamic resource allocation control without job preemption.

Shared libraries have become ubiquitous and are used to achieve great resource efficiencies on many platforms. The same properties that enable efficiencies on time-shared computers and convenience on small clusters prove to be great obstacles to scalability on large clusters and High Performance Computing platforms. In addition, Light Weight operating systems such as Catamount have historically not supported the use of shared libraries specifically because they hinder scalability. In this report we will outline the methods of supporting shared libraries on High Performance Computing platforms using Light Weight kernels that we investigated. The considerations necessary to evaluate utility in this area are many and sometimes conflicting. While our initial path forward has been determined based on this evaluation we consider this effort ongoing and remain prepared to re-evaluate any technology that might provide a scalable solution. This report is an evaluation of a range of possible methods of supporting dynamically linked executables on capability class1 High Performance Computing platforms. Efforts are ongoing and extensive testing at scale is necessary to evaluate performance. While performance is a critical driving factor, supporting whatever method is used in a production environment is an equally important and challenging task.

The rapid adoption of machine learning (ML) has increased the computational and storage demands on high-performance computing (HPC) and cloud datacenters. Unfortunately, deploying ML models on HPC and cloud platforms is challenging because it is often difficult to manage, financially costly, and poses substantial environmental concerns. This dissertation attempts to answer a simple but ambitious question: can we deploy ML models to meet the Quality of Service (QoS) requirements while remaining cost-effective and environmentally sustainable? The dissertation demonstrates that achieving this balance is indeed possible, albeit through employing non-intuitive strategies and user engagements. We show that a combination of fast and slow hardware can outperform the common strategy of exclusively employing the latest and fastest hardware when co-optimizing for both performance and cost. Realizing this potential, however, requires a strategy to create an effective hardware mix, a process rigorously developed within this dissertation. This dissertation further reveals that the carbon footprint of ML models can be managed by opportunistically trading off answer quality transiently and re-configuring/partitioning the GPU hardware resources, depending on the carbon intensity of available energy sources in the datacenter's region. Finally, this dissertation discovers that ML deployments can be made more environmentally sustainable by strategically engaging users and tuning response brevity, especially in the context of large language models (LLMs). The scientific advancements made in this dissertation aim to relieve the community from the cost constraints, hardware inefficiencies, and potential negative environmental impacts of large-scale deployment of ML models on HPC and cloud platforms. We are hopeful that its open-source contributions will promote further innovation and broader engagement toward sustainable ML deployment.--Author's abstract

The development of big remote sensing data related technologies and applications poses a big challenge that massive computing capability is needed to support big data processing. In order to solve this challenge, this paper proposes an architecture of heterogeneous platform of high performance computing, which employs the computer hardware resources to improve the efficiency of big remote sensing data processing by optimizing scheduling strategies and designing high-performance algorithms. Furthermore, the proposed platform can dynamically incorporated with a workflow engine regarding big remote sensing data processing. These algorithms are modular to meet the flexible combination of different processes.

To reproduce eScience, several challenges need to be solved: scientific workflows need to be automated; the involved software versions need to be provided in an unambiguous way; input data needs to be easily accessible; High-Performance Computing (HPC) clusters are often involved and to achieve bit-to-bit reproducibility, it might be even necessary to execute the code on a particular cluster to avoid differences caused by different HPC platforms (and unless this is a scientist's local cluster, it needs to be accessed across (administrative) borders). Preferably, to allow even inexperienced users to (re-)produce results, all should be user-friendly. While some easy-to-use web-based scientific portals support already to access HPC resources, this typically only refers to computing and data resources that are local. By the example of two community-specific portals in the fields of biodiversity and climate research, we present a solution for accessing remote HPC (and cloud) compute and data resources from scientific portals across borders, involving rigorous container-based packaging of the software version and setup automation, thus enhancing reproducibility.