Data Decomposition for Code Parallelization in Practice: What Do the Experts Need?

Scientific collaboration is critical in high performance computational (HPC) environments. Computer security incidents, however, are a constant threat in today's interconnected computational environments. Many sites struggle with finding a balance between the needs of users and the threat from unauthorized usage or access. Understanding the constantly changing battlefield is critical for effective and optimal deployment of computer security resources.This tutorial will provide an underlying knowledge of the current state of computer security, its effects on HPC environments and mitigation strategies. Topics we will address include:1.HPC Security Fundamentals2.HPC Security Protection Fundamentals3.Intrusion Detection Using Bro4.The Changing Computer Protection Environment5.Cross-site Implications of Collaborative Environments6.Keeping One Step Ahead7.Risk Assessment, Mitigation and Compliance8.Components of a Good Protection ProgramWe will also show how described techniques are in use at SC06 to protect SCinet.

The lightweight nature, application portability, and deployment flexibility of containers is driving their widespread adoption in cloud solutions. Data analysis and deep learning (DL)/machine learning (ML) applications have especially benefited from containerization. As such data analysis is adopted in high performance computing (HPC), the need for container support in HPC has become paramount. However, containers face crucial performance and I/O challenges in HPC. One obstacle is that while there have been HPC containers, such solutions have not been thoroughly investigated, especially from the aspect of their impact on the crucial HPC I/O throughput. To this end, this paper provides a first-of-its-kind empirical analysis of state-of-the-art representative container solutions (Docker, Podman, Singularity, and Charliecloud) in HPC environments. We also explore how containers interact with an HPC parallel file system like Lustre. We present the design of an analysis framework that is deployed on all nodes in an HPC environment, and captures CPU, memory, network, and file I/O statistics from the nodes and the storage system. We are able to garner key insights from our analysis, e.g., Charliecloud outperforms other container solutions in terms of container start-up time, while Singularity and Charliecloud are equivalent in I/O throughput. But this comes at a cost, as Charliecloud invokes the most metadata and I/O operations on the underlying Lustre file system. By identifying such trade-offs and optimization opportunities, we can enhance HPC containers performance and the ML/DL applications that increasingly rely on them.

The efficient implementation of a diffraction tomography method for medical imaging is addressed within the framework of High Performance Computing (HPC) environment. A non-linear optimization method for the solution of the inverse scattering problem is implemented on a shared memory model computer. Linear speed-up and significant reduction in the total execution time is achieved when the program is executed in parallel, enabling the feasibility of the method for realistic medical imaging applications.

Purpose The present paper aims to address the description of a numerical optimization procedure, based on mesh morphing, and its application for the improvement of the aerodynamic performance of an industrial glider which suffers of a large separation occurring in the wing–fuselage junction region at high incidence angles. Design/methodology/approach Shape variations were applied to the baseline configuration through a mesh morphing technique founded on the mathematical framework of radial basis functions (RBF). The aerodynamic solutions were obtained coupling an RANS code with the mesh morphing tool RBF Morph™. Two shape modifiers were set up to generate a parametric numerical model. An optimization procedure, based on a design of experiment sampling, was set up implementing the fully automated workflow within a high performance computing (HPC) environment. The optimal candidates maximizing the aerodynamic efficiency were identified by means of a cubic RBF response surface approach. Findings The separation was significantly reduced, modifying the local geometry of fuselage and fairing and maintaining the wing aerofoil unchanged. A relevant aerodynamic efficiency improvement was finally gained. Practical implications The developed procedure proved to be a very powerful and efficient tool in facing aerodynamic design problems. However, it might be computationally very expensive if a large number of design variables are adopted and, in those cases, the method can be suitably used only within the HPC environment. Originality/value Such an optimization study is part of an explorative set of analyses that focused on better addressing the numerical strategies to be used in the development of the EU FP7 Project RBF4AERO.

SummaryThe wide range of RNA-seq applications and their high-computational needs require the development of pipelines orchestrating the entire workflow and optimizing usage of available computational resources. We present aRNApipe, a project-oriented pipeline for processing of RNA-seq data in high-performance cluster environments. aRNApipe is highly modular and can be easily migrated to any high-performance computing (HPC) environment. The current applications included in aRNApipe combine the essential RNA-seq primary analyses, including quality control metrics, transcript alignment, count generation, transcript fusion identification, alternative splicing and sequence variant calling. aRNApipe is project-oriented and dynamic so users can easily update analyses to include or exclude samples or enable additional processing modules. Workflow parameters are easily set using a single configuration file that provides centralized tracking of all analytical processes. Finally, aRNApipe incorporates interactive web reports for sample tracking and a tool for managing the genome assemblies available to perform an analysis.Availability and documentationhttps://github.com/HudsonAlpha/aRNAPipe; DOI: 10.5281/zenodo.202950Supplementary informationSupplementary data are available at Bioinformatics online.

Abstract. Most Earth system models (ESMs) are running under different high-performance computing (HPC) environments. This has several advantages, from allowing different groups to work with the same tool in parallel to leveraging the burden of ensemble climate simulations, but it also offers alternative solutions in the case of shutdown (expected or not) of any of the environments. However, for obvious scientific reasons, it is critical to ensure that ESMs provide identical results under changes in computing environment. While strict bit-for-bit reproducibility is not always guaranteed with ESMs, it is desirable that results obtained under one computing environment are at least statistically indistinguishable from those obtained under another environment, which we term a “replicability” condition following the metrology nomenclature. Here, we develop a protocol to assess the replicability of the EC-Earth ESM. Using two versions of EC-Earth, we present one case of non-replicability and one case of replicability. The non-replicable case occurs with the older version of the model and likely finds its origin in the treatment of river runoff along Antarctic coasts. By contrast, the more recent version of the model provides replicable results. The methodology presented here has been adopted as a standard test by the EC-Earth consortium (27 institutions in Europe) to evaluate the replicability of any new model version across platforms, including for CMIP6 experiments. To a larger extent, it can be used to assess whether other ESMs can safely be ported from one HPC environment to another for studying climate-related questions. Our results and experience with this work suggest that the default assumption should be that ESMs are not replicable under changes in the HPC environment, until proven otherwise.

98P - Standardized and compliant bio-informatics pipeline for neoepitope analysis in oncology clinical trials

Abstract: This chapter explores the design and development of photonic processors for ultra-fast data processing in high-performance computing (HPC) environments. By leveraging light instead of electricity for computation, photonic processors offer significant advantages in terms of speed, bandwidth, and energy efficiency. The chapter covers advancements in photonic materials, device engineering, and optical interconnects that enable scalable photonic systems for real-world applications, including AI, machine learning, and quantum computing. Key topics include the integration of photonic processors with quantum communication networks, the role of photonic multiplexing in parallel data processing, and innovations such as all-optical neurons for neuromorphic computing. The chapter also addresses current challenges, including phase sensitivity and the development of photonic quantum gates, offering solutions for scaling photonic computing to meet the demands of HPC and quantum networks. Keywords: photonic processors, high-performance computing, ultra-fast data processing, optical interconnects, photonic materials, quantum communication, AI, machine learning, photonic multiplexing, neuromorphic computing, phase sensitivity, quantum gates, scalable systems.

Modern High Performance Computing (HPC) applications, such as Earth science simulations, produce large amounts of data due to the surging of computing power, while big data applications have become more compute-intensive due to increasingly sophisticated analysis algorithms. The needs of both HPC and big data technologies for advanced HPC and big data applications create a demand for integrated system support. In this study, we introduce Scientific Data Processing (SciDP) to support both HPC and big data applications via integrated scientific data processing. SciDP can directly process scientific data stored on a Parallel File System (PFS), which is typically deployed in an HPC environment, in a big data programming environment running atop Hadoop Distributed File System (HDFS). SciDP seamlessly integrates PFS, HDFS, and the widely-used R data analysis system to support highly efficient processing of scientific data. It utilizes the merits of both PFS and HDFS for fast data transfer, overlaps computing with data accessing, and integrates R into the data transfer process. Experimental results show that SciDP accelerates analysis and visualization of a production NASA Center for Climate Simulation (NCCS) climate and weather application by 6x to 8x when compared to existing solutions.

Scientific workflows are models composed of activities, data and dependencies whose objective is to represent a computer simulation. Workflows are managed by Scientific Workflow Management System (SWfMS). Such workflows commonly demand for many computational resources once their executions may involve a number of different programs processing a huge volume of data. Thus, the use of High Performance Computing (HPC) environments allied to parallelization techniques provides the support for the execution of such experiments. Some resources provided by clouds can be used to build HPC environments. Although clouds offer advantages such as elasticity and availability, failures are a reality rather than a possibility. Thus, SWfMS must be fault-tolerant. There are several types of fault tolerance techniques used in SWfMS such as checkpoint-restart and replication, but which fault tolerance technique best fits with a specific workflow? This work aims at analyzing several fault tolerance techniques in SWfMSs and recommending the suitable one for the user’s workflow using machine learning techniques and provenance data, thus improving resiliency.

Continuous integration (CI) has become a ubiquitous practice in modern software development, with major code hosting services offering free automation on popular platforms. CI offers major benefits, as it enables detecting bugs in code prior to committing changes. While high-performance computing (HPC) research relies heavily on software, HPC machines are not considered “common” platforms. This presents several challenges that hinder the adoption of CI in HPC environments, making it difficult to maintain bug-free HPC projects, and resulting in adverse effects on the research community. In this article, we explore the challenges that impede HPC CI, such as hardware diversity, security, isolation, administrative policies, and nonstandard authentication, environments, and job submission mechanisms. We propose several solutions that could enhance the quality of HPC software and the experience of developers. Implementing these solutions would require significant changes at HPC centers, but if these changes are made, it would ultimately enable faster and better science.

Containers in High Performance Computing (HPC) provide a flexible and efficient alternative to traditional modular software. They allow researchers and engineers to package applications and their dependencies in a single file, ensuring portability, reproducibility and consistent performance across different computing environments. With the emergence of AI, industrial users and SMEs in HPC, containers have become even more convenient to use, as they can also meet the security requirements for isolating user workloads. As the HPC community embraces container technology, it is ushering in a new era of more efficient, repeatable and scalable research computing. However, adoption of containers in HPC has been relatively slow, largely due to misconceptions about how they work and concerns about potential performance degradation, energy inefficiency and security risks. This article looks at the security considerations when using containers and uses practical examples to show that, with the right configuration and tools, containers can be as secure as any other system application.

The Pasadena Workshop on System Software and Tools for High Performance Computing Environments was held at the Jet Propulsion Laboratory from April 14 through April 16, 1992. The workshop was sponsored by a number of Federal agencies committed to the advancement of high performance computing (HPC) both as a means to advance their respective missions and as a national resource to enhance American productivity and competitiveness. Over a hundred experts in related fields from industry, academia, and government were invited to participate in this effort to assess the current status of software technology in support of HPC systems. The overall objectives of the workshop were to understand the requirements and current limitations of HPC software technology and to contribute to a basis for establishing new directions in research and development for software technology in HPC environments. This report includes reports written by the participants of the workshop`s seven working groups. Materials presented at the workshop are reproduced in appendices. Additional chapters summarize the findings and analyze their implications for future directions in HPC software technology development.

With the advent of myriad different technologies in the modern computing era, the future of high performance computing (HPC) data centers comprising heterogeneous compute servers and networks involves a significantly greater degree of cyber-threats than we are observing currently. As a result, conducting cybersecurity operations at the HPC data centers encompasses a highly cognitive workload environment, where information is rapidly generated, is exceedingly dynamic and is vastly unpredictable. An important challenge faced by cybersecurity specialists is managing the extreme levels of stress and utilizing emerging technologies for monitoring, assessing and defending multiple cyber-attacks in real-time. In this paper, we present an augmented reality based framework for storing, analyzing, visualizing and augmenting data at the HPC centers. This framework is proposed with the goal of improving the data presentation, forecasting, analysis and decision making, thus reducing the time to insight and learning the HPC security operations to combat increasing cyber physical threats.

This chapter gives a comprehensive overview of the current status of accounting and billing for up-todate computing environments. Accounting is the key for the management of information system resources. At this stage of evolution of accounting systems it is adequate not to separate computing environments into High Performance Computing and Grid Computing environments for allowing a “holistic” view showing the different approaches and the state of the art for integrated accounting and billing in distributed computing environments. Requirements resulting from a public survey within all communities of the German Grid infrastructure, as well as from computing centres and resource providers of High Performance Computing resources like HLRN, and ZIVGrid, within the German e-Science framework, have been considered as well as requirements resulting from various information systems and the virtualisation of organisations and resources. Additionally, conceptual, technical, economical, and legal questions also had to be taken into consideration. After the requirements have been consolidated and implementations have been done over one year ago, now the overall results and conclusions are presented in the following sections showing a case study based on the GISIG framework and the Grid- GIS framework. The focus is on how an integrated architecture can be built and used in heterogeneous environments. A prototypical implementation is outlined that is able to manage and visualise relevant accounting and billing information based on suitable monitoring data in a virtual organisation (VO) specific way regarding basic business, economic, and security issues.