Efficient optical interconnect architecture for HPC and data center systems

The rising concern for power consumption of large-scale computer systems puts a research focus on the respective measurement methods. Varying workload patterns and energy efficiency optimizations cause highly dynamic power consumption on today’s compute nodes—a challenge for every measurement infrastructure. We identify five partly contradictory requirements that characterize such infrastructures: temporal granularity, spatial granularity, well-defined accuracy, scalability, and cost. In two projects we push the boundaries for these criteria: a scalable measurement solution for hundreds of nodes at millisecond granularity that is tightly integrated into the HPC system, and a sophisticated single-node instrumentation to measure the power consumption of application events in the microsecond range. Both measurement solutions are calibrated and their accuracy is carefully studied. We discuss scalable processing of the measurements for global monitoring in large-scale systems and use this data for energy efficiency analyses in combination with contextual information such as application performance trace data.

Mission planners are continuously striving to ensure that spacecraft, associated ground systems, communication equipment and all transmitted data are adequately protected against attacks. In this paper, we present an Intellectual Property (IP) Core of Advanced Encryption Standard/ Galois Counter Mode (AES/GCM) with 256-bit key in FPGA suited for space data systems. This is the algorithm recommended by Consultative Committee for Space Data Systems (CCSDS) in order to ensure confidentiality, authenticity and integrity of data. Since space missions are vulnerable to anomalies caused by Single Event Effects (SEE), our implementation does not use memory blocks in FPGA. Sub-field arithmetic is used in the implementation of Substitution Box (S-box) module. Efficient architectures are used in the design of the MixColumn with embedded ShiftRows and Key generation modules using XOR operations and avoiding conditional shifts. Shift and Add method is used to implement the Galois Field (GF) (2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">128</sup> ) Multiplier in the GCM authentication function. The design is ported to the Xilinx Space Grade Virtex-4 device XQR4VLX200. The functionality of AES/GCM implementation is verified using standard test vectors from National Institute of Standards and Technology (NIST). An implementation of AES/GCM using one each of AES-256 and GF(2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">128</sup> ) multiplication cores achieves a high resource efficiency of 18% slices and a maximum frequency of 139.725 MHz

Since today’s HPC and data center systems can comprise hundreds of thousands of servers and beyond, it is crucial to equip them with a network that provides high performance. New topologies proposed to achieve such performance need to be evaluated under different traffic conditions, aiming to closely replicate real-world scenarios. While most optimizations should be guided by common traffic patterns, it is essential to ensure that no pathological traffic pattern can compromise the entire system. Determining synthetic adversarial traffic patterns for a network typically relies on a thorough understanding of its topology and routing. In this paper, we address the problem of identifying a generic adversarial traffic pattern for low-diameter direct interconnection networks. We first focus on Random Regular Graphs (RRGs), which represent a typical case for these networks. Moreover, RRGs have been proposed as topologies for interconnection networks due to their superior scalability and expandability, among other advantages. We introduce Ant Mill, an adversarial traffic pattern for RRGs when using routes of minimal length. Secondly, we demonstrate that the Ant Mill traffic pattern is also adversarial in other low-diameter direct interconnection networks such as Slimfly, Dragonfly, and Projective networks. Ant Mill is thoroughly motivated and evaluated, enabling future studies of low-diameter direct interconnection networks to leverage its findings.

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

The interest in lattice-based cryptography is increasing due to its quantum resistance and its provable security under some worst-case hardness assumptions. As this is a relatively new topic, the search for efficient hardware architectures for lattice-based cryptographic building blocks is still an active area of research. We present area optimizations for the most critical and computationally-intensive operation in lattice-based cryptography: polynomial multiplication with the Number Theoretic Transform (NTT). The proposed methods are implemented on an FPGA for polynomial multiplication over the ideal ℤp[x]〈xn + 1〉. The proposed hardware architectures reduce slice usage, number of utilized memory blocks and total memory accesses by using a simplified address generation, improved memory organization and on-the-fly operand generations. Compared to prior work, with similar performance the proposed hardware architectures can save up to 67% of occupied slices, 80% of used memory blocks and 60% of memory accesses, and can fit into smallest Xilinx Spartan-6 FPGA.

Large-scale storage systems may service millions of clients. Effectively identifying and authenticating these clients can not only prevent illegal accesses but also eliminate unnecessary redundancies of access control. However, existing security solutions have largely ignored the identification and authentication issues or separately consider them from access control mechanisms. As a result, these solutions either have incurred a high cost of access control or led to new insurmountable problems. In this paper, we discuss the important consideration in designing an appropriate authentication scheme for large-scale storage systems. First, we discuss potential authentication techniques for large-scale storage systems. The looming crisis of PKI, a widely used technology for authentication in today's information security area is discussed and potential alternative technologies to PKI are suggested. Second, by merging authentication into access control we develop a decentralized security architecture for large-scale storage systems, designed to improve the scalability of authentication and reduce the overhead of access control. Finally, we experimentally evaluate how much overhead can be caused when various cryptographic algorithms used in an authentication protocol are applied to a large-scale storage system.

After 15 years of exponential improvement in microprocessor clock rates, the physical principles allowing for Dennard scaling, which enabled performance improvements without a commensurate increase in power consumption, have all but ended. Until now, most HPC systems have not focused on power efficiency. However, as the cost of power reaches parity with capital costs, it is increasingly important to compare systems with metrics based on the sustained performance per watt. Therefore we need to establish practical methods to measure power consumption of such systems in- situ in order to support such metrics. Our study provides power measurements for various computational loads on the largest scale HPC systems ever involved in such an assessment. This study demonstrates clearly that, contrary to conventional wisdom, the power consumed while running the high performance Linpack (HPL) benchmark is very close to the power consumed by any subset of a typical compute-intensive scientific workload. Therefore, HPL, which in most cases cannot serve as a suitable workload for performance measurements, can be used for the purposes of power measurement. Furthermore, we show through measurements on a large scale system that the power consumed by smaller subsets of the system can be projected straightforwardly and accurately to estimate the power consumption of the full system. This allows a less invasive approach for determining the power consumption of large-scale systems.

Internet of Things (IoT) technology has been pervasively applied to environmental monitoring, due to the advantages of low cost and flexible deployment of IoT enabled systems. In many large-scale IoT systems, accurate and efficient data sampling and reconstruction is among the most critical requirements, since this can relieve the data rate of trunk link for data uploading while ensure data accuracy. To address the related challenges, we have proposed an unmanned aerial vehicle (UAV) enabled spatial data sampling scheme in this paper using denoising autoencoder (DAE) neural network. More specifically, a UAV-enabled edge-cloud collaborative IoT system architecture is first developed for data processing in large-scale IoT monitoring systems, where UAV is utilized as mobile edge computing device. Based on this system architecture, the UAV-enabled spatial data sampling scheme is further proposed, where the wireless sensor nodes of large-scale IoT systems are clustered by a newly developed bounded-size K-means clustering algorithm. A neural network model, i.e., DAE, is applied to each cluster for data sampling and reconstruction, by exploitation of both linear and nonlinear spatial correlation among data samples. Simulations have been conducted and the results indicate that the proposed scheme has improved data reconstruction accuracy under the sampling ratio without introducing extra complexity, as compared to the compressive sensing-based method.

With the parallel computer systems scaling-up, the measure index for performance of the systems demands a shift from traditional "high performance" to "high productivity." This brings a new challenge to defining a synthetic, yet meaningful, measure index of multiple productivity variables; namely computing performance, reliability, energy consumption, parallel software development, etc. Traditional measures for large-scale parallel computer systems merely focus on computing performance, and are incapable of measuring the multiple productivity variables simultaneously in an effective manner. A recently proposed market-related money model, which pursues high utility/cost ratio, relies on money as a measure to consider the multiple productivity variables. Differing from the previous models, this paper proposes a novel system productivity speedup metric for large-scale parallel computer systems. The metric uses speedup instead of money to comprehensively unify the measures of multiple productivity variables. Finally, we propose a trade-off productivity measurement to weigh different productivity variables, to address different design targets. The measurement can facilitate the system evaluation, expose future technique tendencies, and guide future system design.

The rapid development of cyber-attacks have brought demand on the fore the weaknesses of conventional forms of cyber security protections which heavily depend on static and predetermined rules and fixed network configurations. With the intelligence of adversaries who unfortunate like have progressed incredible persistent threats, (APTs), zero-day exploits, and social engineering, static defenses often fail to mitigate attack which change with time. As such, the need is currently felt more for dynamic and adaptive defense mechanism which could constantly evolve in meeting these threats. One promising solution to this is the concept of Adaptive Deception Networks (ADNs), which is aimed at confusing and misleading attackers by dynamically reigning the behavior of the system. This is an adaptive strategy in case the adversaries try to map the network, which would make their reconnaissance and attack attempts ineffective. A very important part of ADNs is real time topology morphing which is the process of continuously and dynamically changing the topology of a network. By changing the interconnections between nodes, this technique provides a moving target to attackers, which prevents them from getting concept of what the system's architecture is like. Real-time topology morphing makes it more difficult for the attacker in that the attacker now has to constantly adapt to a shifting landscape. This paper examines real-time topology morphing integration into ADNs through cyber-physical systems (CPS), autonomous systems and other critical infrastructures. The research reported in this paper deals with the combination of multi-agent systems (MAS) with ADNs, aim of automating and controlling the changes of the topology. By using so-called event triggered consensus control, several agents can cooperate to ensure that the system adapts in real-time and that it still keeps its synchronization and stability. This paper introduces an overall design in order to form a single consistent structure, which not only can support the functional of real-time topology morphing, but also can guarantee the coherence and scalability of the system. The study also deals with issues involved in implementing such a dynamic approach into large scale systems such as issues such as performance degradation, synchronization of the agents, and resource allocation. The possible findings are part of the ongoing research in adaptive cybersecurity defenses, which can provide insights on how to use dynamic deception strategies to make the critical infrastructure system more resilient.

Recent studies indicate that the energy cost and carbon footprint of data centers have become exorbitant. It is a demanding and challenging task to reduce energy consumption in large-scale storage systems in modern data centers. Most energy conservation techniques inevitably have adverse impacts on parallel disk systems. To address the reliability issues of energy-efficient parallel disks, we propose a reliable energy-efficient RAID system called REED, which aims at improving both energy efficiency and reliability of RAID systems by seamlessly integrating HDDs and SSDs. At the heart of REED is a high-performance cache mechanism powered by SSDs, which are serving popular data. Under light workload conditions, REED spins down HDDs into the low-power mode, thereby offering energy conservation. Importantly, during an I/O access turbulence (i.e., I/O load is dynamically and frequently changing), REED is conducive to reducing the number of disk power-state transitions by keeping HDDs in the low-power mode while serving requests with SSDs. We build a model to quantitatively show that REED is capable of improving the reliability of energy-efficient RAIDs. We implement the REED prototype in a real-world RAID-0 system. Our experimental results demonstrate that REED improves the energy-efficiency of conventional RAID-0 by up to 73% while maintaining good reliability.

To predict application performance on an HPC system is an important technology for designing the computing system and developing applications. However, accurate prediction is a challenge, particularly, in the case of a future coming system with higher performance. In this paper, we present a new method for predicting application performance on HPC systems. This method combines modeling of sequential performance on a single processor and macro-level simulations of applications for parallel performance on the entire system. In the simulation, the execution flow is traced but kernel computations are omitted for reducing the execution time. Validation on a real terascale system showed that the predicted and measured performance agreed within 10% to 20 %. We employed the method in designing a hypothetical petascale system of 32768 SIMD-extended processor cores. For predicting application performance on the petascale system, the macro-level simulation required several hours.

Today, large scale computer systems have become an important component in production and scientific computing and lead to rapid advances in many disciplines. However, the size, and complexity of systems make them very difficult to detect unusual nodes automatically and traditional host monitoring tools are not capable of dealing with the need of anomaly detection in large amount of nodes. In this paper, we introduce a novel tool, LSAND, which could detect anomalistic nodes in a horizontal view of the machines with comparable configuration and tasks running on. We evaluated LSAND in a cluster environment, table the results of our experiment and give a discussion on the effect and show that LSAND is both effective and efficient for detecting anomalistic nodes with high-dimensional features.

To simultaneously address dual demands on the spectral efficiency and the data transmission rate of maritime communications, in this paper, a novel maritime communication technology based on time-delay overlapping modulation (TDOM) is proposed. In TDOM, multiple carriers are delayed in turn and directly overlapped in the time domain to form the communication symbol. In this way, the strict orthogonality between carriers can be broken through, indicating that the frequency interval between carriers can be much narrower than ever. In other words, TDOM can achieve non-orthogonal communication and realize both higher spectral efficiency and a higher maritime communication rate. The system architecture is designed, and its performance is analyzed and compared with that of other typical maritime communication technologies such as the very-high-frequency data exchange system (VDES). It is shown that TDOM can reach up to 20 times the peak communication rate and the spectral efficiency of VDES. As a digital modulation technology implemented in the physical layer, the proposed TDOM can be extended to other communication systems such as satellite communications or 4G/5G based off-shore communications.

Programming models intended to run on exascale systems have a number of challenges to overcome, specially the sheer size of the system as measured by the number of concurrent software entities created and managed by the underlying runtime. It is clear from the size of these systems that any state maintained by the programming model has to be strictly sub-linear in size, in order not to overwhelm memory usage with pure overhead. A principal feature of Partitioned Global Address Space (PGAS) models is providing easy access to global-view distributed data structures. In order to provide efficient access to these distributed data structures, PGAS models must keep track of metadata such as where array sections are located with respect to processes/threads running on the HPC system. As PGAS models and applications become ubiquitous on very large trans-pet scale systems, a key component to their performance and scalability will be efficient and judicious use of memory for model overhead (metadata) compared to application data. We present an evaluation of several strategies to manage PGAS metadata that exhibit different space/time tradeoffs. We use two real-world PGAS applications to capture metadata usage patterns and gain insight into their communication behavior.