Phase I Final Technical Report on Energy-Efficient Reconfigurable Universal Accelerator Interconnect

Silicon photonics has emerged as a promising solution to address the interconnect bottleneck in high performance computing (HPC) systems and data centers. Silicon photonics provides unprecedented I/O bandwidth, enabling ultrahigh aggregated bandwidth (~ 10 Tbps), high bandwidth density (~ Tbps/mm), and high energy efficiency (~ pJ/bit). In addition, silicon photonics also enable optical switching with large radix and short switching time. This talk will provide an overview of the state of the art of silicon photonic switches, with emphasis on new micro-electromechanical-system (MEMS)-actuated switching mechanism. Large scale switches with 64x64 and 128x128 port counts have been demonstrated, as well as wavelength-selective switches with 8x8 ports and 8 wavelengths. Future scaling to even larger port count will be discussed.

SummaryThis paper presents a high‐performance computing (HPC) framework for Monte Carlo (MC) simulations in the ocean color (OC) application domain. The objective is to optimize a parallel MC radiative transfer code named MOX, developed by the authors to create a virtual marine environment for investigating the quality of OC data products derived fromin situmeasurements of in‐water radiometric quantities. A consolidated set of solutions for performance modeling, prediction, and optimization is implemented to enhance the efficiency of MC OC simulations on HPC run‐time infrastructures. HPC, machine learning, and adaptive computing techniques are applied taking into account a clear separation and systematic treatment of accuracy and precision requirements for large‐scale MC OC simulations. The added value of the work is the integration of computational methods and tools for MC OC simulations in the form of an HPC‐oriented problem‐solving environment specifically tailored to investigate data acquisition and reduction methods for OC field measurements. Study results highlight the benefit of close collaboration between HPC and application domain researchers to improve the efficiency and flexibility of computer simulations in the marine optics application domain. © 2016 The Authors.Concurrency and Computation: Practice and ExperiencePublished by John Wiley & Sons, Ltd.

The electrical interconnects in high performance computing (HPC) systems are reaching their bandwidth capacities in supporting data-intensive applications. Currently, communication between compute nodes through these interconnects is the main bottleneck for overall HPC system performance. Optical interconnects based on the emerging silicon photonics (SiP) platform are considered to be a promising replacement to boost the speed of the data transfer with reduced cost and energy consumption compared to electrical interconnects. In this paper, we present a comprehensive analysis of a comb source microring-based SiP link architecture with p-i-n photodetectors. In particular, we direct our focus on improved grating coupler and bus waveguide designs to reduce the link power penalties. Additionally, we map the required performance from the comb laser to provide an aggregated data rate of 1 Tbps under the constraints of free spectral range (FSR) and nonlinearities of the microring resonators (MRRs). We show that a select few comb configurations satisfy these requirements, and energy consumption as low as 3 pJ/bit is achievable.

In this paper we present a computing system that uniquely leverages the bandwidth, density, and latency advantages of silicon photonic interconnects to enable highly compact supercomputerscale systems. We present the details of an optically enabled "macrochip" which is a set of contiguous, optically-interconnected chips that deploy wavelength-division multiplexed (WDM) enabled by silicon photonics. We describe the system architecture and the WDM point-to-point network implementation of a "macrochip" providing bisection bandwidth of 10 TBps and discuss system and device level challenges, constraints, and the critical technologies needed to implement this system. We present a roadmap to lowering the energy-per-bit of a silicon photonic interconnect and highlight recent advances in silicon photonics under the UNIC program that facilitate implementation of a "macrochip" system made of arrayed chips.

Sharing More Widely: Data At The Heart Of Evidence, Policy And Transformation At Defra

Abstract: This chapter explores the design and development of photonic processors for ultra-fast data processing in high-performance computing (HPC) environments. By leveraging light instead of electricity for computation, photonic processors offer significant advantages in terms of speed, bandwidth, and energy efficiency. The chapter covers advancements in photonic materials, device engineering, and optical interconnects that enable scalable photonic systems for real-world applications, including AI, machine learning, and quantum computing. Key topics include the integration of photonic processors with quantum communication networks, the role of photonic multiplexing in parallel data processing, and innovations such as all-optical neurons for neuromorphic computing. The chapter also addresses current challenges, including phase sensitivity and the development of photonic quantum gates, offering solutions for scaling photonic computing to meet the demands of HPC and quantum networks. Keywords: photonic processors, high-performance computing, ultra-fast data processing, optical interconnects, photonic materials, quantum communication, AI, machine learning, photonic multiplexing, neuromorphic computing, phase sensitivity, quantum gates, scalable systems.

Current heterogeneous systems integrate generalpurpose Central Processing Units (CPUs), Graphics Processing Units (GPUs), and Neural Processing Units (NPUs). The efficient use of such systems requires a significant programming effort to distribute computation and synchronize across devices, which usually involves using Atomic Memory Operations (AMOs). Arm recently launched a floating-point Atomic Memory Operations (FAMOs) extension to perform atomic updates on floating-point data types specifically. This work characterizes and models heterogeneous architectures to understand how floating-point AMOs impact graph, Machine Learning (ML), and high-performance computing (HPC) workloads. Our analysis shows that many AMOs are performed on floating-point data, which modern systems execute using inefficient compare-and-swap (CAS) constructs. Therefore, replacing CASbased constructs with FAMOs can improve a wide range of workloads. Moreover, we analyze the trade-offs of executing FAMOs at different memory hierarchy levels, either in private caches (near) or remotely in shared caches (far). We have extended the widely used AMBA CHI protocol to evaluate such FAMO support on a simulated chiplet-based heterogeneous architecture. While near FAMOs achieve an average $1.34 \times$ speed-up, far FAMOs reach an average $1.58 \times$ speed-up. We conclude that FAMOs can bridge the gap between CPU architecture and accelerators and enabling synchronization in key application domains.

Hard Disk Drive (HDD) based storage systems in high performance computing (HPC), such as Lustre, are being confronted with great challenges from massive parallel data-intensive applications, and becomes one of the most critical factors that influence applications performance. In this paper, we analyze the performance of Lustre, and statistically examine the distribution of file sizes, I/O patterns for typical data-intensive applications, as well as the utilization of CPU and memory resources in TH-1A supercomputer from the last five years. We present masFS, a novel file system for HPC that exploits available memory and SSD resources on compute nodes with little interference to applications running on the nodes. The masFS supports POSIX interface with an unique namespace and provides comprehensive file service for applications. We design and implement masFS with memory and SSD, and have deployed and evaluated it on TH-1A. Experimental results show that masFS works as general storage system and can meet the requirement of variety of I/O loads. It can achieve a speedup of 14.5x for read and 8.8x for write respectively in running benchmarks, and an average 7.6x faster I/O time in a real-world data-intensive application compared to Lustre.

With the extraordinary growth in parallelism at all system scales driven by multicore architectures, computing performance is increasingly determined by how efficiently high-bandwidth data is communicated among the numerous compute resources. High-performance systems are especially challenged by the growing energy costs dominated by data movement. As future computing systems aim to realize the Exascale regime-surpassing 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">18</sup> operations per second-achieving energy efficient high-bandwidth communication becomes paramount to scaled performance. Silicon photonics offers the possibility of delivering the needed communication bandwidths to match the growing computing powers of these highly parallel architectures with extremely scalable energy efficiency. However, the insertion of photonic interconnects is not a one-for-one replacement. The lack of practical buffering and the fundamental circuit switched nature of optical data communications require a holistic approach to designing system-wide photonic interconnection networks. New network architectures are required and must include arbitration strategies that incorporate the characteristics of the optical physical layer. This paper reviews the recent progresses in silicon photonic based interconnect devices along with the system level requirements for Exascale. We present a co-design approach for building silicon photonic interconnection networks that leverages the unique optical data movement capabilities and offers a path toward realizing future Exascale systems.

The rapid scaling of microprocessor technologies has led to unprecedented advancements in processing speed and transistor density. However, as device dimensions shrink below the nanometre scale, traditional electrical on-chip interconnects face critical limitations including increased signal delay, power consumption, and reduced bandwidth. These challenges have emerged as major bottlenecks in the performance and energy efficiency of modern microprocessors, especially in multi-core and many-core architectures. To overcome these limitations, optical interconnects have gained significant attention as a high-performance alternative for on-chip and chip-to-chip communication. Optical interconnects offer key advantages such as higher data rates, lower latency, reduced power dissipation, and immunity to electromagnetic interference. The integration of optical components like waveguides, modulators, photodetectors, and silicon photonics within the chip architecture has the potential to revolutionize interconnect design by addressing the shortcomings of conventional electrical wiring. This paper explores the evolution of on-chip interconnects, comparing the performance, scalability, and integration challenges of electrical and optical solutions in scaled microprocessors. It also discusses emerging hybrid interconnect architectures that combine the strengths of both technologies. A comparison is made between electrical and optical interconnects for different design criteria, based on projections about the future of optical devices. Around a tenth of the length of the chip's edge is the crucial dimension beyond which optical connectivity becomes preferable to electrical interconnect at the 22 nm technological node.

In cloud computing, elasticity is defined as degree to which a system is able to adapt to workload changes by provisioning and de-provisioning resources in an autonomic manner, such that at each point in time the available resources match the current demand as closely as possible. Adding elasticity to HPC (High Performance Computing) clusters management systems remains challenging even if we deploy such HPC systems in today's cloud environments. This difficulty is caused by the fact that HPC jobs scheduler needs to rely on a fixed set of resources. Every change of topology (adding or removing computing resources) leads to a global restart of the HPC jobs scheduler. This phenomenon is not a major drawback because it provides a very effective way of sharing a fixed set of resources but we think that it could be complemented by a more elastic approach. Moreover, the elasticity issue should not be reduced to the scaling of resources issues. Clouds also enable access to various technologies that enhance the services offer to users. In this paper, our approach is to use containers technology to instantiate a tailored HPC environment based on the user's reservation constraints. We claim that the introduction and use of containers in HPC job schedulers allow better management of resources, in a more economical way. From the use case of SLURM, we release a methodology for 'containerization' of HPC jobs schedulers which is pervasive i.e. spreading widely throughout any layers of job schedulers. We also provide initial experiments demonstrating that our containerized SLURM system is operational and promising.

Machine learning (ML) and high-performance computing (HPC) applications increasingly combine dense and sparse memory access computations to maximize storage efficiency. However, existing central processing units (CPUs) and graphics processing units (GPUs) struggle to flexibly handle these heterogeneous workloads with consistently high compute efficiency. We present Occamy, a 432-core, 768-DP-GFLOP/s, dual-HBM2E, dual-chiplet RISC-V system with a latency-tolerant hierarchical interconnect and in-core streaming units (SUs) designed to accelerate dense and sparse FP8-to-FP64 ML and HPC workloads. We implement Occamy’s compute chiplets in 12-nm FinFET and its passive interposer, Hedwig, in a 65-nm node. On dense linear algebra (LA), Occamy achieves a competitive floating-point unit (FPU) utilization of 89%. On stencil codes, Occamy reaches an FPU utilization of 83% and a technology-node-normalized compute density of 11.1 DP-GFLOP/s/mm2, leading state-of-the-art (SoA) processors by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.7\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.2\times $ </tex-math></inline-formula>, respectively. On sparse-dense LA, it achieves 42% FPU utilization and a normalized compute density of 5.95 DP-GFLOP/s/mm2, surpassing the SoA by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.2\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11\times $ </tex-math></inline-formula>, respectively. On sparse–sparse LA, Occamy reaches a throughput of up to 187 GCOMP/s at 17.4 GCOMP/s/W and a compute density of 3.63 GCOMP/s/mm2. Finally, we reach up to 75% and 54% FPU utilization on and dense (large language model) and graph-sparse (graph convolutional network) ML inference workloads. Occamy’s register transfer level (RTL) description is freely available under a permissive open-source license.

New machine learning algorithms such as deep neural networks and the availability of large datasets have created a large drive towards new types of hardware capable of executing these algorithms with higher energy-efficiency. Recently, silicon photonics has emerged as a promising hardware platform for neuromorphic computing due to its inherent capability to process linear and non-linear operations and transmit a high bandwidth of data in parallel. At Hewlett Packard Labs, an energy-efficient dense-wavelength division multiplexing (DWDM) silicon photonics platform has been developed as the underlying foundation for innovative neuromorphic computing architectures. The latest research on our silicon photonic neuromorphic platform will be presented and discussed.

We present an optically connected disaggregated system architecture that supports dynamic resource allocation, leveraging the optical spatial switching capability of silicon photonic microring-based interconnects and demonstrate compute nodes on-request access to remote memory and PCIe-based resources.

As communication energy exceeds computation energy in future technologies, traditional on-chip electrical interconnects face fundamental challenges in the many-core era. Photonic interconnects have been proposed as a disruptive technology solution due to superior performance per Watt, distance independent energy consumption and CMOS compatibility for on-chip interconnects. Static power due to the laser being always switched on, varying link utilization due to spatial and temporal traffic fluctuations and thermal sensitivity are some of the critical challenges facing photonics interconnects. In this paper, we propose photonic interconnects for heterogeneous multicores using a checkerboard pattern that clusters CPU-GPU cores together and implements bandwidth reconfiguration using local router information without global coordination. To reduce the static power, we also propose a dynamic laser scaling technique that predicts the power level for the next epoch using the buffer occupancy of previous epoch. To further improve power-performance trade-offs, we also propose a regression-based machine learning technique for scaling the power of the photonic link. Our simulation results demonstrate a 34% performance improvement over a baseline electrical CMESH while consuming 25% less energy per bit when dynamically reallocating bandwidth. When dynamically scaling laser power, our buffer-based reactive and ML-based proactive prediction techniques show 40 - 65% in power savings with 0 - 14% in throughput loss depending on the reservation window size.