Free-space optics for high-speed reconfigurable card-to-card optical interconnects

We are delighted to introduce a Special Issue on Optical Interconnects for the IET Optoelectronics magazine. This issue presents the current status as well as some of the outstanding progress that has been witnessed in the rapidly evolving area of optical interconnects. Today, data centres (DC) and high-performance computers are struggling with limited interconnect capacity and increased power consumption while trying to meet overwhelming bandwidth demands. As photonics technologies have been rapidly maturing, optical interconnects have come into the spotlight as the means to cope with the emerging bandwidth, energy, and capacity needs. As such, a wealth of photonic interconnect solutions across the DC hierarchy has unfolded, covering from electro-optic boards to transceiver development, on-chip and on-board switches and interconnects as well as novel noninvasive monitoring methods for light controlling for compensating thermal crosstalk, fabrication tolerances and unstable environmental conditions. At the same time, the increased penetration of optical interconnects in short-reach communications, such as board and chip levels, has triggered new challenges in photonics technologies calling for new innovations with respect to integration techniques and packaging methods. This Special Issue of IET includes integrated circuits, assembly techniques, systems-level architectures, and controlling methods related to short reach ‘chip-to-chip’, ‘in-the-box’, and ‘box-to-box’ communications. Specific areas covered by this special issue include: Silicon photonic transceiver technologies and flat DC topologies; High-speed electro-optic hosting boards and fast, energy-efficient electronics; DC switches and co-packaged optic (CPO) technologies; and Noninvasive controlling techniques for complex photonic integrated circuits. The four papers in this special section promote high-performance optical interconnect architectures and key technologies to tackle the rising bandwidth demands of DCs. They cover (a) co-packaged DC optics, (b) silicon technology transceiver modules and high-speed hosting boards for multisocket server board (MSB) interconnects and (c) automated controlling architecture for complex photonic systems; Items (b) and (c) draw from impressive achievements accomplished through the recently completed European H2020 Project ICT-STREAMS. This report written by C. Mickenberg tries to shed light on motivating factors behind co-packaged optics (CPO), the trade-offs involved and their potential impact on data centre network architectures. The article reviews the evolution of DC switch application-specific integrated circuit (ASICs) and DC optics, provides a future road map and finally discusses the potential impact of CPO on the network architecture. This topic is covered by two articles. The first, written by M. Morallis-Pegios, proposes using silicon photonics to advance interconnect throughput, socket connectivity and energy efficiency in MSB layouts by enabling a flat-topology wavelength division multiplexing (WDM) point-to-point optical MSB interconnect scheme. WDM silicon photonic transceivers coassembled with their electronic circuits achieving a 50-Gb/s line rate and 400-Gb/s aggregate data transmission are demonstrated. The article also proposes silicon photonic Arrayed Waveguide Grating Router (AWGRs) that can offer collision-less time-of-flight connectivity for up to 16 nodes. The second article covering that area is written by T. Lamprecht and reports on the development of electro-optic hosting boards for on-board transceiver subassemblies which is required to eventually form the Tb/s multi-socket onboard communication. The article also discusses the electrical interface as well as photonic board requirements, including the manufacturing technique and the assembly concept that was followed to deliver terabyte per second multichip optical communication. This article, written by F. Zanetto, introduces a novel dithering technique for real-time control of multiple cascaded photonic integrated devices, proposing a calibration-free strategy to support the development and real-life diffusion of increasingly complex photonic systems. The article elaborates on the implementation of a calibration-free automated control loop exploiting integral controllers to close a real-time feedback and stabilise the optimum working point of the employed photonic devices. The effectiveness of the proposed architectural approach is experimentally validated with the error-free routing and transmission of a 30-Gbit/s optical signal traversing a cascade of micro-ring resonator filters. All of the articles selected for this Special Issue present a wide variety of key topics and significant results addressing optical interconnects in high-performance data communication systems, including new progress on architectures, systems and packaging techniques as well as novel optimised controlling methods. As guest editors, we would like to thank all of the authors for their high-quality submissions that significantly contribute to the optical interconnect field. We would also like to thank the reviewers for their valuable and constructive technical comments that greatly enhance the Special Issue, because their feedback and comments ensure the high quality of the IET Optoelectronics Journal. We hope that this IET Special Issue will provide a useful reference of the state-of-the-art in optical interconnects to readers, and we anticipate such technological perspectives will stimulate new advancements and technological outcomes that will in turn enrich the growing field of optical interconnects. This work was supported by the E.C. through the H2020-ICT-STREAMS research project (688172).

Data rates are continuing to increase for box-to-box, rack-to-rack, board-to-board, and chip-to-chip interconnects for terabit switches and routers, multiprocessor computers and high-end servers. The increase in individual line rates and bandwidth drives the need to replace copper interconnects with optical interconnects. Fiber optics are advantageous for these applications because they allow for longer link lengths, increased bandwidth, smaller cables and connectors, less susceptibility to electromagnetic interference, and potentially lower power dissipation, depending on the data rate and link length. Towards this direction, this thesis aims to design and demonstrate low-cost, low-latency, high throughput, rack-to-rack optical interconnect architectures for exascale (i.e., performing 10^18 floating point operations per second) high-performance computing (HPC) systems and data centers. In particular, a novel, cost-effective, optical interconnect architecture for ultrafast optical switching, based on semiconductor optical amplifiers (SOAs), is studied. The proposed design of a fast N×N all-optical, wavelength-space crossbar switch for optical interconnects uses a minimum number of ON/OFF gates. This thesis compares and proves the superiority of the proposed architecture with respect to its originally-proposed counterpart, both theoretically and experimentally. Additionally, in order to increase the capacity and to minimize the impact of transmission effects (especially self-gain modulation (SGM), cross-gain modulation (XGM), self-phase modulation (SPM), cross-phase modulation (XPM), and polarization dependent gain (PDG)), we investigate the performance of conventional binary intensity modulation (IM), in conjunction with direct detection, as well as of advanced, more resilient, spectrally-efficient modulation formats (e.g., Differential Phase Shift Keying (DPSK), Orthogonal Frequency Division Multiplexing (OFDM), Polarization Division Multiplexed Quadrature Phase Shift Keying (PDM-QPSK), Single (SP)- and PDM- 16-ary Quadrature Amplitude Modulation (16QAM) in conjunction with coherent detection). Finally, as a seperate research activity, we study the performance of point-to-point links based on vertical-cavity surface-emitting lasers (VCSELs) and single- or multi- mode fibers, in conjuction with IM/DD, four-level Pulse Amplitude Modulation (4-PAM), and OFDM, to enable state-of-the-art, high-capacity, low-cost optical interconnects.

Scaling of VLSI technology has been dramatically increasing microelectronic device densities and speeds. However, the interconnection technology between devices does not advance proportionally. Limited available interconnect materials compatible with VLSI and packaging technologies, increased wire resistance as a result of scaling, residual wire capacitance due to fringing fields and fields between interconnect wires are among the factors that prohibit drastic improvement of the electrical interconnect performance. As a result, the performance of VLSI systems become increasingly more dominated by the performance of long interconnects. To overcome this limitation, free-space optical interconnects have been suggested, where long electrical interconnects are replaced by an optical transmitter, a. photodetector, and interconnection optics between them [2, 10. 13–15,19, 22, 27]. This scheme, although devoid of electrical interconnection para.sitics, has its own difficulties. Unavailability of monolithically integrated optical transmitters on silicon imposes hybrid integration schemes with larger parasitic capacitance and increased cost. Transformation of information from electrical to optical domain and vice versa introduces severe inefficiencies into the energy budget. There are voltage incompatibility issues between some transmitter technologies and VLSI technology as well.

In interconnections between bookshelf-assembled asynchronous transfer mode (ATM) switch boards, a large number of micro-optical signal beams can be optically interconnected by a beam direction compensating system using a vertical cavity surface emitting laser, an x-y beam positioning sensor and a beam deflector made of an adjustable liquid prism. This beam direction compensating system can suppress the decrease in coupling efficiency between transmitters and receivers to no more than 15%, even if boards are inserted and extracted repeatedly, and are shocked repeatedly at a high intensity of 100 G. The compensation is very fast (20 ms). Furthermore, various optical interconnections necessary for ATM switching networks can be achieved by beam deflectors of a liquid crystal microprism array. In this preliminary study, we fabricate no more than eight-channel optical interconnections. However, since one surface emitting laser diode can have many channels in a small area (64 channels per 4 mm/sup 2/) and the aperture of this adjustable liquid prism is wide (28 mm in diameter) and uniform, a huge number of optical interconnection channels is possible using this system.

Traditional high performance computing (HPC) centers that operate a single large supercomputer cluster have not required sophisticated mechanisms to manage and enforce network policies. Recently, HPC centers have expanded to support a wide range of computational infrastructure, such as OpenStack-based private clouds and Ceph object stores, each with its own unique characteristics and network security requirements. Network security policies are becoming more complex and harder to manage. To address the challenge, this paper explores ways to define and manage the new network policies required by emerging HPC systems. As the first step, we identify the new types of policies that are required and the technical capabilities needed to support them. We present example policies and discuss ways to implement those policies using emerging programmable networks and intent-based networks. We describe our initial work toward automatically converting human readable network policies into network configurations and programmable network controllers that implement those policies using business rule management systems.

While High Performance Computing (HPC) is critical for scientific breakthroughs and AI development, its growing energy use (often dependent on fossil fuels) requires a detailed environmental impact analysis. This study addresses the ever-increasing ecological challenges posed by high-performance computing (HPC) data centers by developing a model to quantify and predict their energy consumption and carbon emissions. Our model estimates the annual energy consumption of HPC data centers globally using data on the size and capacity of small and massive data centers. It distinguishes between theoretical maximum and actual energy consumption based on average utilization. The results show an upward trend in carbon emissions associated with HPC, which are projected to reach 5.81 x 1017 grams of carbon dioxide by 2030. As a result, we propose a series of recommendations to reduce the environmental impact of HPC at both the technical and policy levels in favor of promoting a healthy and sustainable environment.

Scalability analysis methodology for passive optical interconnects in data center networks using PAM

Current High-Performance Computing (HPC) and data center networks rely on large-radix routers. Hamming graphs (Cartesian products of complete graphs) and dragonflies (two-level direct networks with nodes organized in groups) are some direct topologies proposed for such networks. The original definition of the dragonfly topology is very loose, with several degrees of freedom, such as the inter- and intragroup topology, the specific global connectivity, and the number of parallel links between groups (or trunking level). This work provides a comprehensive analysis of the topological properties of the dragonfly network, providing balancing conditions for network dimensioning, as well as introducing and classifying several alternatives for the global connectivity and trunking level. From a topological study of the network, it is noted that a Hamming graph can be seen as a canonical dragonfly topology with a high level of trunking. Based on this observation and by carefully selecting the global connectivity, the Dimension Order Routing (DOR) mechanism safely used in Hamming graphs is adapted to dragonfly networks with trunking. The resulting routing algorithms approximate the performance of minimal, nonminimal, and adaptive routings typically used in dragonflies but without requiring virtual channels to avoid packet deadlock, thus allowing for lower cost router implementations. This is obtained by properly selecting the link to route between groups based on a graph coloring of network routers. Evaluations show that the proposed mechanisms are competitive with traditional solutions when using the same number of virtual channels and enable for simpler implementations with lower cost. Finally, multilevel dragonflies are discussed, considering how the proposed mechanisms could be adapted to them.

Modern High-Performance Computing (HPC) and data center operators rely more and more on data analytics techniques to improve the efficiency and reliability of their operations. They employ models that ingest time-series monitoring sensor data and transform it into actionable knowledge for system tuning: a process known as Operational Data Analytics (ODA). However, monitoring data has a high dimensionality, is hardware-dependent and difficult to interpret. This, coupled with the strict requirements of ODA, makes most traditional data mining methods impractical and in turn renders this type of data cumbersome to process. Most current ODA solutions use ad-hoc processing methods that are not generic, are sensible to the sensors' features and are not fit for visualization. In this paper we propose a novel method, called Correlation-wise Smoothing (CS), to extract descriptive signatures from time-series monitoring data in a generic and lightweight way. Our CS method exploits correlations between data dimensions to form groups and produces image-like signatures that can be easily manipulated, visualized and compared. We evaluate the CS method on HPC-ODA, a collection of datasets that we release with this work, and show that it leads to the same performance as most state-of-the-art methods while producing signatures that are up to ten times smaller and up to ten times faster, while gaining visualizability, portability across systems and clear scaling properties.

Optical interconnects are being discussed as a replacement for conventional electrical interconnects and are expected to be applied for future generations of high-performance supercomputers and data centers. Benes networks have attracted much attention because they require only 2 × 2 optical switches, which reduce the cost of rearrangeable nonblocking. However, optical power imbalances can significantly challenge receiver sensitivity. In this work, insertion loss (IL) fairness has been proposed and applied to the field of switches to achieve a relative balance of optical path data transmission in Benes networks. Fairness can be achieved when the port count is small (4 × 4) if the IL between ports is balanced. When the number of ports is moderate (8 × 8), we must use a suitable algorithm or determine the appropriate operating wavelength to minimize the power imbalance. An efficient two-step algorithm (ETS) has particular advantages in solving the path fairness problem and mitigating the power imbalance. As the number of ports increases, the switch states and topology jointly deteriorate the power imbalance. Finally, the ETS algorithm narrows the dynamic range requirement to 13.66 dB, with a 2 dB improvement. It achieves an extinction ratio of 24 dB and a bandwidth of 375 GHz, which outperforms the conventional 32 × 32 Benes network, respectively.

Free-space optical interconnects have been identified as a potentially important technology for future massively parallel-computing systems. The development of optoelectronic smart pixels based on InGaAs/AlGaAs multiple-quantum-well modulators and detectors flip-chip solder-bump bonded onto complementary-metal-oxide-semiconductor (CMOS) circuits and the design and construction of an experimental processor in which the devices are linked by free-space optical interconnects are described. For demonstrating the capabilities of the technology, a parallel data-sorting system has been identified as an effective demonstrator. By use of Batcher's bitonic sorting algorithm and exploitation of a perfect-shuffle optical interconnection, the system has the potential to perform a full sort on 1024, 16-bit words in less than 16 mus. We describe the design, testing, and characterization of the smart-pixel devices and free-space optical components. InGaAs-CMOS smart-pixel, chip-to-chip communication has been demonstrated at 50 Mbits/s. It is shown that the initial system specifications can be met by the component technologies.

According to different transmission distances, application scenarios of a data center mainly include intra- and inter-data center optical interconnects. The intra-data center optical interconnect is considered as a few kilometers optical interconnect between servers and racks inside a data center, which accounts for nearly 80% of data traffic of a data center. The other one, inter-data center optical interconnect, is mainly applied in tens of kilometers data transmission among different data centers. Since data exchange in data centers generally occurs between many servers and racks, and a lot of transmitter and receiver components are required, optical interconnects become highly sensitive to component costs. In this paper, we firstly review the development and applications of mainstream transmitter components (e.g., VCSEL, DML, EML, MZM, and monolithic integrated transmitter) and receiver components (e.g., single-end photodetector, Kramers-Kronig receiver, Stokes vector receiver, and monolithic integrated receiver), which have been widely applied in short-reach transmission systems. Then, two types of integrated solutions including simplified detection scheme and transceiver integration scheme are presented in detail. Finally, we summarize and discuss the technological and component options for different transmission distances. We believe that monolithic integrated components, especially transceiver integration, will become a powerful solution for next-generation high-speed short-reach transmission systems.

We present an array of electro-optic Bragg gratings for on-chip optical waveguide interconnect applications. These electro-optical gratings can serve as switchable couplers to couple high-speed optical signals from a vertical cavity surface emitting lasers (VCSEL) to on-chip polymeric waveguides and from waveguides to photodetectors. The unique switching feature provides a promising solution to facilitate dynamically reconfigurable on-chip optical interconnects architecture for intra-chip, chip-to-chip and board-to-board data communications. We demonstrate on-chip switched optical waveguide interconnect using electro-optic grating and polymeric channel waveguides on a silicon substrate.

As the science and engineering demand for high performance computing (HPC) grows beyond leading edge research institutions and communities to encompass routine activities of many disciplines, computing center infrastructures expand in their size, density and power demands. Long-practiced HPC center enhancements are increasingly demanding cooling methods that extend well beyond the capabilities of implementations that have become the staple of facility designs, even beyond those of the now-dominant architecture - commodity-processor-based, air-cooled rack clusters. As compute capacity aggressively increases in HPC centers, the power and cooling requirements, and thus cost of operation of the facilities, continues to rise correspondingly. This growth increasingly taxes the abilities of the hosting organizations to accommodate these demands. To address the challenge of meeting such pervasive demands, this paper examines energy efficiency in existing data centers from a two-pronged approach: employing direct water cooling and optimizing the facility infrastructure with as little capital investment to the building as possible.

As bit rates increase, optical interconnects based high-performance computing (HPC) systems improve performance by increasing the available bandwidth (using wavelength-division multiplexing (WDM) and spacedivision multiplexing (SDM)) and decreasing power dissipation as compared to traditional electrical interconnects. While static allocation of wavelengths (channels) in optical interconnects provide every node with equal opportunity for communication, it can lead to network congestion for non-uniform traffic patterns. In this paper, we propose an opto-electronic interconnect for designing a flexible, highbandwidth, low-latency, dynamically reconfigurable architecture for scalable HPC systems. Reconfigurability is realized by monitoring traffic intensities, and implementing dynamic bandwidth re-allocation (DBR) technique that adapts to changes in communication patterns. We propose a DBR technique - Lock-Step (LS) that balances the load on each communication channel based on past utilization. Simulation results indicate that the reconfigured architecture shows 40% increased throughput and and 20% reduced network latency as compared to HPC electrical networks.