Integrated silicon photonics for high-volume data center applications

We are delighted to introduce a Special Issue on Optical Interconnects for the IET Optoelectronics magazine. This issue presents the current status as well as some of the outstanding progress that has been witnessed in the rapidly evolving area of optical interconnects. Today, data centres (DC) and high-performance computers are struggling with limited interconnect capacity and increased power consumption while trying to meet overwhelming bandwidth demands. As photonics technologies have been rapidly maturing, optical interconnects have come into the spotlight as the means to cope with the emerging bandwidth, energy, and capacity needs. As such, a wealth of photonic interconnect solutions across the DC hierarchy has unfolded, covering from electro-optic boards to transceiver development, on-chip and on-board switches and interconnects as well as novel noninvasive monitoring methods for light controlling for compensating thermal crosstalk, fabrication tolerances and unstable environmental conditions. At the same time, the increased penetration of optical interconnects in short-reach communications, such as board and chip levels, has triggered new challenges in photonics technologies calling for new innovations with respect to integration techniques and packaging methods. This Special Issue of IET includes integrated circuits, assembly techniques, systems-level architectures, and controlling methods related to short reach ‘chip-to-chip’, ‘in-the-box’, and ‘box-to-box’ communications. Specific areas covered by this special issue include: Silicon photonic transceiver technologies and flat DC topologies; High-speed electro-optic hosting boards and fast, energy-efficient electronics; DC switches and co-packaged optic (CPO) technologies; and Noninvasive controlling techniques for complex photonic integrated circuits. The four papers in this special section promote high-performance optical interconnect architectures and key technologies to tackle the rising bandwidth demands of DCs. They cover (a) co-packaged DC optics, (b) silicon technology transceiver modules and high-speed hosting boards for multisocket server board (MSB) interconnects and (c) automated controlling architecture for complex photonic systems; Items (b) and (c) draw from impressive achievements accomplished through the recently completed European H2020 Project ICT-STREAMS. This report written by C. Mickenberg tries to shed light on motivating factors behind co-packaged optics (CPO), the trade-offs involved and their potential impact on data centre network architectures. The article reviews the evolution of DC switch application-specific integrated circuit (ASICs) and DC optics, provides a future road map and finally discusses the potential impact of CPO on the network architecture. This topic is covered by two articles. The first, written by M. Morallis-Pegios, proposes using silicon photonics to advance interconnect throughput, socket connectivity and energy efficiency in MSB layouts by enabling a flat-topology wavelength division multiplexing (WDM) point-to-point optical MSB interconnect scheme. WDM silicon photonic transceivers coassembled with their electronic circuits achieving a 50-Gb/s line rate and 400-Gb/s aggregate data transmission are demonstrated. The article also proposes silicon photonic Arrayed Waveguide Grating Router (AWGRs) that can offer collision-less time-of-flight connectivity for up to 16 nodes. The second article covering that area is written by T. Lamprecht and reports on the development of electro-optic hosting boards for on-board transceiver subassemblies which is required to eventually form the Tb/s multi-socket onboard communication. The article also discusses the electrical interface as well as photonic board requirements, including the manufacturing technique and the assembly concept that was followed to deliver terabyte per second multichip optical communication. This article, written by F. Zanetto, introduces a novel dithering technique for real-time control of multiple cascaded photonic integrated devices, proposing a calibration-free strategy to support the development and real-life diffusion of increasingly complex photonic systems. The article elaborates on the implementation of a calibration-free automated control loop exploiting integral controllers to close a real-time feedback and stabilise the optimum working point of the employed photonic devices. The effectiveness of the proposed architectural approach is experimentally validated with the error-free routing and transmission of a 30-Gbit/s optical signal traversing a cascade of micro-ring resonator filters. All of the articles selected for this Special Issue present a wide variety of key topics and significant results addressing optical interconnects in high-performance data communication systems, including new progress on architectures, systems and packaging techniques as well as novel optimised controlling methods. As guest editors, we would like to thank all of the authors for their high-quality submissions that significantly contribute to the optical interconnect field. We would also like to thank the reviewers for their valuable and constructive technical comments that greatly enhance the Special Issue, because their feedback and comments ensure the high quality of the IET Optoelectronics Journal. We hope that this IET Special Issue will provide a useful reference of the state-of-the-art in optical interconnects to readers, and we anticipate such technological perspectives will stimulate new advancements and technological outcomes that will in turn enrich the growing field of optical interconnects. This work was supported by the E.C. through the H2020-ICT-STREAMS research project (688172).

In this paper we present a computing system that uniquely leverages the bandwidth, density, and latency advantages of silicon photonic interconnects to enable highly compact supercomputerscale systems. We present the details of an optically enabled "macrochip" which is a set of contiguous, optically-interconnected chips that deploy wavelength-division multiplexed (WDM) enabled by silicon photonics. We describe the system architecture and the WDM point-to-point network implementation of a "macrochip" providing bisection bandwidth of 10 TBps and discuss system and device level challenges, constraints, and the critical technologies needed to implement this system. We present a roadmap to lowering the energy-per-bit of a silicon photonic interconnect and highlight recent advances in silicon photonics under the UNIC program that facilitate implementation of a "macrochip" system made of arrayed chips.

Despite significant advances in silicon photonics, optical packaging and interconnection of photonic integrated circuits constitutes a large portion of the associated manufacturing costs. Optical fibers are the dominant approach to optical interconnection. However, techniques that rely on fibers lack scalability, have low efficiency and show poor misalignment tolerance. Proposed alternative solutions using polymer waveguides have shown improved coupling efficiency, misalignment tolerance and scalability. Here we present the design, simulation and the experimental confirmation of an adiabatic silicon nitride taper to polymer waveguide coupler with only 0.52 dB of TE coupling loss at 1550-nm wavelength, excluding material loss. The coupler demonstrates $\pm 4~\mu \text{m}$ of lateral misalignment tolerance with a 1-dB penalty which, to the best of our knowledge, is the highest reported tolerance for silicon nitride to polymer waveguide coupling.

The transition from pluggable optics to co-packaged optics is having an impact on the photonic integrated circuit design automation. Traditionally, chip designers for fiber-optic systems using electronic design automation tools frequently needed a way to model a few photonic components in the same design environment with the electronics. Examples include modeling lasers or modulators with electrical driver circuits, or modeling receiver electronics with photodetector. These designs focused primarily on optimizing electronics and it was typically sufficient to model a few photonic components through electrically equivalent circuit representations using SPICE or Verilog-A. However, with the advances in silicon photonics, the photonic component count in modern PICs is rapidly growing raising serious concerns over efficacy and reliability of treating photonics as electronics. This paper describes an E-O co-design approach that (i) does not require user intervention on RF and photonic domain demarcations, and (ii) supports schematic-driven layout and back-annotation.

Silicon photonics has emerged as a promising solution to address the interconnect bottleneck in high performance computing (HPC) systems and data centers. Silicon photonics provides unprecedented I/O bandwidth, enabling ultrahigh aggregated bandwidth (~ 10 Tbps), high bandwidth density (~ Tbps/mm), and high energy efficiency (~ pJ/bit). In addition, silicon photonics also enable optical switching with large radix and short switching time. This talk will provide an overview of the state of the art of silicon photonic switches, with emphasis on new micro-electromechanical-system (MEMS)-actuated switching mechanism. Large scale switches with 64x64 and 128x128 port counts have been demonstrated, as well as wavelength-selective switches with 8x8 ports and 8 wavelengths. Future scaling to even larger port count will be discussed.

Telecommunication networks are evolving due to rapid growth of internet traffic and the necessity to satisfy the new requirements of emerging packet services. Upgrading of network devices so as they can allow scalable and full line rate traffic aggregation and eliminate any internal performance bottlenecks is crucial. In packet switching devices, since data plane functionalities need to be executed for each incoming packet and power per given bandwidth is strongly related to the amount of processing on this data, packet processing at ultra-high rate is becoming the major challenge. The main trend to address such power consumption and scalability issues is to bypass packet switches by switching traffic at lower layers. This results in the packet optical transport network approach, where packet switching provides flexible end-to-end connectivity based on tunnel encapsulation while wavelength switching, exploiting optical bypass, allows reducing electrical switch size at transit nodes. There are also opportunities consisting in simplifying packet switch functionalities or designing completely different packet switch architectures. In this paper a new Ethernet aggregation and switching solution with potentialities to simplify and scale Ethernet switch forwarding functionality is proposed. This solution, based on a burst-basis transmission compliant with the Ethernet Standard, is able to maintain flexibility and any to any connectivity deriving from the connectionless nature of Ethernet. At the same time, it provides Ethernet technology with efficient aggregation capabilities allowing to reduce processing of transit traffic. This is allowed thanks to an Ethernet burst structure conceived as a variable number of consecutive frames of the same connection preceded by a proprietary burst control frame carrying information necessary for burst data frames classification. As a result, burst control frames experience the conventional Ethernet switch packet processing while data frames are mapped on the corresponding queue/output port according to the result of control frames classification. The proposed solution also provides the possibility to recognize data frames of the same burst through a proprietary inter-frame gap inserted among them; that allows to dynamically adapt burst size to the available bandwidth at transit nodes in order to limit frame delay and jitter and to support intermediate grooming. These features make it a very competitive approach in the context of packet optical transport being able to support dynamic multi-granular switching. The proposed solution has been validated by estimating its efficiency in terms of energy consumption with respect to a commercial packet switch. The impact of burst transmission on packet delay and jitter across ring and mesh networks has been also evaluated through different sets of simulations.

Chapter 7 - Data Center Architectures and Opportunities for Silicon Photonics

Cooling solutions in an operational data centre: A case study

As advanced packaging continues to drive towards heterogeneous integration, new materials need to be developed to address the many challenges faced in the assembly process. This presentation will focus on innovative materials for advanced packaging, particularly for small form factor modules, including novel flux formulations for flip chip attach that are compatible with underfill and molding compound, an adhesive agent for fluxless soldering with a formic acid atmosphere, solders with various temperatures for hierarchical soldering. The technology drives toward increased functionality, better efficiency, and higher power density; advanced packaging silicon chip/die size is now bigger, with densely packed interconnect, tighter pitch, and smaller bumps. Some of the common issues we observed due to these are warpage-induced open joints from the big and thin die, flux residue removal challenges due to the tight spacings, and thermal-induced reliability issues. Novel flux formulations and adhesive agents for wafer/die bonding are needed, and interactions between the flux material and various dielectric materials on the wafer need to be better studied. The shift towards using semiconductor-grade ultra-low residue no-clean flip-chip fluxes, which eliminates the cleaning process, is one of the solutions for overcoming these challenges. Another possible solution, studied in recent years, is using a near-zero residue fluxless adhesive agent to hold the die in place while reflowing in a formic acid reflow atmosphere. The development of new solders is also a critical enabler for heterogeneous integration. If there are different reflow cycles that require a solder hierarchy, alloys with varying melting temperatures may be needed for SIP. In addition, lower melting-temperature solders can provide better warpage control, especially for thin die and substrate, as well as for heat-sensitive sensors, die, or substrate that require low-temperature processing. Alloys with enhanced reliability are also needed for certain SIP modules. In summary, proper selection and application of interconnect materials for various assembly processes is crucial to ensuring high production yield for the assembly process of advanced packaging and SIP application.

In Data Centers each rack uses a Top-of-Rack (ToR) as a first level switch to connect servers to the aggregation switches. The current paper focuses on a hybrid electrical/optical ToR design, which first, adapts the servers Ethernet traffic to the optical TDMA operation of the core network for supporting optical switching in the Data Center's upper layer and second, it employs optical switching of traffic at the rack level. The proposed ToR architecture is based on an Ethernet switch and FPGA port extensions realizing the required functions to support 20 10Gbps connections, exploit the network routing resources and handle effectively virtual queues.

Recently, Fibre Channel over Ethernet (FCoE) has been gaining attention as a solution of LAN/SAN convergence in Data Centers. In FCoE, it is necessary to introduce Forwarder (FCF) and to replace legacy Ethernet switches with lossless Ethernet switches. Current FCoE, specified by FC-BB-5, has a scalability issue due to concentrating all operations of Userplane and Control-plane into FCF. Although FC-BB-6 project aims to solve the issue, it requires FC/FCoE Data Forwarder (FDF), which raises DC cost. In this paper, we propose a novel FCoE system, called Advanced FCoE (AFCoE), to realize large-scale FCoE system at low-cost. In AFCoE, network edge devices have mechanisms to separate FCoE frames into Userplane frames or Control-plane frames. The User-plane traffic is directly forwarded by legacy Ethernet switches to remote edge devices, while the Control-plane traffic is forwarded to a controlling server and processed there. In addition, reliable data transport technology is introduced for frame loss caused in legacy Ethernet switches. We also show the validation results of a prototype AFCoE system.

Data centers often hold tens to hundreds of thousands of servers in order to offer cloud computing services at scale. Ethernet switching and IP routing have their own advantages and limitations in building data center networks. Recent research, such as PortLand and BCube, has proposed scalable data center network designs. A common feature of these designs is that their addressing and routing are customized to specific topologies. In this paper, we propose a generic addressing, routing and forwarding protocol for data center networks, which works on arbitrarily layered'' network topologies. We first form the network as a multi-rooted tree. Each network node (i.e., hosts and switches) is then assigned one or more locators, and each locator encodes a downward path from the roots to this node. Data center networks often have rich path diversity, so tracking all locators of a destination node will cause switches to have very large forwarding tables. We further use a new forwarding model to reduce the forwarding states. In addition, the multiple-locator mechanism brings built-in support for multi-path routing, load balancing and fault tolerance. Evaluations based on simulations and prototype experiments demonstrate that our proposal achieves our design goals.

Silicon photonic integrated circuits for telecommunication and data centers have been well studied in the past decade, and now most related efforts have been progressing toward commercialization. Scaling up the silicon-on-insulator (SOI)-based device dimensions in order to extend the operation wavelength to the short mid-infrared (MIR) range (2–4 μm) is attracting research interest, owing to the host of potential applications in lab-on-chip sensors, free space communications, and much more. Other material systems and technology platforms, including silicon-on-silicon nitride, germanium-on-silicon, germanium-on-SOI, germanium-on-silicon nitride, sapphire-on-silicon, SiGe alloy-on-silicon, and aluminum nitride-on-insulator are explored as well in order to realize low-loss waveguide devices for different MIR wavelengths. In this paper, we will comprehensively review silicon photonics for MIR applications, with regard to the state-of-the-art achievements from various device demonstrations in different material platforms by various groups. We will then introduce in detail of our institute’s research and development efforts on the MIR photonic platforms as one case study. Meanwhile, we will discuss the integration schemes along with remaining challenges in devices (e.g., light source) and integration. A few application-oriented examples will be examined to illustrate the issues needing a critical solution toward the final production path (e.g., gas sensors). Finally, we will provide our assessment of the outlook of potential future research topics and engineering challenges along with opportunities.

Silicon photonics has drawn increasing attention in the past few decades and is a promising key technology for future daily applications due to its various merits including ultra-low cost, high integration density owing to the high refractive index of silicon, and compatibility with current semiconductor fabrication process. Optical interconnects is an important issue in silicon photonic integrated circuits for transmitting light, and fiber-to-chip optical interconnects is vital in application scenarios such as data centers and optical transmission systems. There are mainly two categories of fiber-to-chip optical coupling: off-plane coupling and in-plane coupling. Grating couplers work under the former category, while edge couplers function as in-plane coupling. In this paper, we mainly focus on edge couplers in silicon photonic integrated circuits. We deliver an introduction to the research background, operation mechanisms, and design principles of silicon photonic edge couplers. The state-of-the-art of edge couplers is reviewed according to the different structural configurations of the device, while identifying the performance, fabrication feasibility, and applications. In addition, a brief comparison between edge couplers and grating couplers is conducted. Packaging issues are also discussed, and several prospective techniques for further improvements of edge couplers are proposed.

Integrated Mach-Zehnder (MZ) modulators are important components in silicon photonic devices that rely on a reverse-biased p-n junction to modulate the optical signal via a change of the waveguide refractive index [1]. Reducing their energy consumption is a crucial step towards the application of silicon photonics, especially in connection with the growing traffic volumes in data centers. In this work, we combine band-edge slow light structures consisting of silicon grating waveguides [2] with periodic (interlaced) p-n junctions [3,4] to maximize spatial matching between the optical field mode and the depletion region of the p-n junction. The two effects together will result in an improved modulation efficiency, leading to a strongly reduced energy dissipation per bit.