Optical Interconnects in Next Generation Data Centers: An End to End View

Inside a data center many thousands of powerful servers need to communicate with high speed and low latency posing challenging requirements to the intra-data centers networking. Moreover, scalability, low footprint, high-energy efficiency and low cost are key factors for any new architecture and technology that want to be a candidate for use in data centers. While optical interconnect has already received a large consensus as the best technology for inter- and intra-rack interconnect, switching is today entirely implemented by electrical packet switching devices. This paper aims to review results of the European project IRIS (http://www.ict-iris.eu/), i.e. the design and the achieved results on a CMOS electronic and photonic integrated device for low cost, low power, transparent, mass-manufacturable optical switching. An unprecedented number of integrated photonic components (more than 1000), each individually electronically controlled, allows for the realization of a transponder aggregator device which interconnects up to 8 transponders to a four direction colorless-directionless-contentionless ROADM. Each direction supports twelve 200 GHz spaced wavelengths, which can be independently added or dropped from the network. An electronic ASIC, 3D integrated on top of the photonic chip, controls the switch fabrics to allow a complete and us fast reconfigurability.

As the infrastructure of cloud computing and big data, data centers have been deployed widely. Then, how to make full use of the computing and storage resources in data centers will be the focus. Data center networks are considered important solution for the problem above, which include intra-data center and inter-data center networks. Both of them will depend on the optical networking due to its advantages, such as high bandwidth, low latency, and low energy consumption. Data center interconnected by flexi-grid optical networks is a promising scenario to allocate spectral resources for applications in a dynamic, tunable and efficient control manner. Compared with inter-data center, optical interconnect in intra-data center networks is a more pressing need and promising scenario to accommodate these applications in a dynamic, flexible and efficient manner. OpenFlow based Software Defined Networking (SDN) is considered as a good technology, which is very suitable for data center networks. This paper mainly focuses on the data center optical networks based on software defined networking (SDN), which can control the heterogeneous networks with unified resource interface. Architecture and experimental demonstration of OpenFlow-based optical interconnects in intra-data center Networks and OpenFlow-based flexi-grid optical networks for inter-data center are presented in the paper respectively. Some future works are listed finally.

We are delighted to introduce a Special Issue on Optical Interconnects for the IET Optoelectronics magazine. This issue presents the current status as well as some of the outstanding progress that has been witnessed in the rapidly evolving area of optical interconnects. Today, data centres (DC) and high-performance computers are struggling with limited interconnect capacity and increased power consumption while trying to meet overwhelming bandwidth demands. As photonics technologies have been rapidly maturing, optical interconnects have come into the spotlight as the means to cope with the emerging bandwidth, energy, and capacity needs. As such, a wealth of photonic interconnect solutions across the DC hierarchy has unfolded, covering from electro-optic boards to transceiver development, on-chip and on-board switches and interconnects as well as novel noninvasive monitoring methods for light controlling for compensating thermal crosstalk, fabrication tolerances and unstable environmental conditions. At the same time, the increased penetration of optical interconnects in short-reach communications, such as board and chip levels, has triggered new challenges in photonics technologies calling for new innovations with respect to integration techniques and packaging methods. This Special Issue of IET includes integrated circuits, assembly techniques, systems-level architectures, and controlling methods related to short reach ‘chip-to-chip’, ‘in-the-box’, and ‘box-to-box’ communications. Specific areas covered by this special issue include: Silicon photonic transceiver technologies and flat DC topologies; High-speed electro-optic hosting boards and fast, energy-efficient electronics; DC switches and co-packaged optic (CPO) technologies; and Noninvasive controlling techniques for complex photonic integrated circuits. The four papers in this special section promote high-performance optical interconnect architectures and key technologies to tackle the rising bandwidth demands of DCs. They cover (a) co-packaged DC optics, (b) silicon technology transceiver modules and high-speed hosting boards for multisocket server board (MSB) interconnects and (c) automated controlling architecture for complex photonic systems; Items (b) and (c) draw from impressive achievements accomplished through the recently completed European H2020 Project ICT-STREAMS. This report written by C. Mickenberg tries to shed light on motivating factors behind co-packaged optics (CPO), the trade-offs involved and their potential impact on data centre network architectures. The article reviews the evolution of DC switch application-specific integrated circuit (ASICs) and DC optics, provides a future road map and finally discusses the potential impact of CPO on the network architecture. This topic is covered by two articles. The first, written by M. Morallis-Pegios, proposes using silicon photonics to advance interconnect throughput, socket connectivity and energy efficiency in MSB layouts by enabling a flat-topology wavelength division multiplexing (WDM) point-to-point optical MSB interconnect scheme. WDM silicon photonic transceivers coassembled with their electronic circuits achieving a 50-Gb/s line rate and 400-Gb/s aggregate data transmission are demonstrated. The article also proposes silicon photonic Arrayed Waveguide Grating Router (AWGRs) that can offer collision-less time-of-flight connectivity for up to 16 nodes. The second article covering that area is written by T. Lamprecht and reports on the development of electro-optic hosting boards for on-board transceiver subassemblies which is required to eventually form the Tb/s multi-socket onboard communication. The article also discusses the electrical interface as well as photonic board requirements, including the manufacturing technique and the assembly concept that was followed to deliver terabyte per second multichip optical communication. This article, written by F. Zanetto, introduces a novel dithering technique for real-time control of multiple cascaded photonic integrated devices, proposing a calibration-free strategy to support the development and real-life diffusion of increasingly complex photonic systems. The article elaborates on the implementation of a calibration-free automated control loop exploiting integral controllers to close a real-time feedback and stabilise the optimum working point of the employed photonic devices. The effectiveness of the proposed architectural approach is experimentally validated with the error-free routing and transmission of a 30-Gbit/s optical signal traversing a cascade of micro-ring resonator filters. All of the articles selected for this Special Issue present a wide variety of key topics and significant results addressing optical interconnects in high-performance data communication systems, including new progress on architectures, systems and packaging techniques as well as novel optimised controlling methods. As guest editors, we would like to thank all of the authors for their high-quality submissions that significantly contribute to the optical interconnect field. We would also like to thank the reviewers for their valuable and constructive technical comments that greatly enhance the Special Issue, because their feedback and comments ensure the high quality of the IET Optoelectronics Journal. We hope that this IET Special Issue will provide a useful reference of the state-of-the-art in optical interconnects to readers, and we anticipate such technological perspectives will stimulate new advancements and technological outcomes that will in turn enrich the growing field of optical interconnects. This work was supported by the E.C. through the H2020-ICT-STREAMS research project (688172).

1 - Data center architectures

Data center (DC) and high performance computing (HPC) applications have traditionally used a combination of copper, multimode fiber and single-mode fiber interconnects with relative percentages that depend on factors such as the line rate, reach and connectivity costs. The balance between these transmission media has increasingly shifted towards optical fiber due to the reach constraints of copper at data rates of 10 Gb/s and higher. The percentage of single-mode fiber deployed in the DC has also grown slightly since 2014, coinciding with the emergence of mega DCs with extended distance needs beyond 100 m. This trend will likely continue in the next few years as DCs expand their capacity from 100G to 400G, increase the physical size of their facilities and begin to utilize silicon-photonics transceiver technology. However there is a still a need for the low-cost and high-density connectivity, and this is sustaining the deployment of multimode fiber for links ≤ 100 m. In this paper, we discuss options for single-mode and multimode fibers in DCs and HPCs and introduce a reduced diameter multimode fiber concept which provides intra-and inter-rack connectivity as well as compatibility with silicon-photonic transceivers operating at 1310 nm. We also discuss the trade-offs between single-mode fiber attributes such as bend-insensitivity, attenuation and mode field diameter and their roles in capacity and connectivity in data centers.

Today, there is a generalized standard usage of internet for all.The devices via multiple technologies that facilitates to provide few communication methods to scholars to work with. By forming multiple paths in the data center network, latest generation data centers offer maximum bandwidth with robustness. To utilize this bandwidth, it is necessary that different data flows take separate paths. In brief, a single-path transport seems inappropriate for such networks. By using Multipath TCP, we must reconsider data center networks, with a diverse approach as to the association between topology, transport protocols, routing. Multipath TCP allows certain topologies that single path TCP cannot use. In newer generation data centers, Multipath TCP is already deployable using extensively deployed technologies such as Equal-cost multipath routing. But, major benefits will come when data centers are specifically designed for multipath transports. Due to manifold of technologies like Cloud computing, social networking, and information networks there is a need to deploy the number of large data centers. While Transport Control Protocol is the leading Layer-3 transport protocol in data center networks, the operating conditions like high bandwidth, small-buffered switches, and traffic patterns causes poor performance of TCP. Data Center TCP algorithm has newly been anticipated as a TCP option for data centers which address these limitations. It is worth noting that traditional TCP protocol.

Spectrum engineering in flexible grid data center optical networks

The advantage of Smart Grid and ICT communication brings up the power reliability achieved 99.999%. Data Center availability is critical concern of business continuous and growth in Internet service competitions. Data Center Tier IV requirement is design to accomplish 99.995% available operations. However, this legacy design for availability is coming up with high investment of redundant equipments and connectivity. Understanding of Data Center power distribution system (DCPDS) cascading failures lead to optimal design equipments and connectivity topology. To take advantage of Smart Grid to reduce investment of Data Center, this paper proposes a cognitive approach model of Next Generation Data Center (NGDC) design under Smart Grid ecosystem. Load shedding technique is presented in this research; it is applied to different parts of the power distribution system to protect the critical application servers by shutting down certain pre-arranged electric loads or IT equipments when the Data Center demand response exceeded power supply capacity.

In order to serve the explosive demands of triple play services or multimedia services with high quality of data transmission in Fifth Generation (5G) communication systems which has availability of large spectrum unlike Fourth Generation (4G), low latency, improved transmission, and increased reliability, an access networks with optical fiber distribution network and wireless enabled technology is urgently needed and therefore, a prominent technology known as Radio over Fiber (RoF) is suggested by researcher/Scientists due to a perfect blend of flexible fiber technology and wireless mobility. RoF technology is helping the data transmission systems from last thirty years with great success and has numerous advantages such as scalability, transparency, antenna site complexity reduction, low installation cost, accessibility of radio carrier by diverse antenna sites dynamically. Millimeter waves (Mm waves) integration in 5G wireless RoF systems comes out to be an encouraging candidate to cater the bandwidth hungry current as well as future generation internet application such as high definition TV (HDTV), online games, video conferencing, and voice over IP (VoIP) etc. For the prolonged reach, economical installations, and improved performance, optimal Mm wave generation methods should be incorporated in 5G such as employing multi carrier generation technique (MCG). Deployment of single mode fiber (SMF) in 5G offers high speed, large capacity, improved security, etc., but also has a prominent limitation of nonlinear effects and it is unwanted phenomenon which degrade the system performance by introducing intermodulation distortion, distortion of signal phase, harmonic distortion, and interference in the neighbouring channels. Minimum modifications of architecture and minimum cost should be spent on the compensation of nonlinear effects or Kerr’s effects in RoF systems and recently mode division multiplexing has emerged as an ultimate solution. In this review article, detailed discussion of an evolutionary technology, i.e., RoF is presented with a literature review, present evaluations, and future challenges. Our utmost goal is to enrich the young researchers with precious aspects for RoF technology on single platform and also encourage them to conduct the technology advancement research works in various fields such as nonlinear compensation, Mm wave generation, dispersion compensation, cost reduction, complexity reduction, and security enhancements for 5G RoF systems.

The exponential increase of Internet traffic due to cloud computing and other emerging web applications has created the need for more powerful data centres. These warehouse-scale data centres consist of thousands of racks interconnected with commodity switches and consume vast amount of energy. To face the requirements of the future data centres, new interconnection schemes have to be adopted to provide high throughput and reduced power consumption. Several optical interconnects for future data centre networks have recently appeared that provide high throughput, low latency and reduced power consumption. This paper discusses the need for a shift of the data centres networks to the optical domain to reduce the power consumption and meet the bandwidth requirements. Additionally, this paper presents some indicative optical interconnects for high performance data centres that have appeared recently in the research literature. Finally, it presents a qualitative categorization and comparison of these schemes based on their main features such as performance, connectivity and scalability.

Growing concern for reduced power dissipation, cost and latency demands in next generation Data Centers (DC) motivates us to revisit header optimizations. Headers contribute to about 30–40% of DC traffic and is responsible for equal proportion of consumed power. This amounts to significant overhead on per byte transfer of payload. In the past, highly inflexible switches have limited the focus of header optimizations primarily on end-host or edge network routers. Further, strict compliance to protocols by switches/routers along the path have limited the segments of headers that can be optimized. Recent evolution of Software Defined Network (SDN) brings in several opportunities to control over switches. We believe that if SDN's finer handle to the switching devices is exploited, it can help deal with header information. In this paper, we exploit the capabilities of SDN and introduce a new functionality that can effectively replace the redundant and repetitive header information with shorter unique identity. We present Scissor that trims the headers lower in the protocol stack. As a replacement for routing, we introduce the notion of Flow-ID, where all packets belonging to a flow are identified using this unique Flow-ID. We leverage the capabilities of the SDN to dynamically allow switching devices to route the packets based on the Flow-IDs. Our approach of trimming header at the switching devices leaves the hosts unmodified making it highly adoptable for DC environment. We show that our approach can decrease switch fabric power consumption by a factor of 2.5 compared to an existing L2 switch and reduce the latency by about 30% for a significant fraction (around 30–50%) of the network traffic.

Due to widespread applications and rapid development of cloud computing and Internet services, hyperscale data centers have experienced an unprecedented demand for networking. Data center networks are required not only to handle fast-growing traffic, which doubles almost every one to two years, but also to provide high flexibility and availability to support rapidly changing businesses as well. Therefore, hyperscale data center networks have become one of the main drivers for optical interconnect technologies in recent years. In this article, technologies for scaling optical interconnects for inter-data center interconnects (DCIs) are presented. We first describe hyperscale data center network architectures and requirements, as well as the differences between carrier’s optical transport networks and DCI optical networks, with the main focus on metro-DCI and campus-DCI networks. The scale and fast growth rates of DCI networks require innovations not only in data plane technologies but also in control and management planes as well. In data planes, high-capacity flexible coherent technology, pluggable wavelength-division-multiplexing (WDM) optical transponders, including 100G direct-detection four-level pulse-amplitude modulation (PAM4) and coherent 400ZR/800ZR, and flex-grid and disaggregated reconfigurable optical add-drop multiplexer (ROADM) optical networks are described. Control and management planes are an integral part of optical transport networks and crucial for effectively operating DCI networks. We show that standard protocols, data models, and modular design of a software platform are essential to building a scalable open and disaggregated optical network, which is fundamental to enabling high automation and intelligence in an optical network. An example control and management platform is discussed in detail. With current fiber capacity approaching the nonlinear Shannon limit, challenges to further scale DCI networks and some potential technologies to support ever-increasing traffic growth are discussed at the end.

Cloud computing have created the need for more powerful data centers that can face the growing data traffic. These data centers need high bandwidth interconnects that can sustain the high network traffic inside the data center network. Optical interconnects have gained attention recently as a promising energy efficient solution offering high throughput, low latency and reduced energy consumption compared to current networks based on commodity switches. In this paper, we examine the energy consumption of a novel data center network architecture based on cyclic arrayed waveguide grating device and multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) technology. The proposed architecture is compared with a reference design based on commodity electrical switches and it is shown that the proposed scheme can provide up to 25% lower power consumption.

The rise of Cloud Computing and Big Data has created the need for morepowerful warehouse-scale cloud data centers, which usually comprise hundreds of thousands ofservers with highaccess bandwidth (>10Gb/s) or more, and cost-effective intra- and inter- data center networks(DCNs) to provide switching and interconnect among all servers. However, the current DCNs sufferfrom limited throughput, large latencies and highpower consumption, and will not be able to sustainthe projected future DCN traffic demandswithout consuming excessive amounts of power. Optical interconnection networks capable of supporting high throughput, high bandwidth, low latency, low power consumptionand full-bandwidth communication among all servers have been proposedas a promising solution by performing switchingat the optical domain, thus greatly improving the performance of the cloud data centers.

There is an important class of applications in data centre called online dataintensive (OLDI) application, such as search engine, electronic shopping, and advertising, which has low latency, high throughput, and high burstiness. For better user experience, the query response delay in the data centre is controlled under 300 milliseconds. DCTCP can better meet the requirements of OLDI, but all the switches in the data centre are required to support RED. For data centre that does not fully support RED switch, TCP protocol can only be used. TCP Incast will appear in the application, which is far from the requirements of OLDI application. This paper proposes the TCP protocol of data centre based on window control. It is in existing switches and network devices,and centralized control makes the send window match with the switch queue cache. In the case of ensuring low latency, it keeps the switch queue in a small range of fluctuation, enabling the data centre to meet low latency, high throughput and high burstiness application requirements. Simulation experiments verify that WCTCP congestion control method can effectively solve the above problems. It is 5 times lower than TCPNewReno packet loss rate, 6 times higher than the network utilization rate, and the network performance is close to the DCTCP protocol.