EDM: An Ultra-Low Latency Ethernet Fabric for Memory Disaggregation

Recent advances in networking hardware have led to a new generation of Remote Memory Access (RMA) networks in which processors from different machines can communicate directly, bypassing the operating system and allowing higher performance. Researchers and practitioners have proposed libraries and programming models for RMA to enable the development of applications running on these networks, However, the memory models implied by these RMA libraries and languages are often loosely specified, poorly understood, and differ depending on the underlying network architecture and other factors. Hence, it is difficult to precisely reason about the semantics of RMA programs or how changes in the network architecture affect them. We address this problem with the following contributions: (i) a coreRMA language which serves as a common foundation, formalizing the essential characteristics of RMA programming; (ii) complete axiomatic semantics for that language; (iii) integration of our semantics with an existing constraint solver, enabling us to exhaustively generate coreRMA programs (litmus tests) up to a specified bound and check whether the tests satisfy their specification; and (iv) extensive validation of our semantics on real-world RMA systems. We generated and ran 7441 litmus tests using each of the low-level RMA network APIs: DMAPP, VPI Verbs, and Portals 4. Our results confirmed that our model successfully captures behaviors exhibited by these networks. Moreover, we found RMA programs that behave inconsistently with existing documentation, confirmed by network experts. Our work provides an important step towards understanding existing RMA networks, thus influencing the design of future RMA interfaces and hardware.

Interconnection networks usually consist of a fabric of interconnected routers, which receive packets arriving at their input ports and forward them to appropriate output ports. Unfortunately, network packets moving through these routers are often delayed due to conflicting demand for resources, such as output ports or buffer space. Hence, routers typically employ arbiters that resolve conflicting resource demands to maximize the number of matches between packets waiting at input ports and free output ports. Efficient design and implementation of the algorithm running on these arbiters is critical to maximize network performance.This paper proposes a new arbitration algorithm called SPAA (Simple Pipelined Arbitration Algorithm), which is implemented in the Alpha 21364 processor's on-chip router pipeline. Simulation results show that SPAA significantly outperforms two earlier well-known arbitration algorithms: PIM (Parallel Iterative Matching) and WFA (Wave-Front Arbiter) implemented in the SGI Spider switch. SPAA outperforms PIM and WFA because SPAA exhibits matching capabilities similar to PIM and WFA under realistic conditions when many output ports are busy, incurs fewer clock cycles to perform the arbitration, and can be pipelined effectively. Additionally, we propose a new prioritization policy called the Rotary Rule, which prevents the network's adverse performance degradation from saturation at high network loads by prioritizing packets already in the network over new packets generated by caches or memory.

Interconnection networks usually consist of a fabric of interconnected routers, which receive packets arriving at their input ports and forward them to appropriate output ports. Unfortunately, network packets moving through these routers are often delayed due to conflicting demand for resources, such as output ports or buffer space. Hence, routers typically employ arbiters that resolve conflicting resource demands to maximize the number of matches between packets waiting at input ports and free output ports. Efficient design and implementation of the algorithm running on these arbiters is critical to maximize network performance.This paper proposes a new arbitration algorithm called SPAA (Simple Pipelined Arbitration Algorithm), which is implemented in the Alpha 21364 processor's on-chip router pipeline. Simulation results show that SPAA significantly outperforms two earlier well-known arbitration algorithms: PIM (Parallel Iterative Matching) and WFA (Wave-Front Arbiter) implemented in the SGI Spider switch. SPAA outperforms PIM and WFA because SPAA exhibits matching capabilities similar to PIM and WFA under realistic conditions when many output ports are busy, incurs fewer clock cycles to perform the arbitration, and can be pipelined effectively. Additionally, we propose a new prioritization policy called the Rotary Rule, which prevents the network's adverse performance degradation from saturation at high network loads by prioritizing packets already in the network over new packets generated by caches or memory.

On contemporary cache-coherent Non-Uniform Memory Access (ccNUMA) architectures, applications with a large memory footprint suffer from the cost of the garbage collector (GC), because, as the GC scans the reference graph, it makes many remote memory accesses, saturating the interconnect between memory nodes. We address this problem with NumaGiC, a GC with a mostly-distributed design. In order to maximise memory access locality during collection, a GC thread avoids accessing a different memory node, instead notifying a remote GC thread with a message; nonetheless, NumaGiC avoids the drawbacks of a pure distributed design, which tends to decrease parallelism. We compare NumaGiC with Parallel Scavenge and NAPS on two different ccNUMA architectures running on the Hotspot Java Virtual Machine of OpenJDK 7. On Spark and Neo4j, two industry-strength analytics applications, with heap sizes ranging from 160GB to 350GB, and on SPECjbb2013 and SPECjbb2005, ourgc improves overall performance by up to 45% over NAPS (up to 94% over Parallel Scavenge), and increases the performance of the collector itself by up to 3.6x over NAPS (up to 5.4x over Parallel Scavenge).

Remote Direct Memory Access (RDMA)-based communication has enjoyed considerable growth since the introduction of the Virtual Interface Architecture (VIA) in the late 1990s. This growth has accelerated since the introduction of the Open Fabric Alliance’s (OFA) Verbs interface. The stability of this interface and its independence from the underlying physical network has facilitated significant growth of software applications that exploit the benefits of RDMA. Until recently, a user wishing to utilize RDMA could use either the InfiniBand Architecture, which required the use of the unique link and physical layers defined by the InfiniBand Architecture, or the RDDP protocols, designed to run over an IP network. Recent developments in Ethernet have expanded the available options for providing RDMA services to applications while using Ethernet link and physical layers. Specifically, there is a new proposal for adapting the InfiniBand Architecture transport to run over Converged Enhanced Ethernet (CEE). In this paper we describe this proposal and compare it to two existing methods for delivering RDMA services to an application using existing IP-based transports.

As part of a new MPI Remote Memory Access (RMA) implementation over remote direct memory access, we propose a message scheduling scheme that interleaves inter-node and intra-node data transfers in a way that minimizes the overall latency of the RMA epoch. By doing so, we fulfill inter-node communication/intra-node communication overlapping and show that further latency mitigation is possible when computation length and communication payload sizes are too unbalanced to allow a satisfying level of communication/computation overlapping. Test results show that the proposed scheduling scheme compares favorably against the approaches used in MVAPICH and Open MPI.

We present a novel architecture of a communication engine for non-coherent distributed shared memory systems. The shared memory is composed by a set of nodes exporting their memory. Remote memory access is possible by forwarding local load or store transactions to remote nodes. No software layers are involved in a remote access, neither on origin or target side: a user level process can directly access remote locations without any kind of software involvement. We have implemented the architecture as an FPGA-based prototype in order to demonstrate the functionality of the complete system. This prototype also allows real world measurements in order to show the performance potential of this architecture, in particular for fine grain memory accesses like they are typically used for synchronization tasks.

It is well known that contention is one of the factors that limit the performance of high performance parallel computing systems that implement distributed shared memory (DSM). On cache-coherent DSM multiprocessors, one of the most intrusive forms of contention is the one that occurs at the network interface of a node, due to simultaneous requests for remote memory accesses. This thesis focuses on the issue of resolving remote memory access contention on hardware DSM systems, and on the performance impact of implementing contention resolution algorithms. The Simultaneous Optical Multiprocessor Exchange Bus (SOME-Bus) is a low-latency, high-bandwidth, fiber-optic interconnection network supporting DSM with a unique feature that every processor is connected to the other processor through a dedicated output channel. The thesis presents simple but effective contention resolution algorithms that rely on the information of average channel utilization, average channel waiting time, number of messages in the channel queue or short-term average channel waiting time reported by each node, and uses simulation to examine the performance of the algorithms on the SOMEBus multiprocessor. The algorithms detect potential hot spots and resolve contention using dynamic block migration, which is a novel approach for migrating home nodes on a block-level basis and balancing remote memory accesses across the nodes of the system. The algorithms require minimal support from the operating system and all the necessary message transactions happen on the hardware level. Experiments with eight parallel codes on a 64-processor SOME-Bus show that algorithms yield significant performance improvements such as reduction in the execution times, number of remote memory accesses, average channel waiting times and average network latencies. Modified contention resolution algorithms are developed with the addition of data acknowledge message multicasting. Application of the modified algorithms yields more performance improvements in some codes. Architectural enhancements to the network interface and directory of the SOME-Bus multiprocessor to support the contention resolution algorithms are also presented.

Applying multimedia to Wireless Sensor Network (WSN) adds more challenges due to WSN resource constraints and the strict Quality of Service (QoS) requirements for multimedia transmission. Different multimedia applications may have different QoS requirements, so routing protocols designed for Wireless Multimedia Sensor Network (WMSN) should be conversant of these requirements and challenges in order to ensure the efficient use of resources to transfer multimedia packets in an utmost manner. The majority of solutions proposed for WMSN depends on traditional layered based mechanisms which are inefficient for multimedia transmission. In this paper we propose a cross-layer Ant based Routing Protocol for WMSN (CARPM) by using modified ant colony optimization (ACO) technique to enhance the routing efficiency. The proposed protocol uses an improved ACO to search for the best path that are satisfied with the multimedia traffic requirements. While making best decision the weightage is given to energy consumption, and queuing delay. The proposed cross layer scheme works between the routing, MAC, and physical layers. Since, the remaining power and timestamp metrics are exchanged from physical layer to network layer. Dynamics duty cycle assignment is proposed at MAC layer which changes according to traffic rate. The presented algorithm is simulated using NS2 and is proven to satisfy its goals through a series of simulations.

Remote Memory Access (RMA) programming is one of the core concepts behind modern parallel programming languages such as UPC and Fortran 2008 or high-performance libraries such as MPI-3 One Sided or SHMEM. Many applications have to communicate non-contiguous data due to their data layout in main memory. Previous studies showed that such non-contiguous transfers can reduce communication performance by up to an order of magnitude. In this work, we demonstrate a simple scheme for statically optimizing non-contiguous RMA transfers by combining partial packing, communication overlap, and remote access pipelining. We determine accurate performance models for the various operations to find near-optimal pipeline parameters. The proposed approach is applicable to all RMA languages and does not depend on the availability of special hardware features such as scatter-gather lists or strided copies. We show that our proposed superpipelining leads to significant improvements compared to either full packing or sending each contiguous segment individually. We outline how our approach can be used to optimize non-contiguous data transfers in PGAS programs automatically. We observed a 37 % performance gain over the fastest of either packing or individual sending for a realistic application.

The ever increasing demand for communication bandwidth has led to spectrum scarcity. We need to manage spectrum as a scare resource. Better utilization of spectrum is the key requirement for the radio communication systems. The inefficient usage of the existing spectrum can be improved through opportunistic access to the licensed bands without interfering with the primary users. This introduces the concept of dynamic spectrum access. Cognitive radio is must for providing quality of service in dynamic spectrum access. To deal with the cognition at physical layer, upper layers are also modified to support cognition and adaptation which make a network cognitive network. This book chapter describes the technical challenges and approaches for cognitive radio networks. Spectrum access models from regulatory point of view are presented. It also explains the challenges and solutions from literature for physical, MAC and network layer of cognitive networks. In the end of the chapter, real hope for designing protocols for such networks, i.e. cross layer protocol design is discussed.

Optimizing communication is essential for high-performance computing because synchronization bottlenecks inhibit the overall performance and scalability of parallel applications. Today's cutting-edge computing hardware, as well as networking interfaces like Cray Aries/Gemini, features extremely low latency and high bandwidth remote memory access (RMA) operations for optimized data movement. However for any efficient data movement to occur between two logical processing units, software substrates must be able to properly exploit hardware resources for the underlying fabric. Overheads due to coarse granular synchronization and stalls during irregular access of remote memory regions may hint at two adverse effects of resource under-utilization in time and space. We introduce a uGNI-based distributed remote memory allocator called rmalloc which expands RDMA-enabled memory utilization, and a communication substrate called rpipe that tries to mitigate synchronization bottlenecks. Our UNIX-inspired RMA programming model is simple to use and equally applicable to both higher-level applications as well as lower-level runtime systems for enabling efficient data movement. Our micro-benchmark results suggest that rmalloc default next-fit allocator outperforms MPI-3.0 RMA by 1.5X and up to 6X in most cases, while other variants of rmalloc (i.e. best-fit, worst-fit) reduce external fragmentation and perform comparably or better than the default rmalloc allocator for irregular RMA.

We describe new methods for multi-hop mobile wireless network design, so as to meet performance specifications. We introduce an implicitly defined approximate loss model that couples the physical, MAC and routing layers' effects. The model provides quantitative statistical relations between the loss parameters that are used to characterize multiuser interference and physical path conditions on the one hand and the traffic rates and types between origin-destination pairs on the other. The model considers effects of hidden nodes, node scheduling algorithms, MAC and PHY layer failures and unsuccessful packet transmission attempts at the MAC layer in arbitrary network topologies where multiple paths share nodes. We consider both contention-based as well as scheduling-based MAC protocols. The new methods are based on implicit loss network models developed using fixed point methods. We describe the application of Automatic Differentiation (AD) to these implicit performance models, and develop a methodology for sensitivity analysis and parameter optimization for wireless protocols. We also introduce new methods that utilize on-line feedback from the PHY layer to simplify the joint MAC and routing protocol design. We provide several examples, with realistic scenarios, that demonstrate the benefits of applying these new methods in an integrated manner. We close with current and future work that extends these methods and algorithms towards a Component-Based design, performance analysis and validation of distributed protocols for mobile wireless networks. We argue that this set of ideas provides a very promising approach to 'clean slate' design for such networks and the resulting hybrid broadband Internet.

Multi-rate capabilities are supported by the physical layers of most 802.11 devices. To enhance the network throughput of MANETs, transfer rate adaptation at MAC layer should employ the multi-rate capability at physical and the information of previous transmissions provided by MAC and physical layers. In this paper, we propose a transfer rate adaptation scheme plus back-to-back frame transmissions, and fragmentation at MAC layer, named TRAF. TRAF adopts a bi-direction-based approach with an extended option to select an appropriate rate for frame transmission under fast changing channel conditions. Consecutive back-to-back frame transmissions to fully utilize good channel quality during a coherent time interval and fragmentation algorithm to maintain high throughput under worse channel conditions are recommended in TRAF. Extensive simulation is experimented to evaluate the performance of TRAF. Regarding simulation results, frame delivery ratio, and network throughput of TRAF are significantly improved by comparing to that of other protocols.

We concern parallel computations with communication based on remote direct memory access (RDMA), which provides low level un-buffered access to distributed memory of computational nodes. Fine grain computations involve very frequent transmissions of small messages. For their efficient execution with RDMA communication a special memory infrastructure - rotating buffers (RB) - is introduced. Their organization is adjusted to program needs in advance - before program execution. It allows an intensive use of all communication resources available in the system based on additional synchronization between involved processes. The proposed method is illustrated by an example of a typical fine-grain problem which is the discrete fast Fourier Transform (FFT). "The transpose algorithm" of FFT has been implemented with the RDMA rotating buffers and its efficiency is compared with a solution based on standard message passing library MPI.