Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)

Remote Direct Memory Access (RDMA) is a technology to update a remote machine's memory without intervention at the receiver side. We evaluate where RDMA can be usefully applied and where it is a loss in Object-Oriented DSM systems. RDMA is difficult to use in modern OO-DSMs due to their support for large address spaces, advanced protocols, and heterogeneity. First, a communication pattern that is based on objects reduces the applicability of bulk RDMA. Second, large address spaces (meaning far larger than that of a single machine) and large numbers of machines require an address space translation scheme to map an object at different addresses on different machines. Finally, RDMA usage is hard since without polling (which would require source code modifications), incoming RDMA messages are hard to notice on time. Our results show that even with RDMA, update protocols are slower than invalidation protocols. But RDMA can be successfully applied to fetching of objects in an invalidation protocol and improves performance by 20.6%.

We develop a hybrid TCP/IP offload engine (hybrid TOE) that processes TCP/IP via hardware/software coprocessing based on an FPGA and a general-purpose embedded processor. We also develop an efficient remote direct memory access (RDMA) mechanism that is tightly coupled with the hybrid TOE. In this mechanism, the hybrid TOE performs CRC calculations using hardware modules and supports zero-copy data transmission; the host CPU simply generates and processes RDMA protocol headers. By using the hybrid TOE and the RDMA mechanism, computer systems can achieve good network performance with very low CPU utilizations, and thus they can be expected to show a great improvement in overall performance. In experiments on a gigabit Ethernet network, although the embedded processor operated with a 300 MHz core clock, which was one-seventh the speed of the host CPU’s clock, the hybrid TOE showed a minimum latency of 17.4 μs and a maximum bandwidth of 736 Mbps. The RDMA mechanism exhibited a minimum latency of 20.6 μs and a maximum bandwidth of 642 Mbps. Most importantly, the hybrid TOE and the TOE-based RDMA mechanism gave CPU utilizations of less than 5.6% and 8.4%, respectively—approximately one-tenth the utilizations when TCP/IP and TCP/IP-based RDMA were processed by the host CPU.

Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA) provides the RDMA data transfer capability to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol. An RDMA-Capable Protocol provides RDMA Read and Write services, which enable data to be transferred directly into SCSI I/O Buffers without intermediate data copies. This document describes the extensions to the iSCSI protocol to support RDMA services as provided by an RDMA- Capable Protocol. This document obsoletes RFC 5046.

Remote Direct Memory Access (RDMA) has been widely deployed in datacenters for its high performance. Large-scale high performance cloud services built on geographically distributed datacenters require long-range RDMA for performance requirements. However, existing RDMA solutions can hardly satisfy the stringent requirements of the emerging large-scale high-performance cloud services built on geo-distributed datacenters in terms of throughput and delay. On the one hand, lossless RDMA suffers from a deep buffer and potential suboptimal throughput for inter-datacenter traffic due to delayed response to Priority Flow Control (PFC) messages. On the other hand, lossy RDMA with selective retransmissions suffers from poor performance when multiple flows with different round-trip times (RTTs) coexist in cross-datacenter scenarios. This article proposes <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Swing</small> , which expands the high-performance lossless RDMA to long-distance links through PFC-Relay. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Swing</small> ensures the throughput of long-distance links while minimizing the buffer requirement for long-range RDMA. It enables long-range RDMA without making any modifications to existing in-datacenter networks. The evaluation shows that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Swing</small> can reduce the average flow completion time (FCT) by 14%-66% in a variety of traffic scenarios.

RXIO: Design and implementation of high performance RDMA-capable GridFTP

Remote Direct Memory Access (RDMA) devices are widely deployed in modern data centers. However, existing RDMA usages lead to a dilemma between performance and redesign cost. Server-reply mode directly replaces socket-based send/receive primitives with corresponding RDMA counterparts. It can not fully harness the power of RDMA devices. Server-bypass is distinct mode provided by RDMA for reaching the hardware limits. This mode can totally bypass the server by using one-sided RDMA operations but at the cost of redesigning software. This paper provides a different approach, called RF-RPC (Remote Fetching RPC Paradigm). Different from server-reply, RF-RPC makes client fetch the results from server using one-sided RDMA instead of waiting the results pushed by server. Different from server-bypass, RF-RPC demands server to process client's requests for supporting programming paradigm like RPC. Thus, RF-RPC can achieve high performance without abandoning traditional programming models. An RF-RPC supported in-memory key-value store shows that the performance can be improved by 1.6× comparing to server-reply paradigm and 4× comparing to server-bypass paradigm.

Remote Direct Memory Access (RDMA) is a memory technology that allows remote devices to directly write to and read from each other's memory, bypassing components such as the CPU and operating system. This enables low-latency high-throughput networking, as required for many modern data centres, HPC applications and AI/ML workloads. However, baseline RDMA comprises a highly permissive weak memory model that is difficult to use in practice and has only recently been formalised. In this paper, we introduce the Library of Composable Objects (LOCO), a formally verified library for building multi-node objects on RDMA, filling the gap between shared memory and distributed system programming. LOCO objects are well-encapsulated and take advantage of the strong locality and the weak consistency characteristics of RDMA. They have performance comparable to custom RDMA systems (e.g. distributed maps), but with a far simpler programming model amenable to formal proofs of correctness. To support verification, we develop a novel modular declarative verification framework, called Mowgli, that is flexible enough to model multinode objects and is independent of a memory consistency model. We instantiate Mowgli with the RDMA memory model, and use it to verify correctness of LOCO libraries.

Nowadays more and more data have been generated every day in some enterprises such as facebook and google. These data need to be collected and analyzed in time. So the speed of transmitting data must be very high and the latency must be very low. Hadoop is applied in these enterprises and they use several data centers to store and process these data. But if the amount of data is growing fast or we will use only one data center then the bandwidth of the Ethernet Hadoop Distributed File System (HDFS) using cannot meet the need. The bandwidth of the Ethernet is going to become the performance bottleneck of HDFS. In order to solve this problem we will introduce a relatively new switched fabric communication link–—Infiniband in this paper. Based on Infiniband we have designed a new communication mechanism of HDFS and implemented it by modifying the code of HDFS. We use remote direct memory access (RDMA) to send and receive data rather than socket. The new HDFS will not use original stream mode to transmit data. Instead it will dynamically expand buffer and use changeable threshold. In this way the new HDFS will make CPU idle and improve performance. Unlike IPoIB which only uses Infiniband hardware device, our optimized HDFS is not only based on Infiniband hardware but also changes the code of HDFS to use RDMA. Our HDFS uses socket to transmit control message and RDMA to transmit data to make full use of the bandwidth of Infiniband. So applying the Infiniband with RDMA network bandwidth has not been the performance bottleneck of HDFS any more. According to the experiment results we have found that the network bandwidth of HDFS over Infiniband is 60 percent higher than the Ethernet and our optimized HDFS has much better performance than the HDFS over the Ethernet. On the other hand, the performance of our HDFS is also higher than the one which only use IPoIB.KeywordsInfinibandRDMADFSNetwork bandwidth

Remote Direct Memory Access (RDMA) technique allows the messaging service that directly access the memory on remote machines, which provides low CPU overhead, low latency, and high throughput network transmission. On the other hand, however, due to the limited cache space in RDMA NIC (RNIC), it is still challenging to achieve effective and fair resource sharing across different applications. To address this problem, we present a scalable RDMA as a service to manage resource and deliver fair scheduling to applications' requests. We study the thread contention and preemptive schedule issues at end-hosts, and report the corresponding performance degradation through experiments. Then, we introduce Avatar, a model to manage memory and Queue Pairs (QPs) resource for a large number of connections, which eliminates the lock contention and provides fair data scheduling for applications with different priorities. Finally, we implement Avatar and demonstrate that Avatar can support a thousand of connections, improve the fairness and reduce the requests completion time up to 50% in comparison with the native RDMA.

Several cloud computing applications use Replicated State Machines (RSM) to provide fault-tolerant services, ensuring consistency with consensus protocols. However, these protocols often come with a high latency cost, sometimes even forcing system designers to sacrifice consistency for availability. This latency is due, in part, to unnecessary data copies in the kernel TCP/IP layers. Remote Direct Memory Access (RDMA) bypasses the kernel to provide faster communication and lower CPU overhead through zero-copy data transfer. Recent works have utilized RDMA primitives to improve the performance of consensus protocols. However, integrating RDMA into such protocols and utilizing it efficiently can be a complex task. In this paper, we address this problem by presenting a systematic review of the state-of-the-art approaches for implementing RDMA-based consensus protocols.

Using remote direct memory access (RDMA) to ship data is becoming a very popular technique in network architectures. As these networks are adopted by the broader computing market, new challenges arise in transitioning existing code to use RDMA APIs. One particular class of applications that map poorly to RDMA are those that act as servers of file data. In order to access file data and send it over the network, an application must copy it to user-space buffers, and the operating system must register those buffers with the network adapter. Ordinary sockets-based networks can achieve higher performance by using the "sendfile" mechanism to avoid copying file data into user-space buffers. In this work we revisit time-honored approaches to sending file data, but adapted to RDMA networks. In particular, both pipelining and sendfile can be used, albeit with modifications to handle memory registration issues. However, memory registration is not well- integrated in current operating systems, leading to difficulties in adapting the sendfile mechanism. These two techniques make it feasible to create RDMA-based applications that serve file data and still maintain a high level of performance.

With the development of data center networks, traditional TCP/IP cannot support the demand in data centers. Remote Direct Memory Access (RDMA) technology could improve the performance of DCN significantly because of high throughput and low latency. However, load balancing is a key issue in RDMA which has not been solved distribute. This paper will propose an algorithm to solve the load balance problem in RDMA on application layer without hardware changing. The main idea is to divide data to chunks and data chunks on multiple reachable paths for transmission. However, it is no trivial to find the optimal chunk size and the path number, some empirical values are found by varieties of experiments and tests. Moreover, the chunk allocation scheme also needs to consider the traffic condition in DCNs to find more free paths to transmit. We evaluate the algorithm in application layer with ns3 simulator. The experiment results show that with our algorithm the completion time can decrease 81.03% at most.

We concern parallel computations with communication based on remote direct memory access (RDMA), which provides low level un-buffered access to distributed memory of computational nodes. Fine grain computations involve very frequent transmissions of small messages. For their efficient execution with RDMA communication a special memory infrastructure - rotating buffers (RB) - is introduced. Their organization is adjusted to program needs in advance - before program execution. It allows an intensive use of all communication resources available in the system based on additional synchronization between involved processes. The proposed method is illustrated by an example of a typical fine-grain problem which is the discrete fast Fourier Transform (FFT). "The transpose algorithm" of FFT has been implemented with the RDMA rotating buffers and its efficiency is compared with a solution based on standard message passing library MPI.

It has been well studied that reliable multicast enables consistency protocols, including Byzantine Fault Tolerant protocols, for distributed systems. However, no transport-layer reliable multicast is used today due to limitations with existing switch fabrics and transport-layer protocols. In this paper, we introduce a layer-4 (L4) transport based on remote direct memory access (RDMA) datagram to achieve reliable multicast over a shared optical medium. By connecting a cluster of networking nodes using a passive optical cross-connect fabric enhanced with wavelength division multiplexing, all messages are broadcast to all nodes. This mechanism enables consistency in a distributed system to be maintained at a low latency cost. By further utilizing RDMA datagram as the L4 protocol, we have achieved a low-enough message loss-ratio (better than one in 68 billion) to make a simple Negative Acknowledge (NACK)-based L4 multicast practical to deploy. To our knowledge, it is the first multicast architecture able to demonstrate such low message loss-ratio. Furthermore, with this reliable multicast transport, end-to-end latencies of eight microseconds or less (&lt; 8us) have been routinely achieved using an enhanced software RDMA implementation on a variety of commodity 10G Ethernet network adapters.

Distributed data structures are key to implementing scalable applications for scientific simulations and data analysis. In this paper we look at two implementation styles for distributed data structures: remote direct memory access (RDMA) and remote procedure call (RPC). We focus on operations that require individual accesses to remote portions of a distributed data structure, e.g., accessing a hash table bucket or distributed queue, rather than global operations in which all processors collectively exchange information. We look at the trade-offs between the two styles through microbenchmarks and a performance model that approximates the cost of each. The RDMA operations have direct hardware support in the network and therefore lower latency and overhead, while the RPC operations are more expressive but higher cost and can suffer from lack of attentiveness from the remote side. We also run experiments to compare the real-world performance of RDMA- and RPC-based data structure operations with the predicted performance to evaluate the accuracy of our model, and show that while the model does not always precisely predict running time, it allows us to choose the best implementation in the examples shown. We believe this analysis will assist developers in designing data structures that will perform well on current network architectures, as well as network architects in providing better support for this class of distributed data structures.