Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers

The authors study how continuous media data can be stored and accessed in the Swift distributed input/output (IO) architecture. They provide a scheme for scheduling real-time data transfers that satisfies the strict requirements of continuous-media applications. This scheme allows large data objects to be stored and retrieved concurrently from multiple disks to satisfy the high data rate requirements typical of real-time video and audio data. To do this, data transfer requests are split into smaller requests, which are then handled by the various components by Swift. On-line algorithms are studied that respond to a data request by promising to either satisfy or reject it. Each response must be made before the next request is seen by the algorithm. The authors discuss two different performance measures to evaluate such algorithms and show that no on-line algorithm can optimize these criteria to less than a constant fraction of the optimal. Finally, they propose an algorithm for handling such requests on-line and the related data structures. >

Wavelength Division Multiplexing (WDM) technology multiplexes optical signals that take different wavelengths into the optical fiber, and offers a high bandwidth transmission. As an application of the WDM technology to the Internet, several network architectures have been discussed; packet-based architecture such as IP over WDM and path-based architecture that establishes a lightpath between nodes on-demand basis. In this paper, we consider a packet/path integration for WDM-based optical networks. The main advantage of the packet/path integration is to enjoy strengths of both packet switching paradigm and circuit switching paradigm. We evaluate the performance of packet/path integrated network via computer simulations with respect to the latency, which is defined from when a data transfer request arises to when the data transfer completes. We first show that the latency much depends on the amount of wavelength resources allocated to the packet switched network. We therefore develop a wavelength allocation algorithm that determines the amount of wavelength resources based on the queue length of buffers in packet switched network. Computer simulations show that our algorithm lowers the latency of the packet/path integrated network even when the arrival rate of data transfer requests drastically changes.

As the infrastructure of cloud computing and big data, data centers have been deployed widely. Then, how to make full use of the computing and storage resources in data centers will be the focus. Data center networks are considered important solution for the problem above, which include intra-data center and inter-data center networks. Both of them will depend on the optical networking due to its advantages, such as high bandwidth, low latency, and low energy consumption. Data center interconnected by flexi-grid optical networks is a promising scenario to allocate spectral resources for applications in a dynamic, tunable and efficient control manner. Compared with inter-data center, optical interconnect in intra-data center networks is a more pressing need and promising scenario to accommodate these applications in a dynamic, flexible and efficient manner. OpenFlow based Software Defined Networking (SDN) is considered as a good technology, which is very suitable for data center networks. This paper mainly focuses on the data center optical networks based on software defined networking (SDN), which can control the heterogeneous networks with unified resource interface. Architecture and experimental demonstration of OpenFlow-based optical interconnects in intra-data center Networks and OpenFlow-based flexi-grid optical networks for inter-data center are presented in the paper respectively. Some future works are listed finally.

Software Defined Networks (SDN) enables flexible flow control by installing policy rules into switches. However, one of the challenges is the dependencies between rules, which is generated due to the rules overlapping in filed space with different priorities. To keep the forwarding correctness and avoid complicated scenarios caused by the asynchronous removal, controllers usually adopt a hard timeout mechanism. However, such mechanism is inflexible for evolving and dynamic network flows. A large timeout may waste the switch memory, while a short timeout may cause multiple requests (Packet-in events) to occur for the same flow. To handle such rule dependencies flexibly, we propose a hybrid timeout mechanism. When a table miss occurs, we adaptively assign an idle timeout to the table-miss flow rule, and dependent rules are assigned with no timeout, which allow them to be removed using a proactive eviction strategy. We conduct extensive experiments using real packet traces from data centers. The experimental results show that our hybrid mechanism significantly reduces the number of table misses and the flow table occupation, while adapting quickly to changes of network flows.

The exponential rise of data/information and network resources made most of the organizations investing their resources to develop the biggest data centers (DCs) to accommodate. In the process, controlling the workload at the DCs with minimum response time, there is a need for an effective load-balancing system for which routing applications are playing an important role. Some of the routing applications based on software-defined networking (SDN) available in the market handling the traffic loads of these DCs efficiently. Using the centralized routing allows the network engineers to adjust different types of network elements such as switches, links, and ports dynamically based on the traffic loads. The data flow management at these DCs is managed by the routing applications sometimes face a non-conflicting way for the flow of data and towards the instructions for the switches as well. The security of routing applications plays an important role in safeguarding the control of data flow routing or scheduling from the attackers. The possibility of attackers controlling the traffic, servers, and switches at a DC is a serious threat and someone 2may even shut down the complete system/server. Such attacks may disrupt the routing application and also data flow management. In this chapter, various challenges and security analysis of SDN concerning to routing, cloud, Wi-Fi, and big data will be explored. Different types of design considerations for security requirements are discussed with considerable examples and code analysis. SDN cloud applications and security challenges that are related to the cloud are explained with detailed analysis. Some of the mitigation techniques are proposed for spoofing, data tampering, and repudiation issues. In addition, a detailed discussion on the big data using Apache Hadoop for processing and analyzing the large data sets will be explored.

SummaryWith the development of cloud computing and big data, the internal communication business in data center has increased dramatically, and then the traffic in data center has also significantly increased. The bandwidth of data center is difficult to meet the bandwidth requirements of those intensive applications, and data center is facing a risk of network congestion. Under the background of the development of network intelligence, software‐defined network (SDN) should demonstrate its intelligence as a future network architecture. In this paper, we introduce reinforcement learning into the SDN data center to implement congestion control based on flow. We improve the Q‐learning and Sarsa algorithms and propose two methods of congestion control based on the algorithms. Test results show that these two congestion control methods can control congestion effectively. And Sarsa method has a better performance of link utilization. The average link utilization of the Sarsa method is 2.4% higher than the Q‐learning method and is 4.48% higher than the on‐demand method.

Growing concern for reduced power dissipation, cost and latency demands in next generation Data Centers (DC) motivates us to revisit header optimizations. Headers contribute to about 30–40% of DC traffic and is responsible for equal proportion of consumed power. This amounts to significant overhead on per byte transfer of payload. In the past, highly inflexible switches have limited the focus of header optimizations primarily on end-host or edge network routers. Further, strict compliance to protocols by switches/routers along the path have limited the segments of headers that can be optimized. Recent evolution of Software Defined Network (SDN) brings in several opportunities to control over switches. We believe that if SDN's finer handle to the switching devices is exploited, it can help deal with header information. In this paper, we exploit the capabilities of SDN and introduce a new functionality that can effectively replace the redundant and repetitive header information with shorter unique identity. We present Scissor that trims the headers lower in the protocol stack. As a replacement for routing, we introduce the notion of Flow-ID, where all packets belonging to a flow are identified using this unique Flow-ID. We leverage the capabilities of the SDN to dynamically allow switching devices to route the packets based on the Flow-IDs. Our approach of trimming header at the switching devices leaves the hosts unmodified making it highly adoptable for DC environment. We show that our approach can decrease switch fabric power consumption by a factor of 2.5 compared to an existing L2 switch and reduce the latency by about 30% for a significant fraction (around 30–50%) of the network traffic.

Data centers consume vast amounts of energy, making them a significant indirect source of carbon emissions. The use of renewable energy sources to power data centers is essential to reduce these emissions and is a key strategy to achieve zero-carbon goals. Redesigning data centers to integrate renewable energy has become increasingly important. One approach is to collocate data centers with renewable energy sources, and, when necessary, combine multiple multisite data centers into a single larger entity. We propose the Green Multisite Data Center (GMDC) network infrastructure architecture to replace expensive electrical power transmission lines with network connections. The proposed architecture improves GMDC resiliency by leveraging network features like all-active paths, multi-homing, redundancy, and load balancing capabilities. In addition, it offers a pay-as-you-go model for owners and investors in green data center facilities. The GMDC network is an energy-aware infrastructure that integrates renewable energy sources while mitigating the risks related to their time and location variability. The design prioritizes minimizing initial investment costs while ensuring long-term economic sustainability. In our solution, which combines Software-Defined Networking (SDN) with Ethernet VPN/Virtual eXtensible LAN (EVPN/VXLAN), we simulate the SDN forwarding layer to manage multisite inter-data center traffic. The primary focus is improving performance metrics, particularly total data transfer time, which is vital to establish network infrastructure across geographically dispersed data centers. This paper simulates the data plane and presents strategies for minimizing total workload migration times using Data Plane Development Kit (DPDK) libraries. Testing results indicate a 4.24% reduction in the overall workload migration time between green data centers in a multisite setup.

Various applications in a multi-tenant architecture based cloud computing comprises various requirements and priorities. Severe failure of a critical application may result if the network is congested in a data center. Hence, it is an important to use an efficient resource provisioning technique for ensuring transfer of high priority data to the other traffics still when the network is congested. Therefore, a new framework of Home Cloud architecture is proposed in this paper to provide an effective and flexible delivery of new applications in an edge-cloud architecture which integrates the two complementary technologies of Software-Defined Networking (SDN) and Network Function Virtualization (NFV). In SDN infrastructure, this presents a priority aware resource placement algorithms by taking into consideration of both host and network resources. The Virtual Machines (VMs) of high priority applications are placed at the strongly connected hosts by introducing the priority aware VM allocation (PAVA) algorithm for reducing the possibility of causing network congestion by other tenants. It also guarantees the critical applications for a required bandwidth by means of allocating bandwidth through the setting of priority queues for each network connection within the network of data center handled by SDN controller. The simulation result of the proposed system shows that the combination of proposed approaches will allocate sufficient resources for high-priority applications within a Home Cloud data centre to meet the QoS requirements.

While future terabit networks hold the promise of significantly improving big-data motion among geographically distributed data centers, significant challenges must be overcome even on today's 100 gigabit networks to realize end-to-end performance. Multiple bottlenecks exist along the end-to-end path from source to sink. Data storage infrastructure at both the source and sink and its interplay with the wide-area network are increasingly the bottleneck to achieving high performance. In this paper, we identify the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network environment, and we present a new bulk data movement framework called LADS for terabit networks. LADS exploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS also uses the Common Communication Interface (CCI) in lieu of the sockets interface to use zero-copy, OS-bypass hardware when available. It can further improve data transfer performance under congestion on the end systems using buffering at the source using flash storage. With our evaluations, we show that LADS can avoid congested storage elements within the shared storage resource, improving I/O bandwidth, and data transfer rates across the high speed networks.

Software Defined Networking (SDN) has emerged as a new paradigm that offers the programmability required to dynamically configure and control a network. In this paper, we present a path assignment method based on load balancing(PA-LB) for a centralized controller to calculate optimal end-to-end virtual paths over the underlying network infrastructure, considering multiple requests and load balance simultaneously. In the paper, theoretical analysis shows that the path selection is NP-hard, and strongly polynomial time inapproximable under the assumption P not equal to NP. We also give a constant-factor polynomial time approximation algorithm for this problem under some bounded parameters. Extensive simulation results, over a wide range of underlying network topologies and input parameters, demonstrate that the proposed method outperforms traditional shortest path first (SPF) approaches not only in paths mapping number, but also in network load balance. Under the condition of limited resources, such as bandwidth, the proposed method has a better performance with the increase of the number of nodes. Moreover, optimal load balance can be kept in the network, in order to avoid network congestion and minimize the average packet transmission latency.

Combining edge processing (at data capture site) with analysis carried out while data is enroute from the capture site to a data center offers a variety of different processing models. Such in-transit nodes include network data centers that have generally been used to support content distribution (providing support for data multicast and caching), but have recently started to offer user-defined programmability, through Software Defined Networks (SDN) capability, e.g., OpenFlow and Network Function Visualization (NFV). We demonstrate how this multi-site computational capability can be aggregated to support video analytics, with Quality of Service and cost constraints (e.g., latency-bound analysis). The use of SDN technology enables separation of the data path from the control path, enabling in-network processing capabilities to be supported as data is migrated across the network. We propose to leverage SDN capability to gain control over the data transport service with the purpose of dynamically establishing data routes such that we can opportunistically exploit the latent computational capabilities located along the network path. Using a number of scenarios, we demonstrate the benefits and limitations of this approach for video analysis, comparing this with the baseline scenario of undertaking all such analysis at a data center located at the core of the infrastructure.

New and emerging use cases, such as the interconnection of geographically distributed data centers (DCs), are drawing attention to the requirement for dynamic end-to-end service provisioning, spanning multiple and heterogeneous optical network domains. This heterogeneity is, not only due to the diverse data transmission and switching technologies, but also due to the different options of control plane techniques. In light of this, the problem of heterogeneous control plane interworking needs to be solved, and in particular, the solution must address the specific issues of multi-domain networks, such as limited domain topology visibility, given the scalability and confidentiality constraints. In this article, some of the recent activities regarding the Software-Defined Networking (SDN) orchestration are reviewed to address such a multi-domain control plane interworking problem. Specifically, three different models, including the single SDN controller model, multiple SDN controllers in mesh, and multiple SDN controllers in a hierarchical setting, are presented for the DC interconnection network with multiple SDN/ OpenFlow domains or multiple OpenFlow/ Generalized Multi-Protocol Label Switching (GMPLS) heterogeneous domains. In addition, two concrete implementations of the orchestration architectures are detailed, showing the overall feasibility and procedures of SDN orchestration for the end-to-end service provisioning in multi-domain data center optical networks.

Many large organizations collect massive volumes of data each day in a geographically distributed fashion, at data centers around the globe. Despite their geographically diverse origin the data must be processed and analyzed as a whole to extract insight. We call the problem of supporting large-scale geo-distributed analytics Wide-Area Big Data (WABD). To the best of our knowledge, WABD is currently addressed by copying all the data to a central data center where the analytics are run. This approach consumes expensive cross-data center bandwidth and is incompatible with data sovereignty restrictions that are starting to take shape. We instead propose WANalytics, a system that solves the WABD problem by orchestrating distributed query execution and adjusting data replication across data centers in order to minimize bandwidth usage, while respecting sovereignty requirements. WANalytics achieves an up to 360x reduction in data transfer cost when compared to the centralized approach on both real Microsoft production workloads and standard synthetic benchmarks, including TPC-CH and Berkeley Big-Data. In this demonstration, attendees will interact with a live geo-scale multi-data center deployment of WANalytics, allowing them to experience the data transfer reduction our system achieves, and to explore how it dynamically adapts execution strategy in response to changes in the workload and environment.

The network of traditional data center has been usually designed and constructed for the provision of user's equal access of data centre's resource or data. Therefore, network administrators have a strong tendency to manage user traffic from the viewpoint that the traffic has a similar size and characteristics. But, the emersion of big data begins to make data centers have to deal with 1015 byte-data transfer at once. Such a big data transfer can cause problems in network traffic management in the existed data center. And, the tiered network architecture of the legacy data center magnifies the magnitude of the problems. One of the well-known big data in science is from large hadron collider such as LHC in Swiss CERN. CERN LHC generates multi-peta byte data per year. From our experience of CERN data service, this paper showed the impact of network traffic affected by large-scale data stream using NS2 simulation, and then, suggested the evolution direction based on separating of large-scale data stream for the big data center's network architecture.