Performance Evaluation of Data Intensive Computing In the Cloud

Cloud computing is a relatively new form of computing, which uses virtualized resources and is dynamically scalable and is often provided as pay for use service over the Internet or Intranet or both. With increasing demand for data storage in the cloud, study of data intensive applications is becoming a primary focus. Data intensive applications are those which involve a high CPU usage, processsing large volumes of data typically in size of hundreds of gigabytes, terabytes, or petabytes. This study was conducted on Amazon's Elastic Cloud Compute (EC2) and Amazon Elastic Map Reduce (EMR) using HiBench Hadoop Benchmark Suite. HiBench is a Hadoop benchmark suite and is used for performing and evaluating Hadoop based data intensive computation on both these cloud paltforms. Both quantitative and qualitative comparison was performed on both Amazon EC2 and Amazon EMR, including a study of their pricing models and measures are suggested for future studies and research.

Cloud resources like Amazon EC2 and Google Compute Engine have become an integral part of our life with their pay-per-use pricing models. On the other hand, attempts have also been made to create virtual clouds on the fly using available devices in the locality like smartphones and tablets. Such devices being equipped with powerful processors, this is a novel method of utilizing their normally unused computational capabilities. We attempt to design and develop a pricing model for such resources created for opportunistic computing. Since the values of the resources in these virtual clouds cannot be calculated, the pay-per-resource model has to give way to a pay-per-effort model. We model the scenario as a principal-agent problem with moral hazards and propose an incentive-driven Quality of Service (QoS) to ensure that the agents act appropriately. In view of the short contractual duration, we consider only deterministic contracts which are easy to understand and implement. We show that the mobile devices can be modeled as economic actors in a contractual arrangement and discuss how the optimal decisions are identified.

An increasing demand is seen from enterprises to host and dynamically manage middlebox services in public clouds in order to leverage the same benefits that network functions provide in traditional, in-house deployments. However, today's public clouds provide only a limited view and programmability for tenants that challenges flexible deployment of transparent, software-defined network functions. Moreover, current virtual network functions can't take full advantage of a virtualized cloud environment, limiting scalability and fault tolerance. In this paper we review and evaluate the current infrastructural limitations imposed by public cloud providers and present the design and implementation of GNFC, a cloud-based Network Function Virtualization (NFV) framework that gives tenants the ability to transparently attach stateless, container-based network functions to their services hosted in public clouds. We evaluate the proposed system over three public cloud providers (Amazon EC2, Microsoft Azure and Google Compute Engine) and show the effects on end-to-end latency and throughput using various instance types for NFV hosts.

Data-intensive applications process large volumes of data using a parallel processing method. MapReduce is a programming model designed for data-intensive applications for massive data sets and an execution framework for large-scale data processing on clusters of commodity servers. While fault tolerance, easy programming structure, and high scalability are considered strong points of MapReduce; however its configuration parameters must be fine-tuned to the specific deployment, which makes it more complex in configuration and performance. This paper explains tuning of the Hadoop configuration parameters, which directly affect MapReduce's job workflow performance under various conditions to achieve maximum performance. On the basis of the empirical data we collected, it became apparent that three main methodologies can affect the execution time of MapReduce running on cluster systems. Therefore, in this paper, we present a model that consists of three main modules: (1) Extending a data redistribution technique in order to find the high-performance nodes, (2) Utilizing the number of map/reduce slots in order to make it more efficient in terms of execution time, and (3) Developing a new hybrid routing schedule shuffle phase in order to define the scheduler task while memory management level is reduced. © 2013 Wiley Periodicals, Inc.

We present our experiences using cloud computing to support data-intensive analytics on satellite imagery for commercial applications. Drawing from our background in high-performance computing, we draw parallels between the early days of clustered computing systems and the current state of cloud computing and its potential to disrupt the HPC market. Using our own virtual file system layer on top of cloud remote object storage, we demonstrate aggregate read bandwidth of 230 gigabytes per second using 512 Google Compute Engine (GCE) nodes accessing a USA multi-region standard storage bucket. This figure is comparable to the best HPC storage systems in existence. We also present several of our application results, including the identification of field boundaries in Ukraine, and the generation of a global cloud-free base layer from Landsat imagery.

Cloud Computing has received much attention by the IT-Business world. As compared to the common computing platforms, cloud computing is more flexible in supporting real-time computation and is considered a more powerful model for hosting and delivering services over the Internet. However, since cloud computing is still at its infancy, it faces many challenges that stand against its growth and spread. This article discusses some challenges facing cloud computing growth and conducts a comparison study between Amazon EC2 and Windows Azure in dealing with such challenges. It concludes that Amazon EC2 generally offers better solutions than Windows Azure. Nevertheless, the selection between them depends on the needs of customers.

With the rapid growth of biological technology, large amount of biological data can be produced in few days or months. Many of common-used tools become computation-consuming in analyzing such big biological data. Cloud computing has emerged to provide the huge amount of computing power and play important role in development of bioinformatics tools. We propose a cloud computing framework that is able to easily deploy the bioinformatics tools on cloud virtualization platform based on Hadoop. This framework can work on the public cloud platform vendor such as Amazon EC2 and also private cloud platform. All the tools performed by cloud computing framework as Bioinformatics as a Services are available at http://bioinfo.cs.pu.edu.tw/CBBTS. The tools deployed on cloud platform by the proposed framework are tested in Providence University cloud platform and provided good proportional acceleration when scaled out onto many computational units. In the big biological data era, cloud computing based solutions are important role to develop bioinformatics services over internet. In the work, the proposed framework is able to simply deploy several well-known bioinformatics tools on cloud virtualization platform and as web services. This framework can work on the public cloud platform vendor such as Amazon EC2 and also private cloud platform.

Cloud computing is essential for executing scientific workflows, offering scalable resources to process large volumes of data. However, efficient resource allocation remains a challenge due to fluctuations in demand and inadequate planning, which can increase makespan and thus energy consumption. In this context, infrastructure as a service is primarily used, allowing the rental of virtual machines with different characteristics. This study focuses on identifying the factors that influence the improvement of makespan for executing workflows from different disciplines and characteristics. The factors evaluated in this study are workfloF size, structure, and the number of virtual machines (VMs) required. The algorithms in this study consist of four heuristics and one metaheuristic, specifically a genetic algorithm (GA). Regarding the factors, the most relevant in CC is the number of VMs, where increasing the number of VMs reduces makespan to a certain extent, following the behavior described by the law of diminishing returns.

Cloud computing is a suitable platform for running applications to process large volumes of data. Currently, with the growth of geographic and spatial data volume, conceptualized as Big Geospatial Data, some tools have been developed to allow the processing of this data efficiently. This work presents a cost-efficient method for processing geospatial data, optimizing the number of data nodes in a SpatialHadoop cluster according to dataset size. With this, it is possible to analyse and compare the costs for this type of application on public cloud providers.

Purpose: The accuracy of medical physics calculations is largely limited by the availability of fast computer hardware, which is in turn strongly limited by budget considerations. This research explores the use of the Cloud Computing paradigm to replace or supplement existing computing infrastructure for medical physics calculations. Cloud computing services allow for on‐demand virtual computing clusters with a pay‐as‐you‐go pricing model. This model has the potential to make large scale Monte Carlo simulations routine in clinical and research activities. Methods and materials: To test the feasibility of medical physics calculations in a cloud computing environment, a distributed Monte Carlo dose calculation framework was implemented with Amazon's EC2 cloud computing service, the Fluka 2008.3b Monte Carlo package, a distributed processing framework written in Python, and the Hadoop MapReduce framework. A variety of relevant calculations were performed: depth dose profiles of proton, electron, and photon beams, along with a simple proton treatment plan using a voxel patient geometry (Zubal phantom). The performance of EC2 was tested by running calculations on 1 to 200 virtual nodes with the different distributed processing frameworks. EC2 prices were compared with costs of comparable local hardware. Results: Relevant medical physics calculations were successfully carried out using the EC2 service. Heavy charged particle, electron, and photon depth dose profiles and a simple treatment plan were successfully calculated remotely. Performance data demonstrated the expected 1/n speed‐up as a function of node number. Two different distributed processing schemes were implemented in the cloud. Conclusion: Our implementations demonstrate the power of the existing cloud computing model. The relatively cheap pay‐as‐you‐go pricing for infrastructure on demand offers an extremely promising new paradigm for clinical and research computing, available to anyone with a network connection.

As more and more data can be generated at a faster-than-ever rate nowadays, it becomes a challenge to processing large volumes of data for complex data analysis. In order to address performance and cost issues of big data processing on clouds, we present a novel design of adaptive workflow management system which includes a data mining based prediction model, workflow scheduler, and iteration controls to optimize the data processing via iterative workflow tasks. We proposed a new heuristic algorithm, called Upgrade Fit, which dynamically and continuously reallocates multiple types of cloud resources to fulfill the performance and cost requirements. The iterative workflow tasks can be bursty bags of tasks to be executed repetitively for data processing. A real application of weather forecast workflow has been used to evaluate the capability of our system for large volume image data processing. Experimental system has been set up and the results indicate that the system can effectively handle multiple types of cloud resources and optimize the performance iteratively.

Although cloud computing has become one of the most popular paradigms for executing data-intensive applications (for example, Hadoop), the storage subsystem is not optimized for scientific applications. We believe that when executing scientific applications in the cloud, a node-local distributed storage architecture is a key approach to overcome the challenges from the conventional shared/parallel storage systems. We analyze and evaluate four representative file systems (S3FS, HDFS, Ceph, and FusionFS) on three platforms (Kodiak cluster, Amazon EC2 and FermiCloud) with a variety of benchmarks to explore how well these storage systems can handle metadata intensive, write intensive, and read intensive workloads.

Processing large volumes of data to drive their core business has been the primary objective of many firms and scientific applications in these days. Cloud computing being a large-scale distributed computing paradigm can be used to cater for the needs of data intensive applications. There are various approaches for managing the workload on a data intensive cloud. Live migration of a virtual machine is the most prominent paradigm. Existing approaches to live migration use network attached storage where just the run time state needs to be transferred. Live migration of virtual machines with local persistent storage has been shown to have performance advantages like security, availability and privacy. This paper presents an optimised approach for migration of a virtual machine along with its local storage by considering the locality of storage access. Count map combined with a restricted block transfer mechanism is used to minimise the downtime and overhead. The solution proposed is tested by various parameters like bandwidth, write access patterns and threshold. Results show the improvement in downtime and reduction in overhead.

Cloud computing with virtualization technologies has become an important trend in the information technology industry. Due to its salient features of reliability and cost effectiveness, cloud computing has changed the paradigms of development for mobile pervasive services, effectively permeating the market. While most types of best effort mobile pervasive applications can be seamlessly migrated to cloud computing infrastructures, we need to consider specialized elements to make cloud computing infrastructures more effective in real-time healthcare applications. The client side of those applications dramatically increases its transmission rate whenever it detects an abnormal event. However, the existing server side mechanisms have limitations in adaptively allocating necessary computing resources in order to handle these various data volumes over time. In this article, we propose a novel server-side auto-scaling mechanism to autonomously allocate virtual resources on an on-demand basis. The mechanism is tested in an Amazon EC2, and the results show how the proposed mechanism can efficiently scale up and down the virtual resources, depending on the volume of requested real-time tasks.

Latency-sensitive and data-intensive applications, such as IoT or mobile services, are leveraged by Edge computing, which extends the cloud ecosystem with distributed computational resources in proximity to data providers and consumers. This brings significant benefits in terms of lower latency and higher bandwidth. However, by definition, edge computing has limited resources with respect to cloud counterparts; thus, there exists a trade-off between proximity to users and resource utilization. Moreover, service availability is a significant concern at the edge of the network, where extensive support systems as in cloud data centers are not usually present. To overcome these limitations, we propose a score-based edge service scheduling algorithm that evaluates network, compute, and reliability capabilities of edge nodes. The algorithm outputs the maximum scoring mapping between resources and services with regard to four critical aspects of service quality. Our simulation-based experiments on live video streaming services demonstrate significant improvements in both network delay and service time. Moreover, we compare edge computing with cloud computing and content delivery networks within the context of latency-sensitive and data-intensive applications. The results suggest that our edge-based scheduling algorithm is a viable solution for high service quality and responsiveness in deploying such applications.