Provisioning and Evaluating Multi-domain Networked Clouds for Hadoop-based Applications

This study compares data dispersion and encryption techniques with an emphasis on Hadoop Distributed File System (HDFS) and Advanced Encryption Standard (AES) in secure cloud storage systems. The study uses 29,502 samples using ClincalC with G-power values (α=0.05, β=0.2). Data dispersion was analyzed using synthetic datasets, AES encryption was verified in a controlled cloud environment, and HDFS functioning was assessed using Hadoop clusters. AES was shown through statistical analysis in SPSS to be a successful data security solution that guarantees confidentiality. Scalable and fault-tolerant storage was made possible by Hadoop (HDFS) (82.14% vs. 61.54%, p<0.02), showing strong performance but necessitating more security precautions for complete data protection.

Hadoop and Spark are well-known big data processing platforms. The main technologies of Hadoop are Hadoop Distributed File System and MapReduce processing. Hadoop stores intermediary data on Hadoop Distributed File System, which is a disk-based distributed file system, while Spark stores intermediary data in the memories of distributed computing nodes as Resilient Distributed Dataset. In this paper, we show how memory size affects distributed processing of large volume of data, by comparing the running time of K-means algorithm of HiBench benchmark on Hadoop and Spark clusters, with different size of memories allocated to data nodes. Our results show that Spark cluster is faster than Hadoop cluster as long as the memory size is big enough for the data size. But, with the increase of the data size, Hadoop cluster outperforms Spark cluster. When data size is bigger than memory cache, Spark has to replace disk data with memory cached data, and this situation causes performance degradation.

The usage of cloud systems is at an all-time high, and with more organizations reaching for Big Data the forensic implications must be analyzed. The Hadoop Distributed File System is widely used both as a cloud service and with organizations implementing it themselves. This paper analyzed the forensic viability of a RAM analysis method for Hadoop based investigations and compared it against targeted process data dumping through the Java heap information. The RAM analysis was done through string searching and the use of the RAM analysis tool Volatility. This work found that RAM analysis can be a valuable tool for discovering artefacts of deleted resources from a Hadoop cluster but was unable to discover further information such as the block to node mapping. The targeted process analysis managed to provide some partial information about deleted resources and produce important metadata on the current state of the file system.

MPJ Express Meets YARN: Towards Java HPC on Hadoop Systems

In today's cloud service market, different providers have very different low-level mechanisms to accommodate various types of policies from their users. Enforcement of policies over multiple cloud provider domains is an intrinsically complex problem for both sides. In reality, cloud providers have to either manually update enforcement mechanisms or negotiate adjusted policies with their users for enforcement. To save these high-cost and error-prone manual updates or adjustments, an automatic and flexible solution is desired. This paper proposes a semantic modeling and mapping based approach to help enforce high-level user policies across cloud domain boundaries when applications or IT operations have to span over multiple cloud domains. This approach creates policy models and maps these models across cloud domain boundaries automatically or semi-automatically. Policy rules following these mappings can be tied to multiple enforcement mechanisms in different cloud domains. If a rule cannot be mapped, a manual adjustment solution will be suggested. A case study is also included to demonstrate the efficiency and accuracy of this approach.

Big Data OverviewDriven by the need to generate business value, the enterprise has started to adopt Big Data as a solution, migrating from the classical databases and data stores which lack the flexibility and are not optimized enough [1].The changes in the environment make big data analytics attractive to all types of organizations, while the market conditions make it practical. The combination of simplified models for development, commoditization, a wider palette of data management tools, and low-cost utility computing has effectively lowered the barrier to entry. [2]. The concept addresses large volumes of complex data, rapid growing data sets that may come from different autonomous sources.In recent approaches, Big Data is characterized by principles known as the 4VVolume, Variety, Velocity and Veracity [3]. There are opinions about accepting other principles as Big Data characteristics, such as Value.Each day more businesses realize that Big Data is relevant as the applications generate large volumes of data generated automatically, from different data sources, centralized or autonomous. As traditional databases hit limitations when the need of analyzing this data, dedicated solutions must be considered.Important BigData Solutions:* Apache HBase/Hadoop is based on Google's BigTable distributed storage system, which runs on top of Hadoop as a distributed and scalable big data store. This means that HBase can leverage the distributed processing paradigm of the Hadoop Distributed File System (HDFS) and benefit from Hadoop's MapReduce programming model. It combines the scalability of Hadoop with real-time data access as a key/value store and deep analytic capabilities of Map Reduce [4].HBase allows to query for individual records as well as derive aggregate analytic reports across a massive amount of data. It can host large tables with billions of rows, millions of columns and run across a cluster of commodity hardware. HBase is composed of three types of servers in a master slave type of architecture. Region servers are responsible to serve data for reads and writes. When accessing data, clients communicate with HBase Region Servers directly. Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process.* Apache Cassandra is a distributed database used for the administration and management of large amounts of structured data across multiple servers, while providing highly available service and no single point of failure. It provides features such as continuous availability, linear scale performance, data distribution across multiple data centers and cloud availability zones. Cassandra inherits its data architecture from Google's BigTable and it borrows its distribution mechanisms from Amazon's Dynamo.The nodes in a Cassandra cluster are completely symmetrical, all having identical responsibilities. Cassandra also employs consistent hashing to partition and replicate data. it has the capability of handling large amounts of data and thousands of concurrent users or operations per second across multiple data centers.Cassandra has a hierarchy of caching mechanisms and carefully orchestrated disk I/O which ensures speed and data safety. Write operations are sent first to a persistent commit log (ensuring a durable write), then to a write-back cache called a memTable; when the memTable fills, it is flushed to a sorted string table - SSTable - on disk. A Cassandra cluster is organized as a ring, and it uses a partitioning strategy to distribute data evenly.* Redis represents an in-memory data structure store used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs and geospatial indexes with radius queries. Redis stores all data in RAM, allowing lightning fast reads and writes. It runs extremely efficiently in memory and handles high-velocity data, needing simple standard servers to deliver millions of operations per second with submillisecond latency. …

Distributed storage systems are increasingly transitioning to the use of erasure codes since they offer higher reliability at significantly lower storage costs than data replication. However, these codes tradeoff recovery performance as they require multiple disk reads and network transfers for reconstructing an unavailable data block. As a result, most existing systems use an erasure code either optimized for storage overhead or recovery performance.In this paper, we present HACFS, a new erasure-coded storage system that instead uses two different erasure codes and dynamically adapts to workload changes. It uses a fast code to optimize for recovery performance and a compact code to reduce the storage overhead. A novel conversion mechanism is used to efficiently upcode and downcode data blocks between fast and compact codes. We show that HACFS design techniques are generic and successfully apply it to two different code families: Product and LRC codes.We have implemented HACFS as an extension to the Hadoop Distributed File System (HDFS) and experimentally evaluate it with five different workloads from production clusters. The HACFS system always maintains a low storage overhead and significantly improves the recovery performance as compared to three popular single-code storage systems. It reduces the degraded read latency by up to 46%, and the reconstruction time and disk/network traffic by up to 45%.

Hadoop is an inexpensive analytical tool as compared to the other distributed storage in market as it does not need any standalone machines and works on group of commodity hardware. It is a distributed storage system along with this it achieves parallelization of larger data collections. With MapReduce, HDFS (Hadoop distributed file system) provides solution for the system where processing huge datasets is a requirement. Few of the main reasons of stragglers in assorted Hadoop clusters are load inconsistency during storing, resource friction throughout scheduling tasks, hardware downturn due to excessive usage, as well as software configuration issues while managing the cluster. Hadoop's performance lows down in a heterogeneous network due to the technical heterogeneity. We used a supervised machine learning (ML) technique to identify straggler nodes in an eminently distributed network in this article. The suggested technique identifies the proper slow-running job (Straggler) in the network and assign it to other node in the stack to complete the operation with quick succession. Virtual Machine (VM) identifier, network bandwidth consumption, number of processors and its load, memory load and other parameters included in the full data set are utilized for recognition. Various feature extraction methodologies have been utilized to develop its training system. The whole data set was processed for heterogenous features on the dataset. We analyzed our approach using our suggested classifier after doing comprehensive empirical work. As out-turn, the system outperforms using typical machine learning models in classification performance.

With the rapid development of the Internet of Things (IoT) technology, diversified applications deploy extensive sensors to monitor objects, such as PM2.5 air quality monitoring. The sensors transmit data to the server periodically and continuously. However, a single server cannot provide efficient services for the ever-growing IoT devices and the data they generate. This study bases on the concept of symmetry of architecture and quantities in system design and explores the load balancing issue to improve performance. This study uses the Linux Virtual Server (LVS) and virtualization technology to deploy a virtual machine (VM) cluster. It consists of a front-end server, also a load balancer, to dispatch requests, and several back-end servers to provide services. These receive data from sensors and provide Web services for browsing real-time sensor data. The Hadoop Distributed File System (HDFS) and HBase are used to store the massive amount of received sensor data. Because load-balancing is critical for resource utilization, this study also proposes a new load distribution algorithm for VM-based server clusters that simultaneously provide multiple services, such as sensor services and Web service. It considers the aggregate load of all back-end servers on the same physical server that provides multiple services. It also considers the difference between physical machines and VMs. Algorithms such as those for LVS, which do not consider these factors, can cause load imbalance between physical servers. The experimental results demonstrate that the proposed system is fault tolerant, highly scalable, and offers high availability and high performance.

Every day a huge amount of unstructured and semi structured data is used in all the business sectors. Those data are very complicated to store and process for applying into the decision-making system. Especially, the medical data, clinical data and patient history data are to be accessed in a faster way to bring the feasible solution. In view of this, a high speed, reliable and fault tolerant programming framework is needed [1].Apache Pig is a high level and globally accepted programming language to execute the map reduce tasks over the Hadoop cluster while dealing with unstructured data. This language works on Hadoop Distributed File System (HDFS) and this language is written in Java.In this proposed work, the medical history data of patients are considered to be processed. The existing approaches such as oracle SQL queries and mongo DB based results have been producing the very slower time to process these records. But in pig programming language, these gaps are rectified and produced an efficient result.The data dictionary of this implemented dataset is having 7 fields of records to be process in a phased approach. Each and every phase of analysis, the various fields are considered for further processing and interpretation. With the help of relationship operator’s, the relationship of these dataset fields is identified. Followed by this, the functions are applied to produce the faster segregation on the given dataset. Two types of functions are used here such as math function and evaluation function [2].

Cloud security has gained increasingly emphasis in the research community, with much focus primary concentrated on how to secure the operation system and virtual machine on which cloud system runs on. We take an alternative perspective to consider the problem of building a secure cloud storage service on top of a public cloud infrastructure where the service provider is not completely trusted by the customer. So, it is necessary to put cipher text into the public cloud. We describe an architecture based on Trusted Platform Module and the client of cloud storage system to help manage the symmetric keys used for encrypting data in the public cloud and the asymmetric keys used for encrypting symmetric keys. The key management mechanism includes how to store keys, how to backup keys, and how to share keys. Based on the HDFS (Hadoop Distributed File System), we put a way of key management into practice, and survey the benefits that such an infrastructure will provide to cloud users and providers, and we also survey the time cost it will bring to us.

Hadoop is an open source software framework of distributed processing of big data. There are many kinds of services in Hadoop ecosystem, such as HDFS, Map-Reduce, HBase, Hive, Yarn, Flume, Spark, Storm, Zookeeper, and so on, which increase the complexity of deployment and configuration. It takes plenty of time to construct a Hadoop cluster. Although there are some management tools which help administrators deploy and configure Hadoop clusters automatically, they usually provide a fixed solution. So administrators couldn't construct their Hadoop clusters according to different management requirements by the tools. Software architecture acts as a bridge between requirements and implementations. It has been used to reduce the complexity and cost mainly resulted from the difficulties faced by understanding the large-scale and complex software system. This paper proposes a model based approach to Hadoop deployment and configuration which help administrators construct Hadoop clusters in a simple but powerful enough manner. First, we provide the unified models of Hadoop software architecture, according to the domain knowledge of current Hadoop deployment and configuration. Second, we provide a framework with a set of definable rules for domain experts to describe their solutions to deploy and configure Hadoop clusters. Thus, administrators can use various custom solutions to automatically deploy and configure their Hadoop clusters according to different management requirements. In addition, a real-world experiment demonstrates the feasibility, effectiveness and benefits of the new approach to Hadoop deployment and configuration.

A distributed remote sensing data storage system based on Hadoop platform is designed to solve the problem of low storage efficiency and low security of mass remote sensing data. Uesd virtualization technology to PC as hardware based virtual computing node the Hadoop Cluster required, without increasing the dedicated server clusters and does not change the original PC use conditions, built the Hadoop platform by the virtual machine cluster. Using LandSat-8 satellite image data as the experimental object, through the design of file tree structure, the remote sensing image data according to the band and split into several blocks of image coordinates, combined with the Hadoop distributed file system (HDFS) characteristics of block storage data block, the remote sensing data stored in the distribution of the whole virtual computer cluster in. Build Hadoop platform remote sensing data distributed storage system with low cost.

HyperFlow is a workflow engine that enables to execute tasks of scientific workflows on the available computing resources (e.g. Virtual Machines in a cloud). PaaSage is a model-based cloud platform to provision resources, deploy the application and automatically scale them according to the application demands. One of the challenges is to design the interplay mechanism between the application-specific workflow scheduler in HyperFlow with the generic provisioning and autoscaling components of PaaSage in a generic way. Here we report on our current developments in integrating HyperFlow applications with PaaSage, outline the architecture of the proposed solution, present the CAMEL model of the application and the prototype status. The main conclusion is that thanks to the model-based approach proposed by PaaSage it is possible to modularize the workflow application and make its components easily deployable across multiple clouds.

The recent increase in availability of Infrastructure-as-a-Service (IaaS) computing clouds provides a new way for researchers to run complex scientific applications. However, using cloud resources for a large number of research jobs requires significant effort and expertise. Furthermore, running jobs on many different clouds presents even more difficulty. In order to make it easy for researchers to deploy scientific applications across many cloud resources, we have developed a virtual machine resource manager (Cloud Scheduler) for distributed compute clouds. In response to a user's job submission to a batch system, the Cloud Scheduler manages the distribution and deployment of user-customized virtual machines across multiple clouds. We describe the motivation for and implementation of a distributed cloud using the Cloud Scheduler that is spread across both commercial and dedicated private sites, and present some early results of scientific data analysis using the system.