Implementation of Multi Node Hadoop Virtual Cluster on Open Stack Cloud Environments

In cloud computing, virtualization is accomplished by equipping with a virtual switch to connect between physical machine and virtual machine. The virtual switch which is usually written coded in software provides a flexible configuration and management. In this paper, we use kernel based virtual machine (KVM) to deploy the virtual server cluster and apply Open vSwitch for the virtualized network environment. Our platform provides service with single entry point so that the virtual server cluster is fully transparent to end users. The implementation of load balancing is carried out by using Open vSwitch with an Openflow controller. Openflow controller dynamically manages the flow table in Open vSwitch through secure channel by Openflow protocol. With the platform, we can also deploy a virtual cluster across two Local Area Networks (LANs) through Generic Routing Encapsulation tunnel protocol. Upon a request, the Open vSwitch forwards the traffic to the virtual server cluster upon the loading policy to achieve load balancing.

Virtual organization clusters are systems comprised of virtual machines that provide dedicated computing clusters for each individual virtual organization. The design of these clusters allows individual virtual machines to be independent of the underlying physical hardware, potentially allowing virtual clusters to span multiple grid sites. A major challenge in using virtual organization clusters as a grid computing abstraction arises from the need to schedule and provision physical resources to run the virtual machines. This paper describes a virtual cluster scheduler implementation based on the Condor high throughput computing system. By means of real-time monitoring of the Condor job queue, virtual machines that belong to individual virtual organizations are provisioned and booted. Jobs belonging to each virtual organization are then run on the organization-specific virtual machines, which form a cluster dedicated to the specific organization. Once the queued jobs have executed, the virtual machines are terminated, thereby allowing the physical resources to be re-claimed. Tests of this system were conducted using synthetic workloads, demonstrating that dynamic provisioning of virtual machines preserves system throughput for all but the shortest-running of grid jobs, without undue increase in scheduling latency.

High-performance processing scientific data has enabled the development of digital resources and web-based services that facilitate uses of data beyond those that may have been envisioned by the original data producers. Scientific data processing systems must handle scientific data coming from real time, high-throughput applications. Timely processing of scientific data is important and requires sufficient available resources to achieve high throughput and deliver accurate output results. Cloud Computing provides a low-priced way for small and medium sized enterprises to process scientific data. Based on Hadoop MapReduce of cloud computing, we propose the detailed procedure of scientific data processing algorithm which can improve the overall performance under the shared environment while retaining compatibility with the native Hadoop MapReduce in this paper.

As server consolidations based on the virtualization techniques become popular and cloud services continue to grow rapidly, more and more data centers are being built to accommodate a number of virtual clusters running various workloads. Since these virtual clusters often share the resources provided by physical machines (PMs), it is more likely that the interferences between virtual machines (VMs) affect the performance of applications running on top of the virtual clusters. While a lot of studies have proposed different virtual machine placement algorithms to investigate this issue, the problem caused by network performance variability still remains as a challenging issue. Since they usually ignore the CPU overhead to process the communications between VMs, the network bandwidth allocated to a VM cannot be fully utilized when a PM has not enough CPU resources to cover the CPU overhead for VM networking functions. This results in unpredictable application performance running on the virtual clusters. This paper proposes a virtual machine placement algorithm that considers the CPU overhead incurred to reserve network bandwidth in a virtualized data center environment. In order to decide the CPU overhead necessary to guarantee the network bandwidth allocated to a VM, a performance model based on standard linear regression using the data collected from a real environment is used. By comparing the amount of CPU resource available in the driver domain with the CPU overhead obtained from the performance model, the proposed algorithm decides whether the network bandwidth for the VM can be provided or not and selects an appropriate location for VM placement. The benchmarking results show that the proposed algorithm guarantees the network bandwidth allocated to each VM without violations when the CPU resources are shared by multiple VMs.

The quality of the scheduling method for virtualization clusters directly affects the performance of the entire cluster. Improper scheduling can even eliminate the advantages of virtualization clusters and reduce the performance of the entire system. Aiming at the problems of low resource utilization and poor performance caused by improper resource scheduling in cluster mode, a comprehensive heuristic resource scheduling method based on virtualization cluster - Resource Type Priority List Scheduling (RTPLS) algorithm is proposed. The entire algorithm includes two stages: resource monitoring and virtual machine scheduling. On the basis of considering multiple threshold triggering conditions such as CPU, memory, and network in the host load judgment during virtual machine scheduling, the host’s resource oscillation granularity and resource weight are introduced to comprehensively evaluate the resource load situation of the entire host, avoiding frequent migration of virtual machines caused by the oscillation of host resource load. In the selection of target virtual machine migration, in order to minimize the number of virtual machine migrations as much as possible, the compatibility coefficient λ of virtual machines is introduced, and the migration is sorted according to the priority list of λ to reduce the server load and energy consumption caused by virtual machine migration with the minimum number of virtual machine migrations. When selecting the target server, in order to minimize the load on the target virtual machine, target server relevance is introduced. Finally, by testing the algorithm in practical environments, it was found that the RTPLS algorithm can significantly improve the efficiency of host load balancing adjustments and enhance resource utilization.

Cloud computing and big data: Technologies and applications

Presently maintaining the load is the most challenging issue for a researcher in a cloud environment. In the IT sector, cloud computing is offering a variety of resources and free resources based on robust technology and Internet access in cloud computing. Users can also keep doing, gives a very reliable and inexpensive. Therefore, users of cloud computing are now evolving for every department of the organization, government and education for increased use of cloud services. Then, when they come in a variety of user demand for cloud resources, we can use load balancing technology to meet user needs. The use of the load balancing process is useful for distributing load over load to another node to adjust. The aim of load balancing is to balance virtual machines that should not be under load or overload. In this paper, propose two algorithms for the balance of load balancing algorithm including the workload of cloud system. In our work, there are basically low load systems and virtual machine demand will increase. The minimum finish time would be to put everything on one virtual machine and see the lowest number of high priority virtual machines. Improvements will be made to the importance of our research system, better use of resources and reduced full-time. Propose paper want the ClaudSim cloud platform to use a simulator. Also use unit response time in a VM as a metric, full implementation, and operation. Events demonstrate the features of the proposed algorithm running in a virtual machine to be used on experimental scale and for better application performance as possible.

Cloud computing is one of the well developing fields in Computer Technology. Now days cloud computing is one of the fast growing technology because of online, cheap and pay as use scheme. Cloud Computing involves the concepts of parallel processing and distributed computing in order to provide the shared resources by means of Virtual Machines(VMs) hosted by physical servers. It is a service oriented design that reduces the cost of access to gather the information of the clients offer greater flexibility and demand based services. Cloud computing is emerging fastly and no doubt it is the next generation technology where humans will be using anywhere and anytime. In this internet world cloud computing is raising high by providing everything incense the required resources, applications, software, hardware, computing power to computing infrastructure, business process to control collaboration. Apart of its popularity it has some concerns which are becoming huddles for its wider adoption. In this paper a study has been made on virtualization concerns. In this paper, we present a complete survey of cloud computing and virtual machine migration.

The author presents the architecture of a new family of parallel operating systems based on an alternative design approach when compared to multiprogramming, namely, generic construction of parallel virtual machines. This construction is made according to needs of models for parallel programming which shall be supported as subsystems by the parallel operating system kernel. The author presents the PAROS architecture and its generic kernel ParX which has been designed for message passing parallel computers. Several parallel virtual machines are also described, such as for instance a diffusion virtual machine, and a message passing-shared data virtual machine. >

In present days cloud computing is one of the greatest platform which provides storage of data in very lower cost and available for all time over the internet.But it has more critical issue like security, load management and fault tolerance. In this paper we are discussing Load Balancing approach. Resource scheduling management design on Cloud computing is an important problem. Scheduling model, cost, quality of service, time, and conditions of the request for access to services are factors to be focused. A good task scheduler should adapt its scheduling strategy to the changing environment and load balancing Cloud task scheduling policy. Cloud Computing is high utility software having the ability to change the IT software industry and making the software even more attractive. It has also changed the way IT companies used to buy and design hardware. The elasticity of resources without paying a premium for large scale is unprecedented in the history of IT industry. The increase in web traffic and different services are increasing day by day making load balancing a big research topic. Cloud computing is a new technology which uses virtual machine instead of physical machine to host, store and network the different components. Load balancers are used for assigning load to different virtual machines in such a way that none of the nodes gets loaded heavily or lightly. The load balancing needs to be done properly because failure in any one of the node can lead to unavailability of data.

Cloud Computing and Big Data continue to be disruptive forces in computing and have made inroads in the Computer Science curriculum, with courses in Cloud Computing and Big Data being routinely offered at the graduate and undergraduate level. One major challenge in offering courses in Big Data and Cloud Computing is resources. The question is how to provide students with authentic experiences making use of current Cloud and Big Data resources and tools and do so in a cost effective manner. Historically, three options, namely physical clusters, virtual clusters and cloud-based clusters, have been used to support Big Data and Cloud Computing courses. Virtual clusters and cloud-based options are those that institutions have typically adopted and many arguments in favor of these options exist in the literature, citing cost and performance. Here we argue that teaching Big Data and Cloud Computing courses can be done making use of a physical cluster and that many of the existing arguments fail to take into account many important factors in their calculations. These factors include the flexibility and control of a physical cluster in responding to changes in industry, the ability to work with much larger datasets, and the synergy and broad applicability of an appropriately equipped physical cluster for courses such as Cloud Computing, Big Data and Data Mining. We present three possible configurations of a physical cluster which span the spectrum in terms of cost and provide cost comparisons of these configurations against virtual and cloud-based options, taking into account the unique requirements of an academic setting. While limitations do exist with a physical cluster and it is not an option for all situations, our analysis and experience indicates that there is great value in using a physical cluster to support teaching Cloud Computing and Big Data courses and it should not be dismissed.

Recently, virtualization of hardware resources to run multiple instances of independent virtual machines over physical hosts has gained popularity due to an industry-wide focus on the need to reduce the cost of operation of an enterprise computing infrastructure. Xen is an open source hypervisor that provides a virtual machine abstraction layer which is very similar to the underlying physical machine. Using multiple physical hosts, each hosting multiple virtual machines over a VMM like Xen, system administrators can setup a high-availability virtual cluster to meet the ever-increasing demands of their data centers. In such an environment, the Xen hypervisor enables live migration of individual virtual machine instances from one physical node to another without significantly affecting the performance of the applications running on a target virtual machine. This paper describes a scalable Virtual Cluster Manager that provides such application agnostic cluster management capabilities to the system administrators maintaining virtual clusters over Xen powered virtual nodes.

Cloud computing, as defined by United States National Institute of Standards and Technology (NIST), “... is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.” Goal of the study is to highlight the positive economical impacts of cloud computing for companies, which can be assessed by focusing on their business goals to make more profit. Under the cloud computing model, firms can rent as many virtual machines as they need at any given time, and either design or use off-the-shelf solutions to integrate company-wide data and then easily distribute access to users within and outside of the company. Hence, cloud computing converts fixed capital costs to variable costs, prevents under or over provisioning, and allows minute by minute flexibility. Hence, the companies concentrate on their own business goals on top on a stable and effective information technology backbone. Most literature on cloud computing reside in computer science resources. However, there is little work focusing on the social and economic significance of cloud computing in the economics literature. Hence, this paper first discusses the basics of the cloud computing along with its implications on information technology (IT) field, with paying special attention to economical aspects. Then, empirical results and elaborations are to be given to yield a conclusion.

Recently, virtualization has become more and more important in the cloud computing to support efficient flexible resource provisioning. However, performance interference among virtual machines (VMs) has become a challenge which may affect the effectiveness of resource provisioning. In a virtual cluster which runs the MapReduce applications, the performance interference can also affect the performance of the Map and Reduce tasks and thus cause a performance degradation of the MapReduce job. Accordingly, this paper presents a MapReduce scheduling framework to mitigate this performance degradation caused by the performance interference. The framework includes a performance interference prediction module and an interference aware scheduling algorithm. To verify its effectiveness, we have done a set of experiments on a 24-node virtual MapReduce cluster. The experiments illustrate that the proposed framework can achieve a performance improvement in the virtualized environment compared with other MapReduce schedulers.

In recent years, the number of users and service providers are increasing in using cloud services so the accessibility and the effective management of the required resources, irrespective of the time and place, seem to be of great importance for both sides.Improving the performance and utilization of the cloud systems are gained by the auto-scaling of the applications; this is because of the fact that, some approaches have been proposed for auto scaling.This paper seeks to checking some value, based on the learning automata, for the scalability of the web applications, which combines virtual machine clusters and the learning automata in order to provide the best possible way for the scaling up and scaling down of the virtual machines.The results of this study indicate how an increased capacity of virtual machine which have been done by the value of thresholds could effect on SLA and overhead of responding.