Virtual Cluster Management with Xen

The quality of the scheduling method for virtualization clusters directly affects the performance of the entire cluster. Improper scheduling can even eliminate the advantages of virtualization clusters and reduce the performance of the entire system. Aiming at the problems of low resource utilization and poor performance caused by improper resource scheduling in cluster mode, a comprehensive heuristic resource scheduling method based on virtualization cluster - Resource Type Priority List Scheduling (RTPLS) algorithm is proposed. The entire algorithm includes two stages: resource monitoring and virtual machine scheduling. On the basis of considering multiple threshold triggering conditions such as CPU, memory, and network in the host load judgment during virtual machine scheduling, the host’s resource oscillation granularity and resource weight are introduced to comprehensively evaluate the resource load situation of the entire host, avoiding frequent migration of virtual machines caused by the oscillation of host resource load. In the selection of target virtual machine migration, in order to minimize the number of virtual machine migrations as much as possible, the compatibility coefficient λ of virtual machines is introduced, and the migration is sorted according to the priority list of λ to reduce the server load and energy consumption caused by virtual machine migration with the minimum number of virtual machine migrations. When selecting the target server, in order to minimize the load on the target virtual machine, target server relevance is introduced. Finally, by testing the algorithm in practical environments, it was found that the RTPLS algorithm can significantly improve the efficiency of host load balancing adjustments and enhance resource utilization.

Virtual organization clusters are systems comprised of virtual machines that provide dedicated computing clusters for each individual virtual organization. The design of these clusters allows individual virtual machines to be independent of the underlying physical hardware, potentially allowing virtual clusters to span multiple grid sites. A major challenge in using virtual organization clusters as a grid computing abstraction arises from the need to schedule and provision physical resources to run the virtual machines. This paper describes a virtual cluster scheduler implementation based on the Condor high throughput computing system. By means of real-time monitoring of the Condor job queue, virtual machines that belong to individual virtual organizations are provisioned and booted. Jobs belonging to each virtual organization are then run on the organization-specific virtual machines, which form a cluster dedicated to the specific organization. Once the queued jobs have executed, the virtual machines are terminated, thereby allowing the physical resources to be re-claimed. Tests of this system were conducted using synthetic workloads, demonstrating that dynamic provisioning of virtual machines preserves system throughput for all but the shortest-running of grid jobs, without undue increase in scheduling latency.

Cloud providers generally run one or more Virtual Machine (VM) instances on the same physical machine. Though it increases data center utilization, it exposes VM to a co-location attack. VM placement and migration are the two strategies adopted for mitigating co-locations. Current methods for VM placement or VM migration consider only security as decision criteria and do not consider other factors like Quality-of-Service degradation, data center utilization, etc. This work proposes a placement and migration strategy for mitigation of co-location attacks with joint consideration of multi objectives like QoS, data center utilization, energy consumption, and security risks. A security-driven multi-criteria optimization -based VM placement policy is proposed. A joint consideration of multiobjective performance optimization along with co-location security risk minimization is done to design a novel VM placement policy based on user categorization. The policy can reduce the likelihood of co-location target VM with attacker VM without much degradation to the performance of VM and data center utilization. The solution mitigates co-location risks without much compromise to the performance of VM and data center resource utilization. The co-residence risk is mitigated by the categorization of users into three levels i.e. unlabeled, risky, and safe, and physical machines into two groups as safe and unsafe. The PMs available in data center is grouped into three different VM placement policies, they are undecided pool, safe pool and unsafe pool.

As server consolidations based on the virtualization techniques become popular and cloud services continue to grow rapidly, more and more data centers are being built to accommodate a number of virtual clusters running various workloads. Since these virtual clusters often share the resources provided by physical machines (PMs), it is more likely that the interferences between virtual machines (VMs) affect the performance of applications running on top of the virtual clusters. While a lot of studies have proposed different virtual machine placement algorithms to investigate this issue, the problem caused by network performance variability still remains as a challenging issue. Since they usually ignore the CPU overhead to process the communications between VMs, the network bandwidth allocated to a VM cannot be fully utilized when a PM has not enough CPU resources to cover the CPU overhead for VM networking functions. This results in unpredictable application performance running on the virtual clusters. This paper proposes a virtual machine placement algorithm that considers the CPU overhead incurred to reserve network bandwidth in a virtualized data center environment. In order to decide the CPU overhead necessary to guarantee the network bandwidth allocated to a VM, a performance model based on standard linear regression using the data collected from a real environment is used. By comparing the amount of CPU resource available in the driver domain with the CPU overhead obtained from the performance model, the proposed algorithm decides whether the network bandwidth for the VM can be provided or not and selects an appropriate location for VM placement. The benchmarking results show that the proposed algorithm guarantees the network bandwidth allocated to each VM without violations when the CPU resources are shared by multiple VMs.

In Infrastructure-as-a-Service (IaaS) clouds, users manage the systems in virtual machines (VMs) called user VMs through remote management systems (RMSes). To allow users to manage their VMs during failures inside the VMs, IaaS usually provides out-of-band remote management. This management is performed indirectly via an RMS server in a privileged VM called the management VM. However, it is discontinued when a user VM is migrated. This is because an RMS server in the management VM at the source host is terminated on VM migration. Even worse, pending data is lost between an RMS client and a user VM. In this paper, we propose D-MORE for continuing out-of-band remote management across VM migration. D-MORE provides a privileged and migratable VM called DomR and performs out-of-band remote management of a user VM via DomR. During VM migration, it synchronously co-migrates DomR and its target VM and transparently maintains the connections between an RMS client, DomR, and its target VM. We have implemented D-MORE in Xen and confirmed that a remote user could manage his VM via DomR after the VM has been migrated. Our experiments showed that input data was not lost during VM migration and the overhead of D-MORE was acceptable.

Current cloud computing providers allocate their virtual machine (VM) instances via fixed price-based or auction-like mechanisms. However, these mechanisms have one limitation, they are all offline mechanisms, therefore they need to collect information and be invoked periodically. In this paper, we address this limitation by designing an online mechanism for dynamic provisioning and allocation of VM instances in clouds. Our proposed mechanism, MOVMPA, is invoked as soon as a user places a request or some VM instances already allocated become available again. When invoked, the mechanism selects users who would be allocated VM instances for the period they requested for, and ensures that those users will continue using those VMs for the entire period requested. We prove that the mechanism is incentive compatible and also investigate its performance through extensive simulation experiments.

Cloud providers provision their various resources such as CPUs, memory, and storage in the form of Virtual Machine (VM) instances which are then allocated to the users. We design online mechanisms for VM provisioning and allocation in clouds that consider several types of available resources. Our proposed online mechanisms make no assumptions about future demand of VMs, which is the case in real cloud settings. The proposed mechanisms are invoked as soon as a user places a request or some of the allocated resources are released and become available. The mechanisms allocate VM instances to selected users for the period they are requested for, and ensure that the users will continue using their VM instances for the entire requested period. In addition, the mechanisms determine the payment the users have to pay for using the allocated resources. We prove that the mechanisms are incentive-compatible, that is, they give incentives to the users to reveal their true valuations for their requested bundles of VM instances. We investigate the performance of our proposed mechanisms through extensive experiments.

Infrastructure as a service offered by the cloud computing is one of the most important service. It allows physical machines to get virtualized by creating many instances of virtual machines. Mapping virtual machines on physical machine has become the major challenge in cloud data centres. The dynamic virtual machine placement methods are used to solve this issue with objectives like maximizing the resource utilization, minimizing the energy consumption and maximizing the scalability of data centres. In this paper a virtual machine placement-based bin packaging algorithm is proposed and analysed with four different fitness strategies to obtain the optimal solution. The unimodal (Sphere, Step) and multimodal (Graywang and Rastridge) benchmark functions are used with proposed algorithm for the analysis and obtain the quantitative measurements. The results show that optimizing the mass of particles using the best fitting strategy reduces the energy consumption, resource utilization and improved the scalability of data centres.

To address the virtual machine cluster deployment issues in cloud computing environment, a novel MCSA (Min-cut segmentation algorithm) of virtual machine cluster is proposed with resource and communication bandwidth constraints. In this paper, the basic idea is based on the fully consideration on the CPU, memory, hard-disk and other resource constraints between virtual machine cluster and physical host, as well as the communication bandwidth constraints between the virtual machine. We quantified the virtual machine cluster constructed an undirected graph. In the undirected graph, the nodes represent the virtual machine, so the weight of a node represents the value of resources, and the edges represent the communication bandwidth, so the weight of the edge represents the value of communication bandwidth. Base on the above transformations, the resources and bandwidth constrained optimization problem is transformed into the graph segmentation problem. Next we segment the undirected graph by minimum cut algorithm, and computing the matching degree of physical machines. Last we obtained the approximate solution. To validate the effectiveness of the new algorithm, we carried out extensive experiments based on the CloudSim platform.

In cloud computing, virtualization is accomplished by equipping with a virtual switch to connect between physical machine and virtual machine. The virtual switch which is usually written coded in software provides a flexible configuration and management. In this paper, we use kernel based virtual machine (KVM) to deploy the virtual server cluster and apply Open vSwitch for the virtualized network environment. Our platform provides service with single entry point so that the virtual server cluster is fully transparent to end users. The implementation of load balancing is carried out by using Open vSwitch with an Openflow controller. Openflow controller dynamically manages the flow table in Open vSwitch through secure channel by Openflow protocol. With the platform, we can also deploy a virtual cluster across two Local Area Networks (LANs) through Generic Routing Encapsulation tunnel protocol. Upon a request, the Open vSwitch forwards the traffic to the virtual server cluster upon the loading policy to achieve load balancing.

SummaryVirtual clusters (VCs) have exhibited various advantages over traditional cluster computing platforms by virtue of their extensibility, reconfigurability, and maintainability. As such, they have become a major execution environment for cloud‐based cluster applications. However, compared with traditional clusters, their distributed‐memory programming paradigm still remains largely unchanged, which implies that cluster applications cannot be efficiently deployed in VCs, especially when virtual machines (VMs) are running in different physical hosts. Recently, some efforts have been made to improve inter‐VM communication, resulting in many studies on how cluster applications could take advantages of VCs. However, most of them mainly focus on the situation that the VMs are all coresident on the same physical machine where the message passing mechanism is usually optimized away by exploiting the host's shared memory. In this paper, we present a design and implementation of Naplus, a kernel‐based virtual machine approach to the inter‐VM communications that are across different physical hosts. Naplus is based on Nahanni, a mechanism for shared‐memory communication in virtual environments. As such, it not only inherits the major merits of Nahanni with respect to flexible data structures and efficient synchronization but also achieves a shared‐memory paradigm among VMs. With Naplus, we enable the size of shared space to be maximized as large as the sum of each machine's local memory to accommodate cluster applications with large memory footprints. We prototype Naplus in a dual‐host system where an empirical study is conducted to show the effectiveness of the Naplus approach in achieving distributed shared memory for VCs in data centers. Copyright © 2017 John Wiley & Sons, Ltd.

Improving the energy efficiency while guaranteeing quality of services (QoS) is one of the main challenges of efficient resource management of large-scale data centers. Dynamic virtual machine (VM) consolidation is a promising approach that aims to reduce the energy consumption by reallocating VMs to hosts dynamically. Previous works mostly have considered only the current utilization of resources in the dynamic VM consolidation procedure, which imposes unnecessary migrations and host power mode transitions. Moreover, they select the destinations of VM migrations with conservative approaches to keep the service-level agreements , which is not in line with packing VMs on fewer physical hosts. In this paper, we propose a regression-based approach that predicts the resource utilization of the VMs and hosts based on their historical data and uses the predictions in different problems of the whole process. Predicting future utilization provides the opportunity of selecting the host with higher utilization for the destination of a VM migration, which leads to a better VMs placement from the viewpoint of VM consolidation. Results show that our proposed approach reduces the energy consumption of the modeled data center by up to 38% compared to other works in the area, guaranteeing the same QoS. Moreover, the results show a better scalability than all other approaches. Our proposed approach improves the energy efficiency even for the largest simulated benchmarks and takes less than 5% time overhead to execute for a data center with 7600 physical hosts.

SummaryVirtual desktop infrastructure (VDI) solutions seek to provide a satisfactory user experience at the client side when accessing remote desktop applications, even from mobile devices with limited capabilities. This paper presents a new approach, improving on previous work by the authors, in which a combination of Virtual Network Computing (VNC) and streaming protocols allowed efficient remote web access to virtualized applications within a cloud architecture. The new approach simplifies virtual machine templates, from which virtual machine instances are deployed, by centralizing software modules, greatly simplifying their management. Our new contribution consists of an integrated solution with specific WebM video encoding modules in charge of application visual output processing, an Hypertext Transfer Protocol (HTTP) streaming server, and a VNC server. The solution can be installed in the hypervisor of the host machines instead of replicating the servers and modules throughout the guest (virtual) machines that run the virtualized applications. Consequently, their implementations are unique and independent of the operating system of the virtual machines. In short, it is not necessary to provide different implementations for different operating systems, which reduces the complexity of virtual machine templates and greatly simplies platform management. To demonstrate our solution, we have modified the Quick Emulator (QEMU)‐Kernel‐based Virtual Machine (KVM) hypervisor source code accordingly. We also present qualitative and quantitative analyses that demonstrate that the new approach is advantageous in terms of software management and quality of experience, compared with our previous work and other well‐known thin clients, contributing to the enhancement of VDI systems. Copyright © 2015 John Wiley & Sons, Ltd.

The distributed nature and growing demand for open source cloud makes the system an ideal target for malicious attacks and unauthorised file transfers. Requirements of provenance cognition scheme can come forward to solve the problem. However, such mechanisms of provenance detection has been considered to a limited extent for open source cloud computing. ProvIntSec is a novel mechanism that ensures effective collection of provenance information from a large pool of virtual machine (VM) instances on open source cloud platform. ProvIntSec captures critical system journals from VM instances and pattern matches those with predefined signatures to detect the presence of malicious activities. In addition, ProvIntSec identifies the Linux process trees to determine unauthorised file movements across different nodes. The experiments were executed in OpenStack Essex cloud environment running on real life system, and standard metrics were used to calculate the results. The obtained results show average precision values of 92.81% and 81.24% for malware detection and unauthorised file transfers respectively. At the same time, cumulative performance gains of 0.3991 and 8.77 are obtained. Upon comparison of the obtained results with benchmarks, ProvIntSec shows desirable gain in performance.

As cloud computing models have evolved from clusters to large-scale data centers, reducing the energy consumption, which is a large part of the overall operating expense of data centers, has received much attention lately. From a cluster-level viewpoint, the most popular method for an energy efficient cloud is Dynamic Right Sizing (DRS), which turns off idle servers that do not have any virtual resources running. To maximize the energy efficiency with DRS, one of the primary adaptive resource management strategies is a Virtual Machine (VM) migration which consolidates VM instances into as few servers as possible. In this paper, we propose a Two Phase based Adaptive Resource Management (TP-ARM) scheme that migrates VM instances from under-utilized servers that are supposed to be turned off to sustainable ones based on their monitored resource utilizations in real time. In addition, we designed a Self-Adjusting Workload Prediction (SAWP) method to improve the forecasting accuracy of resource utilization even under irregular demand patterns. From the experimental results using real cloud servers, we show that our proposed schemes provide the superior performance of energy consumption, resource utilization and job completion time over existing resource allocation schemes.