Load balancing on PC clusters with the super-programming model

In Cloud Computing (CC) environment, load balancing refers to the process of optimizing resources of virtual machines. Load balancing in the CC environment is one of the analytical approaches utilized to ensure indistinguishable workload distribution and effective utilization of resources. This is because only by ensuring effective balance of dynamic workload results in higher user satisfaction and optimal allocation of resource, therefore improve cloud application performance. Moreover, a paramount objective of load balancing is task scheduling because surges in the number of clients utilizing cloud lead to inappropriate job scheduling. Hence, issues encircling task scheduling has to be addressed. In this work a method called, Ford Fulkerson and Newey West Regression-based Dynamic Load Balancing (FF-NWRDLB) in CC environment is proposed. The FF-NWRDLB method is split into two sections, namely, task scheduling and dynamic load balancing. First, Ford Fulkerson-based Task Scheduling is applied to the cloud user requested tasks obtained from Personal Cloud Dataset. Here, employing Ford Fulkerson function based on the flow of tasks, energy-efficient task scheduling is ensured. The execution of asymmetrical scientific applications can be smoothly influenced by an unbalanced workload distribution between computing resources. In this context load balancing signifies as one of the most significant solution to enhance utilization of resources. However, selecting the best accomplishing load balancing technique is not an insignificant piece of work. For example, selecting a load balancing model does not work in circumstances with dynamic behavior. In this context, a machine learning technique called, Newey West Regression-based dynamic load balancer is designed to balance the load in a dynamic manner at run time, therefore ensuring accurate data communication. The FF-NWRDLB method has been compared to recent algorithms that use the markov optimization and the prediction scheme to achieve load balancing. Our experimental results show that our proposed FF-NWRDLB method outperforms other state of the art schemes in terms of energy consumption, throughput, delay, bandwidth and task scheduling efficiency in CC environment.

Dear SCPE Reader, We present a selection of papers which are extensions of papers presented at the 7-th International Symposium on Parallel and Distributed Computing, 1–5 July 2008, in Krakow, Poland. The motivation for publishing the selection in the SCPE Journal was, on the one hand, to present the flavour of the research reported at the conference and on the other hand to present some of the most relevant topics currently focused on the research on parallel and distributed computing in general. The selection contains only 6 papers out of about 60 presented at the conference, and thus, is far from covering all relevant topics represented at the ISPDC 2008. This is because not all of the invited authors were patient enough to accept a fairly long paper publishing process. Nevertheless, we hope that the presented papers will bring you closer to the research covered by the ISPDC conferences and will encourage you to participate in future ISPDC editions. The first paper ``The Impact of Workload Variability on Load Balancing Algorithms'' is by Marta Beltran and Antonio Guzman from King Juan Carlos University in Spain. It concerns an important topic of load balancing in cluster systems, namely adaptativity of the load balancing algorithms to changes of the workload in the system. Adequate accounting for additional load in the hosting system is of great relevance for correct optimization effects. The paper presents a thorough formal analysis of the workload variability metrics and their influence on the quality of load balancing algorithms. Four basic activities appearing in load balancing algorithms are identified, and based on them some algorithmic solutions are proposed to correctly deal with workload variability in system load balancing. The problem of dynamic load balancing algorithms robustness has been discussed. Two different robustness metrics sensitive to the applied type of opimization: local task-oriented or a global one enable selecting task remote execution or migration as load balancing operations. The proposed approach is illustrated with experiments. The second paper ``Model-Driven Engineering and Formal Validation of High-Performance Embedded Systems'' is by Abdoulaye Gamatie, Eric Rutten, Huafeng Yu, Pierre Boulet, Jean-Luc Dekeyser, from University of Lille and INRIA in France. The paper is concerned with a very advanced methodology of designing correct parallel embedded systems for intensive data-parallel computing. In their previous research, the authors of the paper designed the GASPARD embedded system design framework. It is based on the hardware/software co-design approach through model-driven engineering. The framework is based on an UML-like model specification language in which hardware and software elements are modelled using a component approach with special mechanisms for repetitive structures. This paper tries to combine the modelling framework of GASPARD with the mechanisms of synchronous languages to achieve design verifiability provided for such languages. The paper shows how GASPARD models can be translated into synchronous models based on data flow equations in order to formally check their correctness. The proposed approach is illustrated with an example of a video processing system. The third paper ``Relations Between Several Parallel Computational Models'' is by Stephan Bruda and Yuanqiao Zhang from Bishop’s University in Canada. The paper is concerned with theoretical aspects of shared memory systems described by the parallel random access machine PRAM model and aims in studying performance properties of different types of PRAM systems. The attention is focussed on analysing the computational power of two more sophisticated PRAM models (Combining CRCW and Broadcast Selective Reduction), which include data reduction in case of concurrent writes. The paper shows that these two models have equivalent computational power, which is a new result comparing the existing literature. The performance of both models applied to reconfigurable multiple bus machines was studied as a possible architectural solution for current VLSI processor implementations. It was shown that in such systems under reasonable assumptions concurrent-write does not enhance performance comparing the exclusive-write model. Another result important for the VLSI technology is that the Combining CRCW PRAM model (in which data of concurrent writes are arthmetically or logically combined before write) and the exclusive-write on directed reconfigurable busses perform in equivalent way under strong real-time requirements. The fourth paper ``Experiences with Mesh-Like Computations Using Prediction Binary Trees'' is by Gennaro Cordasco, Biagio Cosenza, Rosario de Chiara, Ugo Erra and Vittorio Scarano from the University ``degli Studi'' of Salerno and the University ``degli Studi della Biasilicata'' of Potenza in Italy. The paper concerns optimization methods for mesh-like computations in clusters of processors. The computations are perfomed assuming a phase-like program execution control using a tiling approach which reduces inter-processor communication. A temporal coherence is also assumed, which means that task sizes provide similar execution times in consecutive phases. Temporary coherent computations are structured in a Prediction Binary Tree, in which leaves represent computing tiles to be mapped to processors. A phase-by-phase semi-static load balancing is introduced to the scheduling algorithm. The scheduling algorithm is equipped with a predictor, which estimates the computation time of next phase tiles based on previous execution times and modifies the tiles to achieve balanced execution in phases. For this, two heuristics are used to leverage on data locality in processors. The proposed approach is illustrated by the example of interactive rendering with Parallel Ray Tracing algorithm. The fifth paper ``The Influence of the IBM pSeries Servers Virtualization Mechanism on Dynamic Resource Allocation in AIX 5L'' is by Maciej Mlynski from ASpartner Limited in Poland. The paper concerns a very up-to-date problem of system virtualization and presents the results of research carried on IBM pSeries servers. IBM is strongly developing the virtualization technique especially on IBM pSeries servers enabling an improved and flexible sharing of system resources between applications. The paper investigates novel facilities for dynamic resource management such as micro-partitioning and partition load manager. They enable dynamic creation of workload logical partitions of system resources and their dynamic mangement. It includes run-time resource re-alocation between logical partitions including setting of sharing specifications as well as run-time adding/removing/setting parameters of resources in the system. It remains an open question how to properly tune parameters of the operating system using the provided virtualization facilities to obtain the best efficiency for a given application program. The paper presents the results of experiments which study the effects of tuning the disk subsystem parameters under the IBM AIX 5L operating system with the use of the provided virtualization facilities on the resulting application execution performance. The results show that even small deterioration in the resource pool status requires an immediate adaptation of the operating system parameters to maintain the required performance. The sixth paper ``HeteroPBLAS: A Set of Parallel Basic Linear Algebra Subprograms Optimized for Heterogeneous Computational Clusters'' is by Ravi Reddy, Alexey Lastovetsky and Pedro Alonso from University College Dublin in Ireland and Polytechnic University of Valencia in Spain. The paper concerns the methodology for parallelization of linear algebra computations for execution in heterogeneous cluster environments. The design of the HeteroPBLAS library (Parallel Basic Linear Algebra Subprograms) for heterogeneous computational clusters is presented. The main contribution of the paper is the automation of the parallelization and optimization of the PBLAS, which is done by means of a special user interface and the underlying set of functions. An important element is here a performance model that is based on program code instrumentation, which determines parameters of the application and the executive heterogeneous platform relevant for execution performance of parallel code. The parameter values specified for or returned by execution of the performance model functions are next used for generation and optimal mapping of the parallel code of the library subroutines. The proposed approach is illustrated by experimental results of execution of optimized HeteroPBLAS programs on homogeneous and heterogeneous computing clusters. Marek Tudruj

Load balancing is performed to achieve the optimal use of the existing computational resources as much as possible whereby none of the resources remains idle while some other resources are being utilized. Balanced load distribution can be achieved by the immigration of the load from the source nodes which have surplus workload to the comparatively lightly loaded destination nodes. Applying load balancing during run time is called dynamic load balancing (DLB). DLB can be realized both in a direct or iterative manner according to the execution node selection. In iterative methods, the final destination node is determined through several iteration steps, while in direct methods it is selected in one step. This paper presents the randomized hydrodynamic load balancing (RHLB) method which is a hybrid method that takes advantage of both direct and iterative methods. Using random load migration as a direct method, RHLB approach intends to solve the problems derived from the exceptional instantaneous load rises, and diffuse the surplus workload to relatively free resources. Besides, using hydrodynamic approach as an iterative method, RHLB aims to consume minimum possible system resources to balance the common workload distributions. The results of the experiments designate that, RHLB outruns other iterative based methods in terms of both balance quality and the total time of the load balancing process.

Dominant trends in parallel and distributed computing (parallel computers, parallel algorithms) aim to connect a number of powerful workstations (WS) based on single personal computers (PC) or symmetrical multiprocessor system (SMP) to solve complex problems in a parallel way. Typical representatives of these trends are network of workstation (NOW) and Grid as a high integrated network of NOW modules. In relation to it there is a bad need of unified modelling in traditionally evolved parallel computing and distributed computing as two separate research disciplines. Likewise current trends in high performance computing (HPC) aim to use NOW modules based on SMP parallel computers as cheaper alternatives to the traditionally used massive parallel multiprocessors or supercomputers and to profit from unifying modelling in parallel and distributed computing too. To exploit the parallel processing capability of such cluster, any application program must be paralleled. A way how to do it (decomposition model) belongs to a most important step in developing process of effective parallel algorithm. This article at first discusses the development of effective (optimised) iterative parallel algorithms (IPA) to solve partial differential equations (PDE). By carefully choosing an illustrative typical problem of PDE (Laplace equation) the proposed research has demonstrated the most important complexities in the process of complete modelling of substantial complexities and their optimisation in order to develop an effective PA. Such approach could be used as illustrative example for a complete modelling and effective solution of even other more complex PDE´s in a parallel way.

Analysis of Load Average and its Relationship to Program Run Time on Networks of Workstations

Modern standards in High Performance Computing (HPC) have started to consider energy consumption and power draw as a limiting factor. New and more complex architectures have been introduced in HPC systems to afford these new restrictions, and include coprocessors such as GPGPUs for intensive computational tasks. As systems increase in heterogeneity, workload distribution becomes a more core problem to achieve the maximum efficiency in every computational component. We present a Multi-Objective Dynamic Load Balancing (DLB) approach where several objectives can be applied to tune an application. These objectives can be dynamically exchanged during the execution of an algorithm to better adapt to the resources available in a system. We have implemented the Multi-Objective DLB together with a generic heuristic engine, designed to perform multiple strategies for DLB in iterative problems. We also present Ull Multiobjective Framework (UllMF), an open-source tool that implements the Multi-Objective generic approach. UllMF separates metric gathering, objective functions to be optimized and load balancing algorithms, and improves code portability using a simple interface to reduce the costs of new implementations. We illustrate how performance and energy consumption are improved for the implemented techniques, and analyze their quality using different DLB techniques from the literature.

A domain partition approach to parallel adaptive simulation of dynamic threshold voltage MOSFET

The advance of the GPU hardware architecture has made GPUs attractive devices for general- purpose computing. Modern GPUs are equipped with an increasing number of cores, a flexible memory hierarchy, and a large memory capacity. While the computational power of modern GPU devices has allowed their introduction in high-performance computing (HPC) clusters and the efficient processing of ever larger workloads, existing software components for HPC clusters still offer basic support for hardware heterogeneity and often cause performance limitations in the presence of GPU devices. In particular, two kinds of limitations are associated with these software components: runtime support and programmability. We found that these limitations are due to the fact that existing software frameworks for heterogeneous clusters treat GPUs as dedicated coprocessor devices. In this dissertation, we propose two software frameworks for addressing the performance and hardware underutilization issues found in heterogeneous CPU-GPU clusters as well as increasing their programmability. Our frameworks provide a uniform view of compute resources and treat CPUs and GPUs equally as first-class resources, allowing efficient management of heterogeneous compute resources. First, we propose a hierarchical scheduling framework consisting of a node-level runtime and a cluster-level scheduler that provides abstraction of heterogeneous compute resources at different granularities. This hierarchical framework targets existing applications and does not require their modification. In the node-level runtime, we identify and design mechanisms, such as virtual GPUs, GPU virtual memory, dynamic load balancing and pre-emption, which are necessary to support efficient sharing and load balancing schemes for GPUs within a compute node. In the cluster-level scheduler, we introduce mechanisms to abstract compute nodes and perform load balancing in concert with the node-level runtime. Our hierarchical scheduling framework allows supporting different load balancing policies and does not require additional inputs (such as profiling information) from users. Second, we propose a programming framework based on a novel memory and execution model. Our memory model hides disjoint addressing spaces (corresponding to different CPUs, GPUs and compute nodes) and provides a view of a single virtual memory space that can be accessed by all compute resources in a heterogeneous cluster. Our execution model provides uniform access to compute resources and allows our framework to treat all CPUs and GPUs equally and to access data in the virtual memory space.

Any micro service-based programme, including alumni management software, must include load balancing. The effective use of resources is made possible by load balancing, which also ensures that application performance is unaffected by unexpected increases in requests or user usage. In an alumni management application, load balancing can be achieved with a combination of techniques such as round-robin scheduling, weighted load balancing, and dynamic scheduling. These techniques help to ensure that the application remains up and running, even during peak usage times. Additionally, load balancing can be used to ensure users get the best possible experience when using the application. By distributing traffic among multiple back-end services, users can expect faster response times and more reliable performance.Load balancing in a micro services-based alumni management application is a way of distributing incoming web traffic across multiple back end servers in order to efficiently manage traffic, reduce latency, and improve overall performance. Load balancing works by distributing incoming requests across a cluster of servers, and can be done either statically or dynamically. Static load balancing assigns each request to a specific server, while dynamic load balancing uses algorithms to decide which server should handle each request. In the alumni management application, load balancing is important for ensuring that requests are handled efficiently and that the system is not overwhelmed by too many requests. This is especially important when multiple users are accessing the application simultaneously. By distributing requests across multiple servers, the application can remain responsive and reliable even under heavy load.

Resource Virtualization and Load Balancing are main objectives to reduce the power consumption and to improve the performance of large data centers (DC). The management of Cloud Data Centers (CDC) requires an accurate planning and an efficient use of system resources in order to save energy consumption (greening), to provide Quality of Service (QoS), and to meet negotiated Service Level Agreements (SLA). This contribution addresses the question of modeling and the development of generic queuing models for energy-efficient use of resources for dynamic load balancing in virtualized CDCs. Performance models are developed for energy efficiency through automatic Server Consolidation, Dynamic Voltage and Frequency Scaling (DVFS) under Static Load Balancing; Dynamic Load Balancing can be achieved through Virtual Machine (VM) migrations. The analysis of such models provides quantitative performance figures upon which the system operation can be optimized with respect to guaranteed real-time performance and energy efficiency under prescribed SLAs.

New parallel reservoir simulator software designed for Linux clusters enable to overcome hardware limitation and to simulate models with large amount of data. Reservoir engineering industry is very interested in using ever growing dataset with more and more complex physics and detailed models. The key issue still remains running simulations in an acceptable CPU time. As, the trend in hardware technologies is not to improve drastically the performance of individual CPUs but to facilitate the aggregation of computation facilities (with high bandwidth network, multi-core architectures …), the challenge is to improve the efficiency of reservoir simulation software on a large number of processors. New numerical difficulties and performance problems appear when the number of cells and the number of processors are growing. As a matter of fact, the architecture of Linux clusters is very sensible to memory distribution and load balancing: the cost of parallel solver algorithm is usually sensible to the size of the reservoir model (lack of scalability) and the consequences on CPU performance can no more be neglected;the domain decomposition algorithms used to distribute data between processors have a great influence on the computing load balancing between processors;using adaptive numerical schemes with dynamic space criteria (AIM schemes, flash algorithms based on the thermodynamic state of each cell) is a source of unbalance that cannot statically be resolved;simulation result storages on irregular data structures, such as unstructured grids, multilateral smart wells and perforated cells, lead to store an important amount of information during the simulation. With the variety of IO subsystems found on Linux clusters the simulator must be able to adapt its IO strategy to the underlying IO library/file system and hardware. In this paper, we present different approaches to overcome these kinds of problems. We discuss technical choices such like: advanced scalable linear solver algorithm;load balancing issue with different domain decomposition strategies;dynamic space criteria, mesh partitioner strategy and parallel solver performance management;flexible IO strategy from simple file system to more complex parallel file system or database. We have developed and benchmarked these different solutions on published reference large scale problems and actual case studies with several tens millions of cells. We analyze the results and discuss the efficiency of each solution to overcome the scalability difficulties and performance limitations due to load unbalance.

We study the problem of balancing the load on processors in an arbitrary network. If jobs arrive or depart during the process of load balancing, we have the dynamic load balancing problem; otherwise, we have the static load balancing problem. While static load balancing on a number of different networks has been well-studied, analytical work on dynamic load balancing is limited. The difficulty lies in modeling the arrivals and departures of jobs in a clean manner. In this paper, we introduce a new framework for dynamic load balancing in which the job traffic is modeled by an adversary. Our main result is that a simple local load balancing algorithm maintains the load of the network within a stable level against a powerful adversary. Our result holds for different models of traffic patterns and processor communication.

This paper describes load balancing techniques for the Time Warp distributed synchronization technique for object-oriented simulation. The Time Warp system distributes objects across nodes and provides optimistic concurrency control. Our implementation is Lisp-based and executes on a network of UNIX-based workstations, where system load varies depending on the number of users and processes. This technique achieves a balanced load by first determining if the load is unbalanced. Then if it is unbalanced, the system must determine which machine is the most underloaded and migrate an object to that machine. Several object-migration techniques were attempted before arriving at a method efficient enough to make dynamic load balancing viable. We discuss a static balancing technique to start the system with a balanced load, but variance in object behavior and individual workstation load require a dynamic load monitor. Time Warp simulations executed under dynamic balancing versus static load balancing are five to 10 times faster.

Routing is a very complex process in the field programmable gate array (FPGA) CAD flow. The increase of both FPGA size and design complexity leads to a long routing time hindering the productivity. In this article, we propose a more effective parallel router that combines static and dynamic load balance in parallel routing for FPGAs. First, we explore hierarchical region partitioning to assign routing tasks to different cores for static load balance. Then, we coordinate message propagation and task migration at runtime so that load balance between cores can be dynamically maintained in parallel routing. Finally, we combine static and dynamic load balance in the parallel routing for a higher degree of parallelism. Our parallel router performs on the multicore distributed-memory systems and the communication between cores is through message passing interface messages. We demonstrate the effectiveness of our parallel router using large-scale Titan designs. On average, our parallel router can scale up to 32 cores to achieve about 17× speedup with slight loss of quality, compared with the latest VTR 8 router.

This paper studies cluster load balancing policies and system support for fine-grain network services. Load balancing on a cluster of machines has been studied extensively in the literature, mainly focusing on coarse-grain distributed computation. Fine-grain services introduce additional challenges because system states fluctuate rapidly,for those services and system performance is highly sensitive to various overhead. The main contribution of our work is to identify effective load balancing schemes for fine-grain services through simulations and empirical evaluations on synthetic workload and real traces. Another contribution is the design and implementation of a load balancing system in a Linux cluster that strikes a balance between acquiring enough load information and minimizing system overhead. Our study concludes that: 1) Random polling based load-balancing policies are well-suited for fine-grain network services: 2) A small poll size provides sufficient information for load balancing, while an excessively large poll size may in fact degrade the performance clue to polling overhead; 3) Discarding slow-responding polls can further improve system performance.