Scaling Synchronization in Multicore Programs

The fundamental problems in dynamic load balancing and job scheduling in parallel and distributed computers involve moving load between processors. In this paper, we consider a new model for load movement in synchronous parallel and distributed machines. In each step of our model, each processor can transfer load to at most one neighbor; also, any amount of load can be moved along a communication link between two processors in one step. This is a reasonable model for load movement in significant classes of dynamic load balancing problems.We derive efficient algorithms for a number of task reallocation problems under our model of load movement. These include dynamic load balancing on processor networks, adaptive mesh re-partitioning such as those in finite element methods, and progressive job migration under dynamic generation and consumption of load.To obtain the above-mentioned results, we introduce and solve the abstract problem of Incremental Weight Migration (IWM) on arbitrary graphs. Our main result is a simple, randomized, algorithm for this problem which provably results in asymptotically optimal convergence towards the state where weights on the nodes of the graph are all equal. This algorithm utilizes an appropriate random set of edges forming a matching. Our algorithm for the IWM problem is used in deriving efficient algorithms for all the problems mentioned above.Our results are very general. The algorithms we derive are local, and hence, scalable. They work for arbitrary load distributions and for networks of arbitrary topology which can possibly undergo link failures. Of independent interest is our proof technique which we use to lower bound the convergence of our algorithms in terms of the eigenstructure of the underlying graph.Finally, we present preliminary experimental results analyzing issues in load balancing related to our algorithms.

This paper proposes a self‐routing Benes network controlled distributively by dynamic load balancing. This Benes network consists of a distribution network and a routing network. The distribution network is controlled by a proposed method called “Dynamic Load Balance,” where load information is fed back to the input side via each unit switch so that loads at intermediate unit switches are equalized.The routing network is self‐routed by an output tag. The computer simulation results show that mean time delay vs. throughput characteristic of the proposed dynamic load balance Benes network is superior to those of conventional networks.

As multi-core computers becoming the main computation force of the computer network, dynamic load balancing for parallel applications on multi-core computers needs to be studied. The paper describes a method for extending the dynamic load balancing tool to support multi-core computers. The method scales all cost function information into per CPU core unit and performs dynamic load balancing in terms of CPU cores. The parallel processes are assigned to each computer (not each CPU core) assuming that the operating system assigns the parallel processes equally among all CPU cores. The proposed tool has been successfully tested for supporting parallel CFD applications.

Routing is a very complex process in the field programmable gate array (FPGA) CAD flow. The increase of both FPGA size and design complexity leads to a long routing time hindering the productivity. In this article, we propose a more effective parallel router that combines static and dynamic load balance in parallel routing for FPGAs. First, we explore hierarchical region partitioning to assign routing tasks to different cores for static load balance. Then, we coordinate message propagation and task migration at runtime so that load balance between cores can be dynamically maintained in parallel routing. Finally, we combine static and dynamic load balance in the parallel routing for a higher degree of parallelism. Our parallel router performs on the multicore distributed-memory systems and the communication between cores is through message passing interface messages. We demonstrate the effectiveness of our parallel router using large-scale Titan designs. On average, our parallel router can scale up to 32 cores to achieve about 17× speedup with slight loss of quality, compared with the latest VTR 8 router.

Modern standards in High Performance Computing (HPC) have started to consider energy consumption and power draw as a limiting factor. New and more complex architectures have been introduced in HPC systems to afford these new restrictions, and include coprocessors such as GPGPUs for intensive computational tasks. As systems increase in heterogeneity, workload distribution becomes a more core problem to achieve the maximum efficiency in every computational component. We present a Multi-Objective Dynamic Load Balancing (DLB) approach where several objectives can be applied to tune an application. These objectives can be dynamically exchanged during the execution of an algorithm to better adapt to the resources available in a system. We have implemented the Multi-Objective DLB together with a generic heuristic engine, designed to perform multiple strategies for DLB in iterative problems. We also present Ull Multiobjective Framework (UllMF), an open-source tool that implements the Multi-Objective generic approach. UllMF separates metric gathering, objective functions to be optimized and load balancing algorithms, and improves code portability using a simple interface to reduce the costs of new implementations. We illustrate how performance and energy consumption are improved for the implemented techniques, and analyze their quality using different DLB techniques from the literature.

Increasingly large-scale clusters of PC/WS continue to become majority platform in HPC field. Such a commodity cluster environment, there may be incremental upgrade due to several reasons, such as rapid progress in processor technologies, or user needs and it may cause the performance heterogeneity between nodes from which the application programmer will suffer as load imbalances. To overcome these problems, some dynamic load balancing mechanisms are needed. In this paper, we report our ongoing work on dynamic load balancing extension to Omni/SCASH which is an implementation of OpenMP on Software Distributed Shared Memory, SLASH. Using our dynamic load balancing mechanisms, we expect that programmers can have load imbalances adjusted automatically by the runtime system without explicit definition of data and task placements in a commodity cluster environment with possibly heterogeneous performance nodes.

The capacity of wireless networks can be increased via dynamic load balancing/sharing by employing overlay networks on top of the existing cellular networks. One such recently proposed system is the integrated cellular and ad hoc relay (iCAR) system, where an overlay ad hoc network is employed to use the resources efficiently by dynamically balancing the load of the hot spots in the cellular network, and to provide quality-of-service to subscribers, no matter where they are located and when the request is made. It is assumed that this overlay network operates in the 2.4-GHz Industrial, Scientific, and Medical (ISM) band and, hence, the number of available ISM-band relay channels used for load balancing will be limited due to other users' interference at a given point in time. In this paper, the impact of ISM-band interference on the performance of iCAR systems, which is a representative hybrid wireless network, is studied, and it is shown that dynamic load balancing and sharing capabilities of iCAR systems are strictly dependent on the availability of the ISM-band relay channels. In addition to quantifying the impact of the number of available relay channels on the performance of iCAR systems, a simple channel assignment scheme to reduce the performance degradation due to other users' interference is also provided. Results show that this interference avoidance technique can improve the realistic performance of iCAR-like hybrid wireless networks by 12%-23% when the interferers are uniformly distributed in the ISM-band.

Large scale computing requires parallelization in order to arrive at solution at reasonable time. Today parallelization is a standard in fluid problems simulation. On the other hand adaptation is a technique that allows for dynamic modification of the mesh as the need for locally higher resolution arises. Adaptation used during parallel simulation leads to unbalanced numerical load. This in turn decreases the efficiency of parallelization. Dynamic load balancing strategies should be applied in order to ensure proper parallelization efficiency. The paper presents the potential benefits of applying the dynamic load balancing to adaptive flow problems simulated in parallel environments.

Feng, X.; Wan, M., and Li, S., 2020. The design of marine remote sensing image fusion model based on dynamic load balancing strategy. In: Yang, Y.; Mi, C.; Zhao, L., and Lam, S. (eds.), Global Topics and New Trends in Coastal Research: Port, Coastal and Ocean Engineering. Journal of Coastal Research, Special Issue No. 103, pp. 943–946. Coconut Creek (Florida), ISSN 0749-0208.In this work, a model of marine remote sensing image fusion based on dynamic load balancing is proposed. The remote sensing image acquisition system based on Android intelligent device obtains the marine remote sensing image, and enhances and registers the image based on the fuzzy theory and template matching, respectively, to realize the image preprocessing. Based on the preprocessing results, considering the small number of objects in the marine remote sensing image, the target area is divided by the idea of quad tree structure. At the same time, the task allocation strategy of dynamic load balancing and the idea of parallel computing are used to propose the task allocation algorithm processing model of the cluster system for the fusion processing of the image in the target area, so as to achieve the fusion of marine remote sensing image. It can be known that the proposed model takes less time and has superior average gradient and clarity of image.

In Cloud Computing (CC) environment, load balancing refers to the process of optimizing resources of virtual machines. Load balancing in the CC environment is one of the analytical approaches utilized to ensure indistinguishable workload distribution and effective utilization of resources. This is because only by ensuring effective balance of dynamic workload results in higher user satisfaction and optimal allocation of resource, therefore improve cloud application performance. Moreover, a paramount objective of load balancing is task scheduling because surges in the number of clients utilizing cloud lead to inappropriate job scheduling. Hence, issues encircling task scheduling has to be addressed. In this work a method called, Ford Fulkerson and Newey West Regression-based Dynamic Load Balancing (FF-NWRDLB) in CC environment is proposed. The FF-NWRDLB method is split into two sections, namely, task scheduling and dynamic load balancing. First, Ford Fulkerson-based Task Scheduling is applied to the cloud user requested tasks obtained from Personal Cloud Dataset. Here, employing Ford Fulkerson function based on the flow of tasks, energy-efficient task scheduling is ensured. The execution of asymmetrical scientific applications can be smoothly influenced by an unbalanced workload distribution between computing resources. In this context load balancing signifies as one of the most significant solution to enhance utilization of resources. However, selecting the best accomplishing load balancing technique is not an insignificant piece of work. For example, selecting a load balancing model does not work in circumstances with dynamic behavior. In this context, a machine learning technique called, Newey West Regression-based dynamic load balancer is designed to balance the load in a dynamic manner at run time, therefore ensuring accurate data communication. The FF-NWRDLB method has been compared to recent algorithms that use the markov optimization and the prediction scheme to achieve load balancing. Our experimental results show that our proposed FF-NWRDLB method outperforms other state of the art schemes in terms of energy consumption, throughput, delay, bandwidth and task scheduling efficiency in CC environment.

Parallelization and dynamic load balancing of the two-dimensional and three-dimensional NPARC flow codes are presented. A previously developed parallel database package, GPAR, and a dynamic load balancer program, DLB, are used for both the two-dimensional and three-dimensional versions of NPARC. Performance characteristics of the implemented algorithms in two-dimensional and three-dimensional internal flow configurations are explored. Dynamic load balancing studies are carried out with the two parallel codes for an engine inlet configuration. The benchmark cases consist of a two-dimensional case with 4592 grid points and 2 three-dimensional cases, one with 50,950 grid points and the other with 240,000 grid points. The grids are decomposed into solution blocks and parallel computations are carried out with varying number of processors. The pressure response to unsteady perturbations of the inlet temperature is calculated using a variable time-stepping approach specifically developed for parallel computations, which takes into account the time-step variations in blocks with selective communication between the blocks. Time accuracy is maintained with the benefits of increased speedup

In the era of petascale supercomputing, the importance of load balancing is crucial. Although dynamic load balancing is widespread, it is increasingly difficult to implement effectively with thousands of processors or more, prompting a second look at static load-balancing techniques even though the optimal allocation of tasks to processors is an NP-hard problem. We propose a heuristic static load-balancing algorithm, employing fitted benchmarking data, as an alternative to dynamic load balancing. The problem of allocating CPU cores to tasks is formulated as a mixed-integer nonlinear optimization problem, which is solved by using an optimization solver. On 163,840 cores of Blue Gene/P, we achieved a parallel efficiency of 80% for an execution of the fragment molecular orbital method applied to model protein-ligand complexes quantum-mechanically. The obtained allocation is shown to outperform dynamic load balancing by at least a factor of 2, thus motivating the use of this approach on other coarse-grained applications.

A parallel tree code for large N-body simulation: dynamic load balance and data distribution on a CRAY T3D system

There are two basic concerns for supporting multi-dimensional range query in P2P overlay networks. The first is to preserve data locality in the process of data space partitioning, and the second is the maintenance of data locality among data items with an exponentially expanding rate and an exponentially extending rate. The first problem has been well addressed by using recursive decomposition schemes, such as Quad tree, K-d tree, Z-order, and Hilbert curve. While the second problem was recently identified by our novel data structure: HD Tree. This paper is a follow-up to our previous work in HD Tree. In this paper, we explore how data locality can be easily maintained, and how range query can be efficiently supported in HD Tree. This is done by introducing two basic routing strategies, hierarchical routing and distributed routing. Although hierarchical routing can be applied to any two nodes in the system, it generates high volume traffic towards nodes near the root, and has very limited options to cope with a node failure. On the other hand, distributed routing concerns source and destination pairs only at the same depth, but traffic load is bound to some nodes at two neighboring depths, and multiple options can be found to redirect a routing request. Because HD Tree supports multiple routes between any two nodes in the system, routing in HD Tree is very flexible, and can be designed for many purposes, like fault tolerance, or dynamic load balancing. Distributed Routing Oriented Combined Routing algorithm is one such routing strategies implemented so far. It is a hybrid algorithm combining advantages from both the hierarchical routing and the distributed routing. The experimental results show that the DROCR algorithm achieves considerable performance gain over the equivalent tree routing at the highest depth examined. In the experiment of supporting multi-dimensional range query, we employ the Z-order space filling curve over the HD Tree overlay layer. We are expecting that the performance of range query will vary proportionally with the change of range size, and reasonably with the increase of dimensionality.

This work presents a GPU-based backtracking algorithm for permutation combinatorial problems based on the Integer-Vector-Matrix (IVM) data structure. IVM is a data structure dedicated to permutation combinatorial optimization problems. In this algorithm, the load balancing is performed without intervention of the CPU, inside a work stealing phase invoked after each node expansion phase. The proposed work stealing approach uses a virtual n-dimensional hypercube topology and a triggering mechanism to reduce the overhead incurred by dynamic load balancing. We have implemented this new algorithm for solving instances of the Asymmetric Travelling Salesman Problem by implicit enumeration, a scenario where the cost of node evaluation is low, compared to the overall search procedure. Experimental results show that the dynamically load balanced IVM-algorithm reaches speed-ups up to 17\(\times \) over a serial implementation using a bitset-data structure and up to 2\(\times \) over its GPU counterpart.