A Load Distribution Algorithm Based on Profiling for Heterogeneous GPU Clusters

These years energy consumption has become the main constraint for the further improvement of computing ability of data centers. In this work, the energy efficient resource management strategies for web servers (one typical class of data centers) are discussed. We firstly introduce the daily pattern of requests sent to web servers which indicates that a simple off-line resource provisioning method would-be effective for web servers. Then, the energy efficiency of two load distribution algorithms are compared by replaying subset of a real-world Wikipedia trace in our local web service environment. The results show that, compared with the relative load distribution (RLB) algorithm, the adaptive load distribution (ALD) algorithm proposed in our previous work is more energy efficient. Finally, we demonstrate a potential that much energy can be saved by shutting down the idle nodes.

Finding exact hitting set solutions for systems biology applications using heterogeneous GPU clusters

Polling-based load distribution (LD) algorithms suffer from two weaknesses: (i) load information exchanged during a polling session is confined to the two negotiating nodes only; (ii) as the distributed system grows in size (in terms of the number of constituent nodes), a larger number of polling sessions, and thus a higher amount of network bandwidth consumption and CPU overhead, are needed. We propose a new LD algorithm which is based on anti-tasks and load state vectors. This new algorithm avoids the above weaknesses of polling-based LD algorithms. Anti-tasks are composite agents which travel around a distributed system to facilitate the pairing up of task senders and receivers, as well as the collection and dissemination of load information. Time-stamped load information of processing nodes is stored in load state vectors which, when used together with anti-tasks, encourage mutual sharing of load information among processing nodes. Anti-tasks, which make use of load state vectors to decide their traveling paths, are spontaneously directed towards processing nodes having high transient workload, thus allowing their surplus workload to be relocated quickly. Using simulations, we evaluate the performance of our new algorithm by comparing its performance with a number of well-known polling-based load distribution algorithms. We found that our algorithm provides significant reduction of mean task response time over a large range of system sizes. The cost of achieving this performance gain in terms of CPU overhead and channel bandwidth consumption is generally comparable to the other algorithms we studied. © 1998 John Wiley & Sons, Ltd.

Simulation of spiking neural networks — architectures and implementations

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents HAP, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. HAP jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A*-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, HAP explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that HAP achieves up to 2.41x speed-up on heterogeneous clusters. • Computing methodologies → Distributed computing methodologies.

HASE: Hardware-Aware Scheduling for Inference Tasks in Heterogeneous GPU Clusters

Electrical stimulation of neural systems is a key tool for understanding neural dynamics and ultimately for developing clinical treatments. Many applications of electrical stimulation affect large populations of neurons. However, computational models of large networks of spiking neurons are inherently hard to simulate and analyze. We evaluate a reduced mean-field model of excitatory and inhibitory adaptive exponential integrate-and-fire (AdEx) neurons which can be used to efficiently study the effects of electrical stimulation on large neural populations. The rich dynamical properties of this basic cortical model are described in detail and validated using large network simulations. Bifurcation diagrams reflecting the network's state reveal asynchronous up- and down-states, bistable regimes, and oscillatory regions corresponding to fast excitation-inhibition and slow excitation-adaptation feedback loops. The biophysical parameters of the AdEx neuron can be coupled to an electric field with realistic field strengths which then can be propagated up to the population description. We show how on the edge of bifurcation, direct electrical inputs cause network state transitions, such as turning on and off oscillations of the population rate. Oscillatory input can frequency-entrain and phase-lock endogenous oscillations. Relatively weak electric field strengths on the order of 1 V/m are able to produce these effects, indicating that field effects are strongly amplified in the network. The effects of time-varying external stimulation are well-predicted by the mean-field model, further underpinning the utility of low-dimensional neural mass models.

This paper presents a parallel algorithm for the simulation of large complex interconnect networks with frequency-dependent parameters. The equivalent transmission line model, which is suitable for time-domain simulation, is introduced for the modeling of dispersive interconnects. By means of the equivalent multi-port model, the interconnects are separated from other circuit elements, and thus makes the parallel simulation feasible. Based on the proposed technique, a program named FdSPICE is developed for parallel simulation of large interconnect networks. The accuracy and efficiency of FdSPICE are demonstrated by numerical examples.

As the deep learning model grows larger, training model with a single computational resource becomes impractical. To solve this, hybrid parallelism, which combines data and pipeline parallelism emerges to train large models with multiple GPUs. In practice, using heterogeneous GPU clusters to train large models is a common need due to the upgrade of a part of hardware. However, existing hybrid parallelism approaches in the heterogeneous environment do not work well in communication efficacy, workload balance among GPUs and utilizing the memory constrained GPU. To address these problems, we present a parallel DNN training approach, Hybrid Parallelism on Heterogeneous clusters (HPH). In HPH, we propose a topology designer that minimizes the communication time cost. Furthermore, HPH uses a partition algorithm that automatically partitions DNN layers among workers to maximize throughput. Besides, HPH adopts recomputation-aware scheduling to reduce memory consumption and further reschedule the pipeline to eliminate the extra time overhead of recomputation. Our experimental results on a 32-GPU heterogeneous cluster show that HPH achieves up to 1.42x training speed-ups compared with the state-of-the-art approach.

Neuronal network simulators are central to computational neuroscience, enabling the study of the nervous system through in-silico experiments. Through the utilization of high-performance computing resources, these simulators are capable of simulating increasingly complex and large networks of neurons today. Yet, the increased capabilities introduce a challenge to the analysis and visualization of the simulation results. In this work, we propose a pipeline for in-transit analysis and visualization of data produced by neuronal network simulators. The pipeline is able to couple with simulators, enabling querying, filtering, and merging data from multiple simulation instances. Additionally, the architecture allows user-defined plugins that perform analysis tasks in the pipeline. The pipeline applies traditional REST API paradigms and utilizes data formats such as JSON to provide easy access to the generated data for visualization and further processing. We present and assess the proposed architecture in the context of neuronal network simulations generated by the NEST simulator.

Parallel processing is a promising technique for reducing the execution time of a large and complex wireless network simulation. In this article, a parallel processing technique is presented for time-driven simulations of large and complex wireless networks. The technique explicitly considers the physical-layer details of wireless network simulators such as shadowing and co-channel interference. The technique uses multiple processors interconnected with highspeed links. Principles and issues are elaborated, including workload partitioning, synchronization, and database design. Numerical results from two different wireless network simulators employing the technique are presented to demonstrate the performance of the technique. The results indicate that the technique can significantly improve the run-time performance of time-driven simulations of large and complicated wireless networks of different types.

The study of energy filtering management process for microgrid based on the dynamic response model of vanadium redox flow battery

A new wearable robot for upper-extremity was designed for handling heavy weights. The human-robot integrated structure constitutes a closed-chain mechanism which allows a proper load distribution between the robot and the human. The kinematic and dynamic models for this human-robot integrated exoskeleton system were presented. Through the load distribution algorithm between the human and the robot, it was shown that the robot could withstand majority of the payload. Copyright ?? 2012 IEEE.

The authors describe various approaches to construction of an algorithm for the solution of the problem of load distribution at a combined heat and power (CHP) plant with the complex mix of the equipment and complex schemes of heat and electrical energy supply on the basis of which the software system has been developed. Methods of obtaining energy characteristics of the equipment used for solving the problem of load distribution were studied. The results of the implementation of the software system for load distribution at the CHP-23 plant belonging to OAO Mosenergo are given. Realization of recommendations on maintaining an operational mode of the equipment with due regard for its optimal loading makes it possible to obtain fuel savings of up to 1%.

Trailer type mobile robots have been studied from the viewpoint of control and design of nonholonomic mechanical systems in literature. However, researches for its detailed closed form kinematic and dynamic modeling are rarely being progressed. Though the trailer type mobile robot is shaped like a kinematically redundant system consisting of surplus joints, its actual kinematic characteristic is similar to a closed chain system since the motion of every trailer is constrained to the ground and the following trailers' motion is dependent on the leading trailer's motion. The exact kinematic relationship between the leading trailer and the following trailers is derived by the transfer method using the augmented generalized coordinates. A general dynamic model is derived using the Lagrangian approach and load distribution algorithm is suggested. It can be seen that the proposed approach facilitates distribution of the system load to all trailers. Forward dynamic simulation is conducted to verify the effectiveness of the suggested kinematic and dynamic models.