Loop scheduling for heterogeneity

In this paper, we consider the problem of scheduling and mapping precedence-constrained tasks to a network of heterogeneous processors. In such systems, processors are usually physically distributed, implying that the communication cost is considerably higher than in tightly coupled multiprocessors. Therefore, scheduling and mapping algorithms for such systems must schedule the tasks as well as the communication traffic by treating both the processors and communication links as important resources. We propose an algorithm that achieves these objectives and adapts its tasks scheduling and mapping decisions according to the given network topology. Just like tasks, messages are also scheduled and mapped to suitable links during the minimization of the finish times of tasks. Heterogeneity of processors is exploited by scheduling critical tasks to the fastest processors. Our extensive experimental study has demonstrated that the proposed algorithm is efficient, robust, and yields consistent performance over a wide range of scheduling parameters.

This paper presents a performance estimator, a prototype of which is implemented within the parallel programming environment PCASE. The estimation is based on static performance prediction, using not only the design information of target machines but also benchmarking results. Additionally, communication costs are estimated based on a hierarchical memory machine model, which enables users to understand all the underlying communication costs on distributed memory machines for each parallel loop. With this performance suggestion, it is possible to interactively optimize data distribution and appropriately select vectorized or parallelized loops. Moreover, the skeleton profiling method is presented. It makes high speed trace (execution count of each statement) generation possible by deleting statements which do not affect the execution path of a program.KeywordsExecution PathExecution CostTarget MachineMemory MachineVector InstructionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

High-level data-parallel programming with distributed object sets eases many aspects of parallel programming. In this paper we describe the design and implementation of Taco, a template library that provides higher-order operations on polymorphic regular and irregular distributed object sets by means of reusable topology classes and C++ function templates.Keywordsdynamic collectionsdistributed data-parallel processingglobal communication

The first session of the 18th International Real Time Ada Workshop discussed two aspects of parallel programming in real-time systems, the use of executors in parallel systems, and syntax to guide the reduction of parallel computations to return a correct single answer. This report captures the discussions held and the decisions and recommendations of the workshop on these topics.

Grid computing integrates scattered clusters, servers, storages and networks in different geographic locations to form a virtual super-computer. Along with the development of grid computing, dealing with the data distribution requires a method which is faster and more effective for parallel applications in order to reduce data exchange between clusters. In this paper, we present two methods to reduce inter-cluster communication cost based on the consideration to different kinds of communication cost and a simple logic mapping technology. Our theoretical analyses and simulation results show the proposed methods are better than the methods without reordering processor and considering the communication cost. The performance evaluation shows that the proposed methods not only reduce communication cost successfully but also achieve a great improvement.

Language constructs that support parallel computing are relatively well recognized at this point, with features such as parallel loops (optionally with reduction operators), divide-and-conquer parallelism, and general parallel blocks. But what language features would make distributed computing safer and more productive? Is it helpful to be able to specify on what node a computation should take place, and on what node data should reside, or is that overspecification? We don't normally expect a user of a parallel programming language to specify what core is used for a given iteration of a loop, nor which data should be moved into which core's cache. Generally the compiler and the run-time manage the allocation of cores, and the hardware worries about the cache. But in a distributed world, communication costs can easily outweigh computation costs in a poorly designed application. This panel will discuss various language features, some of which already exist to support parallel computing, and how they could be enhanced or generalized to support distributed computing safely and efficiently.

Considers at run-time t the dynamic load redistribution of the m(t) interacting heterogeneous unexecuted modules of a given workload over the p(t) heterogeneous processors available at time t. Average-valued parameters are used to model the workload parameters and its execution on the system processors. We seek to minimize a multi-metric objective function representing a combination of workload completion time, communication cost, resource utilization cost, and processor idle-time cost. The efficacy of a processor is defined as a composite measure of its CPU speed, memory speed, and the averaged workload parameters. The optimal load distribution is found to be the apportionment of the total load among a subset of the available processors, composed of the q(t)/spl les/p(t) most efficacious processors, in direct proportion to their efficacies. >

Large pre-trained models (LPMs) provide essential technical support for downstream Artificial Intelligence (AI) tasks spawning under the intelligent evolution of wireless networks. Using parameter-efficient fine-tuning (PEFT) technology, LPM can effectively capture task-specific PEFT knowledge and generalize to new tasks. However, the inherent system heterogeneity of wireless networks and data across user equipments (UEs) cause the heterogeneous PEFT models. This in turn brings difficulties in combining the PEFT knowledge of users, and thus degrades the global generalization capability of LPM. Meanwhile, additional computational and communication costs will be incurred if conventional heterogeneous model training methods are used for LPM to capture the global PEFT knowledge. Therefore, effective PEFT knowledge sharing becomes crucial for exploiting the global generalization capability of LPM. In this paper, we propose a PEFT knowledge sharing based distributed fine-tuning (DFT) scheme, by aggregating heterogeneous PEFT models. In DFT, the PEFT knowledge of users is captured with low-rank adaptive (LoRA) model, where the LoRA models can be heterogeneous for different users with specific resource constraints. On this basis, we further design an aggregation and iterative learning strategy for the heterogeneous LoRA models to capture the global PEFT knowledge from downstream users. Specifically, the aggregation process only requires simple alignment of heterogeneous LoRA models, without introducing additional computation cost for either users or aggregation server. Meanwhile, users perform iterative learning based on individual reinitialized local LoRA models, so as to avoid additional communication cost in aggregating and using a global LoRA model. We theoretically analyze the convergence bound of our proposed DFT scheme. Moreover, we conduct comprehensive simulation experiments based on the Databricks-dolly-15k dataset and state-of-the-art LPMs. The numerical results verify the effectiveness of our proposed DFT scheme in enhancing the global generalization capability of LPM while significantly reducing communication cost.

With the proliferation of workstation clusters connected by high-speed networks, providing efficient system support for concurrent applications engaging in nontrivial interaction has become an important problem. Two principal barriers to harnessing parallelism are: efficient mechanisms that achieve transparent dependency maintenance while preserving semantic correctness; and scheduling algorithms that match coupled processes to distributed resources while explicitly incorporating their communication costs. This paper describes a set of performance features, their properties, and implementation in a system support environment called DUNES that achieves transparent dependency maintenance-IPC, file access, memory access, process creation/termination, process relationships-under dynamic load balancing. The two principal performance features are push/pull-based active and passive end-point caching and communication-sensitive load balancing. Collectively, they mitigate the overhead introduced by the transparent dependency maintenance mechanisms. Communication-sensitive load balancing, in addition, affects the scheduling of distributed resources to application processes where both communication and computation costs are explicitly taken into account. The DUNES architecture endows commodity operating systems with distributed operating system functionality while achieving transparency with respect to their existing application base. DUNES also preserves semantic correctness with respect to single processor semantics.

Graph computing plays an important role in mining data at large scale. Partition is the primary step when we process large graph in a distributed system. A good partition has less communication and memory cost as well as more balanced load to take advantage of the whole system. Traditional edge cut methods introduce large communication cost for realistic power law graphs. Current vertex cut methods perform poorly with little consideration on load balance especially for online streaming vertex cut partition. In this paper, we formulate the total cost (partition cost, communication cost and computing cost) of graph computing especially that in iterating algorithms and analyze the cost of current partitioning methods. In addition, we explore a novel vertex cut method to ensure lower total cost. It has more balanced load with fewer communications. Experiments show that our method outperforms in state of the art graph computing frameworks at an average of 10 percent.

Programmers of sequential and parallel computers share the objective of developing fast, reliable programs as efficiently as possible. Already, considerable effort has been made to develop tools and techniques for sequential computers. Several approaches have been devised to meet the parallel programming challenge. Sequential languages have been adapted for parallel computers; new parallel languages designed; and tools developed which aid the programmer in detecting parallel code. A parallel programming assistant can aid the user in several ways: it can carry out tasks that are simple but mundane to the user, it can make comprehension easier for the user by summarizing program information; and it can cooperate with the user in doing a complex task, thus combining its own knowledge with that of the user's.

In this paper, based on the analyzed emerging challenges encountered by the future radio network, we focus on several candidate techniques targeting at a more spectrum-efficient and cost-effective solution. Among them, spectrum management is one special topic covering fields of technique, regulation and politics. We point out the high tendency of spectrum liberation in terms of spectrum allocation for services and radio technologies even between different network providers. In the network planning topic, we address a number of solutions w.r.t. some typical dominating factors. A solution reducing the risk of high expenditure in conjunction with the flexibility of spectrum management is also given. To further enhance the efficiency, advanced radio resource management allowing interworking among heterogeneous radio technologies is also presented. I. Introduction Furthering the big success of the GSM system (Global System forMobile communications) with the population of subscribers reaches over 250 million worldwide, the international mobile and wireless market turns out to the evolution of third Generation Mobile Communication System termed as the UMTS (Universal MobileT elecommunication System) and the system beyond it. Future radio network faces practical challenges such as the higher throughput and more elastic traffic demand based on the exploiting multimedia services, the scarcity and deployment difficulty, the heterogeneity of different radio air interface and the high cost of network deployment and operation. To achieve high spectrum efficiency with high coverage is always the goal to design a radio network. After the significant contribution from Shannon in the late 50’s, people research on radio transmission technology targeting at approaching the Shannon bounds for different scenarios like AWGN noise-limited channel, fading channel, multi-user communication environment, etc. Several technologies achieve significant improvement of spectrum efficiency, such as the advanced channel coding schemes, diversity approach, digital modulation schemes, joint source-channel coding schemes, accurate channel estimation and joint detection techniques, antenna techniques like MIMO (Multiple Input and Multiple Output), etc. From higher layer mechanism viewpoint, one can see the performance improvements given by a number of mechanisms like Medium Access Control P rotocol (MAC), Data Link Control P rotocols (DLC), Radio Link Control P rotocols (RLC) as well as Radio Resource Control P rotocols (RRC). In the network layer and beyond, mechanisms enhancing the performance for TCP (T ransport control protocol) and UDP (User Diagram protocol) over IP protocol are required for the wireless communication. From the system level, advanced network planning and management techniques are needed catering for the heterogeneity and therefore facilitating the interoperability of the co-existing Radio Access T echnologies (RATs). Apart from the cellular technologies, kinds of Broadband W ireless Access (BWA) technology have been emerged for years. Typical examples are the WLAN (W ireless Local Area Network) represented by IEEE 802.11 family, Hiperlan in Europe and WiMAX (W orldwide interoperability for Microwave Access) represented by IEEE 802.16 family. In order to offer ubiquitous wireless access, the interworking and harmonization among heterogeneous networks are considered as primary important in the coming years. One aspect of it is to design the wireless local network as a complementarity to the cellular radio network, which must be considered in the network planning phase for such heterogeneous networks. Different to the classical network planning scheme, network co-planning for heterogeneous networks is not only based on the carrier strength that the mobile terminals can receive, traffic distribution, networks’ deployment costs, their interoperability, and the terminal capabilities and spectrum management are tightly associated topics.

The main function of parallelizing compilers is to analyze sequential programs, in particular the loop structure, to detect hidden parallelism and automatically restructure sequential programs into parallel subtasks that are executed on a multiprocessor. This article describes the design and implementation of an efficient parallelizing compiler to parallelize loops and achieve high speedup rates on multiprocessor systems. It is well known that the execution efficiency of a loop can be enhanced if the loop is executed in parallel or partially parallel, such as in a DOALL or DOACROSS loop. This article also reviews a practical parallel loop detector (PPD) that is implemented in our PFPC on finding the parallelism in loops. The PPD can extract the potential DOALL and DOACROSS loops in a program by verifying array subscripts. In addition, a new model by using knowledge‐based approach is proposed to exploit more loop parallelisms in this paper. The knowledge‐based approach integrates existing loop transformations and loop scheduling algorithms to make good use of their ability to extract loop parallelisms. Two rule‐based systems, called the KPLT and IPLS, are then developed using repertory grid analysis and attribute‐ordering tables respectively, to construct the knowledge bases. These systems can choose an appropriate transform and loop schedule, and then apply the resulting methods to perform loop parallelization and obtain a high speedup rate. For example, the IPLS system can choose an appropriate loop schedule for running on multiprocessor systems. Finally, a runtime technique based on the inspector/executor scheme is proposed in this article for finding available parallelism on loops. Our inspector can determine the wavefronts of a loop with any complex indirected array‐indexing pattern by building a DEF‐USE table. The inspector is fully parallel without any synchronization. Experimental results show that the new method can resolve any complex data dependence patterns where no previous research can. One of the ultimate goals is to construct a high‐performance and portable FORTRAN parallelizing compiler on shared‐memory multiprocessors. We believe that our research may provide more insight into the development of a high‐performance parallelizing compiler. Copyright © 2001 John Wiley & Sons, Ltd.

Mobile communication plays an important role in the future of communication systems. To meet personalized and intelligent requirements, mobile communication has evolved from single wireless cellular network to heterogeneous mobile communication network, including wireless cellular network, wireless local network, and wireless personal network. In the heterogeneous communication system, to entirely fulfill the spectrum resource complementary advantages of such a heterogeneous wireless network, the spectrum resource trade algorithm has attracted tremendous research efforts. In this paper, the advantages and disadvantages of dynamic spectrum allocation and load balancing are discussed and spectrum auction takes the place of spectrum allocation to maximize the operation revenue. The authors design a joint spectrum auction and load balancing algorithm (SALB) based on heterogeneous network environment to develop a more flexible and efficient spectrum management. The simulation result shows that SALB increases the operation profit significantly.

This paper describes a dynamic load balancing tool intended for computational investigators who have little familiarity with programming for a message-passing environment. Motivated by the PAR DOALL directive available in some compilers for shared-memory systems, the tool is designed to simplify the manual conversion of sequential programs containing computationally intensive loops with independent iterates into parallel programs that execute with high efficiency on general-purpose clusters. The tool implements a dynamic loop scheduling strategy to address load imbalance which may be induced by the non-uniformity of loop iterate times, and by the heterogeneity of processors. The tool is based on the Message Passing Interface library for wide availability. Timings of a nontrivial application that utilize the tool on a Linux cluster are presented to demonstrate sample achievable performance.