Turbo: Efficient Communication Framework for Large-scale Data Processing Cluster

Hadoop has become one popular framework to process massive data sets in a large scale cluster. However, it is observed that the detection of the failed worker is delayed, which may result in a significant increase in the completion time of jobs with different workload. To cope with it, we present two mechanisms: Adaptive interval and Reputation-based Detector that support Hadoop to detect the failed worker in the shortest time. The Adaptive interval is trying to dynamically configure the expiration time which is adaptive to the job size. The Reputation-based Detector is trying to evaluate the reputation of each worker. Once the reputation of a worker is lower than a threshold, then the worker will be considered as a failed worker. In our experiments, we demonstrate that both of these strategies have achieved great improvement in the detection of the failed worker. Specifically, the Adaptive interval has a relatively better performance with small jobs, while the Reputation-based Detector is more suitable for large jobs.

It has become routine for large volumes of data to be generated, stored, and processed across geographically distributed datacenters. To run a single data analytic job on such geo-distributed data, recent research proposed to distribute its tasks across datacenters, considering both data locality and network bandwidth across datacenters. Yet, it remains an open problem in the more general case, where multiple analytic jobs need to fairly share the resources at these geo-distributed datacenters. In this paper, we focus on the problem of assigning tasks belonging to multiple jobs across datacenters, with the specific objective of achieving max-min fairness across jobs sharing these datacenters, in terms of their job completion times. We formulate this problem as a lexicographical minimization problem, which is challenging to solve in practice due to its inherent multi-objective and discrete nature. To address these challenges, we iteratively solve its single-objective subproblems, which can be transformed to equivalent linear programming (LP) problems to be efficiently solved, thanks to their favorable properties. As a highlight of this paper, we have designed and implemented our proposed solution as a fair job scheduler based on Apache Spark, a modern data processing framework. With extensive evaluations of our real-world implementation on Amazon EC2 and large-scale simulations, we have shown convincing evidence that max-min fairness has been achieved and the worst job completion time has been significantly improved using our new job scheduler.

Big data systems handle vast volumes of structured and unstructured information using distributed computing frameworks like Apache Hadoop, Spark, and Flink. These platforms rely heavily on computational resources, and CPU utilization becomes a critical factor in determining the performance and efficiency of data processing operations. Effective CPU usage directly impacts throughput, latency, and the ability to meet service-level objectives. In large-scale clusters, workloads vary from CPU-intensive tasks such as real-time analytics, data mining, and machine learning to more I/O-driven operations like data ingestion or storage management. Poor CPU distribution can lead to overloaded nodes while others remain underutilized, ultimately reducing system efficiency. Traditional scheduling mechanisms in big data platforms are often rule-based and may not respond well to dynamic or unpredictable workloads, which contributes to suboptimal CPU usage. When CPU utilization remains consistently high, it can trigger job slowdowns, task failures, and increased power consumption, negatively affecting overall performance. Conversely, low CPU utilization indicates underused resources, leading to waste in computational capacity and energy. Additionally, container technologies and control groups (cgroups) support fine-tuned CPU allocation and isolation, ensuring fair usage across concurrent users or tasks in multi-tenant environments. In more advanced implementations, machine learning algorithms are used to forecast CPU needs and schedule jobs more efficiently by learning from historical patterns and usage behavior. Reinforcement learning approaches have also shown potential in achieving balanced CPU usage by adapting policies based on feedback and environment changes. These intelligent mechanisms allow big data systems to optimize CPU allocation continuously, adapting to fluctuating demand without manual intervention. Efficient CPU management contributes significantly to faster job execution, reduced operational costs, and higher cluster reliability. As data volumes and complexity continue to grow, maintaining optimal CPU utilization remains a priority for organizations aiming to derive timely insights while maximizing infrastructure value. Basic CPU utilization approaches are exhibiting performance limitations. This paper addresses performance limitations related to CPU utilization.

Industrial automation is expected to benefit from the capabilities of Beyond 5G (B5G)/6G, particularly in improving real-time decision making and control in information-driven scenarios, supporting improved efficiency, precision and safety in connected manufacturing environments. While not all aspects of industrial automation are equally affected, certain areas such as real-time monitoring, wireless control and predictive maintenance can benefit significantly from the high reliability, ultra-low latency and scalability of next-generation wireless systems. Motivated by the increasing convergence between operational technology (OT) and advanced communication infrastructures, this paper surveys how B5G/6G can serve as an enabler for data-centric industrial ecosystems. The main contribution lies in analysing evolving industrial architectures, the integration of OT systems and application domains including digital twins, cloud robotics and smart devices. At the end of the paper, we also offer strategic insights into enabling technologies and future challenges, providing a comprehensive overview of how B5G/6G can support the evolution of Industry 5.0.

The influence of interest on job completion time, ordinary and worthy job performance, and job satisfaction is investigated following the provision of job-related prior knowledge. This quasiexperimental posttest research compared studies of low- and high-value jobs. In Study 1, participants (n = 75) completed a dressing job (low value). In Study 2, participants (n = 71) completed a computer job (high value). Interest in the low-value job was found to trigger significant job performance and job satisfaction, but no significant impact was noted on job completion time. However, worthy job performance (in which performance is evaluated by total cost spent to acquire desired output) of the lowvalue job was nonsignificant and inverse to job performance. Interest in the highvalue job was seen to significantly affect job completion time and job satisfaction, but not job performance. In contrast to Study 1, worthy job performance was consistent with job performance and seen as significant. The results are discussed, and further research studies are recommended.

Two important characteristics encountered in many real-world scheduling problems are heterogeneous processors and a certain degree of uncertainty about the sizes of jobs. In this paper we address both, and study for the first time a scheduling problem that combines the classical unrelated machine scheduling model with stochastic processing times of jobs. Here, the processing time of job j on machine i is governed by random variable P_{ij} , and its realization becomes known only upon job completion. With w_j being the given weight of job j, we study the objective to minimize the expected total weighted completion time E[Sum w_j.C_j] , where C_j is the completion time of job j. By means of a novel time-indexed linear programming relaxation, we compute in polynomial time a scheduling policy with performance guarantee (3+D)/2+e. Here, e>0 is arbitrarily small, and D is an upper bound on the squared coefficient of variation of the processing times. When jobs also have individual release dates r_{ij}, our bound is (2+D)+e. We also show that the dependence of the performance guarantees on D is tight. Via D=0, currently best known bounds for deterministic scheduling on unrelated machines are contained as special case.

Pooling idle computing resources from stationary and computationally powerful vehicles at a nearby parking lot is one way to meet increasing computation demands from mobile devices at the network's edge. This new computing paradigm is called vehicular cloud (VC). Despite the research in performance modeling of VC having gained increasing interest in recent years, a tractable solution for job completion time under concurrent task execution has yet to be seen. Thus, this work investigates the computation capabilities of a VC by analytically determining job completion time when the VC simultaneously executes several tasks. Under several stochastics assumptions, we derive the probability density functions of completion times of the tasks, which also include the job completion time and their first moments. Finally, we present the numerical results for the analysis and the simulation results to show the correctness of the work.

It has been recently discovered that on an unreliable server, the job completion time distribution function (df) can be heavy-tailed (HT) even when job size df is light-tailed (LT) [1, 5]. A key to this phenomenon is the RESTART feature where if a job is interrupted in the middle of its processing, the entire job needs to restart from the beginning, i.e., the work that is partially completed is lost.
\n
\nA standard mechanism for reducing the job completion
\ntime in an unreliable service environment is checkpointing
\n[3, 4, 6]. We view checkpointing as a job fragmentation operation, where the server processes one fragment of the job at a time. If the server becomes unavailable, say due to failure, then only the work corresponding to the fragment being processed at the time of failure is lost. In this paper, we are motivated by the question: Can fragmentation ‘lighten’ the tail df of the completion time? In Section 3, we provide sufficient conditions on the fragmentation policy that gives rise to LT completion time so long as the job size df is LT. We then characterize the optimal fragmentation policy seeking to minimize the expected job completion time. This policy
\nrequires a priori knowledge of the job size. We then describe a sub-optimal fragmentation policy that is blind to the job size and is provably very close to optimal. We also describe the asymptotic tail behavior of the job completion time df under both policies. Assuming the server unavailability periods are LT, both policies produce LT completion times when the job size df is LT. For the case of regularly varying job size df, the job completion time under both policies is regularly varying with the same degree - this is the lightest possible asymptotic tail behavior (in the degree sense).

This paper presents a low-complexity algorithm for online job scheduling at workcenters along a given route in a manufacturing system. At each workcenter, the job has to be processed by any one of a given set of identical machines. Each machine has a preset schedule of operations, leaving out time-windows during which the job's processing must be scheduled. The manufacturing system has no internal buffers and the job cannot wait between two consecutive operations. There is some flexibility in the job's processing times, which must be confined to given time intervals. The scheduling algorithm minimizes the job's completion time, and it is executable in real time whenever the job requirement is generated.

In this paper, we study the problem of job dispatching and scheduling, where each job consists of a set of tasks. Each task is processed by a set of machines simultaneously. We consider two important performance metrics, the average job completion time (JCT), and the number of deadline-aware jobs that meet their deadlines. The goal is to minimize the former and maximize the latter. We first propose OneJ to minimize the job completion time (JCT) when there is exactly one single job in the system. Then, we propose an online algorithm called MultiJ, taking OneJ as a subroutine, to minimize the average JCT, and prove it has a good competitive ratio. We then derive another online algorithm QuickJ to maximize the number of jobs that can meet their deadlines. We show that QuickJ is competitive via a worst case analysis. We also conjecture that the competitive ratio of QuickJ is likely to be the best one that any deterministic algorithm can achieve. We also shed light on several important merits of MultiJ and QuickJ, such as no severe coordination overhead, scalability, work conservation, and no job starvation.

As MapReduce is becoming ubiquitous in large-scale data analysis, many recent studies have shown that the performance of MapReduce could be improved by different job scheduling approaches, e.g., Fair Scheduler and Capacity Scheduler. However, most exiting MapReduce job schedulers focus on the scenario that MapReduce cluster is stable and pay little attention to the MapReduce cluster with dynamic resource availability. In fact, MapReduce cluster resources may fluctuate as there is a growing number of Hadoop clusters deployed on hybrid systems, e.g., infrastructure powered by mix of traditional and renewable energy, and cloud platforms hosting heterogeneous workloads. Thus, there is a growing need for providing predictable services to users who have strict requirements on job completion times in such dynamic environments. In this paper, we propose, RDS , a Resource and Deadline-aware Hadoop job Scheduler that takes future resource availability into consideration when minimizing job deadline misses. We formulate the job scheduling problem as an online optimization problem and solve it using an efficient receding horizon control algorithm. To aid the control, we design a self-learning model to estimate job completion times. We further extend the design of RDS scheduler to support flexible performance goals in various dynamic clusters. In particular, we use flexible deadline time bounds instead of the single fixed job completion deadline. We have implemented RDS in the open-source Hadoop implementation and performed evaluations with various benchmark workloads. Experimental results show that RDS substantially reduces the penalty of deadline misses by at least 36 and 10 percent compared with Fair Scheduler and Earliest Deadline First (EDF) scheduler, respectively. In a Hadoop cluster running partially on renewable energy, the experimental result shows the green power based resource prediction approach can further reduce the penalty of deadline misses by 16 percent compared to Auto-Regressive Integrated Moving Average (ARIMA) prediction approach.

Modern cluster management systems, such as Kubernetes, support heterogeneous workloads and resources. However, existing resource schedulers in these systems do not differentiate between heterogeneous G PU resources-which are becoming a norm-and do not support GPU sharing-which is necessary to support emerging collocation of jobs and multi-tenant applications. Thus the systems suffer from low GPU resource utilization, higher queuing delays, and an increase in application makespan, i.e., the duration between the arrival of the first job and the completion of the last job of a workflow. This is especially a problem in supporting crucial deep learning (DL) applications. To this end, in this paper, we profile and analyze DL jobs on heterogeneous GPUs, investigate the interference caused by collocating jobs on GPUs, and use this information to predict the GPU memory demand and job completion times. We propose SCHEDTUNE, a machine-learning-based heterogeneity-aware scheduler that ensures higher GPU memory utilization and reduced out-of-memory (OOM) failures, while supporting improved makespan. Our evaluation shows that SCHEDTUNE GPU memory predictors and scheduler outperform the state-of-the-art predictors by achieving 81% higher GPU memory utilization, 100% detection and avoidance of OOM errors, and 17.5% reduction in makespan compared to the default Kubernetes scheduler.

Modeling NaTech-related domino effects in process clusters: A network-based approach

The existing resource scheduling algorithms for virtual machines usually use serial job deployment ways which easily lead to the job completion time overlong and the system load unbalance. To solve the problems, an Improved Potential Capacity (IPC) based resource scheduling algorithm for virtual machines is proposed, which comprehensively considers the overall job completion time and system load balancing, and applies a new metric to dynamically estimate the resource remaining capacities of virtual machines, and thus reduce the inexact matching between jobs and virtual machines. A batch job deployment method is also proposed to execute the batch job deployment. Many simulation experimental results show that the proposed algorithm can effectively decrease the overall job completion time and improve the load balancing of a cloud system.

In this paper, we consider bicriterion scheduling problems involving job completion times and job waiting times in a nonpreemptive single machine environment. As an example, we seek to determine a schedule that yields both a low mean and a low variation of completion time. Such a schedule is attractive when the system has the dual objectives of minimizing in-process inventory and providing different jobs the same quality of service. Similarly, we seek a schedule that performs well on both the mean and variation of waiting time. As the measure of variation, we use the sum of absolute pairwise differences. We present an efficient algorithm for minimizing the variation of waiting time. Next, we assume that the total scheduling cost is a function of the mean and variation of completion time alone, and is linear. We give an efficient algorithm for minimizing this total cost. Finally, we consider parametric analysis of this cost function and present simple procedures for generating all or some of the optimal schedules. All the results are extended to the case where job waiting times are substituted for job completion times.