Piranha

It is common to find a mixture of both long batch jobs and latency-sensitive short jobs in enterprise data centers. Recently hybrid job schedulers emerge as attractive alternatives of conventional centralized job schedulers. In this paper, we conduct trace-driven experiments to study the job-completion-delay performance of two representative hybrid job schedulers (Hawk and Eagle), and find that short jobs still encounter long latency issues due to fluctuating bursty nature of workloads. To this end, we propose Dice, a general performance optimization framework for hybrid job schedulers, to alleviate the high job-completion-delay problem of short jobs. Dice is composed of two simple yet effective techniques: Elastic Sizing and Opportunistic Preemption. Both Elastic Sizing and Opportunistic Preemption keep track of the task waiting times of short jobs. When the mean task waiting time of short jobs is high, Elastic Sizing dynamically and adaptively increases the short partition size to prioritize short jobs over long jobs. On the other hand, Opportunistic Preemption preempts resources from long tasks running in the general partition on demand, so as to mitigate the head-of-line blocking problem of short jobs. We enhance the two schedulers with Dice and evaluate Dice performance improvement in our prototype implementation. Experiment results show that Dice achieves 50.9%, 54.5%, and 43.5% improvement on 50th-percentile, 75th-percentile, and 90th-percentile job completion delays of short jobs in Hawk respectively, as well as 33.2%, 74.1%, and 85.3% improvement on those in Eagle respectively under the Google trace, at low performance costs to long jobs.

Many large-scale data analytics infrastructures are employed for a wide variety of jobs, ranging from short interactive queries to large data analysis jobs that may take hours or even days to complete. As a consequence, data-processing frameworks like MapReduce may have workloads consisting of jobs with heavy-tailed processing requirements. With such workloads, short jobs may experience slowdowns that are an order of magnitude larger than large jobs do, while the users may expect slowdowns that are more in proportion with the job sizes. To address this problem of large job slowdown variability in MapReduce frameworks, we design a scheduling system called TYREX that is inspired by the well-known TAGS task assignment policy in distributed-server systems. In particular, TYREX partitions the resources of a MapReduce framework, allowing any job running in any partition to read data stored on any machine, imposes runtime limits in the partitions, and successively executes parts of jobs in a work-conserving way in these partitions until they can run to completion. We develop a statistical model for dynamically setting the runtime limits that achieves near optimal job slowdown performance, and we empirically evaluate TYREX on a cluster system with workloads consisting of both synthetic and real-world benchmarks. We find that TYREX cuts in half the job slowdown variability while preserving the median job slowdown when compared to state-of-the-art MapReduce schedulers such as FIFO and FAIR. Furthermore, TYREX reduces the job slowdown at the 95th percentile by more than 50% when compared to FIFO and by 20-40% when compared to FAIR.

The specific choice of workload task schedulers for Hadoop MapReduce applications can have a dramatic effect on job workload latency. The Hadoop Fair Scheduler (FairS) assigns resources to jobs such that all jobs get, on average, an equal share of resources over time. Thus, it addresses the problem with a FIFO scheduler when short jobs have to wait for long running jobs to complete. We show that even for the FairS, jobs are still forced to wait significantly when the MapReduce system assigns equal sharing of resources due to dependencies between Map, Shuffle, Sort, Reduce phases. We propose a Hybrid Scheduler (HybS) algorithm based on dynamic priority in order to reduce the latency for variable length concurrent jobs, while maintaining data locality. The dynamic priorities can accommodate multiple task lengths, job sizes, and job waiting times by applying a greedy fractional knapsack algorithm for job task processor assignment. The estimated runtime of Map and Reduce tasks are provided to the HybS dynamic priorities from the historical Hadoop log files. In addition to dynamic priority, we implement a reordering of task processor assignment to account for data availability to automatically maintain the benefits of data locality in this environment. We evaluate our approach by running concurrent workloads consisting of the Word-count and Terasort benchmarks, and a satellite scientific data processing workload and developing a simulator. Our evaluation shows the HybS system improves the average response time for the workloads approximately 2.1x faster over the Hadoop FairS with a standard deviation of 1.4x.

Efficient multiprocessor scheduling is pivotal in optimizing the performance of parallel computing systems. This paper leverages the power of Petri nets and the tool GPenSIM to model and simulate a variety of multiprocessor scheduling algorithms (the basic algorithms such as first come first serve, shortest job first, and round robin, and more sophisticated schedulers like multi-level feedback queue and Linux’s completely fair scheduler). This paper presents the evaluation of three crucial performance metrics in multiprocessor scheduling (such as turnaround time, response time, and throughput) under various scheduling algorithms. However, the primary focus of the paper is to develop a robust simulation platform consisting of Petri Modules to facilitate the dynamic representation of concurrent processes, enabling us to explore the real-time interactions and dependencies in a multiprocessor environment; more advanced and newer schedulers can be tested with the simulation platform presented in this paper.

Scheduling has a great influence in improving computer system performance. Many recent system designs use policies which give priority to short jobs as it is often the case that the majority of the jobs are short. SRPT (Shortest Remaining Processing Time First), which servers the jobs that needs the least amount of service to complete, is known to produce optimum mean response time. However SRPT requires having the knowledge of the individual service time exactly in advance, which is not practical. FCFS and PS which shares the system capacity equally among all the jobs can be used if the CPU time requirement of each job is not available beforehand. However, FCFS is detrimental even when there is a moderate variability in job service times. In practice, however, PS must be implemented by RR with time-slicing, which incurs non-negligible job switching overhead for small time-slices. Over time, in addition to PS strategies, for example, LCFSPR (Last Come First Serve with Pre-emptive Resume), LAT (Least-Attained-Time), SRT (Shortest Residual-Time), etc have been presented. A common feature of these strategies is that preemption is employed to favour possibly short jobs. Among these strategies, some are simple to use and incur less overhead, while others incur more overhead. For example, RR and LCFSPR only require insertion of jobs at the front/back of queue (with complexity O(1)), whereas LAT and SRT need to maintain a sorted queue (with complexity O(log(n))), and keep track of the attained times and residual times of individual jobs. LCFSPR also yields M/M/1 result, but it leads to situations where short jobs can get stuck behind long jobs. In this paper, we mainly consider the issue of handling newly arrived jobs in implementing RR strategy. A research issue is then how time-slicing performs if large time-slices have to be used. In this paper, we investigate several RR variants through Discrete Event Simulation. Our results show that, by favouring newly arrived jobs, the performance of RR with large time-slices could be improved than that of ideal PS. The simple immediate preemption scheme, which serves the new jobs immediately by preempting the current active job, is shown to further improve the performance of RR.

Load balancing in cloud computing assigning the all load over each node of a mutual or shared system to make resource usage effective and to perform the response time to the job. This has used Round Robin (RR) and shortest job first (SJF) algorithm. Scheduling is the main part of cloud computing and jobs or workflow scheduling under the platform as a service. The work under infrastructure as a service work to virtual machine scheduling which machine is decided by scheduler should go on which job or virtual machine. Simply RR is a time quantum-based preemptive algorithm, and SJF is a burst time-based non-preemptive and preemptive algorithm. But RR work in two-phase like expected finish time adapted for each process. Each process with generally minimum completion times scheduled. This algorithm result as no longer waiting for the smallest task.

Data warehouse is an inseparable part of modern business but the amount of data it stores is huge and growing so fast, the queries of the business leaders it processes have become very sophisticated as well. Data warehouse needs to response to those queries with high quality data having less response time but the major challenge for that is the scale of data and nature of the queries involved. It’s very hard to access the required data set with minimum response time in data processing as the data warehouse needs to perform analytical operation on a large set of data analyzing sophisticated queries. This paper introduces a new data warehousing schema that constructs a multidimensional relation tree hierarchy, changing the traditional use of relational database, with partitioned database in its every level instead of any individual data. This schema allows data warehouse to partition and process a large data part by part in a small batch to minimize the response time and stores summarized data to increase data availability in query processing. Allowing easy and fast access to the required set of data, this schema improves the performance and the decision making ability of the data warehouse.

Data warehouse is designed for answering analytical queries. Data warehouse saves historical data. In the data warehouse, the response time to analytical queries is long. So reducing the response time is a critical problem. There are a lot of algorithms to solve the problem. Some of them, materialize frequent views. The previously posed queries have important information that will be used in the future. This paper proposes an algorithm for view materialization. The proposed algorithm finds proper views using previous queries and materializes them. The views are able to answer future queries. The view selection algorithm has four steps. At first, it clusters previous queries by SOM method. Then frequent queries are found by Apriori algorithm. In the third step the problem is converted to 0/1 knapsack equations and finally, optimal queries are joined to create only one view for each cluster. This paper improves the first and third step. This paper uses the SOM algorithm for clustering previous queries in the first step and it solves the 0/1 knapsack equations according to shuffled frog leaping algorithm in the third step. Experimental results show that it improves the previous view selection algorithms according to response time and storage space factor.

In this manuscript, we present a prototype of a modular data platform that is able to continuously ingest, process, retain, and analyse large amounts of network telemetry data in a scalable and straightforward manner. It follows a recently proposed Data Lakehouse architectural pattern, which is an evolution of two well-known approaches used in this area – data warehouses and data lakes. The platform is based on open standards and open-source components, and it follows cloud native principles in order to be able to run in modern computing environments such as public, private, and hybrid clouds. The primary focus of the prototype is network telemetry and analytics over traffic flows and infrastructure logs for the purposes of cyber-security digital forensics and incident response. During the demonstration part, we will further describe internal workings of the presented data platform and showcase its capabilities and possible applications on a public dataset.

The primary objective of this proposal is to minimize the battery drain and to maximize the compatibility of the device so that the complexity of the device is much more reduced to enhance the battery period considerably. In this paper we adopted a method for increasing the response time using a priority based algorithm along with Field Programmable Gate Array (FPGA) in Reconfigurable Computing System (RCS) to handle complex applications and to increase the efficiency and throughput of the system. A primary comparison was made with all existing algorithms namely First Come First Serve (FCFS), Round Robin, Shortest Job First (SJF) for a given set of processes by taking process delay as a core parameter. The result obtained from this analysis gave a clear picture that the objective can be achieved by decreasing the waiting time and by increasing the throughput. By achieving this task, multitasking between the applications or processes can be done frequently. By adopting these kind of scheduling algorithms with programmable gate array, there is a considerable improvement and performance of the device. Also, the life span of the device is increased which is considered as most important criteria for any users in computing environment.

Schema versioning in data warehouses: Enabling cross-version querying via schema augmentation

Chapter 9 - A Data Warehouse Strategy for on-Demand Multiscale Mapping

A data warehouse is designed for answering analytical queries. These queries are usually long, complex and ad hoc in nature and their response time is high when processed against an exponentially growing data warehouse. Materialising views has been found to be an effective tool for reducing this response time. All views cannot be materialised on account of resource constraints. Further, optimal view selection is shown to be an NP-complete problem. This necessitates selection of an appropriate set of views, from amongst all possible views that reduce the query response time. Most of the view selection algorithms are greedy or evolutionary. In this paper, a differential evolution view selection algorithm (DEVSA) that selects the Top-K views from a multi-dimensional lattice is proposed. Further, it is shown that DEVSA when compared to the greedy, genetic and memetic-based view selection algorithms selects comparatively better quality views for higher dimensional datasets.

With advances in mobile technology and mobile Internet applications, smart mobile devices, such as smartphones and tablets, have become increasingly popular, and the number of Internet users worldwide continues to grow. In the Internet era, the amount of data is growing exponentially and companies must be able to harness the value of the vast amount of data. Data platforms must integrate massive amounts of data collection, storage, computation and analysis to meet these opportunities and challenges. In this study, the log data of Internet users browsing websites are analyzed and the technologies used in the platform are briefly described. Finally, a draft platform for analyzing offline Internet user behavior data is proposed, taking into account the current common needs of different industries, while incorporating some innovations. Three modules are designed and implemented: data collection, data warehouse and data visualization. The user's data is mainly collected by the data collection module. The data warehouse is mainly responsible for cleaning, modeling and analyzing the data. As part of the data visualization module, the result data from the ADS layer is used as a template to create tables in MySQL, export the results to MySQL periodically using the Sqoop tool, and visualize the data using the data visualization tool. With Flume, Kafka and Sqoop, HDFS is used as the data storage framework, Hive is used as the storage tool, and Spark is used as the Hive computation engine to build the platform in a large context to analyze Internet user behavior.

As the rapid development of hardware and network technology, cloud computing has become an important research topic. For applications of large-scale data processing, such as data warehouse, Map Reduce is the most famous platform for parallel data processing in cloud computing. To support the star-join queries in data warehouse, Scatter-Gather-Merge (SGM) proposes an efficient algorithm on the Map Reduce framework. However, SGM supports only the equi-join queries. Nonequi-join queries may cause SGM to fail. In this paper, we propose a method to cope with theta-join queries, i.e., both equi-join and nonequi-join queries. Our proposed method uses a novel manipulation of keys for partitioning data. The key manipulation matches up the Map Reduce paradigm, and makes theta-join queries workable on the Map Reduce platform. Our experimental results show that the proposed method achieves similar performance to SGM, but our method supports more join-query types. Our method performs even better than SGM in some query types of high data selectivity.