Comparative Study of Scheduling Algorithms in Heterogeneous Distributed Computing Systems

Loosely coupled applications composed of a potentially very large number (from tens of thousands to even billions) of tasks are commonly used in high-throughput computing and many-task computing paradigms. To efficiently execute large-scale computations which can exceed the capability in a single type of computing resources within expected time, we should be able to effectively integrate resources from heterogeneous distributed computing (HDC) systems such as clusters, grids, and clouds. In this paper, we quantitatively analyze the performance of three different real scientific applications consisting of many tasks on top of HDC systems based on a partnership of distributed computing clusters, grids, and clouds to understand the application and resource characteristics, and show practical issues that normal scientific users can face during the course of leveraging these systems. Our experimental study shows that the performance of a loosely coupled application can be significantly affected by the characteristics of a HDC system, along with hardware specification of a node, and their impacts on the performance can vary widely depending on the resource usage pattern of each application. We then devise a preference-based scheduling algorithm that can reflect characteristics and resource usage patterns of various loosely coupled applications running on top of HDC systems from our experimental study. Our preference-based scheduling algorithm can allocate the resources from different HDC systems to loosely coupled applications based on the preferences of the applications for the HDC systems. We evaluate the overall system performance over various preference types, using trace-based simulations, which can be determined based on different factors such as CPU specifications and application throughputs. Our simulation results demonstrate the importance of understanding the application and resource characteristics on effective scheduling of loosely coupled applications on the HDC systems.

Executing complicated computations in parallel increases the speed of computing and brings user delight to the system. Decomposing the program into several small programs and running multiple parallel processors are modeled by Directed Acyclic Graph. Scheduling nodes to execute this task graph is an important problem that will speed up computations. Since task scheduling in this graph belongs to NP-hard problems, various algorithms were developed for node scheduling to contribute to quality service delivery. The present study brought a heuristic algorithm named looking ahead sequencing algorithm (LASA) to cope with static scheduling in heterogeneous distributed computing systems with the intention of minimizing the schedule length of the user application. In the algorithm proposed here, looking ahead is considered as a criterion for prioritizing tasks. Also, a property called Emphasized Processor has been added to the algorithm to emphasize the task execution on a particular processor. The effectiveness of the algorithm was shown on few workflow type applications and the results of the algorithm implementation were compared with two more heuristic and meta-heuristic algorithms.

A hybrid heuristic–genetic algorithm for task scheduling in heterogeneous processor networks

The paper presents a task allocation technique for multiple applications onto heterogeneous distributed computing system to minimize the overall makespan. An existing critical path based algorithm for scheduling of tasks of single application has been used to allocate tasks of multiple applications onto heterogeneous distributed computing system. The paper discusses how a composite application is given for multiple applications and how critical path based algorithm is applied efficiently for the allocation of tasks of different applications.

Big Data is a collection of data with a large and complex size, consisting of various data types and obtained from various sources, overgrowing quickly. Some of the problems that will arise when processing big data, among others, are related to the storage and access of big data, which consists of various types of data with high complexity that are not able to be handled by the relational model. One technology that can solve the problem of storing and accessing big data is Hadoop. Hadoop is a technology that can store and process big data by distributing big data into several data partitions (data blocks). Problems arise when an analysis process requires all data spread out into one data entity, for example, in the data clustering process. One alternative solution is to do a parallel and scattered analysis, then perform a centralized analysis of the results of the scattered analysis. This study examines and analyzes two methods, namely K-Medoids Mapreduce and K-Modes without Mapreduce. The dataset used is a dataset about cars consisting of 3.5 million rows of data with 400MB distributed in a Hadoop Cluster (consisting of more than one engine). Hadoop has a MapReduce feature, consisting of 2 functions, namely map and reduce. The map function performs a selection to retrieve a key, value pairs, and returns a value in the form of a collection of key value pairs, and then the reduce function combines all key value pairs from several map functions. The results of the cluster quality evaluation are tested using the Silhouette Coefficient testing metric. The K-Medoids MapReduce algorithm for the car dataset gives a silhouette value of 0.99 with a total of 2 clusters.

PurposeBig data has posed problems for businesses, the Information Technology (IT) sector and the science community. The problems posed by big data can be effectively addressed using cloud computing and associated distributed computing technology. Cloud computing and big data are two significant past-year problems that allow high-efficiency and competitive computing tools to be delivered as IT services. The paper aims to examine the role of the cloud as a tool for managing big data in various aspects to help businesses.Design/methodology/approachThis paper delivers solutions in the cloud for storing, compressing, analyzing and processing big data. Hence, articles were divided into four categories: articles on big data storage, articles on big data processing, articles on analyzing and finally, articles on data compression in cloud computing. This article is based on a systematic literature review. Also, it is based on a review of 19 published papers on big data.FindingsFrom the results, it can be inferred that cloud computing technology has features that can be useful for big data management. Challenging issues are raised in each section. For example, in storing big data, privacy and security issues are challenging.Research limitations/implicationsThere were limitations to this systematic review. The first limitation is that only English articles were reviewed. Also, articles that matched the keywords were used. Finally, in this review, authoritative articles were reviewed, and slides and tutorials were avoided.Practical implicationsThe research presents new insight into the business value of cloud computing in interfirm collaborations.Originality/valuePrevious research has often examined other aspects of big data in the cloud. This article takes a new approach to the subject. It allows big data researchers to comprehend the various aspects of big data management in the cloud. In addition, setting an agenda for future research saves time and effort for readers searching for topics within big data.

To break the in-memory bottleneck and facilitate online sampling in cluster computing frameworks, we propose a new sampling-based system for approximate big data analysis on computing clusters. We address both computational and statistical aspects of big data across the main layers of cluster computing frameworks: big data storage, big data management, big data online sampling, big data processing, and big data exploration and analysis. We use the new Random Sample Partition (RSP) distributed data model to store a big data set as a set of ready-to-use random sample data blocks in Hadoop Distributed File System (HDFS), called RSP blocks. With this system, only a few RSP blocks are selected and processed using a sequential algorithm in a distributed data-parallel manner to produce approximate results for the entire data set. In this paper, we present a prototype RSP-based system and demonstrate its advantages. Our experiments show that RSP blocks can be used to get approximate models and summary statistics as well as estimate the proportions of inconsistent values without computing the entire data or running expensive online sampling operations. This new system enables big data exploration and analysis where the entire data set cannot be computed.

Machine learning for Big Data analytics in plants.

While the potential benefits of Big Data adoption are significant, and some initial successes have already been realized, there remain many research and technical challenges that must be addressed to fully realize this potential. The Big Data processing, storage and analytics, of course, are major challenges that are most easily recognized. However, there are additional challenges related for instance to Big Data collection, integration, and quality enforcement. This paper proposes a hybrid approach to Big Data quality evaluation across the Big Data value chain. It consists of assessing first the quality of Big Data itself, which involve processes such as cleansing, filtering and approximation. Then, assessing the quality of process handling this Big Data, which involve for example processing and analytics process. We conduct a set of experiments to evaluate Quality of Data prior and after its pre-processing, and the Quality of the pre-processing and processing on a large dataset. Quality metrics have been measured to access three Big Data quality dimensions: accuracy, completeness, and consistency. The results proved that combination of data-driven and process-driven quality evaluation lead to improved quality enforcement across the Big Data value chain. Hence, we recorded high prediction accuracy and low processing time after we evaluate 6 well-known classification algorithms as part of processing and analytics phase of Big Data value chain.

The prosperity of modern human civilization is attributed to the huge amount of resources and energy. With the increasing population and technological advancements, thedemand for energy will definitely continue to increase. Howto save energy has become an important issue. In this work, we proposed a system to collect the electricity usage data in campus buildings through smart meters and environmental sensors, and process the huge amount of data by big data processing techniques. Therefore, we introduced cloud computing and big data processing architecture as solutions to build a real-time energy monitoring system for smart campus. In this work, we used Hadoop ecosystem which is built on big data processing architecture to improve the capacity of big data storage and processing for our system. The proposedsystem has been implemented in Tunghai University. Finally, the system interface vividly displays the electricity usage states in campus buildings; thus, users can monitor the electricity usage in the campus and historical data at any time and any place.

Over the past few years, data has been growing significantly due to the advent of new connected devices, availability of bandwidth, and the emergence of new applications which utilize cloud computing infrastructure in the data centers. This increased amount of data faces many problems in terms of storage, transmission, management, and processing, etc. Therefore, the term big data has gained significant attention from researchers in recent years. The rapidly growing quantity, velocity, and variety of data require more probable and logical tools for its storage. For this purpose, the industry is highly emphasizing the development of more viable tools for the storage of big data. The traditional big data storage tools are unsuccessful in storing an enormous amount of data. Hence, the structural modifications of management mechanisms of conventional storage systems such as SQL databases to NoSQL databases technology are necessary to cope up with drastically increasing requirements of big data storage. The primary objective of this paper is to concentrate exclusively on designing a road map for NoSQL big data storage technologies, evaluate current evidence, research progresses in NoSQL data storage systems and their applications in various domains. We conducted a systematic literature review (SLR) of various studies published in recent years. We propose a framework to classify selected articles on the basis of various factors such as motivations behind big data storage, NoSQL techniques used for storing big data, and significant applications of big data in different domains. Furthermore, we also discuss research issues and define an outline for future research in the big data storage domain for NoSQL databases.

Hadoop framework has been evolved to manage big data in cloud. Hadoop distributed file system and MapReduce, the vital components of this framework, provide scalable and fault-tolerant big data storage and processing services at a lower cost. However, Hadoop does not provide any robust authentication mechanism for principals’ authentication. In fact, the existing state-of-the-art authentication protocols are vulnerable to various security threats, such as man-in-the-middle, replay, password guessing, stolen-verifier, privileged-insider, identity compromization, impersonation, denial-of-service, online/off-line dictionary, chosen plaintext, workstation compromization, and server-side compromisation attacks. Beside these threats, the state-of-the-art mechanisms lack to address the server-side data integrity and confidentiality issues. In addition to this, most of the existing authentication protocols follow a single-server-based user authentication strategy, which, in fact, originates single point of failure and single point of vulnerability issues. To address these limitations, in this paper, we propose a fault-tolerant authentication protocol suitable for the Hadoop framework, which is called the efficient authentication protocol for Hadoop (HEAP). HEAP alleviates the major issues of the existing state-of-the-art authentication mechanisms, namely operating-system-based authentication, password-based approach, and delegated token-based schemes, respectively, which are presently deployed in Hadoop. HEAP follows two-server-based authentication mechanism. HEAP authenticates the principal based on digital signature generation and verification strategy utilizing both advanced encryption standard and elliptic curve cryptography. The security analysis using both the formal security using the broadly accepted real-or-random (ROR) model and the informal (non-mathematical) security shows that HEAP protects several well-known attacks. In addition, the formal security verification using the widely used automated validation of Internet security protocols and applications ensures that HEAP is resilient against replay and man-in-the-middle attacks. Finally, the performance study contemplates that the overheads incurred in HEAP is reasonable and is also comparable to that of other existing state-of-the-art authentication protocols. High security along with comparable overheads makes HEAP to be robust and practical for a secure access to the big data storage and processing services.

Scheduling algorithm is an important part of CPU work and plays a key role in the program system. Successful scheduling algorithms can effectively improve system efficiency and stability. This paper aims to analyze the characteristics of several scheduling algorithms comparatively through some general criteria to help determine the applicable algorithms in different situations. This paper first introduces the important role of scheduling algorithms. It then discusses and analyzes in detail the complexity of scheduling algorithms, allocation methods, average waiting time, and starvation. Finally, it draws the advantages and disadvantages of different scheduling algorithms through comparison. The types of scheduling algorithms applicable to batch systems and real-time/interactive systems respectively are concluded. The study results provide a theoretical basis for system designers to achieve the optimal utilization of CPU resources in the system. By analyzing and comparing the advantages and disadvantages of different scheduling algorithms, this paper lays a solid foundation for the future research with more scheduling methods.

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.