Modeling MapReduce with CSP

Some Thoughts About Data Type, Distribution, and Statistical Significance

Machine learning algorithms such as support vector machine (SVM) have been widely used to detect brain tumors in big data environments. However, the SVM classifier is unsuitable for a large dataset as the complexity involved is found to be high. Therefore, in this article, a MapReduce model is introduced with SVM to handle large-scale data and deal with this issue. In this article, a framework called mutual informative MapReduce and minimum quadrangle classification (MIMR-MQC) is introduced for brain tumor detection to handle challenges associated with big data classification. Here, preprocessing is performed using MIMR, which removes unwanted and redundant attributes in the brain tumor dataset. This technique reduces the computation complexity and time using a big dataset for detecting the brain tumors. Then, the minimum quadrangle support vector machine model is created using Lagrange multipliers and radial basis kernel function for improving the efficiency of the classification process. The MIMR-MQC framework is validated on a standard dataset called Central Brain tumor Registry of the United States (CBTRUS). Results show that the proposed model observed 21% of higher detection accuracy by minimizing the computational complexity and detection time by 37% and 27%, respectively in comparison with existing models. A comparison with state-of-the-art machine learning techniques, the MIMR-MQC framework performs better in terms of brain tumor detection time and accuracy due to the better distribution of data.

Processing huge volume of data opened new opportunities in ecommerce, engineering, business and large computing applications. MapReduce programming model is a parallel data processing approach for execution on computer clusters. This model provides an abstraction to design scalable computing algorithm for big data processing. For batch processing types of data processing, MapReduce model provides faster computation. The key/value pair generation of MapReduce program creates memory overhead and deserialization overhead due to data redundancy. Redundancy of data is one of the most important factors that consumes space and affect system performance while using large set of data. This overhead can be avoided considerably by using a novel approach that we developed named Data Triggered Multithreaded Programming (DTMP) model. In this paper, we demonstrate the use of DTMP model using a large dataset with author details and his publications. The Data Triggered Multithreaded Programming can dynamically allocate the resources and can identify the data repetition occurring during computation. DTMP model when applied to the MapReduce programming model brings performance improvement to the system. The major contributions of this work are a simple, scalable and powerful processing of text data that enables automatic parallelization and distribution of large-scale computations.

There are plentiful and diverse applications of graph data management and mining techniques in the real-world scientific research and business activities. As one of the most basic operations, uniform path pattern query processing on graph data faces three big challenges. In this paper, we deal with these challenges by the following points. Firstly, a new query language on graph, called G-Path, is presented, which focuses on complex path pattern query processing on a very large graph. Also, the design of a system called Para-G is proposed, which is based on a BSP-like model as well as MapReduce model, and can effectively handle distributed graph data operations and queries. Secondly, the implementation of Para-G on the de facto cloud platform -- Hadoop -- is brought forward. Based on the concept of distributed path finite state automaton, the query processing of a G-Path statement in Para-G is detailed. In addition, as the query optimization of G-Path queries, several tricks are utilized to dramatically improve the performance of query execution. Finally, extensive experiments on several graph data sets are conducted to show the usability of the G-Path query language and the effectiveness of Para-G.

It is expected that the first exascale supercomputer will be deployed within the next 10 years, but which programming model will allow easy development and yet scalable and efficient programs is still not known. One of the programming models considered to be feasible is the so-called partitioned global address space~(PGAS) model, which allows easy development by providing one common memory address space across all cluster nodes. In this paper we compare remote memory access and memory consistency of current PGAS programming languages and describe how synchronization can generated unneeded network transfers. We furthermore introduce our variation of the PGAS model that allows for implicit fine-grained pair-wise synchronization among the nodes. Efficient and easy to use synchronization is necessary to keep all the processors of upcoming supercomputers busy. We furthermore offer easy deployment of RDMA transfers and use communication algorithms commonly used in MPI collective operations, but lift the requirement of the operations to be collective. Our model is based on single assignment variables and uses a data-flow like synchronization mechanism. Reading uninitialized variables results in the reading thread to be blocked until data are made available by another thread. That way synchronization is done implicitly when data are read. Broadcast, scatter and gather are modeled based on data distribution among the nodes, whereas for reduction and scan we follow a combining PRAM approach of having multiple threads write to the same memory location. We discuss both a Gaus-Seidel stencil and bitonic sort in our model. We implemented a proof-of-concept library showing the usability and scalability of the model. With this library the Gaus-Seidel stencil scaled well in initial experiments on an 8-node machine.

Based on the MapReduce model and Hadoop Distributed File System (HDFS), Hadoop enables the distributed processing of large data sets across clusters with scalability and fault tolerance. Many data-intensive applications involve continuous and incremental updates of data. Understanding the scalability and cost of a Hadoop platform to handle small and independent updates of data sets sheds light on the design of scalable and cost-effective data-intensive applications. In this chapter, we introduce a motivating movie recommendation application implemented in the MapReduce model and deployed on Amazon Elastic MapReduce (EMR), a Hadoop CONTENTS 2.

One of the central problems in machine learning and pattern recognition is how to deal with high-dimensional data either for visualization or for classification and clustering. Most of dimensionality reduction technologies, designed to cope with the curse of dimensionality, are based on Euclidean distance metric. In this work, we propose an unsupervised nonlinear dimensionality reduction method which attempt to preserve the distribution of input data, called distribution preserving embedding (DPE). It is done by minimizing the dissimilarity between the densities estimated in the original and embedded spaces. In theory, patterns in data can effectively be described by the distribution of the data. Therefore, DPE is able to discover the intrinsic pattern (structure) of data, including the global structures and the local structures. Additionally, DPE can be extended to cope with out-of-sample problem naturally. Extensive experiments on different data sets compared with other competing methods are reported to demonstrate the effectiveness of the proposed approach.

Traffic assignment is a fundamental tool to evaluate and analyse the travel behaviour of network users in the transportation network. Although many extensions have been developed, the principle of user equilibrium (UE) is still the cornerstone for solving traffic equilibrium problems. Applications of UE in large-scale transportation networks have been largely limited due to the overwhelming computation burden. Therefore, with the recent advances in parallel computing, this paper proposes an efficient parallel-computing framework based on Map-Reduce to solve the UE problem. This Map-Reduce model provides a concise abstraction, Map and Reduce, for separable computational tasks. We incorporate this parallel programming model into the Frank–Wolfe algorithm and gradient projection algorithm to achieve efficient in-memory computations on large clusters in a fault-tolerant manner. The proposed parallel-computing algorithms are applied to large-scale transportation networks to examine its computation efficiency. This acceleration approach is found to significantly reduce the execution time.

Remote sensing retrieval of saline and alkaline land based on reflectance spectroscopy and RV-MELM in Zhenlai County

In recent years there has been an extraordinary growth of large-scale data processing and related technologies in both, industry and academic communities. This trend is mostly driven by the need to explore the increasingly large amounts of information that global companies and communities are able to gather, and has lead the introduction of new tools and models, most of which are designed around the idea of handling huge amounts of data. A good example of this trend towards improved large-scale data processing is MapReduce, a programming model intended to ease the development of massively parallel applications, and which has been widely adopted to process large datasets thanks to its simplicity. While the MapReduce model was originally used primarily for batch data processing in large static clusters, nowadays it is mostly deployed along with other kinds of workloads in shared environments in which multiple users may be submitting concurrent jobs with completely different priorities and needs: from small, almost interactive, executions, to very long applications that take hours to complete. Scheduling and selecting tasks for execution is extremely relevant in MapReduce environments since it governs a job's opportunity to make progress and determines its performance. However, only basic primitives to prioritize between jobs are available at the moment, constantly causing either under or over-provisioning, as the amount of resources needed to complete a particular job are not obvious a priori. This thesis aims to address both, the lack of management capabilities and the increased complexity of the environments in which MapReduce is executed. To that end, new models and techniques are introduced in order to improve the scheduling of MapReduce in the presence of different constraints found in real-world scenarios, such as completion time goals, data locality, hardware heterogeneity, or availability of resources. The focus is on improving the integration of MapReduce with the computing infrastructures in which it usually runs, allowing alternative techniques for dynamic management and provisioning of resources. More specifically, it is focused in three scenarios that are incremental in its scope. First, it studies the prospects of using high-level performance criteria to manage and drive the performance of MapReduce applications, taking advantage of the fact that MapReduce is executed in controlled environments in which the status of the cluster is known. Second, it examines the feasibility and benefits of making the MapReduce runtime more aware of the underlying hardware and the characteristics of applications. And finally, it also considers the interaction between MapReduce and other kinds of workloads, proposing new techniques to handle these increasingly complex environments. Following these three items described above, this thesis contributes to the management of MapReduce workloads by 1) proposing a performance model for MapReduce workloads and a scheduling algorithm that leverages the proposed model and is able to adapt depending on the various needs of its users in the presence of completion time constraints; 2) proposing a new resource model for MapReduce and a placement algorithm aware of the underlying hardware as well as the characteristics of the applications, capable of improving cluster utilization while still being guided by job performance metrics; and 3) proposing a model for shared environments in which MapReduce is executed along with other kinds of workloads such as transactional applications, and a scheduler aware of these workloads and its expected demand of resources, capable of improving resource utilization across machines while observing completion time goals.

In this work, a practical implementation of two parallel thinning algorithms on a multicomputer system are described. The solution has been conceived for a multiprocessor using the SPMD (single program multiple data) programming model. Our main goal is intended to describe our experiences on data partition/distribution among processors for parallel thinning algorithms as a representative type of algorithm where communications take place between neighbor processors and the work load for each processor depends on the input data. It will be shown how the efficiency of the parallel implementation can be improved through the application of a preprocess. This preprocess is based on the analysis of the work load balance. An analysis of the communication cost is also made. Although the results shown here are concerned with the implementations of two parallel thinning algorithms we think that our proposal about data distribution is general and useful for a wide set of algorithms in the field of image processing. Copyright © 2000 John Wiley & Sons, Ltd.

This paper describes THEMIS, a programming model and run-time library being designed to support cross-component performance optimization through explicit manipulation of the computation's iteration space at run-time. Each is augmented with component dependence metadata, which characterizes the constraints on its execution order, data distribution and memory access order. We show how this supports dynamic adaptation of each to exploit the available resources, the context in which its operands are generated, and results are used, and the evolution of the problem instance. Using a computational fluid dynamics visualization example as motivation, we show how dependence metadata provides a framework in which a number of interesting optimizations become possible. Examples include data placement optimization, loop fusion, tiling, memoization, checkpointing and incrementalization.

Fuzzy quasi-linear SVM classifier: Design and analysis

Article history: Received December 10, 2013 Received in revised format 12 March 2014 Accepted April 24 2014 Available online April 25 2014 This article addresses multi-level lot sizing and scheduling problem in capacitated, dynamic and deterministic cases of a job shop manufacturing system with sequence-dependent setup times and costs assumptions. A new mixed-integer programing (MIP) model with big bucket time approach is provided to the problem formulation. It is well known that the capacitated lot sizing and scheduling problem (CLSP) is NP-hard. The problem of this paper that it is an extent of the CLSP is even more complicated; consequently, it necessitates the use of approximated methods to solve this problem. Hence, two new mixed integer programming-based approaches with rolling horizon framework have been used to solve this model. To evaluate the performance of the proposed model and algorithms, some numerical experiments are conducted. The comparative results indicate the superiority of the second heuristic. © 2014 Growing Science Ltd. All rights reserved.

The generation of RDF data has accelerated to the point where many data sets need to be partitioned across multiple machines in order to achieve reasonable performance when querying the data. Although tremendous progress has been made in the Semantic Web community for achieving high performance data management on a single node, current solutions that allow the data to be partitioned across multiple machines are highly inefficient. In this paper, we introduce a scalable RDF data management system that is up to three orders of magnitude more efficient than popular multi-node RDF data management systems. In so doing, we introduce techniques for (1) leveraging state-of-the-art single node RDF-store technology (2) partitioning the data across nodes in a manner that helps accelerate query processing through locality optimizations and (3) decomposing SPARQL queries into high performance fragments that take advantage of how data is partitioned in a cluster.